Db Devs has developed numerous crawlers for a rather big number of sports website. Some of these websites are Coral, Ladbrokes, William Hill, Paddy Power, Bet365, SkyBet, 888sport, Betfair, Boylesports, Racebets, Stanjames, Titanbet, Betvictor, Williamhill, Winnersports, Betdaq, Matchbook, etc.
For scanning the websites either TOR network Proxy addresses were used or Proxy addressed collected with the help of our internally developed Proxy Collection service. The service scans Free Proxy available on the web with the frequency predefined, checks the possibility of them to be used at each target website and saves these Proxy addresses into the database for further use.
For some of the websites Browser Emulator for clicking on the website pages was used.
At some of the websites it was required to solve captcha, for which either Tesseract (optical symbols recognition software) or http://www.deathbycaptcha.com service were used.
The scraped data is processed, compared, analyzed and visualized, etc. in various ways.
Tools and Technologies