We have developed crawlers for hundreds of retail websites. Among them there are crawlers for http://www.currys.co.uk/, http://www.lowes.com/, http://www.walmart.com/, as well as https://www.amazon.com/ and http://www.ebay.com/ for various countries, etc.
The crawlers for these and many other websites are integrated into complex and multifunctional web-based solutions, meanwhile some of them are used as independent desktop crawlers designed for some particular purpose.
In the case of complex web-based solutions with a big number of crawlers in them crawlers are run on regular basis with the frequency predefined. The scraped data is processed and analyzed in the real time. The result data could be compared, visualized, used as input onto other websites and third-party systems, etc.
APIs of the scraped websites are used when possible and when sufficient. In all other cases crawlers created with the help of our existing custom web-scraping applications are used.
For scanning the websites either TOR network Proxy addresses were used or Proxy addressed collected with the help of our internally developed Proxy Collection Service.
For some of the websites Browser Emulator for clicking on the website pages was used.
Tools and Technologies