AbotX adds advanced functionality, shortcuts and configurations to the rock solid Abot C# Web Crawler. It consists of the two main entry points of CrawlerX and ParallelCrawlerEngine. CrawlerX is a single crawler instance while ParallelCrawlerEngine creates and manages multiple instances of CrawlerX. If you want to just crawl a single site then CrawlerX is where you want to start. If you want to crawl a configurable number of sites concurrently within the same process then the ParallelCrawlerEngine is what you are after.

Technical Details

  • Written In C#
  • Targets .NET Framework 4.0