Most websites you crawl cannot or will not handle the load of a web crawler. Auto Throttling automatically slows down the crawl speed if the website being crawled is showing signs of stress or unwillingness to respond to the frequency of http requests.
Using CrawlerX (single instance of a crawler)
Using ParallelCrawlerEngine (multiple instances of crawlers)
|config.AutoThrottling.IsEnabled||Whether to enable the AutoThrottling feature||CrawlerX|
|config.AutoThrottling.ThresholdHigh||The number of "stressed" requests before considering a crawl as under high stress||CrawlerX|
|config.AutoThrottling.ThresholdMed||The number of "stressed" requests before considering a crawl as under medium stress||CrawlerX|
|config.AutoThrottling.ThresholdTimeInMilliseconds||The number of elapsed milliseconds in response time that would consider the response "stressed"||CrawlerX|
|config.AutoThrottling.MinAdjustmentWaitTimeInSecs||The minimum number of seconds since the last throttled request to wait before attempting to check/adjust throttling again. We want to give the last adjustment a chance to work before adjusting again.||CrawlerX|
|config.Decelerator.ConcurrentSiteCrawlsDecrement||The number to decrement the MaxConcurrentSiteCrawls for each call the the SlowDown() method. This deals with site crawl concurrency, NOT the number of concurrent http requests to a single site crawl.||ParallelCrawlerEngine|
|config.Decelerator.ConcurrentRequestDecrement||The number to decrement the MaxConcurrentThreads for each call the the SlowDown() method. This deals with the number of concurrent http requests for a single crawl.||CrawlerX|
|config.Decelerator.DelayIncrementInMilliseconds||If there is a configured (manual or programatically determined) delay in between requests to a site, this is the amount of milliseconds to add to that configured value on every call to the SlowDown() method||CrawlerX|
|config.Decelerator.MaxDelayInMilliseconds||The maximum value the delay can be.||CrawlerX|
|config.Decelerator.ConcurrentSiteCrawlsMin||The minimum amount of concurrent site crawls to allow no matter how many calls to the SlowDown() method.||ParallelCrawlerEngine|
|config.Decelerator.ConcurrentRequestMin||The minimum amount of concurrent http requests to a single site no matter how many calls to the SlowDown() method.||CrawlerX|
Like other Abot/AbotX features, you can configure this through code or xml. To see the full AbotX xml config examples click here.