This post has moved to Building a Polite & Fast Web Crawler.