Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Just one IP for now. You are right to worry about being blocked from crawling however, it has happened to me already on a few sites. The key things to help mitigate against this are:

1. Always identify your crawler via a consistent user-agent string, that explains its a web search crawler and not a generic web browser.

2. Always obey the directives in robots.txt.

3. Make sure your crawler is not too aggressive (low frequency of requests).

(updated for formatting)



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: