Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
Geezus_42
3 months ago
|
parent
|
context
|
favorite
| on:
So You Want to Define a Well-Known URI
> add a robots.txt
Which bots will then ignore.
Ferret7446
89 days ago
[–]
I didn't expect I'd get to use this so soon. Obligatory canned advisory:
https://www.felesatra.moe/blog/2026/06/19/robots
Geezus_42
89 days ago
|
parent
[–]
They still ignore it and scan everything even though you have told them not to scan certain paths.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
Which bots will then ignore.