I've not written a crawler before, but did something similar. I needed to mirror websites ala `wget -r` and there doesn't seem to be a tool or library that aside from wget that does it, so I translated the wget -r algorithm by reading the source, as best as I could, in Go. It's not parallelised or anything as that looked complicated, but was handy when integrating it into a backend project that needed that functionality. Was a fun learning experience and I found it a bit of a complex project due to interpreting the links in HTML, so I imagine doing a crawler is even more difficult. Also found Go HTML parser not that great.