Yes, this is similar to how I would like to handle issues: just an issues directory full of yaml-markdown files. That way, issues are a part of the same history as the code. They can be specific to a branch, they can diverge as different solutions are attempted in parallel, and they can be deleted and retrieved after resolution.

I've actually been fine with to-do lines in files. Have a metric for the number you have, and don't abuse it, but they are easy to find and literally next to the code they relate to.