Right, my bad; I know they claim to not store _all_ prompts and responses, that was a bit cynical/hyperbolical of me. However, regulation could force them to do it — with all the downsides that come with that.

Your privacy argument, on the other hand, makes total sense. Not even security measures like homomorphic encryption or locality-sensitive hashing could fix that basic issue.