Ah, this is where you need the all-purpose rejoinder, "what's it to you?" That way you don't need to quickly come up with The Clever Retort, and it puts them on the defensive, because they then have to justify, if even to themselves, why they should be cranked enough about a random person going on with your little life to make an uncalled for snarky comment.

I like to say it with puzzled expression, to drill in bizarreness of making such a comment. It leaves the rude person spluttering every time.