This is such a great question. I've been getting dirty looks from women in the bathroom since my son was just 3! I'd like to hear their thoughts on what I might do instead? Are they volunteering to wait outside the bathroom with him while I go? Because he's too young to wait for me alone. Find a strange man to take him into the mens room so he can go? Because, he's still having issues getting his pants up and down on his own, and get on and off those giant toilets by himself.
As was already said, women's bathrooms are all stalls anyway, and a little boy isn't going to see anything questionable, unless you've got your pants down outside the stall, in which case, not my problem to deal with. Also, I still walk around naked in front of my son (he's now 4), which I know is pushing the boundaries, but I'm not about to start shaming my body or his, or anyone else's for that matter. Even if he sees someone sitting on the toilet, he's not liable to see as much as he does when I walk from the shower to the bedroom every morning. So, what's the issue with a little boy being in the women's restroom?
At my son's swim school, there are men's and women's locker rooms for changing. There's a sign that clearly states that their cut off is age 6. Boys over age 6 change alone in the men's room, girls over age six change alone in the women's room (if accompanied by a parent of the opposite sex). I think that's a good age to have as a target. Old enough to properly dress alone, old enough to know not to go anywhere or do anything with a stranger. It depends heavily on the maturity of the child too, so if your 4 year old is capable of all this, my hat is off to you. We, however, will be going to the restroom together for a while yet.