There's a difference between talking and awkward, unwanted, sexist advances. If you can't tell the difference, then you're right, you probably shouldn't tell anyone off. But just because the victim doesn't say something doesn't mean they don't want something to be said.