I agree. In fact, I would argue that open source developer conferences actually have fewer drunken revels than more conventional industry trade shows (which, in my experience, also have more issues with exclusion and sexual harassment). However, that doesn't mean that we can't still try for a better balance.
I've planned quite a few parties. In fact, I used to plan them professionally for the San Francisco Opera, and that's about as high-maintenance as a party gets. I've also helped organize a dozen open source conferences.
My point is not that planning parties is easy, but rather that planning a boring party (i.e. hotel ballroom or bar, bad DJ and some free booze) is easier than planning alternative activities. Scheduling BOFs, game nights, or the kind of party at LinuxCon are all significantly more work than that.
And, in my experience, the majority of parties at confs are, in fact, bad parties which require little planning.