They're doing it because a) they're jerks and b) no one has told them not to.  With these kind of people, nothing will ever get done until you say something.  And no matter how polite you are, they will be offended.  If you want your pool back, say something!