My in laws had neighbors like that. They would let their kids play on the trampoline and swim in the pool whenever even if no one was home. They had to go over there and tell them point blank that no one can play in their yard or anything when no one is home especially without an invitation. They said it was just too much of a liability. Sometimes you have to straight to the point.