You don't have to stay in the village. The US basketball team isn't, so I'm pretty sure some of the minors don't stay there. Coaches can stay because I saw a picture of the US swim team having lunch with the coaches in the dining hall.