Why is this even an issue? You didn't see anything and it's not how you know him. Just go about your day. Maybe it's true, maybe it's not, but either way, unless he acts like that around you why worry about it?