I would be annoyed too. I don't like that shit. 95% of the time, when someone sees that their ex is seeing someone else, they freak out and try to get back with them. You can't really say that he shouldn't go see her because nothing is official with you guys, but you can definitely tell him you didn't like it.