When a guy blames his precious 'self-esteem', then it's out the door for me. There's just not such excuse to be honest, so I never buy crap on emotional or mental reasons etc.
I'm sorry however about what happened. You dealt with it just fine. A jerk never knows he's a jerk..