That's a tough situation. I don't think I could hear something like that and then forget about it. I'd look at the person differently and I know it would affect the way I interacted with them. So maybe it would be best to just address it directly rather than pretending everything's okay while I'm secretly waiting for this person to out himself as a racist.