You are not alone. It took me forever to realize they were related, but I really have no interest in him so there was really no reason for me to make the connection. I actually completely forgot --- again --- until Blurred Lines because the topic came up again.

And I have no idea what song that is and obviously had no idea that was his mother. It all makes perfect sense now.