I think he didn't adopt Harvey because Harvey's biological dad (who isn't really in the picture) wouldn't agree.