It's not a red flag. Like someone else said, it's only a red flag if he wouldn't talk about marriage with you at all. However, he's being completely upfront about not wanting to get married.

If it were me, I wouldn't wait around for the next few years hoping he would change his mind. If marriage is something you want, you will probably be better off finding someone who wants the same things as you.