You find a way to cut ties because it's obvious the temptation can't be resisted.

OR you both decide being unhappy with your partner and living that lie with them isn't what they deserve, so you decide to face scrutiny, divorce, and whatnot to be with someone who could make you happy. Even if it means they all will hate you for either awhile or always.