He married Duchesse Louise Marie Adélaïde de Bourbon on 5 April 1769.
It is during the summer of 1772, a few months after his wife had given birth to a stillborn daughter, that began Philippe's secret liaison with one of her ladies-in-waiting, Stéphanie Félicité Ducrest de St-Albin, comtesse de Genlis, the niece of Madame de Montesson, the morganatic wife of Philippe's father. Passionate at first, the liaison cooled within a few months and, by the spring of 1773, was reported to be "dead". After the romantic affair was over, Félicité remained in the service of Marie-Adélaïde at the Palais-Royal, a trusted friend to both Marie-Adélaïde and Philippe. They both appreciated her intelligence and, in July 1779, she became the governess of the couple's twin daughters born in 1777.