Harriet had numerous lovers during her marriage: as she once remarked "I can never love a little". Among her more notable lovers were Richard Brinsley Sheridan, the playwright and Whig politician, and Granville Leveson-Gower, 1st Earl Granville, who became her most enduring lover.
Her affair with Granville produced two illegitimate children: Harriet Emma Arundel Stewart, wife of George Godolphin Osborne, 8th Duke of Leeds (though she died in 1852 before he succeeded to the title) and George Stewart, yet somehow Harriet managed to hide her pregnancies from her husband. Granville would later on marry Harriet's niece, Lady Harriet Cavendish ('Harryo'). As Harriet sadly remarked, for seventeen years she had "loved to idolatry" the man who she came to believe loved her least of all the men in her life.