In 1964, she embarked on a decade-long extramarital affair with the entrepreneur Sir James Goldsmith, a member of the prominent Goldsmith family. Attracted by his flashiness, she said, she was "Mad about Jimmy. Loved Mark desperately." Though both she and Goldsmith, who was then married to his second wife Ginette Lery, believed that the affair would be a passing fling, it soon gained her notoriety in London's gossip columns as a modern mistress. For years, she repeatedly refused Goldsmith's suggestion of starting a family, stating, "I didn't want to hurt Mark any more than I had." She was eventually coaxed into having his children by their mutual friend John Aspinall, an estranged former friend of Mark Birley who introduced her to Goldsmith.
While still legally married to Birley, she gave birth to Jemima (b. 30 January 1974) and Zac (b. 20 January 1975). Her last child Ben Goldsmith was born on 28 October 1980 at 46, after two consecutive miscarriages. In 1978, Goldsmith and Lady Annabel married solely to legitimise their children. "We didn't marry as a great act of passion. More to make sure that the children's name would be Goldsmith when they went to school", she explained.