On 9 February 1940, Lord Harlech married Sylvia Thomas, daughter of Hugh Lloyd Thomas and Hon Guendaline Ada Bellew. Before Lady Harlech's death in an automobile accident on 30 May 1967, they had five children:
Hon Julian Hugh Ormsby-Gore (1940–1974), who died of gunshot wounds, an apparent suicide.
Hon Jane Teresa Denyse Ormsby-Gore (b. 1942), who was said to have had an affair with Mick Jagger during the 1960s; some consider the Rolling Stones song "Lady Jane" to be about her.
Hon Victoria Mary Ormsby-Gore (b. 1946)
Hon Alice Magdalen Sarah Ormsby-Gore (1952–1995), who became engaged to rock guitarist Eric Clapton in 1969. She and Clapton lived together for five years, but did not marry. She subsequently died of a heroin overdose in 1995, on the day before her 43rd birthday.
Francis David Ormsby-Gore, 6th Baron Harlech (1954–2016), who married Amanda Jane Grieve (b. 1959), daughter of Alan Grieve.