Gilmore met writer Terry Southern on the set of The Loved One (1965) and abandoned Hollywood in 1966 to live with him in New York then Connecticut where she remained his longtime companion until his death thirty years later.
Not long after arriving in Los Angeles, Southern met a young Canadian-born actress and dancer, Gail Gerber, on the MGM backlot. Gerber was working as a dancer on an Elvis Presley movie, and she also had a non-speaking role in The Loved One. Southern and Gerber soon began an affair. The relationship intensified during July/August 1964, and after Southern's wife and son went back to East Canaan, Southern and Gerber moved in together in a suite at the Chateau Marmont hotel.
Southern and Gail Gerber moved to London in early 1966, when Southern was hired to work on the screenplay of Casino Royale. The episodic "quasi-psychedelic burlesque" proved to be a chaotic production, stitched together from segments variously directed or co-directed by a team that included Joseph McGrath, Robert Parrish, Val Guest, John Huston, Richard Talmadge, and Ken Hughes. Many planned scenes could not be filmed due to the feud between Orson Welles and star Peter Sellers, which climaxed with Sellers walking out during the filming of the casino scenes and refusing to return. Many writers contributed to the screenplay, including Southern (who wrote most of the dialogue for Sellers), Woody Allen, Wolf Mankowitz, Michael Sayers, Frank Buxton, Joseph Heller, Ben Hecht, Mickey Rose, and Billy Wilder.
Southern had been introduced to Robert Fraser by Dennis Hopper, and when he went to London to work on Casino Royale he and Gail became part of Fraser's "jet-set" salon that included the Beatles, the Rolling Stones, photographer Michael Cooper, interior designer Christopher Gibbs, model-actress Anita Pallenberg, filmmaker Nicolas Roeg, painter Francis Bacon, producer Sandy Lieberson, Guinness heir Tara Browne, and model Donyale Luna. Southern became close friends with photographer Michael Cooper, who was part of the Rolling Stones' inner circle and who shot the cover photos for the Beatles' Sgt Pepper's Lonely Hearts Club Band LP.
Although he continued to reside in northern Connecticut "beyond the commuter belt", Southern maintained his social life in New York with diligence; longtime girlfriend Gail Gerber would often chauffeur him to Studio 54 (where he cultivated a convivial acquaintance with co-owner Steve Rubell), parties hosted by George Plimpton, and other engagements.