"Larry Harvey was also bisexual, and unlike many other leading men who feared it might damage their careers, never hid that fact. Though many rumors circulated that his long-time lover was British film producer James Woolf, I never asked Larry about his personal life, taking the view that ignorance was my best line of defense if I was ever quizzed on the subject."
- Jerry Pam, Laurence Harvey's publicist in his memoirs.
In his account of being Frank Sinatra's valet, Mr. S: My Life with Frank Sinatra (2003), George Jacobs writes that Harvey often made passes at him while visiting Sinatra. According to Jacobs, Sinatra was aware of Harvey's sexuality. In his autobiography Close Up (2004), British actor John Fraser claimed Harvey was gay and that his long-term lover was Harvey's manager James Woolf, who had cast Harvey in several of the films he produced in the 1950s.