In 1818, Buchanan met Anne Caroline Coleman at a grand ball in the White Swan Inn, Lancaster, and the two began courting. Anne was the daughter of the wealthy iron manufacturing businessman (and protective father) Robert Coleman and sister-in-law of Philadelphia judge Joseph Hemphill, one of Buchanan's colleagues from the House of Representatives. By 1819, the two were engaged, although both were beyond marrying age for the time. Buchanan spent little time with her during the courtship: he was extremely busy with his law firm and political projects during the Panic of 1819, which took him away from Coleman for weeks at a time. Conflicting rumors abounded, suggesting that he was marrying her for her money, because his own family was less affluent, or that he was involved with other women. Buchanan never publicly spoke of his motives or feelings, but letters from Anne revealed she was paying heed to the rumors.
After Buchanan paid a visit to the wife of a friend, Coleman broke off the engagement. She died suddenly soon afterward, on December 9, 1819. The records of a Dr. Chapman, who looked after her in her final hours, and who said just after her death that this was "the first instance he ever knew of hysteria producing death", reveal that he theorized, despite the absence of any valid evidence, the woman's demise was caused by an overdose of laudanum, a concentrated tincture of opium. Buchanan was prevented from attending the funeral service. In a letter to her father he wrote however, that "I feel happiness has fled from me forever."
After Coleman's death, Buchanan never courted any more women or seemed to show any emotional or physical interest. A rumor, without basis, circulated of an affair with the widow of President James K. Polk. It has been suggested that Anne's death in fact served to deflect awkward questions about his sexuality and bachelorhood. While Buchanan may have been asexual or celibate, there are many indicators that suggest he was homosexual. The argument has been put forward by biographer Jean Baker, supported by Shelley Ross.