Royster and Poe were neighbors in Richmond, Virginia when they began their relationship in 1825, when she was 15 years old and he was 16. They discussed marriage, though Royster`s father vocally disapproved. They were secretly engaged as Poe began classes at the University of Virginia in 1826; however, Royster`s father intercepted and destroyed all of Poe`s letters to his daughter. Royster wrote later that his disapproval was only because of their young age but he likely also considered Poe unsuitable due to social and financial status as a poor orphan.
Poe and Royster would meet again in July 1848, over a year after the death of Poe`s wife Virginia Clemm. His visit was unannounced. As Royster described it: "I was ready to go to church and a servant told me that a gentleman in the parlor wanted to see me. I went down and was amazed to see him—but knew him instantly". By this time, Royster was very religious, having been baptized as an adult at St. John`s Episcopal Church. She was 39 and living with her 19-year old daughter Ann and 10-year old son Southall.A friend described her as being very attractive around this time:
"Her eyes were a deep blue, her hair brown, touched with grey, her nose thin and patrician... Her voice was very low, soft and sweet, her manners exquisitely refined, and intellectually she was a woman of education and force of character. Her distinguishing qualities were gentleness and womanliness."
Royster attended Poe`s lecture in Richmond, sitting in the front row. Poe and Royster rekindled their relationship somewhat and discussed marriage. Her children disapproved, however, and her dead husband`s will stipulated that remarriage would remove three-quarters of her estate. Poe visited Richmond on September 17, 1849, and stayed with Royster for the evening. He wrote: "I think she loves me more devotedly than any one I ever knew... I cannot help loving her in return." Poe hoped to be married before he left Richmond and pushed her to respond. She wanted time to consider: "I told him if he would not take a positive denial he must give me time to consider it". Royster may have been reticent because of the rumors of Poe`s drinking and, because of this, may have inspired Poe into joining the Richmond chapter of the Sons of Temperance. Poe`s lecture tour then brought him to Norfolk, Virginia and Old Point Comfort. It is unclear if the couple was ever officially engaged but most biographers agree that they came to an "understanding" by late September
The wedding never took place; after Poe said goodbye to her, he left Richmond on September 27, 1849, and died mysteriously only two weeks later in Baltimore. Royster recalled her last moments with him: "He came up to my house on the evening of the 26th Sept. to take leave of me.–He was very sad, and complained of being quite sick... I felt so wretched about him all of that night, that I went up early the next morning to enquire after him, when, much to my regret, he had left in the boat for Baltimore." On his deathbed, Poe mentioned a wife he had in Richmond, possibly referring to Royster. Biographer John Evangelist Walsh suggests that Royster`s brothers were responsible for Poe`s mysterious death.
Royster later said that she would not "have married him under any circumstances". A letter Royster wrote to Poe`s mother-in-law Maria Clemm, however announced that she was ready to accept her as her own mother-in-law. In her letter, she also referred to Poe as "the dearest object on earth" to her.