From March to August 1812, Lady Caroline embarked on a well-publicized affair with Lord Byron. He was 24 and she 26. She had spurned the attention of the poet on their first meeting, subsequently giving Byron what became his lasting epitaph when she described him as "mad, bad, and dangerous to know." His response was to pursue her passionately.
Lady Caroline and Lord Byron publicly decried each other as they privately pledged their love over the following months. Byron referred to Lamb by the hypocorism "Caro", which she adopted as her public nickname. After Byron broke things off, her husband took the disgraced and desolate Lady Caroline to Ireland. The distance did not cool Lady Caroline's interest in the poet; she and Byron corresponded constantly during her exile. When Lady Caroline returned to London in 1813; however, Byron made it clear he had no intention of restarting their relationship. This spurred increasingly public attempts to reunite with her former lover.
Lady Caroline's obsession with Byron would define much of her later life, as well as influence both her and Byron's works. They would write poems in the style of each other, about each other, and even embed overt messages to one another in their verse. After a thwarted visit to Byron's home, Lady Caroline wrote "Remember Me!" into the flyleaf of one of Byron's books. He responded with the hate poem: "Remember thee! Remember thee!; Till Lethe quench life's burning stream; Remorse and shame shall cling to thee, And haunt thee like a feverish dream! Remember thee! Ay, doubt it not. Thy husband too shall think of thee! By neither shalt thou be forgot, Thou false to him, thou fiend to me!"
Her cousin Harriet (by then Lady Granville), with whom Lady Caroline's relationship had deteriorated after childhood, visited her in December 1816 and was so incredulous at her unrepentant behavior that she ended her description of the visit in a letter to her sister with: "I mean my visit to be annual."