In 1525, Anne married her first husband, Edward Grey, 3rd Baron Grey of Powys (1503–1551). The marriage was not successful, and Anne was unhappy. In 1537, she left him for a lover, Randal Haworth, after Grey had already taken a mistress, Jane Orwell, by whom he had a total of four illegitimate children, and upon whom he would later entail the greater part of his estates comprising the barony of Powys. Her father, with the assistance of Thomas Cromwell, forced Grey to support her, and succeeded in obtaining for her an annuity of £100 . He also asked Cromwell to help discipline Anne so that she would "live after such an honest sort as shall be to her honor and mine". In 1540, Grey petitioned the Privy Council to punish Anne for adultery, and also accused Anne and Haworth of conspiring to murder him. No action was taken against Anne, and she remained with her lover; however, this scandalous arrangement caused her to be excluded from Charles Brandon's will, which contained generous bequests to her two half-sisters, Frances and Eleanor.
Sometime between 1545 and 1551, Anne conspired with a corrupt judge in the Court of Chancery to obtain lands with forged documents, which defrauded Henry Grey, Marquess of Dorset, the husband of her half-sister, Frances, who quickly succeeded the Brandons' father as Duke of Suffolk of a new creation. When the affair came to light in 1552, the judge was arrested, but Anne Brandon went unpunished. By that time, her first husband had died, and she had married Randal Haworth.
Both her marriages were childless, and Anne died in 1558. She was buried in Westminster Abbey or in the adjacent St Margaret's Church.