He later married his model Emma Hill, who appears in many of his paintings and is the wife in The Last of England. His son with Emma, Oliver Madox Brown (1855–1874) showed promise both as an artist and poet, but died of blood poisoning. Their daughter, Catherine, married Francis Hueffer; through Catherine, Brown was the grandfather of novelist Ford Madox Ford and great-grandfather of Labour Home Secretary Frank Soskice.