Grey married Mary Elizabeth Ponsonby (1776 – 1861), only daughter of William Ponsonby, 1st Baron Ponsonby and Hon. Louisa Molesworth in 1794. The marriage was a fruitful one; between 1796 and 1819 the couple had ten sons and six daughters:
[a dau.] Grey (stillborn, 1796)
Lady Louisa Elizabeth Grey (7 April 1797 – 26 November 1841); married John Lambton, 1st Earl of Durham, he of the Durham report
Lady Elizabeth Grey (10 July 1798 – 8 November 1880); married John Crocker Bulteel (d. 10 September 1843). Their daughter, Louisa Emily Charlotte Bulteel, is one of the great-great-great-grandmothers of Diana, Princess of Wales, through her marriage with Edward Baring, 1st Baron Revelstoke
Lady Caroline Grey (30 August 1799 – 28 April 1875); married Capt. The Hon. George Barrington
Lady Georgiana Grey (17 February 1801 – 1900); never married
Henry George Grey, 3rd Earl Grey (28 December 1802 – 9 October 1894), eldest son, who became a politician like his father
General Sir Charles Grey (15 March 1804 – 31 March 1870), father of Albert Grey, 4th Earl Grey
Admiral Sir Frederick William Grey (23 August 1805 – 2 May 1878)
Lady Mary Grey (2 May 1807 – 6 July 1884); married Charles Wood, 1st Viscount Halifax
Hon. William Grey (13 May 1808 – 11 Feb 1815)
Admiral The Hon. George Grey (16 May 1809 – 3 October 1891)
Hon. Thomas Grey (29 Dec 1810 – 8 Jul 1826)
Rev. Hon. John Grey (2 March 1812 – 11 November 1895)
Rev. Hon. Sir Francis Richard Grey (31 March 1813 – 22 March 1890) married Lady Elizabeth Howard (1816–1891), daughter of George Howard, 6th Earl of Carlisle and Georgiana Cavendish (daughter of Georgiana Cavendish, Duchess of Devonshire).
Hon. Henry Cavendish Grey (16 October 1814 – 5 September 1880)
Hon. William George Grey (15 February 1819 – 19 December 1865)