On 24 October 1931, he married Lady May Cambridge, at St. Mary's Church, Balcombe, Sussex. She was born Princess May of Teck, and was a niece of King George V and Queen Mary, and a great-granddaughter of Queen Victoria. They had three children:
Anne Mary Sibylla Abel Smith (born 28 July 1932); she married David Liddell-Grainger (26 January 1930 – 12 March 2007) on 14 December 1957; they were divorced in 1981 but have five children and eight grandchildren. Her eldest son is the Conservative Member of Parliament Ian Liddell-Grainger.
Colonel Richard Francis Abel Smith (11 October 1933–23 December 2004); he married Marcia Kendrew (born 27 March 1940) on 28 April 1960; they have one daughter and four grandchildren.
Elizabeth Alice Abel Smith (born 5 September 1936); she married Peter Wise (born 29 December 1929) on 29 April 1965; they divorced in 1975 and had one daughter who died as an infant from a barbiturate overdose.
Sir Henry was appointed Governor of Queensland from 18 March 1958, serving until 18 March 1966. He was highly respected in the office of governor. People lined the streets as a mark of high esteem to farewell him and Lady May Abel Smith on their final day in office on their way to Brisbane Airport.
Sir Henry died at Barton Lodge, Winkfield, Berkshire, on 24 January 1993. His funeral service took place at St. George's Chapel, Windsor Castle. His cremated remains were buried at the Royal Burial Ground, Frogmore. His wife survived him by only sixteen months.