Lady Harriet Mordaunt was married to Sir Charles Mordaunt, but when, in June 1868, he prepared for an annual fishing trip to Norway, she encouraged him to go on his own. Arrangements were made for her to remain at Walton Hall in the company of a sister and another lady. However, when Sir Charles returned early from Norway, he found his wife alone. One of her maids later testified that, during Sir Charles's absence, Lady Mordaunt had been visited in London by Viscount Cole, who, after dinner, had "remained alone with her until a very late hour"; on another occasion, he had travelled with her by train from Paddington station to Reading, where he alighted from a carriage of which they had been the only occupants. Other servants, who seem to have resented Lady Mordaunt's behaviour, added their own accounts of Lord Cole’s visits.
Parliament rose on 28 July 1868, and in November Sir Charles did not stand for re-election. The Liberal Party swept to power at the first general election since Disraeli's Reform Act of 1867. By then, Lady Mordaunt was five months pregnant. On 28 February 1869 she gave birth prematurely to a daughter, Violet Caroline. The timing was significant, in view of Sir Charles's absence on a fishing trip the previous year. Doctors initially feared that the child might be blind, causing Lady Mordaunt to become hysterical, imagining that this had been brought about by a hereditary sexually transmitted disease. (At the time, gossip surrounding Freddy Johnstone, a close friend of the Prince of Wales, whom Lady Mordaunt shortly afterwards claimed to have been one of her lovers, was that he suffered from such a disease.) Violet’s eye infection was successfully treated and no venereal infection was found in either mother or child. However, following this episode, Lady Mordaunt not only declared to her husband, "Charlie, I have deceived you; the child is not yours; it’s Lord Cole’s", but claimed to have committed adultery with "Lord Cole, Sir Frederick Johnstone, the Prince of Wales and others, often and in open day" As one of Princess Alexandra’s biographers put it, "the ensuing scandal was immense".
In 1870 he was cited as one of two co-respondents in the case for divorce brought by Sir Charles Mordaunt, 10th Bt., a former M.P., against his wife, Harriet, in which Prince Albert Edward, The Prince of Wales (later King Edward VII), was called to give evidence. The divorce was denied as Lady Mordaunt was judged to be insane, but was finally granted in 1875 when Cole did not contest the claim that he was the father of Lady Mordaunt's daughter, Violet (1869–1928), later Marchioness of Bath.