In 1265, his marriage to Teresa Gil de Vidaure began breaking apart. James took another mistress, his cousin Berenguela Alfonso. The pope admonished him severely for this incestuous relationship. The king wished to end the marriage and repudiated Vidaure on the pretext that she had contracted leprosy. Both wrote to Clement IV regarding the annulment of the union; the king to request it and his wife to prevent it. The pope ruled that, although it was not sanctioned by the Church, the marriage was consummated and thus indissoluble. He sharply chastised the king for even requesting the annulment.
Clement IV died in 1268, and Andrés Albalat, bishop of Valencia, declared the marriage null and void. Clement IV's successor, Pope Gregory X, allowed the case to be treated in Rome and confirmed his predecessor's decision. James attempted procuring an annulment again in 1275, this time claiming that he had had sexual relations with Vidaure's cousin before their marriage (which would have rendered the marriage invalid), but the argument was no more successful than the previous one.