RELATED: The Palace Has Issued This “Warning Shot” to Harry & Meghan, Say Sources. According to the Express, Charles finds Prince Andrew and his ex-wife, Sarah Ferguson’s living arrangements at Royal Lodge “outrageous” and a major embarrassment to the Royal Family now that Virginia Roberts Giuffre is suing the Duke of York in a New York court for sexual assault and battery. The Prince of Wales has never approved of Andrew and Sarah’s living arrangements, which are unconventional by royal standards. But he believes their relationship is an even bigger problem for the Royal Family at a time when the House of Windsor is engulfed in such a sensational scandal. The Duke and Duchess of York, who divorced in 1996, still live together at the 30-room mansion, just three miles away from Queen Elizabeth at Windsor Castle. Sources claim Charles has been angry for some time about Sarah living at the royal property as a permanent resident. One insider told the Express: “Andrew and Sarah might try to argue that she is only a guest, but the reality is that she lives at Royal Lodge and has done so for some time.” A royal source told Best Life: “This new lawsuit, which truly does imperil the Royal Family, has finally brought that conversation to a head. Charles is outraged it’s taken this long, but he knows that something drastic needs to happen now.” RELATED: The One Promise the Queen Made That Prince Charles Could Break, Sources Say. Andrew took over Royal Lodge, the former home of the Queen Mother, in 2004 and retains a 75-year lease on the property. He and his then-wife Sarah raised their two daughters, Princess Beatrice and Princess Eugenie, in the house. Now that both Beatrice and Eugenie are married with their own families (Beatrice is expecting her first child with Edoardo Mapelli Mozzi; Eugenie and Jack Brooksbank have a six-month-old son, August), Andrew and Sarah are the sole residents of the Lodge. “The general consensus is that because Andrew is no longer a working royal and the princesses have both moved out, it’s time to make a change regardless,” the insider told Best Life. “The Duke and Duchess of York need to make other arrangements.” Royal expert Dan Wootton quoted a source in the Daily Mail as saying: “The ongoing personal scandal involving Prince Andrew and Jeffrey Epstein is a source of much heartache for Charles and [Prince] William who are trying to protect the future of the monarchy.“ae0fcc31ae342fd3a1346ebb1f342fcb The insider also revealed there may be another royal ready to move into Royal Lodge once Andrew moves out. “Prince William is thought to have expressed an interest on taking over the Lodge at some point in the future and it’s unlikely that Charles would stand in his way,” the source said. It’s not clear where Andrew and Sarah would live if they were forced to move. The Queen gifted Sunninghill Park in Ascot to the couple as a wedding present in 1986, but the Duke of York sold the mansion in 2007 to oil tycoon Timur Kulibayev. And for more royals news sent right to your inbox, sign up for our daily newsletter. While promoting her first novel, Her Heart For a Compass, which was released earlier this month, Sarah has been giving interviews to multiple news outlets. She chose to conduct her interview with Town and Country at Royal Lodge, where writer Andrew Goldman set the stage: “It’s a spring afternoon in England, and Ferguson is perched in a corner of Royal Lodge, the sprawling cottage orné built in 1662 that was once inhabited by King George IV and that she has shared with her ex-husband, Prince Andrew, since 2008.” Sarah moved throughout the house during the interview, even taking Goldman into a bedroom during their conversation. “The whole idea that Fergie would be using Royal Lodge to promote her book and be depicted as ’lady of the manor’ is a mistake,” one royal source told Best Life. “[Andrew and Sarah’s] relationship has always been the source of concern and criticism among the senior royals. Prince Philip refused to be in the same house with her. Andrew and Sarah have always relied on the great affection the Queen has for them both to protect them, but Charles understands in their current situation they are a liability to the very future of the monarchy.” RELATED: This Royal’s Support of Harry & Meghan Could Be a “Game Changer,” Say Sources. Diane Clehane is a New York-based journalist and author of Imagining Diana and Diana: The Secrets of Her Style.