As controversy continues to swirl around Serena Williams’s U.S. Open loss, umpires are reportedly considering boycotting the tennis star’s matches.

After losing 6-2, 6-4, to Naomi Osaka at the U.S. Open championship Sunday, Williams accused the chair umpire, Carlos Ramos, of sexism. He had given her a warning for coaching, then a point penalty for smashing her racket and, after she repeatedly expressed frustration, including calling him a “thief,” Ramos levied the game penalty for verbal abuse. After the match, Williams said that Ramos would not have treated a male player in the same way.

U.S. Tennis Association President Katrina Adams issued a statement Saturday evening praising Williams as a “true champion” and an “inspiration,” and said Tuesday that a double standard exists in how male and female players are treated by chair umpires. The CEO of the Women’s Tennis Association, Steve Simon, also quickly issued a statement of support for Serena, saying Sunday that his organization “believes that there should be no difference in the standards of tolerance provided to the emotions expressed by men vs. women” and that it did “not believe that this was done last night.”

Now, stung by what they perceive as a lack of institutional support for Ramos, other umpires are reportedly discussing the possibility of boycotting Williams’s matches. Top umpires are also considering the formation of a union, according to a report Tuesday, in part because they are not allowed to discuss specific matches.

‘Thrown to the wolves’

The International Tennis Federation, the sport’s governing body, came to Ramos’s defense Monday, saying that his “decisions were in accordance with the relevant rules” and that he “acted at all times with professionalism and integrity.” However, that show of support reportedly was too slow in the making to mollify the umpires’ colleagues.

A report by The Times of London cited an anonymous official who claimed that umpires felt they were frequently “not supported” by the USTA and that Ramos was “thrown to the wolves for simply doing his job and was not willing to be abused for it.” That has led to talk of a boycott, with the Guardian reporting that umpires feel Ramos was “hung out to dry” while “no one is standing up for officials,” lending renewed energy to long-standing discussions about unionizing.

“Umpires don’t have any independent means of representation and are employed by the governing bodies,” a source told The Guardian. “If talking to the media is not allowed, and governing bodies are speaking out against them, what are umpires supposed to do?”

“The umpiring fraternity is thoroughly disturbed at being abandoned by the WTA,” Richard Ings, a former top-level umpire who is now retired, told ESPN on Tuesday. “They are all fearful that they could be the next Ramos. They feel that no one has their back when they have to make unpopular calls.”

Ramos himself did offer some comments to a newspaper in his native Portugal, saying (per USA Today), “I’m fine, given the circumstances,” and “It’s a delicate situation, but ‘a la carte’ arbitration does not exist. Do not worry about me!”

Ramos, who was selected Tuesday by the ITF to umpire Davis Cup semifinal matches this week between the United States and Croatia, has also received some support from major individual figures in the world of tennis. Sportscaster and former pro Mary Carillo described him Monday as “very, very respected,” and she said that Williams occasionally “acts like a bully.”

Martina Navratilova agreed that Williams deserved much of the blame for the incident, writing in a New York Times op-ed piece that, in complaining that Ramos would not have reacted the same way to an argumentative male player, Williams was “missing the point” and would have been better served conducting herself with “respect for the sport we love so dearly.” The 18-time Grand Slam winner said that “we cannot measure ourselves by what we think we should also be able to get away with,” adding, “In fact, this is the sort of behavior that no one should be engaging in on the court.”

Other tennis luminaries, though, including Chris Evert and Novak Djokovic, said that Ramos should have exercised better discretion in his handling of the heated situation. In a guest column last week for The Washington Post, women’s rights activist and former world No. 1 Billie Jean King said: “Did Ramos treat Williams differently than male players have been treated? I think he did.”

Williams was fined $17,000 Sunday by the tournament referee’s office for her coaching, racket destruction and verbal abuse infractions. She won $1.85 million as the runner-up to Osaka, who notched the first Grand Slam title of her career and became the first Japanese tennis player to do so.

‘A blow to all girls’: Why the latest decision in the USWNT’s equal pay lawsuit matters

A judge Friday dismissed the U.S. women’s team’s claim of gender discrimination

Women runners were ‘falling through the cracks.’ Then Mary Cain spoke out.

Now the sport is facing a reckoning