What looked like an uncomfortable on-air exchange with Kelly Ripa in 2006 turned into a moment Clay Aiken says had lasting consequences. Nearly two decades later, he’s opening up about the fallout and how it affected his life and career. The interview revisits the infamous appearance that made headlines at the time, including the media attention that followed and the personal challenges Aiken says came afterward.

Clay Aiken is reflecting on a tense moment with Kelly Ripa two decades later.

The American Idol alum, 47, appeared on the Hollywood Raw podcast on Wednesday, June 24, where he was asked about his guest co-host gig with Ripa in 2006 during her Live with Regis and Kelly days.

During the episode, Aiken and Ripa, now 55, had an uncomfortable exchange when he placed his hand over her mouth to keep her from talking after she asked their guests a series of questions without allowing him to chime in.

“Is that a no-no?” Aiken asked at the time.

“Oh, that’s a no, no, no,” Ripa responded before adding, “I don’t know where that hand’s been, honey.”

“There was a lack of respect, and when you’re a certain person, you’re in the public eye, you have to be respectful of other people. And I don’t think that he was respectful in any way,” she said at the time. “If that upsets his fans, I’m sorry to hear that, but you don’t put your hands over somebody’s face and mouth when they’re conducting an interview, even if it’s for a laugh. And that’s all I’m gonna say.”

Later, The View covered the exchange with then-co-host Rosie O’Donnell calling Ripa’s response “homophobic.” Ripa subsequently called into The View during the broadcast to defend herself against O’Donnell’s “outrageous” remarks, saying, “I have three kids. He’s shaking hands with everybody in the audience. I mean, it’s cold and flu season. That’s what I meant. To imply that it’s anything homophobic is outrageous, Rosie, and you know better. You should be more responsible.”

Now, 20 years later, Aiken called the situation a “huge blowup,” adding it was “probably the most catastrophic week of my life, to be honest.”

Adding that while he wasn’t the one who had the problem, Aiken noted that since the incident, he has seen Ripa at several events but they’ve never spoken about the fallout.

“Maybe I should feel bad that I have not gone up and said something myself, but she hasn’t either,” he noted, sharing that he has vivid memories of that time “because I was the one with the scars.”

The singer said he was offered the guest co-host role at the time because his team was in talks with Buena Vista about him possibly hosting his own show.

Related Stories

Clay Aiken Recalls Simon Cowell Saying He Didn’t Look ‘Like a Pop Star’ During ‘American Idol’ Audition: ‘He Said I Was Ugly’

What Happened to Clay Aiken After ‘American Idol’? All About His Life 23 Years After Almost Winning the Competition Show

“I’m on the show because I wanted to show them I could handle this myself; I could host something and I wasn’t getting to talk,” Aiken explained. “Like, there were cue cards. They had my name on them. So I tried to be funny, and I did what I did, which was incredibly innocent. And then it got a little cold in the room.”

After thinking it had blown over, Aiken then saw Ripa’s discussion with Philbin on the show the following week, which lasted several minutes during the live broadcast. But it was the exchange on The View that became the most upsetting to the musician, who had not yet come out as gay publicly.

“I was not out at the time. I was to Rosie … I was out to people who I knew, but I wasn’t publicly out about it,” Aiken said. “I was not out to my grandparents and the people in my family. And not only does Rosie call it a homophobic remark — which I think she accidentally said, and then she tried to walk it back a little bit — but then Kelly calls in live to The View that day and they get in an argument over it. Not only did she out me, but they had a big argument about whether I was gay and who talked about me being gay.”

Aiken, who publicly came out on the cover of PEOPLE in 2008, said that the experience of his sexuality being discussed on broadcast television was upsetting to him. “I was caving in on myself,” he recalled. “It was a lot.”

After poking fun at the exchange with Tori Spelling at the American Music Awards shortly after and his team locating and releasing several clips of Ripa putting her own hand over Philbin’s mouth, Aiken then sent flowers to Ripa to smooth things over. He said she later called him to thank him.

“She apologized for it becoming blown out of proportion and everything,” he said. “She said, ‘I’d love to come back on the show anytime, etcetera, etcetera.’ But I do know we tried [to come back], and we were not allowed. So I just haven’t tried since. It’s not a big deal. I mean, I honestly have not spoken about it until right now.”

Reps for Ripa and O’Donnell did not respond to PEOPLE’s requests for comment.