"We are disposable to Dicko," Holden told the Herald Sun.
"The reality is he will dispose of us and drop us the minute something better comes along.
"Coming back has definitely not helped. In fact I think it has hurt the show."
Almost every incarnation of Idol features the standard three-judge panel, however the Australian version allowed Dickson to return as a fourth judge.
"I am not loving the four judges," Holden continued.
"I am all for formats being bent, but in this particular case, I think three judges are right."
There's no doubting that Dickson's return has heightened tensions on the judge's panel.
Ratings are also at a five-year low for the reality series.
Last week, Marcia Hines threatened to quit after breaking down live to air.
The former Queen of Pop lost her cool in a fist-pounding burst, declaring she "had the s****" with fellow judge Mark Holden over his harsh criticism of contestant Ben McKenzie.
This is another blow to the most troubled series of Idol yet.
Questions have already been raised over the credibility of the voting system.
Three of the final six hopefuls are member of the controversial Hillsong church, who have been accused of “vote-stacking” the phone lines.
Obviously fearing the claims, Channel 10 has since placed a ban on all religious questions at Idol public appearances.
"[Channel 10 bosses] have promised the Idols they wouldn't have to talk about Christianity," a spokeswoman for Channel Ten told The Sydney Morning Herald.
"They've all been really upset by the stories that are floating around. The interview can't go ahead if you ask them about religion," she said.