The group have given a deadline of Thursday to arrange a meeting with senior officials - or else they will withdraw support.
Mr Lavery said: “We have become collateral damage. I want to go back to Emmerson. He kept us on board.”
Mr Lavery supplied correspondence to The Telegraph showing how Mr Emmerson had helped him when he was considering suicide last year.
“He was the solitary light in the inquiry and it’s gone out. Without his humanity I could have died,” said Mr Lavery.
Some survivors are deeply suspicious of Professor Jay because of her background as a social worker and despite her success in presiding over a child abuse inquiry in Rotherham.
Mr Lavery said social workers had been to blame in instances where abuse had been overlooked in care homes and on occasions had been abusers.
The inquiry’s junior counsel Elizabeth Prochaska, who was Mr Emmerson’s deputy, has also quit.
Their resignations followed the departures of three previous chairmen that has seen the inquiry lurch from one crisis to another.
The inquiry has insisted it is not “in crisis” but has accepted that the upheavals “are unsettling, particularly for victims and survivors of child sexual abuse”.
It said the experiences of victims was critical in learning from past mistakes, adding: “Failure to listen to victims and survivors in the past is the reason we are here today.”