Fiona Woolf, the second person to lead the government’s inquiry into child abuse, is facing parliamentary and legal pressure to stand down after it emerged she was on “dinner-party terms” with Lord Brittan, who was home secretary when a dossier about alleged Westminster paedophiles went missing from his department.

Woolf, a QC and lord mayor of London, replaced the government’s initial choice, Lady Butler-Sloss, who resigned soon after the inquiry was set up when it emerged that her late brother Lord Havers was attorney general at the time of some of the historical allegations.

Now a second conflict of interest row is growing as lawyers representing victims of the abuse insist that Woolf should resign after it emerged that the Tory peer was one of her neighbours, with whom she had dined five times since 2008.

She also faces a legal challenge over her appointment and a parliamentary motion calling for her replacement.

The shadow energy secretary, Caroline Flint, told BBC’s Daily Politics programme: “I think it’s really difficult for her to stay.” But No 10 insisted the prime minister had “full confidence” in Woolf.

Andi Lavery, a victim of abuse at a Catholic boarding school, said Woolf personified the establishment as lord mayor. Speaking to the Guardian, he said: “I’m angry and appalled – it’s a cover up.”

He said he had seen a copy of a judicial review challenging Woolf’s appointment. “There’s a legal challenge from Ian McFadyen [who was abused at Caldicott prep school] and an unnamed lady. She [Woolf] is not fit and proper under the Inquiries Act,” Lavery said.

Woolf had also had coffee on “a small number of occasions” with Brittan’s wife (most recently last year), sat on a prize-giving panel with her and sponsored her for a fun run.

Alison Millar, partner at the solicitors Leigh Day which represents a number of victims, said her clients regarded Woolf’s links to Brittan as “beyond the pale”.

Speaking to BBC Radio 4’s Today programme on Wednesday, Millar said: “She has to be seen to be independent, and somebody who seems to be on dinner-party terms with a senior political figure whose knowledge this inquiry will be scrutinising is somebody who, from the perspective of my clients, does not have the necessary independence.”

Asked to state the single reason why Woolf lacked credibility to chair the inquiry, Milllar said: “This evidence of dinner parties with Lord Brittan puts her beyond the pale, in terms of her credibility with my clients.”

On Tuesday, Woolf had told the House of Commons home affairs select committee that she had “no close association” with the Tory peer and she believed her account of contact with him would “lay to rest” any fears.

Stressing that she was “not a member of the establishment”, she said: “I’ve gone the extra distance to make sure that I’ve dug out every possible connection … to make sure that it absolutely settles all the concerns that may be out there.”

Woolf appeared unwilling to say Lord and Lady Brittan were not friends, and could not remember whether she sent them a Christmas card. “My Christmas card list last year had about 3,000 people on it. To be honest I don’t know whether they were on it or not,” she said.

She added that Brittan’s phone number was not stored in her mobile phone.

However, victims of abuse have continued to questioned Woolf’s credibility. Phil Frampton, a former Barnardo’s boy who campaigns for the victims of abuse, said he was appalled by Woolf’s appointment. “It’s like putting Wayne Rooney in charge of an investigation of the nuclear energy industry,” he told the BBC.

The early day motion tabled by Lib Dem MP John Leech, and supported so far by three other MPs, calls on government “to find a new chair of the inquiry who has palpably demonstrated its willingness to challenge all quarters of the establishment to ensure that it can achieve its aims of providing justice to the victims of historic child abuse”.

The deputy prime minister, Nick Clegg, said he had no evidence that Woolf had not been thoroughly vetted, but appeared to stop short of offering full support. Answering questions after delivering a speech in south London, the Liberal Democrat leader said: “We all need to have confidence that the decisions taken by the home secretary … were thorough. I have not heard anything that suggests to me the process by which Theresa May made the recommendation is anything other than thorough.”

Woolf’s role was given only qualified backing by Sharon Evans, a journalist, campaigner and victim of child abuse, who is on the inquiry’s panel of experts.

She said she understood the anger of victims but was “confident” the panel would properly investigate the allegations.

“There has been so much focus on Fiona Woolf, which I understand, but she is just the head of the panel. There are nine people with an enormous background and expertise in this field,” Evans told Today.

“The whole panel are aware that we have got off to a very difficult start, but … we are satisfied that Fiona Woolf has the skills of a solicitor … We do need balance … I would like people to be reassured that there are victims on this panel.”

Evans added: “We are determined to get to the bottom of this … I will give you my pledge as a journalist and victim that I will not let anybody get away with things that I think are being covered up.”

The role of Brittan is key to the inquiry because it will investigate what happened to a dossier handed to him by the late MP Geoffrey Dickens, which later went missing. Last year, Brittan said he could not remember getting the dossier, but after further questions were put to him in July, he released a statement saying he could now recollect a meeting with Dickens. He said he had asked officials to look into the claims but could not remember hearing any more about it.

However, a Home Office review from last year found Brittan had written to Dickens in 1984 saying the material had been assessed by the director of public prosecutions as worth pursuing and “passed to the appropriate authorities”. Brittan released a second statement saying he had only just been made aware of last summer’s review, which proved that appropriate action had been taken.

Downing Street has backed Woolf, saying it was confident she would ensure there was “no stone left unturned” as head of the inquiry. Theresa May, the home secretary, also gave her support, saying: “Fiona Woolf has a long and distinguished career throughout which she has demonstrated the highest standards of integrity.

“I am confident that she will lead the work of the panel with authority, and that under her leadership the panel will get to the truth of these issues.”