Downing Street said that it was “her decision” to stand down but insisted that the Prime Minister still believes she was the right person to head up the inquiry.
Her decision will prove embarrassing for the Government and comes amid allegations of a Westminster “cover-up” over historic paedophilia allegations.
Theresa May, the Home Secretary, appointed Lady Butler-Sloss to lead the investigation despite the fact that her late brother, Sir Michael Havers, was Attorney General at the time of the alleged abuses in the 1980s.
Sir Michael, father of the actor Nigel Havers, reportedly tried to stop Tory MP Geoffrey Dickens from using parliamentary privilege to name diplomat Sir Peter Hayman as a paedophile. He was accused of a "whitewash" after failing to prosecute Sir Peter.
Lady Butler-Sloss, 80, has said that she "knew absolutely nothing" about her brother's role in the controversy.
Mrs May is understood to have spoken to Lady Butler-Sloss over the weekend about her decision to step down.
The Home Secretary said that she was "saddened" by Lady Butler-Sloss' decision.
Lady Butler-Sloss said: "I have come to the conclusion that I should not chair this inquiry and have so informed the Home Secretary."
She admitted that there was a "widespread perception, particularly among victim and survivor groups, that I am not the right person to chair the inquiry".
She said: "I was honoured to be invited by the Home Secretary to chair the wide-ranging inquiry about child sexual abuse and hoped I could make a useful contribution.
“It has become apparent over the last few days, however, that there is a widespread perception, particularly among victim and survivor groups, that I am not the right person to chair the inquiry. It has also become clear to me that I did not sufficiently consider whether my background and the fact my brother had been Attorney General would cause difficulties."
Mrs May said: "I am deeply saddened by Baroness Butler-Sloss’s decision to withdraw but understand and respect her reasons."
The Prime Minister's spokesman said: "The decision to step down is entirely Lady Butler's decision, it was her decision."
He added: "The Prime Minister and the Home Secretary's views about her suitability is entirely unchanged."
Asked whether her appointment had turned into a shambles, the spokesman said: "I wouldn't characterise it that way, she was appointed the reasons for her appointment still absolutely stand in terms of her professional expertise, I think her integrity, which I don't think has been questioned from any quarter whatsoever."
The process of selecting a replacement is beginning immediately, but a new name is not expected to be announced for some days.
Discussions on the make-up of the panel, including legal and child protection experts, who will conduct the inquiry, are not expected to begin until a new chair is appointed, in order to allow the new inquiry head to take part in the process.