Education Secretary Angela Constance said the probe was necessary to help fully understand what happened and in turn "ensure a brighter future for every child".
The inquiry will have the power to compel witnesses to attend and give evidence, she confirmed.
Ms Constance also pledged that where crimes are uncovered, the "full force of the law" would be used to bring those responsible to justice.
The Scottish Government had been considering if a national inquiry into historic abuse cases was the correct way to help survivors.
Announcing its decision, Ms Constance said: "This Parliament must always be on the side of the victims of abuse. We must have the truth of what happened to them and how those organisations and individuals into whose care the children were entrusted failed them so catastrophically.
"And to get to that truth we will be establishing a national public inquiry into historical abuse of children in institutional care."
She continued: "To ensure justice is done I can tell this chamber that where crimes are exposed the full force of the law will be available to bring perpetrators to account."
The Education Secretary said Scotland's top prosecutor, Lord Advocate Frank Mulholland, had been consulted and that "measures will be put in place to ensure that the inquiry does not compromise or interfere with ongoing criminal investigations and prosecutions".
She stated: "As a society we have an opportunity to confront the mistakes of our past and to learn from them.
"It will not be easy but only by shining a light on the darkest recesses of our recent history will we fully understand the failures of the past, enabling us to prevent them happening again and ensure a brighter future for every child and young person in Scotland, today and in the future."
Survivors will be consulted on both the remit of the inquiry and who should lead it, Ms Constance told MSPs at Holyrood.
That decision was made after the two people appointed to chair a similar UK inquiry both stood down.
UK Home Secretary Theresa May had ordered the probe as part of her commitment to uncover the truth about long-standing claims of child sex abuse by powerful figures.
But Baroness Butler-Sloss stood down as chairwoman in July amid questions over the role played by her late brother, Lord Havers, who was attorney general in the 1980s. Her replacement Fiona Woolf, the then Lord Mayor of London, resigned in October following a barrage of criticism over her ''establishment links''.
Ms Constance said: "We have witnessed the pitfalls when an administration rushes to make decisions about an inquiry, without involving the people who will be most affected by it.
"We are not a Government that believes in haste at the expense of sense."
She insisted: "We will not make the same mistakes as others by rushing out with names before we have consulted with survivors and relevant organisations about the attributes of a chair or panel."
This process should be completed by the end of April, Ms Constance said.
The Education Secretary confirmed that work by the Government to develop a support fund for abuse survives would continue, alongside plans for an appropriate commemoration.
Various groups have also been invited to consider how the civil justice system can be made more accessible to abuse survivors, including the operation of the "time bar" system which means some cases cannot be brought forward if they happened too long ago.
Assistant Chief Constable Malcolm Graham, the lead officer for major crime and public protection at Police Scotland, said the force would "fully co-operate with any inquiry which is convened".
He added: "We have an enduring commitment to investigating child abuse whenever offences occurred. Our approach is absolutely centred on victims of abuse while ensuring those who commit such crimes are subject to rigorous investigation.
"Our work with a wide range of partners at national and local level is essential to providing the right framework in which to conduct investigations into such complex and challenging issues.
"Since April 2013, Police Scotland has worked to introduce even greater consistency and co-ordination to child abuse investigations and how we respond to serious sexual crime.
"In the new year, we will introduce a National Child Abuse Investigation Unit which will see specialist investigators working with local officers on the most serious cases. This will further heighten our capabilities in keeping people safe."
Labour MSP Iain Gray welcomed the decision to hold an inquiry but said it "should have happened sooner".
He said: "I understand the points that the Cabinet Secretary made about the care and the lack of haste required in coming to a view, but it is 10 years since the former first minister Jack McConnell apologised on behalf of the Scottish people to the survivors of institutional child abuse.
"For a moral imperative, this has proceeded and progressed too slowly. This next step has taken too long, but we are taking it today."
He said the most important thing was for survivors to have faith in the process.
Conservative MSP Nanette Milne said she hoped the inquiry would "provide the opportunity to expose the perpetrators of such hideous crimes against children in Scotland and to learn lessons to prevent this abuse of children in care ever happening again."
She emphasised the importance of ensuring that the inquiry is as accessible as possible to survivors of abuse and highlighted that many have chosen to remain silent "and have no voice".
"I'm concerned that this may remain the case unless there's very practical help and support available for the brave people who come forward, because only an inquiry that supports survivors can truly deliver the justice that victims deserve," she said.
Liberal Democrat MSP Alison McInnes said: "I too welcome the announcement of a public inquiry with statutory powers. Victims and survivors have long cried out for that.
"The trauma of victims and survivors must always however be to the forefront."
She echoed calls for assurances on the support that will be made available to victims to "ensure getting to truth doesn't compound that damage".
The inquiry was also welcomed outside Holyrood, with Tam Baillie, Scotland's Commissioner for Children and Young People, stating: "The effects of abuse are long lasting and very much in the present for those who have suffered, no matter how long ago it took place.
"This inquiry must ensure we fully comprehend the extent of historic institutional abuse and that we work together to ensure that we never witness these systemic failings again."
He added: "The inquiry's main priorities must be that the survivors of abuse are given the opportunity to speak and receive as much support as they need. We must use this process to fully understand the mistakes of the past."
Martin Crewe, director of Barnardo's Scotland, also gave the inquiry his full support and said: "Organisations that provide residential care must do everything they can to tackle child abuse, past, present and future.
"We have worked closely with survivors through the 'InterAction' process and it is essential that this inquiry has the confidence of the victims of abuse. We also need to reassure the public that no-one is above the law when it comes to child abuse.
"Any allegations, whether they are current or historical, must be taken seriously and fully investigated by the police. Anyone who has used their power to escape punishment must be brought to justice."
Professor Alan Miller, chair of the Scottish Human Rights Commission, hailed the inquiry as a "major step forward in securing justice for survivors of abuse".
He also said it "provides a clear opportunity to ensure accountability for past abuses and to learn lessons to prevent future abuse".
Prof Miller stressed: "Survivors of abuse - those whose rights have been breached and who the state failed in its duty to protect - must be at the heart of the process of developing the inquiry's terms of reference and its future work."