A nun told a Ballarat abuse victim what had happened was "God's secret", the child abuse royal commission has heard.

The man, who was also abused by three Christian Brothers at Ballarat schools, said the nun abused him when he was in grade one.

"I don't remember how long the incident of abuse lasted but when she was done the nun took me off her lap and said what had just happened was God's secret," the man told the royal commission.

"She told me if I told anyone what had happened I would go to hell."

The victim told the royal commission he was also abused by Brother Gerald Leo Fitzgerald and Brother Robert Charles Best at St Alipius primary school.

He said he was in constant fear at school.

"You never knew what was going to happen next every day because if it wasn't you it was someone else," he said.

The man thought he would be safe when as a 12-year-old he went to secondary school at St Patrick's College.

"I thought it was good that I was away from St Alipius. That was until I met Brother Edward Dowlan," he said, breaking down.

Best and Dowlan are both convicted pedophiles.

The Christian Brothers admit Fitzgerald, who died in 1987, was also a pedophile.