Police began the search at the site, which was once home to the Ballarat Orphanage, the Ballarat Children's Homes and Damascus College, on Monday.
It was sold to developers in 2011, who applied to have it rezoned for commercial and mixed use.
Phylis Read and Edith Orr raised the possibility of children being buried on the site at a Ballarat Council meeting two years later, when it was considering plans to redevelop the site.
Ms Read, who spent time as a child at the orphanage, said it was common practice for children to be buried at the site.
"When I first came in as a small child, when we were small children, it was just usual. There was none of this no one knowing or not allowed to talk about. It was just normal," she said.
The orphanage had a history of sexual and physical abuse.
Superintendent Andrew Allen from Victoria Police said they had not yet found anything at the site.
He confirmed the investigation was triggered by the concerns raised by Ms Read and Ms Orr.
Care Leavers Australasia Network (CLAN) vice-president Frank Golding lived at the orphanage between 1942 and 1953 and said he was not surprised by the allegations.
"I think that part of it is that children just disappeared overnight, sometimes they would be ill and they simply wouldn't be there at roll call the following morning," Mr Golding said.
"We had no information on what happened to those children, there were never any formal farewells... so I think in an era where children were also quite viciously punished at times and had illnesses which were neglected, that leads to speculation, and it's not just confined to the Ballarat Orphanage."
Mr Golding said he believed Ms Orr and Ms Read, who are Aboriginal women, were particularly concerned about Aboriginal children's remains.
He said during his time at the orphanage 10-15 per cent of the children there were members of the stolen generation, whose parents did not know where they were.
From the ABC online news site.