Special Education in Ballarat started in 1955 as an annex at Pleasant Street Primary School site. There were 38 students on the first day of schooling in 1955.

The school community moved to Inkerman Street in 1963 and had 42 students. In 1982 the 90 students at Inkerman Street moved to the Gillies Street Campus.

In 1989 the school's name was changed to Begonia Park, and in 1993 the McCallum House Special Development School moved from the Department of Health to the Department of Education and physically moved to share the Gillies Street site with Begonia Park.

The two schools operated independently between 1993 to 1997 on the Gillies Street campus. On April 1st of 1997, the two schools officially merged and became the Ballarat Specialist School. Ballarat Specialist School has grown from 87 students in 1997 to over 450 in 2020.

In 2008 the school acquired the FARM campus and between 2008 to 2018, it operated as the FARM campus.

With growing enrolments, the decision to move the senior school to the FARM campus was made.