‘Brisbane House’
Above Lavender Bay, north of Clark Park (Demolished mid-1920s)

‘Brisbane House’ was possibly the first substantial stone house built in North Sydney.

It was constructed above Lavender Bay in 1831 by James Milson, himself one of the first recipients of land in the area. The story of the house is bound up with the history and politics of early land grants and ultimately the power of the state to take back – to resume – what was once given away freely.

Free settler and farmer, Milson had been granted 350 acres around present-day Milsons Point and Kirribilli by Governor Thomas Brisbane in 1824. In deference to his benefactor, Milson called his first house ‘Brisbane Cottage’. Destruction of this dwelling and title papers in a bush fire in the late 1820s, however, led to a contestation of ownership with merchant Robert Campbell. In 1828, before the matter with Campbell was concluded, it was decided by the colonial administration that the granting of waterfront around Sydney Harbour should stop, for such land might be useful for public wharfage, defence or other purposes. With Governor Brisbane long gone, Milson was allowed to keep just 50 acres of the original grant – a parcel that sat above Lavender Bay. It was on that land that he built his second home. The name, ‘Brisbane House’, acknowledged again the generosity of Milson’s benefactor and the somewhat grander proportions of the second stone dwelling.

The land extended down to the water and Milson expected to claim the foreshore for wharfage. Instead the small beach there had become a popular unofficial bathing site by the 1860s. Milson's attempts to secure his title to the waterfront failed and the swimming spot was formalised as a place of public recreation in 1868. It became the Lavender Bay Baths.

Milson had, by then, given the house to his daughter Elizabeth after her marriage to Richard Thomas Hall in 1855. She died in 1865 but, as befitting its grandeur and ideal location across from Circular Quay, ‘Brisbane House’ was already being tenanted to a succession of senior public servants and academics including Chief Justice Sir Alfred Stephens, Charles Badham, Professor of Classics at the University of Sydney, and John Whitton, the Engineer in Chief of the New South Wales Railways. A leasing advertisement in 1864 described the house as having ‘13 apartments... kitchen, laundry, coach house and stables, &c, in the rear; a pleasant garden and grounds and fine views of Sydney and the Harbour’ (Sydney Morning Herald, 27/8/1864).

In 1886 ‘Brisbane House’ became a school, ‘Brisbane College for Young Ladies’. It was about this time that ‘The Towers was acquired by Sydney Church of England Grammar School for similar purposes. The conversion of large homes into exclusive schools would become a feature of North Sydney’s development.

The original house was probably a two-storey Georgian structure, symmetrical and simply elegant with ground floor rooms accessed from the porch by French doors, and bedrooms upstairs. Milson and his builder would have had many other similar homes upon which to model ‘Brisbane House’ – not least the designs of Sydney's foremost domestic architect John Verge whose work was found to the east and west of Sydney town. A photograph in the 1920s shows a substantial stone structure winged by timber additions and fronted by a large two storey verandah – both possibly added during the house’s incarnation as a school.

In the 1920s, just short of its centenary, ‘Brisbane House’ and its large garden were resumed and demolished by the State so that Junction Street could be widened to form the lower section of the Pacific Highway leading to the new Sydney Harbour Bridge.