Government House was the official residence of the lieutenant governor of Upper Canada and Ontario, Canada. Four buildings were used for this purpose, none of which exist today, making Ontario one of four provinces not to have an official vice-regal residence.

The colony's first Lieutenant Governor, John Graves Simcoe, occupied a couple of residences during his tenure. Upon his arrival in Upper Canada in 1792, he used one of the buildings at Navy Hall in Niagara-on-the-Lake as a residence, sharing the space with Upper Canada’s legislature. When Simcoe moved the colonial capital to York (present-day Toronto) in 1793, he built a summer residence, Castle Frank, north of the settlement in 1794. Simcoe's successor and the colony's second Lieutenant Governor, Peter Hunter, initially continued to reside in his own home, Russell Abbey, located at the south-west corner of Princess and Front streets.

The first official government house was a one-storey, U-shaped frame house built at Fort York in 1800, designed by Captain Robert Pilkington and first occupied by Hunter. The structure was destroyed when a nearby powder magazine exploded in 1813 during the War of 1812.

After the destruction of the Fort York house, York did not have another Government House until after the War of 1812. In 1815, the government purchased Elmsley House, a more commodious Georgian residence, for its Lieutenant Governor. The new Government House was located in a wooded area to the west of the settled portion of the (then) Town of York, roughly midway on the block now occupied by Roy Thomson Hall and Metro Hall in downtown Toronto.

Built in 1798, the residence had been the home of the Chief Justice and Speaker of the Legislative Assembly, John Elmsley, and it served as the colony's Government House from 1815 to 1841 (and intermittently from 1841 to 1858, during some of the times when Toronto served as the capital of the Province of Canada). From 1847 to 1849 it was home to the Toronto Normal School.

For many years after its purchase by the government, the residence was still known by the name of its former owner, with the correspondence of the Lieutenant Governor typically dated from "Elmsley House". In 1846, the grounds were used for the first annual Provincial Agricultural Fair.

Beginning in 1849, Lord Elgin, the Governor General of the then united Province of Canada, resided for two years at the similarly-named Elmsley Villa, located near what is today the intersection of Bay and Grosvenor Streets (northwest corner), rather than at Elmsley House. Elmsley Villa was a two-storey Georgian structure that stood until at least the 1860s.

Elmsley House was destroyed by fire in 1862.