The Octagon House, also known as the Colonel John Tayloe III House, is a house located at 1799 New York Avenue, Northwest in the Foggy Bottom neighborhood of Washington, D.C. It was built in 1799 for John Tayloe III, the wealthiest planter in the country, at the behest of his new family member George Washington, as his sister Sarah Tayloe married William Augustine Washington, son of Gen. Washington’s half brother Augustine Washington Jr. In September 1814, after British forces burnt the White House during the War of 1812, Tayloe, for six months, lent the Octagon House to United States president James Madison and first lady Dolley Madison to serve as the Executive Mansion. It is one of only five houses to serve as the presidential residence in the history of the United States of America, and one of only three, along with the White House and Blair House, that still stand.

John Tayloe III was born at Mount Airy, which he later inherited. The colonial estate was built by his father, John Tayloe II, on the north bank of the Rappahannock River across from Tappahannock, Virginia. By this time, it was the centerpiece of a roughly 60,000 acre department of interdependent plantation farms known as the Mount Airy department, located approximately one hundred miles south of Washington, D.C., in Richmond County, Virginia. He was educated at Eton and Christ's College, Cambridge University in England, served in the Virginia state legislature, and ran unsuccessfully for Congress in 1800.

John Tayloe III married Ann Ogle, daughter of Benjamin Ogle and granddaughter to Samuel Ogle of Ogle Hall in Annapolis, Maryland, in 1792 at her family's country home Belair Mansion. Ann was only a year younger than her husband. Tayloe was reputed to be the richest Virginian planter of his time, and built the house in Washington at the suggestion of George Washington on land purchased from Gustavus W. Scott or Benjamin Stoddert, first Secretary of the Navy. The Octagon was originally constructed to be a winter residence for the Tayloe family, but they lived in the house year-round from 1818 to 1855. The Octagon property originally included a number of outbuildings, including a smokehouse, laundry, stables, carriage house, slave quarters, and an ice house (the only surviving outbuilding). The Tayloes were involved in shipbuilding, horse breeding and racing, and owned several iron foundries—they were fairly diversified for a plantation family. The Tayloes enslaved hundreds of people, and had between 12 and 18 who worked at the Octagon. Since the 1960s when it was designated a National Historic Landmark, it has served as a house museum, meeting and event space, operated by the American Institute of Architects Foundation.

John and Anne (Ogle) Tayloe were considering Philadelphia as a place to build a town house, since Baltimore and Philadelphia were the nearest metropolitan centers to Mount Airy. George Washington, whose half-brother Augustine Washington Jr.'s son, Capt William Augustine Washington married Tayloe's sister, Sarah 'Sally' Tayloe, on May 11, 1799, found out and persuaded the Tayloes to build their house in the new capital city in an outlying section. The plan was to establish a node of development to stimulate fill-in growth. Col. Tayloe had been considering building his new home in Philadelphia under the expert hands of architect Benjamin Latrobe. Instead, he chose the primitive wilds of the new federal city and the architect William Thornton, the man who designed the new United States Capitol with the help of James Hoban, who was awarded the job of designing the White House. Tayloe went with the wishes of George Washington.

On April 19, 1797, Tayloe paid $1,000 (~$23,280 in 2024) to Gustavus W. Scott for lot 8 in Square 170, at the corner of New York Avenue and 18th Street NW, as laid out in a plan of the city by Pierre Charles L'Enfant and surveyed by Andrew Ellicott. Scott was one of the first purchasers of lots in the newly platted capital. The lot was in open country west of the partly built President's House, about 1 mi. from Georgetown, and about .5 mi. northeast of Hamburgh, which was absorbed into the new city plan. During this time he established the Washington Jockey Club's on a mile track which extended from the rear of what is now the site of Decatur House at H Street and Jackson Place, crossing 17th Street and Pennsylvania Avenue to 20th Street with Charles Carnan Ridgely, at the present site of the Eisenhower Executive Office Building.

On April 19, 1799, Dr. William Thornton wrote to George Washington, "Mr. J. Tayloe, of Virginia, has contracted to build a house in the City near the President's Square of $13,000 value." Thoronton was a self-trained architect who had won the United States Capitol competition. His first problem was to plan a house that would fit the lot, the south side of which was cut away on the bias by the diagonal of New York Avenue. If the house were built to face either of the bordering streets, it would be at an ungainly angle in relation to the other street, and outbuildings and wells had to be fitted in also. He dealt with the problem by relating the house equally to both streets, which put the two walls at a 70 degree angle from each other. The house actually has six sides, but was called "The Octagon" by the Tayloes. It had closets on every floor, an innovative feature for its time. The house is well built of brick trimmed with Aquia Creek sandstone. The lot is triangular and fenced in by a high brick wall. The kitchen, stable and outhouses are built of brick and accommodated a large number of both enslaved people and horses. The interior is elaborately finished, the doors of the first story being of mahogany. All the work in the circular vestibule coincides with the circumference of the tower, the doors, sash and glass being made on the circle. The parlor mantle is made of a fine cement composition painted white. The remains of gold leaf show in some of the relieved portions. Leading into the back hall and dining-room are two secret doors in which the washboards and chairboards run across the door, being ingeniously cut some distance from the actual door, no key holes, hinges or openings showing on the blind side. The knobs and shutter-buttons are of brass and evidently of a special pattern.

The Tayloes had 15 children, 13 of whom survived to adulthood (2 died in infancy: Anne, born and died in 1800, and Lloyd, born 1815, died 1816). The children were all born between 1793 and 1815. The oldest son, John Tayloe IV, served in the US Navy during the War of 1812 aboard the USS Constitution. His early death in 1824 was possibly connected to wounds received during the war. His parents provided for his wife and child after his death. Edward Thornton Tayloe, George Plater Tayloe and Henry Augustine Tayloe were all born at the Octagon.

John Tayloe III was a Federalist, and not terribly supportive of President James Madison and the war with England that began in 1812, but he was active in the Virginia militia and commanded a regiment of DC cavalry. When British forces marched into Washington in August 1814, there was a French Flag flying outside the Octagon. Ann Ogle Tayloe had offered the house to the French consul, in the hopes of sparing the house from being burnt, who was occupying the house when the British arrived in the city. The house probably would have been spared even if it hadn't been effectively a "diplomatic residence", since the British were under strict orders not to damage private property. When First Lady Dolley Madison fled the city as the British approached, she sent her pet parrot to the French consulate at the Octagon for safekeeping.