Visalia, California is the oldest continuously inhabited inland European settlement between Stockton and Los Angeles. It was commonly known in the 1850s as Four Creeks, after the waterways through the area. The area was also briefly known as Buena Vista by the Tulare County board of supervisors, before Visalia formally became the county seat on March 11, 1854. The city played an important role in the American colonization of the San Joaquin Valley as the county seat of Old Tulare County, an expansive region comprising most if not all of modern-day Fresno, Kings, and Kern counties.

Before California became a part of the United States, there were roughly 20,000 indigenous people, mainly the Yokuts, living in what is now Visalia. In 1825 or 1826, a crew of 16 people, led by preacher Jedediah Smith, became the first Americans to cross through the region. Over the next few decades, fur trappers and other explorers continued to traverse the area, resulting in increased and occasionally violent contact with the Yokuts. This contact accelerated after the California gold rush of 1849. During the fall of 1850, a crew of 13 or 14 people, led by John Woods, built a log cabin and started cultivating an area between the Kaweah and the St. John's rivers, located about 8 mi (13 km) east of modern-day Visalia. In December 1850, the Kaweah Yokuts, led by an escaped native named Francisco, gave Woods an ultimatum – either leave or suffer the consequences. Ten days later, they returned, possibly with as many as 1,000 people, killing eleven of men, including Woods, who was flayed and the skin nailed to a tree. Despite the massacre, western settlers continued to move to the region. In December 1851, Nathaniel and Abner Vise built a log cabin in a forested area on the north bank of the Mill Creek, located in what is now Visalia. At the time, the area was known as Four Creeks, after the watershed of creeks and rivers flowing from the Sierra Nevadas.

On April 20, 1852, Tulare County was separated from the larger Mariposa County. At the time, the settlement of Woodsville was temporarily the county seat, located about 8 mi (13 km) west of Visalia. Nathaniel Vise persuaded settlers to move near him due to good soil conditions. His early influence led to the area being named Visalia, named for Vise and his ancestral hometown in Kentucky. Early residents built a stockade, holding a wagon and supplies in case the area was attacked. In August 1853, the Tulare County board of supervisors established the township of Visalia, initially naming it Buena Vista, which was last used in February 1854 in favor of Visalia. By order from the California legislature, Tulare County held an election on September 7, 1853, to determine the permanent county seat. The result narrowly favored Visalia 44 votes, compared to 41 votes for Woodsville; residents of Woodsville alleged two or three people were bribed to sway the close election. On March 11, 1854, the Tulare County board of supervisors formally designated Visalia as the county seat. However, Visalia originally had no formal boundaries at the time, and it was not formally incorporated.

By 1854, settlers had built a school, a mill race, a gristmill, a sawmill, a saloon, and a store. By that time, the first white child was born in Visalia – Commodore Murray. Early in its history, Visalia was largely a jungle surrounded by a swamp. After gold was discovered in what is now Kern County, a road was built between Visalia and Keyesville. Between 1856 and 1860, an estimated 5–6,000 people passed through town, either seeking gold or returning from the mountains. In 1856, a conflict broke out between the Yokuts and white settlers, after rumors of cattle theft and arson. Residents in Visalia let in a few natives after overhearing the plan for attack. In the ensuing battles, the U.S. Army battalion killed about 100 Yokuts, leading to the retreat of the natives. On January 9, 1857, the entire region experienced an earthquake. In August 1857, the Tulare County board of supervisors petitioned the United States Congress to preempt Visalia from securing a proper town title. The clerk was ordered to file with the land office in San Francisco; however, the paperwork was never received, and Visalia remained unincorporated at the time. Also in 1857, Tulare County's first school, courthouse, and cemetery began construction in Visalia. On October 8, 1858, the city was first visited by John Butterfield's Overland Stage mail route, from St. Louis to San Francisco, and was officially added the following year. A new two-story courthouse was built in 1859, with a jail. Also in 1859, the town's first newspaper, the Delta, was established. On June 18, 1860, the Atlantic and Pacific Telegraph Company entered Visalia and established service, which was celebrated by a procession of every horse and carriage in town. Also in 1860, a majority of Visalia residents voted no on a measure to incorporate the city.

By the time the American Civil War began in April 1861, the town was largely populated by Confederate sympathizers. Competing factions supporting both sides were formed in the city, including the pro-secessionist newspaper Equal Rights Expositor which began in 1862. In local elections that year, pro-secessionist parties won on a platform of "anti-coercion". On October 8, 1862, a company of 59 Union Army soldiers, members of the 2nd California Cavalry, arrived in Visalia and established Camp Babbitt. The soldiers arrested Confederate sympathizers, including the owners of the Equal Rights Expositor, forcing loyalty pledges before releasing them. On March 5, 1863, a group of at least 70 men from Camp Babbitt sacked the newspaper headquarters, destroying the equipment, windows, and doors. Although the newspaper owners successfully petitioned the California legislature to compensate them, California governor Henry Huntly Haight vetoed the bill, stating that the actions were done by soldiers of the federal government. Although largely remembered locally for its role in putting down Confederate sympathizers, Camp Babbitt was also a strategic base in the Owens Valley Indian War of 1862–1863, notably for its association with the Keyesville Massacre. It was abandoned in 1866.

While the Civil War was ongoing, Visalia was inundated twice during the Great Flood of 1862, in which 42 homes were destroyed, mostly those made of adobe. Main Street flooded to 2 ft (0.61 m) deep from January 20–23, and four bridges in and around the town were destroyed. Most water wells in town were contaminated, forcing residents to either collect rainwater or rely on the town's single water pump. The 1862 floods created the St. John's River as a tributary from the Kaweah River, which functioned as the primary water outlet to Tulare Lake across the delta that Visalia was situation on. As the town rebuilt after the floods, most buildings in the floodplain were either wooden or brick. On February 27, 1867, the California legislature formally designated Visalia as a city. It was in this charter that a common council and an ex-officio Mayor and President were established, a system which—alongside at-large elections—is still in use today.

Visalia has been called a one-time "capital" of the buckaroos or vaqueros, the California cowboys.

Four Creeks is the only place that the fish Luxilis occidentalis is known to have lived except for Poso Creek in Kern County. It was collected here in 1855 by Dr A.L. Heerman.