The West Side Line was built by the Hudson River Railroad, which completed the forty miles (64 km) to Peekskill on September 29, 1849, opened to Poughkeepsie by the end of that year, and extended to Albany (Rensselaer) in 1851. The city terminus was at the junction of Chambers and Hudson Streets; the track was laid along Hudson, Canal, and West Streets, to Tenth Avenue, which it followed to the upper city station at 34th Street. Over this part of the right-of-way, the rails were laid at grade along the streets, and since the Hudson River Railroad's regulations did not allow locomotives to draw cars through streets, the cars were drawn by a dummy engine. While passing through the city the train of cars was preceded by a man on horseback known as a "West Side cowboy" or "Tenth Avenue cowboy" who gave notice of its approach by blowing a horn.^[1][2]

At 34th Street, the right-of-way curved into Eleventh Avenue, the dummy engine was detached, and the regular locomotive took the train. As far as 60th Street, the track was at street level. The first cut was at Fort Washington Point. The railroad crossed Spuyten Duyvil Creek on a drawbridge; a fatal wreck occurred there on January 13, 1882, when the Atlantic Express, stopped on the line, was rear-ended by a local train, telescoping the last two palace cars, where the stoves and lamps were upset and ignited the woodwork and upholstery.^[3]

In 1867, the New York Central Railroad and Hudson River Railroad were united by Cornelius Vanderbilt, being merged in 1869 to form the New York Central and Hudson River Railroad. The railroad acquired the former Episcopal church's St. John's Park property and built a large freight depot at Beach and Varick streets, which opened in 1868. The tracks south to Chambers Street were then removed.^[4] In 1871, the Spuyten Duyvil and Port Morris Railroad (Hudson Line) opened, and most passenger trains were rerouted into the new Grand Central Depot via that line along the northeast bank of the Harlem River and the New York and Harlem Railroad (Harlem Line), also part of the New York Central system. The old line south of Spuyten Duyvil remained for freight to the docks along Manhattan's west side and minimal passenger service to the West Side station on Chambers Street (used until 1916).

As the city grew, congestion worsened on the west side. Eventually, plans were drawn up for a grade-separated line. The West Side Elevated Highway was built with the line's grade separation in the 1930s. Work on the highway – named for Manhattan Borough President Julius Miller, who championed it – began in 1925, and the first section was dedicated on June 28, 1934. This included a new, elevated eight-track freight terminal called St. John's Terminal, located several blocks north of the old one at St. John's Park, with its southern edge at Spring Street. North of there, an elevated structure (the present-day High Line) carried two tracks north on the west side of Washington Street, curving onto the east side of Tenth Avenue at 14th Street, then crossing Tenth Avenue at 17th Street and heading north along its west side. Just south of the Pennsylvania Station rail yard at 31st–33rd Streets, the line turned west on the north side of 30th Street, then north just east of the West Side Highway. The northernmost bridge crossed 34th Street, and a ramp took it back to Eleventh Avenue south of 35th Street. The elevated line was built through the second or third floors of several buildings along the route; others were served directly by elevated sidings.^[5][6]

In 1937, the tracks along Eleventh Avenue were bypassed by a below-grade line, passing under the 35th Street intersection and running north just west of Tenth Avenue before slowly curving northwest, passing under Eleventh Avenue at 59th Street and rejoining the original alignment. There were three sections that remained open, one at 37th Street, one at 45th Street, and one at 49th Street. The one at 37th Street was covered over in the mid-2010s, but the openings at 45th and 49th Streets remain to this day.^[5][6]

Around the same time, New York City Parks Commissioner Robert Moses covered the line from 72nd Street north to 120th Street with an expansion of Riverside Park. His project, called the West Side Improvement, was twice as expensive as the Hoover Dam and created the Henry Hudson Parkway, as well as a railroad tunnel under the park. The large 60th Street Yard served as the dividing point between the two-track realignment and a wider four-track line to the north. North of 123rd Street, the line became elevated between the Henry Hudson Parkway and Riverside Drive before returning to the surface and crossing under the Parkway to its west side near 159th Street. It continues along the shore of the Hudson River to the Spuyten Duyvil Bridge, a swing bridge across the Harlem Ship Canal (Spuyten Duyvil Creek), before merging with the Hudson Line just north of the bridge.^[6]

In addition to serving the industrial and dock areas of the Lower West Side, the line was the primary route for produce and meat into New York, serving warehouses in the West Village, Chelsea and the Meatpacking District, as well as serving the James Farley Post Office and private freight services.^[7]

The New York Central Railroad was merged into Penn Central in 1968. In 1976, the combined Penn Central, following a bankruptcy and then a merger, became the largest part of Conrail. Conrail continued to operate freight along the West Side Line until 1980, when the line north of 31st Street was acquired by Amtrak.

Donald Trump optioned the 60th Street Yard in 1974.^[8] Riverside South, the development project he ultimately began there, was then the city's biggest private residential development; it faced opposition from many people living on the Upper West Side.^[9] To obtain approval of his project, Trump agreed to substantially reduce the size of his ambitions,^[10] build Riverside Park South on 23 acres (9.3 ha) of the yard,^[11] and donate the park and the right-of-way for a relocated highway to the city.^[12]

As a result of construction to redevelop the yard was redeveloped into Riverside South in the late 1990s and 2000s, the West Side Line tracks were covered. This joined the two tunnel sections created in the 1930s to the north and south, forming one long almost-contiguous tunnel south of 120th Street.^[13]

The Empire Connection (or West Side Connection)^[14] is a rail line that allows Amtrak passenger trains operating on the Empire Corridor to access Penn Station in New York City. The line is used by Amtrak services operating between Penn Station and destinations including Albany, Buffalo, Niagara Falls, Toronto, Montreal, Burlington, and Chicago.

Before its construction, Empire Service trains came into Grand Central Terminal, requiring passengers bound for Northeast Corridor trains to transfer to Penn Station via shuttle bus, taxicab, subway or on foot.^[15] This was a legacy of the fact that the Empire Service lines had previously been part of the New York Central Railroad, which built and owned Grand Central, while Penn Station was owned by the Pennsylvania Railroad. The two stations had never been connected, even after the PRR and New York Central merged as Penn Central in 1968 and after Amtrak took over intercity passenger rail service in 1971. The Penn Central had planned to construct a track connection from Penn Station to the ex–New York Central West Side Line as part of their merger plans in the 1960s, especially as it would allow them to move their intercity trains from Grand Central Terminal to Penn Station as it would allow them to leave the former to state-funded commuter service, but ran out of funds to construct the connection.^[16]

When the West Side Yard for the Long Island Rail Road was built on the west side of Manhattan in 1986, a tunnel was built under it connecting Penn Station to the West Side Line just west of Eleventh Avenue, near the Javits Center. The project severed the southernmost part of the West Side Line, the High Line viaduct, from the rest of the West Side Line.^[17] When additional funding later became available, one track along the northern part of the West Side Line was rebuilt for passenger service and named the Empire Connection. A short section of single track into Penn Station was electrified using third rail and overhead catenary, since diesel locomotives are not allowed in Penn Station's tunnels. North of 39th Street, the single track expands into two tracks and electrification on the line ends. A wye was constructed next to Track 2 (the westernmost track) to allow diesels to turn around. South of 48th Street, there is a crossover from Track 1 (the easternmost track) to Track 2, and another siding splits off Track 2 just south of 48th Street, extending about 300 feet and ending at bumper block at 49th Street. The Empire Connection was double-tracked north of 39th Street to south of the Spuyten Duyvil Bridge in the mid-1990s.

On April 7, 1991, all of Amtrak's trains departing for or arriving from Albany and points north began using the Empire Connection into Penn Station, ending Amtrak service to Grand Central.^[18] Transportation planners had long envisioned consolidating all intercity service to New York at Penn Station, but those efforts did not go beyond the planning stages until the 1980s. Besides being more convenient for passengers, many of whom had balked at taking the train to New York City due to the difficulty of transferring between stations, this saved Amtrak the expense of operating two stations in New York City. Additionally, Amtrak had to pay $600,000 per year to the Metropolitan Transportation Authority, operator of Grand Central, to use that station's tracks.^[19][20] Despite warnings by officials in March 1991, inadequate fencing along the line allowed a 3-year-old boy to enter the tracks, where he was struck and killed.^[21][22]

Under the Penn Station Access project, Metro-North Railroad is studying ways it could also serve Penn Station. One alternative being studied would run some Hudson Line commuter trains into Penn Station via the Empire Connection, possibly with new station stops at West 125th and 62nd Streets.

The southernmost part of the elevated portion (south of Bank Street) was removed in the 1960s.^[23] By the late 1970s, freight traffic on the southern portion of the line had become nearly non-existent.

The northernmost block of the High Line viaduct, between 34th and 35th Streets, was rerouted to the south of 34th Street in the 1980s because the ramp to 35th Street was demolished as part of the Javits Center's construction. However, the realigned ramp was never used.^[24] The tracks were closed for a significant period of time as the line was reconfigured. Even after the line reopened, freight traffic never returned, and the elevated viaducts in Manhattan stood abandoned for over thirty years.

The structure from Bank to Gansevoort Streets was removed between the mid-1980s and early 1990s.^[25]

By mid-2005, the rest of the High Line was owned by CSX, which acquired it after the 1999 breakup of Conrail.^[26] The elevated viaducts were transformed into a 1.45-mile-long (2.33 km) elevated linear park and greenway called the High Line starting in 2006 and opening in phases during 2009, 2011, 2014, and 2019. Since opening in June 2009, the High Line has become an icon of American contemporary landscape architecture. The park became a tourist attraction and spurred real estate development in adjacent neighborhoods, increasing real-estate values and prices along the route.

• East Side Access, companion project to Penn Station Access
• Freedom Tunnel