The James Watson House, at 7 State Street between Pearl and Water Streets in the Financial District of Manhattan, New York City, was built in 1793 and extended in 1806, and is now the rectory of the Shrine of St. Elizabeth Ann Bayley Seton. It is located near the southern tip of Manhattan Island, across from Battery Park.

James Watson was the first Speaker of the New York State Assembly and a Federalist member of the New York and United States Senates. He was a Yale University graduate who became a prosperous importer-exporter. Once part of a row of late-eighteenth-century mansions, the building recalls the time when New York's merchant families lived at Manhattan's southern tip, near the river, in order to have an unobstructed harbor view and to be in close proximity to their shipping interests. At that time it was numbered 6 State Street.

In 1806 Watson sold the house to Moses Rogers and the address was changed to 7 State Street. Rogers was the brother-in-law of shipping magnate Archibald Gracie, who built the spacious home on the northeast side of Manhattan that came to be known as Gracie Mansion. Moses Rogers combined his house with the residence next door which sat significantly back due to the curve of the street. In order to create a unified facade, a colonnaded portico was added, reportedly using masts from his fleet of merchant ships which he was converting to steam power. The architect of the eastern (original) half is unknown, but the western extension, next to the church, is attributed to John McComb, Jr.

With the exodus of wealthy families northward, the building housed the office of the Ithaca Line. The United States government took it over during the Civil War. Afterwards it became the headquarters of the Harbor's Pilot Commissioners.

A bad harvest in Ireland in 1879, combined with Irish political turmoil, led to much emigration to America. Between 1856 and 1921, 3.6 million emigrants left Ireland for North America; a majority of them were women. For every eight Irishmen who left between 1871 and 1951, ten Irishwomen emigrated. Eighty-nine percent of those women were single and younger than twenty-four.

On a visit to Queenstown, the main port of embarkation, Charlotte Grace O'Brien was appalled at the conditions faced by immigrant women, who encountered overcrowded, overpriced lodgings and robbery. In April 1882 she opened the 105-bed "O'Brien Emigrants Home", a boarding-house for the reception and protection of girls on the point of emigrating. She also visited the ships for which her lodgers were destined, along with a medical officer. Finally, O'Brien sailed to New York to investigate conditions upon arrival.

O'Brien found little effort to provide food or drink or accommodation at the Castle Garden entry facility and illiterate young women being tricked into prostitution through spurious offers of employment. She contacted Archbishop John Ireland, who through his contacts with the Irish Catholic Association, arranged the establishment of an information bureau at Castle Garden. Ireland also contacted Cardinal John McCloskey, Archbishop of New York, about providing a chaplain specifically for immigrants arriving at Castle Garden.

The Mission of Our Lady of the Rosary for the Protection of Irish Immigrant Girls opened on January 1, 1884, with the appointment of Rev. John J. Riordan as the first chaplain at Castle Garden. Immigrant girls needing accommodation were placed in local boarding houses until May 1 when a Home for Immigrant Girls was opened at 7 Broadway. In 1885, the Watson House at 7 State Street was purchased from Isabella Wallace for the Mission of Our Lady of the Rosary to serve as a way station for young immigrant women.