Morse married Lucretia Pickering Walker in 1818. They had three children: Susan Walker Morse (1819), Charles Walker Morse (1823), and James Edward Finley Morse (1825).

Lucretia Morse died unexpectedly shortly after the birth of their third child. Morse was devastated by her death, which occurred while he was away on a painting commission.

Morse married Sarah Elizabeth Griswold in 1848. They had four children: Samuel Arthur Breese Morse (1849), Cornelia Livingston Morse (1851), William Goodrich Morse (1853), and Edward Lind Morse (1857).

Samuel Morse died in New York City at the age of 80. He was buried in Green-Wood Cemetery in Brooklyn, New York.