Samuel Finley Breese Morse was born in Charlestown, Massachusetts, the first child of Jedidiah Morse, a Calvinist pastor and geographer, and Elizabeth Ann Finley Breese Morse. His upbringing was steeped in religious and intellectual pursuits.

Morse attended Phillips Academy in Andover, Massachusetts. While his academic performance was reportedly unremarkable, he displayed an early aptitude for art, particularly painting miniature portraits.

Morse studied at Yale College, where he attended lectures on electricity and chemistry. He continued painting portraits to earn money, showcasing his entrepreneurial spirit and artistic talent even during his academic years. He graduated with honors.