The first wooden castle was built in the 13th century by a family of Rus' princes. In 1240, the fortress was destroyed during a Tartar invasion.^[1]

The next castle was built in the 16th century by the Jeło-Maliński family in the Renaissance style.^[1] After the Maliński family, the owner of the castle became Michał Radziwiłł, the Lithuanian equerry, Great Hetman of the Crown, and father of Karol the Lord Lover, during whose time it was still inhabited. Prince Michał, and later Prince Karol, came down here often for the hunting they did in the vast Shumsk forests.^[2] In the 18th century, Prince Józef Sapieha castle, which belonged to him at the time, had a large gallery of paintings with works by Sylwestr de Mirys, among others.^[1] At the end of the 19th century, at the town's exit, on a clump of land splashed with water on three sides, stood the ruins of a castle, probably still founded by the Maliński family. The perimeter walls and the inner regulation of the chambers have survived.^[2]