The Pomesanian Sztum was conquered by Teutonic Knights in 1236. Using stone and brick, they transformed the wooden town into a defensive stronghold. The Castle’s structure differed from the majority of 14th-Century Teutonic seats, as it was erected on an irregular pentagon plan, rather than the usual square. Its shape resembled the island on which Sztum lay in the Middle Ages. The Castle hill was surrounded by a moat and a wall incorporating a gate tower and two donjons. The funds granted by Archduke Albrecht III Habsburg facilitated the construction of a grand Castle tower, from that time on called the Tower of Albrecht.

From 1468 the Castle was the seat of Polish Starosts. During the 17^th-Century wars the Swedish invaders destroyed the stronghold, looting all the precious furnishings. Today, the best preserved parts of the entire Castle complex is the southern wing, the walls with one tower and a tower with a gate.