The castle overlooking the centre of Bielsko-Biała is the oldest and the largest historic building located within the area of the historical city of Bielsko. Legend has it that in the place where it now stands, there used to be a settlement of bandits attacking merchants who were travelling nearby. Casimir, Duke of Opole († 1229/30), is said to have captured that fortalice, killing all the ruffians and building there a hunting lodge which later was developed into a grand castle providing a foundation for the city of Bielsko.
Its oldest parts come from the 14th century followed by years of its gradual expansion and further development. This is a city castle which from the beginning formed part of the fortifications of Bielsko, simultaneously constituting the strongest element of its stockade. For centuries it functioned as a Silesian border fortalice, protecting first the boundary between the feudal states of Cieszyn and Oświęcim, then the national Polish and Bohemian (Czech) border from the second half of the 15th century and finally the Polish and Austrian border from 1526. Since the end of the 16th century the defensive function of the castle was gradually diminishing while the castle itself was changing more and more into a noble residence. The contemporary look of the building is a result of the last large reconstruction works from the second half of the 19th century which effaced the old features of its old style. In the period between 1899 and 1973, in the place of a brick retaining wall, visible now on the eastern side, there used to a series of the so-called bazaars, forming a showy architectural basis for the structure of the castle. Unfortunately, the bazaars were demolished while extending the Zamkowa street.
The castle built by the Piast dynasty ruling the Duchy of Cieszyn for over two centuries was used as one of their seats. Since 1572 it was an administrative and commercial centre of the independent state country of Bielsko governed by representatives of the noble families of Promnitz, Schaffgotsch, Sunnegh, Solms and Haugwitz. In 1752 the state country rose in importance to become a duchy and the property of the Sulkowski family. The Duchy of Bielsko existed till 1849 when upon the introduction of a modern administration system in Austria eliminating the old feudal structures, its territory was included in the county government of Bielsko. The castle itself and its numerous properties located in the vicinity of the city remained in the hands of the Sulkowski family until 1945. After the II World War the castle was taken over by the State and was used as a seat of many cultural institutions, including the Museum. From 1983 it has been used exclusively by the Museum which is the Cultural Institution of the Silesian Local Government. From 2001 it had been called the Museum in Bielsko-Biała and since the end of 2013 it has been named the Historical Museum in Bielsko-Biała.