The origins of the collection date back to the reign of Rudolf II, Holy Roman Emperor, King of Bohemia and Archduke of Austria. From 1583, when Rudolf II asked the Czech Estates for a contribution to the improvement of Prague Castle, not only were paintings and sculptures purchased throughout Europe and works of art commissioned for the collection, but also several halls were built to house them. In 1612, when Rudolf II died, the collection numbered over 3,000 works.

During the following Thirty Years' War, some of the paintings were taken to Vienna. A substantial part of the collection became a prey of the Swedish troops, ordered by Queen Christina at the instigation of Jan Amos Comenius, who wanted to protect the treasures of the Bohemian kingdom from the Catholic Habsburgs. More than 500 paintings thus found their way into Swedish collections, where most of them remain to this day.

As Prague Castle was to continue to serve the Habsburgs as a representative residence, part of the paintings were transported back from Vienna and other paintings were purchased. By 1656 at the latest, the restored picture gallery contained about 600 paintings, and although it was much smaller than the original, its artistic value was not inferior to that of the Viennese collection. The picture gallery as such remained preserved until the reconstruction of Prague Castle under Maria Theresa, and was closed as a separate space in 1762. The paintings continued to serve only as decoration for the royal apartments. A large collection of paintings was sold to Dresden, other paintings were sold at auction. 

After 1918, the newly established Czechoslovak Republic demanded the return of the paintings from Austria. The Austrian side acknowledged this claim, but offered only inferior works for return, so no agreement was reached. Subsequently, the premises of Prague Castle were redecorated with paintings purchased from the so-called Masaryk National Fund and part of the old collections were lent to the National Gallery. However, there was no separate picture gallery at the Castle. In 1961, another part of the paintings with then "inappropriate" religious themes was handed over to the State Institute for Heritage Protection. This completed the fragmentation and reduction of the original collections.

The change of atmosphere in the 1960s, however, contributed positively to the fact that, after a long effort, the original stables under the Spanish Hall were converted into a space for the new Picture Gallery of Prague Castle in 1964-1965. Under the leadership of Professor Jaromír Neumann, by decision of the then President Antonín Novotný, the original intention of Emperor Rudolf II was restored. The exhibition, which is only a fraction of the original collections in terms of numbers, but is significant in the quality of the individual works of art, returned to Prague Castle and became accessible to the public. 

After 1989, a new face of Prague Castle was sought. Areas that were previously closed are being opened to visitors, many things are being reconstructed and changed. The desire to give the Castle a newer, modern look also led to the decision to transform the Picture Gallery. In 1995-1998, all the rooms were substantially renovated technically to modern safety and air-conditioning standards. According to Bořek Šípek's project, the exhibition spaces were modified, the entrance was redesigned and the interiors were equipped with seating furniture. It was also possible to purchase several paintings from Rudolf's original collection. The most important pieces of historical and modern painting were selected for the permanent exhibition and individual works were arranged according to their affiliation with local painting schools. Paintings by Rudolphine masters and works from the collection of Emperor Rudolf II have their own space.

The author of the latest concept of the Picture Gallery is art historian Eliška Fučíková, a former collaborator of Professor Neumann, who participated in research on the history of the Prague Castle collections. The collection of the new Picture Gallery received international recognition in 2001, when it was exhibited as a whole in Maastricht.