The origins of the collection trace back to the reign of Rudolf II – the Holy Roman Emperor, King of Bohemia, and Archduke of Austria. Starting in 1583, when Rudolf II requested funding from the Czech Estates to enhance Prague Castle, paintings and sculptures were acquired throughout Europe, and works of art were also commissioned, with several halls built to house them. By 1612, the year of Rudolf II’s death, the collection had grown to over 3,000 works.

During the subsequent Thirty Years' War, some of the paintings were relocated to Vienna. A significant portion of the collection was taken for safekeeping by Swedish troops, under orders from Queen Christina at the instigation of John Amos Comenius, who sought to protect the treasures of the Kingdom of Bohemia from the Catholic Habsburgs. More than 500 paintings thus found their way into Swedish collections, where most of them remain to this day.

As Prague Castle continued to serve as an official residence for the Habsburgs, some paintings were returned from Vienna and new ones were acquired. By 1656, the restored Picture Gallery housed about 600 paintings. Although smaller than the original collection, its artistic value was by no means inferior to that of the Viennese collection. The gallery remained as such until the reconstruction of Prague Castle under Maria Theresa, closing as a separate space in 1762. The paintings then served solely as decoration for the royal apartments, with a substantial portion sold to Dresden or auctioned off. 

After 1918, the newly established Czechoslovak Republic sought the return of paintings from Austria. While Austria acknowledged the claim, it offered only inferior works, leading to a stalemate. Consequently, the premises of Prague Castle were redecorated with paintings purchased from the Masaryk National Fund, and parts of the old collections were lent to the National Gallery. However, there was no separate picture gallery at the castle. In 1961, another portion of paintings, considered "inappropriate" for their religious themes, was handed over to the State Institute for Heritage Protection, further fragmenting the original collections.

The changing atmosphere of the 1960s, however, led to a positive development. In 1964–1965, after much effort, the original stables under the Spanish Hall were converted into a new Picture Gallery for Prague Castle. Under the leadership of Professor Jaromír Neumann and by order of President Antonín Novotný, the original intent of Emperor Rudolf II was revived, making the exhibition accessible to the public at Prague Castle. Though it now includes only a fraction of the original collection, it is notable for the high quality of its individual works of art. 

After 1989, efforts to modernize Prague Castle included opening previously closed areas to visitors, undertaking extensive reconstructions, and transforming the Picture Gallery to give the castle a newer, more modern look. From 1995 to 1998, all of the rooms in the gallery underwent substantial technical renovations to meet modern safety and air-conditioning standards. Architect Bořek Šípek's project included modifying the exhibition spaces, redesigning the entrance, and furnishing the interiors with seating. Several paintings from Rudolf's original collection were also repurchased. The permanent exhibition now features the most important historical and modern paintings, arranged by their respective schools, with a dedicated space for works by Rudolphine masters and pieces from Emperor Rudolf II’s collection.

The latest concept for the Picture Gallery was developed by art historian Eliška Fučíková, a former collaborator of Professor Neumann and researcher of the history of the Prague Castle collections. The collection of the new Picture Gallery gained international recognition in 2001, when it was exhibited in its entirety in Maastricht in the Netherlands.