The palace is the result of a series of architectural vicissitudes that lasted four centuries. It is among the few palaces of Rome to be still occupied by its owners and it preserves an extraordinary collection of furniture and works of art. The core of the palace dates back to mid-15th century and was built by Cardinal Nicolò Acciapacci. 

Between 1505 and 1507 Cardinal Fazio Santorio rebuilt the palace entirely but he was forced by pope Julius II to give it to the Della Rovere family. The building passed to the Aldobrandinis in 1601 and then, when Olimpia Aldobrandini married Camillo Pamphilj senior in 1647, to the latter. 

The most notable part of the building is that worked on by the architect Gabriele Valvassori between 1731 and 1734. He enriched the façade on the Corso and closed the upper loggia of Bramante’s courtyard to obtain four wings. One wing was turned into the Gallery of Mirrors. In 1651, Pamphilj elected pope as Innocent X, sanctioned the birth of the Gallery. 

The collection at this time already included the famous portrait of the pope commissioned from Velasquez in 1650. The direct Pamphilj family line ended in 1760 and the Doria Pamphilj branch inherited the palace. 

The Gallery was declared indivisible and inalienable in 1871, along with the other ex-trustee collections.