Located on the corner of Viale delle Mura and Via San Giacomo, the palace features a C-shaped plan opening onto the panorama of the Lower City. All facades, except the one on Via San Giacomo, feature a high, rusticated base that conceals the difference in elevation between the two streets.

The upper part of the facades is enclosed within a grid of giant Ionic pilasters, which are replaced by half-columns with an upper balustrade adorned with statues and a clock by sculptor Antonio Gelpi in the projecting portion of the facade facing Porta San Giacomo. Above the windows of the piano nobile, five bas-relief panels depicting scenes from "Jerusalem Delivered" are incorporated, crafted by Giovanni Maria Benzoni in 1848. In the facade on the walls, three arches with an overhanging terrace connect the two lateral wings and define the internal courtyard.

The side along Via San Giacomo is more sober, articulated by a dense sequence of windows with white marble profiles.

From the portal, a long vestibule that runs along the interior chapel leads to the spacious portico of the courtyard, where a monumental staircase with opposite ramps ascends to the piano nobile. Here, corresponding to the portico below, is the grand hall, surrounded by smaller rooms, a gallery, and cabinets, decorated by Luigi Deleidi, Filippo Comerio, and Vincenzo Bonomini.

In the last stretch of Via San Giacomo, overlooking the plain, the palace of Gerolamo Poncino (or Poncini) was built in 1519. The construction was overseen by Pietro Isabello based on a design by Andrea Ziliolo. Remaining largely unchanged for almost two centuries, the building was sold in 1703 by Lodovica Poncino to Francesco Vailetti, who made it his residence. No documented work related to the palace is known until 1783, when Count Luigi Vailetti, great-grandson of Francesco, decreed its almost complete demolition to build a new one designed by Simone Cantoni. The architect enlisted the collaboration of master mason Giovanni Francesco Lucchini, who oversaw the construction that started before the summer of 1783 and concluded five years later.

Inside, the rooms were decorated by Filippo Comerio, Luigi Deleidi, and Vincenzo Bonomini, while the bas-reliefs on the western facade depicting scenes from "Jerusalem Delivered" are by sculptor Giovanni Maria Benzoni (1848).

After the Vailetti family died out, the palace was acquired by the Municipality of Bergamo in 1835 with the intention of housing the high school there, a choice that ultimately proved impossible due to the difficulties of adapting the premises to the new function. In 1841, it was then purchased by Count Giacomo Medolago Albani, who oversaw its restoration. Subject to ministerial protection since 1910, it has been the subject of interventions on the facades in 1984 and 2003.