Transformed into a palazzo by joining several medieval buildings together, among which, documents mention at least four lofts, four workshops and a tower, this palazzo was built between the late 14th century and early 15th centuries, but was only acquired by Mariano and Pietro Agostini, silk merchants, in 1496, who gave it its name.
The façade, totally clad in Gothic terracotta mouldings, in the shapes of plants, human heads and heraldic designs, with double and triple mullioned windows, and the open loggia (later closed) on the top floor, make it one of the most important examples of domestic Gothic architecture in Tuscany. The use of terracotta is probably due to the trade of the Astai family, brickmakers, who owned the palazzo until the 15th century. The roof was added during renovation in 1895.

At street level is the Caffè dell’Ussero founded in 1775 (the palazzo is also known in Pisa as Palazzo dell’Ussero); it was the meeting place of famous scientists and erudite Italians and foreigners. In 1839 the first Congress of Italian Scientists was held in the Caffè, and it is still the seat of the Accademia Nazionale dell’Ussero. It was requisitioned by the Americans in 1944 and renovated and re-opened in 1959.

Behind the Palazzo is a hanging garden with citrus fruits, mentioned in the15th century, and the Cinema Lumìere, one of the most ancient cinemas in Italy, opened in 1905 in part of the Caffè, and the Medieval church of Sant’Ilario in Porta Aurea.