The castle was besieged six times in the 13th century, succumbing on three occasions.
Bernard de Neufmarch defeated Rhys ap Tewdwr and took control of Brecon in 1093. He emphasized his control by erecting a motte and bailey fortress at the confluence of the rivers Usk and Honddu. He then went on to found the Benedictine Priory and St John's church (now Brecon Cathedral).
Llewelyn ap Iortwerth destroyed the wooden castle in 1231, but it was quickly rebuilt. He attacked again two years later and again destroyed the castle.
The castle was rebuilt in stone by Humphrey de Bohun in the early 13th century. The most prominent of de Bohun's structure is the great hall, now in the grounds of a hotel.
The castle gradually fell into disrepair, and by the late Georgian period it was little more than a ruin. In the early 19th century it was converted to use as a hotel, a position it still fulfils. The castle itself may be visited independently from the hotel, however.