The most imposing part of the structure is the stone keep, which dates to about 1300. That was built by Robert Clifford, and it was Clifford's grandson Roger who built the hall and main living quarters around 1360.
The castle was enlarged on several occasions over the centuries, with additions of more and more comfortable living quarters.
Brough Castle was attacked by Scots raiding parties on several occasions, notably in 1174, but the greatest damage done to the castle was not from an attack, but from a fire that started following a great banquet in 1521.
That fire left the castle in a ruinous state until it was restored by Lady Anne Clifford in the mid 17th century. After Lady Anne's death, the castle fell into disrepair, and the many of the stones were taken for other local building projects.
Today, the castle is an imposing ruin, with walls standing to a good height in several locations around the perimeter of a courtyard. At the west end of the site, the large tower still stands.
In the courtyard within the castle walls, the foundations of domestic buildings are easily seen. The views are excellent from this hilltop site, and it is easy to see why the Normans (and the Romans) chose this location.
Note: Despite what the English Heritage website says, Brough Castle is an open-access site (at least it was when we visited). There are no charges or official opening times.