Only one wall of the keep stands to any height now, reaching about 22 feet, and the site is about 36 feet across. There was once a bailey surrounding the keep, but this was later filled in. The castle was apparently abandoned voluntarily in the 13th or 14th century.
The castle would once have stood within curtain walls taking up the entire hillside. Old photographs also show another tower which has since vanished. The great tower which we can see now would have stood to three stories in height. You can still see recesses in the walls which would have been used as latrines, or possibly as cupboards.
The castle and manor of Sutton was once in the hands of Simon de Montfort, Earl of Leicester. De Montfort rebelled against the crown, and for a time held Henry III prisoner. When the Earl was killed at the Battle of Evesham in 1265 Henry granted the manor of Sutton to his half-brother, William de Valence, Earl of Pembroke. It is from the Earl that the village became known as Sutton Valence.
William's son Aymer is known to have stayed in the castle after he inherited the estate in 1315. We also know that William's widow, Joan, spent a month at the castle between 1296 and 1297. After this time we know practically nothing of the castle until it was finally abandoned, probably sometime in the mid 14th century.
The site is accessible at any time.