In 1215 King John gave land at Hadleigh to de Burgh, one of his most trusted advisors and custodian of Windsor Castle and Dover Castle. During the childhood of John's heir, Henry III, Hubert was the de facto ruler of England, and he poured money into Hadleigh Castle as a statement of his wealth and power. Unfortunately, de Burgh quarrelled with the king in 1239, and he was forced to cede all his lands, including Hadleigh, to the crown.
The castle remained in royal control. During the Hundred Years War, Edward III saw the strategic importance of Hadleigh in his attempts to defend the Thames estuary against French attacks. Edward extended Hadleigh, building a new gateway and tower, chapel, and living quarters. He added the twin drum towers which still stand today. He also built Queenborough Castle on the Kent shore of the estuary opposite Hadleigh.
Hadleigh became one of Edward's favourite residences, but his successors did not share his enthusiasm for the castle, and it was rented to a succession of tenants before finally being sold to Lord Riche in 1551. Lord Riche dismantled the castle and sold the stone for building materials.
The major remains include a pair of drum towers and a barbican gate. One of the towers was used by Georgian revenue men as a lookout in their war against local smugglers.