Following the Dissolution of the Monasteries under Henry VIII the priory was owned by Sir Henry Jerringham. In 1547 Jerringham incorporated the monastic buildings into a new manor house on the site. This house lasted until 1784.
In 1823 stone from St Olave's was used to refurbish the parish church at Herringfleet. Two years later the refectory undercroft was converted into a cottage, which was inhabited until 1902.
Of the original monastery, only parts of the refectory, cloisters, and church survive. To the north of these buildings, part of Jerringham's 16th-century mansion can be discerned. The 13th-century church, to the south of the cloister, consists of 5 bays. Parts of the columns that supported the roof of the cloister walk still stand, as do sections of the west wall of the cloister range.