The Perpendicular parish church of St Mary's has a memorial to Sir Joseph and Sir William Hooker, famed botanists who served as the first two heads of Kew Botanic Gardens in London. The 15th-century font has carvings of Suffolk woodwoses, wild men bearing clubs.
Across the lane from the church is a row of almshouses dated to the late 17th century. These were built with money left in the will of William Cary (d. 1686), who wanted to provide a home for a dozen poor single men and women. Cary's will prohibited inmates from smoking, but renovations to the property revealed a cache of smoking implements hidden in a wall cavity.
The almshouses are home to the Halesworth and District Museum, which includes displays on the witchcraft trials here in 1645 when 4 residents of Halesworth were hung. On the upper floor of the almshouses is an art gallery.
The area around the market place retains some lovely old buildings, one of which bears a carving of a mortar and pestle; a reminder of the days when Halesworth was famed for its apothecaries. Nearby is the Angel Hotel, a coaching inn dating to the 16th century that later acted as a posting house for coaches on the road between London and Norwich.
The Halesworth website has an interesting town trail, taking in some of the major buildings of historic interest.