The area to the north of Bury St Edmunds is not noted for its natural beauty. Here beyond the industrial area is where Bury's warehousing is pushed along the sleeves of the roads to Mildenhall, Thetford and Diss, and the dull Breckland scrub with its scattering of pine copses is home to acres and acres of pigs, the great agricultural enthusiasm of West Suffolk in these early years of the 21st Century. Perhaps pigs and warehouses are all that make money these days. But you don't need to go far from the main roads to find another West Suffolk, that of secretive, self-contained little villages, with beautiful little churches. Ampton is one.

I have never been very fortunate with visits to this church. In half a dozen attempts over a period of ten years or so, I had never found it open, and never had an answer to my calls to the keyholders. In any case, the key is several miles off in another village, and when you are on a bike that's a long way to call on the chance of someone being in, especially when the phone isn't being answered. However, coming here in early 2008, I took a note of the address, and found the house quite by chance later that day while visiting another church.

Some weeks later, I came back, and went to the house first. A very kind man came and opened up for me, and so it was that I was able to see inside this beautiful building for the very first time.

The church is small, and sits pointing towards the village street. Opposite, behind the wall, is the massive pile of Ampton Hall, rebuilt after a fire in the 1890s. St Peter and St Paul is attractive enough, especially in Holy Week with the daffodils pushing up around the cross of a grave to the east of the church, but on walking around it you discover something quite unusual, a chantry chapel set off of the north side of the nave. It dates from the later years of the 15th century, and was established by John Coket. It was served by its own Priest, who would often celebrate while the Parish mass was being said in the chancel, an insight into the way in which many medieval churches operate, although most of those with chantries were larger than this one.

St Peter and St Paul underwent a very early Victorian restoration, at the hands of the maverick Samuel Teulon. The first impression on stepping inside is that not much has happened since. This is a delightfuly ramshackle interior, with the rustic smell of age and an endearing shabbiness. So many parish churches must have been like this once.

What is most striking about the interior, and this is accentuated because it is such a small church, is the number and size of memorials. Mostly, there are to members of the Calthorpe family, successors to the Cokets at Ampton Hall. Dorothy Calthorpe, on the south wall, kneels in piety behind her prayer desk, like Mary at the Annunciation. Her inscription records that this virgin foundress of the almshouse left this life for a better... in the 45th year of her age. A virgin votary is oft in snares, this safely vow'd, and made the poor her heirs.

The two grandest memorials are on the north wall of the chancel. The most spectacular is to Henry Calthorpe and his wife, and a prominently placed cartouche records in Latin that John and Matthias, the Christmas Brothers, made it in 1638. Mr and Mrs Calthorpe sit patiently as if in a box at the opera, but it is the smaller figures beneath that catch the eye. Their children kneel either side of a prayer desk, weeping in sorrow - but almost every single one of them carries a skull, denoting that they died before their parents. What a sobering thought.

Beside it is the memorial by Nicholas Stone to William Whetall. He was the father of Mrs Calthorpe, and died at about the same time, in 1630.

Memories of modern Ampton are maintained by the east window, and the entrance to the Coket chantry. The window tracery dates from Teulon's restoration, but the glass records the destruction of Ampton Hall by fire, and the inscription tells us that it was given in grateful memory of January 3rd 1885, because, although the Hall was completely destroyed, no lives were lost. The three sons of the family are shown as Shadrach, Meshach, and Abednego, alive in the Fiery Furnace. The window depicts the risen Christ flanked by four Saints, St Peter and St Paul, the church patrons, and the East Anglians St Edmund and St Etheldreda. Mortlock says it is by Burlison and Grylls, and it is very good, looking the work of perhaps twenty years later.

The chantry chapel entrance is surmounted by the inscription Capella Perpetue Cantarie Joh'is Coket - the perpetual Chantry Capel for John Coket. But he could not know that it would all come to an end in a few short decades. The flanking niches were set with beautiful mosaics in the early 1920s. That to the west depicts St Christopher, and remembers that the rebuilt Ampton Hall was used as a military hospital during World War One. More than six and a half thousand wounded were treated there; the names of the forty who died are recorded on the north wall of the chantry chapel. That to the east depicts St George, and is in memory of Lieutenant Bernard Wickham MC, the only son of the Rector. He won his Military Cross on the Somme, and was killed in the Third Battle of Ypres in April 1917. He was just 22 years old.

The chantry chapel itself retains some medieval glass in the west window, including a roundel with a sacred monogram. There is a curiosity to the east of the entrance, a square recess. Couldit have been a reliquary to the chantry altar?

Set in the tower arch is an unusual fretwork royal arms for Charles I, roughly contemporary with the work of Christmas and Stone in the chancel. Either side of the tower arch are two elaborate consecation crosses peeping from behind broken plaster.

Between the two ends of the church, however, St Peter and St Paul has more of a flavour of rural Victorians, and of the great Anglo-catholic enthusiasms of the early years of the 20th century. It is at once a time capsule and a treasure house. Presumably, some day, there will be an enthusiastic churchwarden who will take it to task. He'll have the walls painted, the old iron stove removed, the floor tiles relaid flat and the windows cleaned, and the shades and shadows will all flee away to be replaced by a dull domesticity. So enjoy it while you can, it is wonderful.