A return to Rushbrooke, and I knew it would be the same. Only the landscape changes as the seasons turn, and I have been here in all seasons, both in the heat of high summer and with snow on the ground. Now, at the end of September, autumn was mellowing out, and a breeze from the east was beginning to tug at the goldening leaves. And I remembered another visit, a good ten years previously, one bright Saturday in early spring. That day, I do not think I had ever seen so many hares before, but they were in every field as I headed south-east out of Bury St Edmunds up the rambling, hilly lane to Rushbrooke. I'd seen half a dozen before I was even a mile from the turn-off. Two of them were having fisticuffs near the pig fields, and I hauled my bike off into the verge to watch.

I must have stood there for twenty minutes, gazing through the hedgerow, and they must have known I was there - the pigs certainly did. But spring madness had infected them, and they had other priorities than worrying about me. I got even closer to one old fellow. He was as big as a dog, and lay sullenly in the furrow of a ploughed hillside barely twenty feet from the road. Perhaps he was sulking that, at his age, he was missing out on all the excitement. Eventually he lifted himself out of the rut, and hauled himself up the bank, his powerful legs kicking back slowly behind him, as if he owned the place. Which, of course, he did. And so I got back on my bike and, with the lunatic hares oblivious to the way they lifted my heart, I had free-wheeled my way down into Rushbrooke.

And today it was good to come back again. If you like something a little out of the ordinary, then Rushbrooke is exactly the kind of place you'd find interesting. On its ridge above the valley, this little settlement is quite unlike any other. Rushbrooke Hall, home to generations of the Jermyn family, was Suffolk's largest and finest moated Tudor mansion. Used for housing troops during the war, it suffered a mysterious fire, and was demolished without permission in 1961. Pevsner called it a capital loss, a tragic disruption of the post-war Suffolk landscape.

There is a late medieval brick well house at a turn in the lane, set between rows of extraordinarily good farmworkers' cottages. They were built in the early 1960s, by the Llewelyn-Davies and Weekes Partnership. St Nicholas is just beyond, looking reassuringly familiar, its 14th century tower and 15th century everything-else all plastered. Mortlock tells us that, beneath this skin, the tower is flint, but the rest brick. Indeed, brick surrounds are evident on several windows. There are fine headstops on the west window, and on the windows of the south aisle. There seems to have been an early 16th Century makeover, which brought the porch and south doorway, and perhaps the crow-stepped gables, the height of which show that the church was once thatched.

If I tell you that this church was extensively restored in the 19th century, you might imagine a well-polished but anonymous interior, all Minton tiles and deal benches. You'd expect recut stonework and a garish reredos. It would all be pleasantly ordered for late 19th century sacramental worship, rather out of date now perhaps, but obviously cared for, as at thousands of other English village churches.

If you thought that, then you would be in for a mighty surprise. For this is perhaps the most extraordinary of all Suffolk church interiors. It is the work of, and a testimony to, one of the great Suffolk eccentrics. The first time you realise that something decidedly odd has taken place is when you step from the porch into what you believe to be the south aisle. Instead, you find yourself in a most unusual vestibule. To the east, your right, is a solid partition wall with a door in it, blocking off the south aisle. A few feet ahead of you is a wooden partition, less substantial, and not going all the way up to the ceiling. The intention seems to be a baptistery, and there are two 19th Century fonts, a stone one set on a medieval base, and beside it a wooden one.

I'm not sure where the stone one came from. It was brought here in the 1980s, and reunited with its column. It may even have been placed on the original base, which had been removed in the 19th century and obviously spent a century or so out of doors, to judge by the weathering. But the wooden font is more interesting, a piece of Victorian curiosa. If you look inside, beneath where the bowl would be placed, an inscription tells us that it was presented to St Nicholas Church Rushbrooke by Henry Sturgeon the senior churchwarden of the parish for 35 years. Behind the font, low down on the wall, is a monument to the man responsible for Rushbrooke church being the way it is today, Colonel Rushbrooke, who lived at the Hall in the early 19th century. If you have already visited nearby Nowton, you will have seen there the 84 roundels of Flemish glass plundered by the Rushbrookes from Belgian monasteries. Here, Colonel Rushbrooke recycled panelling from the Hall and other sources to create a glorious Gothick Fantasy.

For the best first impression, ignore the entrance into the main part of the church ahead of you, turn left, go past the fonts and enter the body of the church from the west. As you pull aside the curtain and step inside, you may be unbalanced slightly for a moment by the sheer lack of familiarity. Colonel Rushbrooke, inspired perhaps by happy memories of his youth, recreated here a college chapel quire, along the lines of Peterhouse, Cambridge. Banked dark seats face inwards, awaiting choral scholars. Old bench ends were pressed into use, but the arcaded carving in the 15th Century style along the fronts is all the work of the Colonel, the spandrels depicting pairs of sturgeons, shrouded skulls, dragons and even rabbits.

At the west end, above your head as you enter, there is a grand array of organ pipes. In fact, they are an elaborate fantasy, a conceit connected to no instrument, purely for decoration. Turning east, the tympanum is still in place, and on it is something even more remarkable, which we'll return to in a moment.

Beyond, the chancel is full of light, a balance to the serious gloom of the quire. Medieval glass is offset by blue surrounding panes, and the banners of Jermyns and Rushbrookes hang down in front. If you turn southwards, you see that the part of the south aisle partitioned off from the entrance. It forms a funerary chapel to the Jermyn family. Of several monuments, the best is to Thomas, the last of the Jermyns. He died in 1692, in a boating accident on the River Thames. He was just 15 years old. A mast collapsed, and landed on his head. And so, after centuries, a great landed family became extinct.

Colonel Rushbrooke's vision is at once absurd and splendid. And yet, perhaps he thought that it wasn't quite triumphant enough, and did not articulate sufficiently the Establishment of the Church. So, on the tympanum at the east end of the nave, he put in place the great royal arms of Henry VIII, the only Henrician set of arms in all England.

Various claims have been made that the arms are, in fact, genuine. The church guidebook still insists on this, pointing out that an installation could have happened at the time of the rebuilding of the roof in the 1530s. Before scoffing, it is worth exploring further. The set of arms was not here in 1840. Its placing in the church therefore roughly coincided with Colonel Rushbrooke's reordering. Thus far, simple enough. And yet, the coat of arms is a rather more primitive piece of work than the furnishings of the chapel below. Simply, it looks older. In raised lettering on the former rood beam below it is the motto Dieu et mon Droict.

We know that Henrician coats of arms were put into churches. At the time of the Marian restoration in the 1550s they would have been removed and destroyed. It is simply beyond all theological and political credibility that a set could have survived in situ. No symbol of loyalty to the crown could be used to express disloyalty to that crown. They were all replaced by crucifixes - or, at least, that was the intention.

At Ludham in Norfolk, the tympanum was retained, and the rood painted on it. Elsewhere, roods seem to have been reconstructed enthusiastically, but Mary died before her counter-reformation was solidly in place. Also apparently destroyed, of course, were Edward VI arms, although in practice these must have been few and far between. If most churches obeyed the order to install the royal arms of Henry in the previous reign, we may assume that those which didn't would hardly have been disposed to install them as an act of submission to the lunatic policies of his young son's advisers. There were no Marian arms, and Suffolk's only set of Elizabeth I royal arms is at Preston, a magnificent object. There are three more in Norfolk of about a dozen surviving Elizabethan sets in all England, that's all.

So, where did this set of arms come from? Is it possible that it could be genuine, and removed from a church by the order of 1553, it survived the 290 years in storage somewhere? And then, Colonel Rushbrooke found it, bought it, and installed it here? Is it even possible that it came originally from this church, and was stored at Rushbrooke Hall?

Or is it possible that the arms are something wholly different, and were never designed for a church? The arms of Henry VIII are also those of Henry VII. Many were produced in the late 15th and early 16th centuries to further the hegemony of the Tudor cause. Rushbrooke was an enthusiastic collector, and might have tracked this set down in a public building, for instance. He was enough of an antiquarian to know that a Henrician set in a church would be unique.

Or perhaps this is all complete speculation, and he made them specially. Cautley hedged his bets, although his posthumous editors scoffed. Mortlock was also tongue-in-cheek about it, but I don't suppose that we will ever really know.

In the years I have been coming here, I have grown very fond of this utterly unusual little church. It is quirky in so many respects, from the aspects already described to the splendid little Flemish roundels of unicorns, which didn't make it into Nowton, the skulls carved deep into the walls of the south aisle, and the great hatchment hanging a few feet above the floor. It was good to see it all again.

Stepping outside into the churchyard, you can wander across to the enclosure of Rushbrooke graves. In this lonely spot they seem grander than they are, and it seems odd to think of the major landed families who have this church and churchyard as a touchstone. As well as the Rushbrookes, there are the Jermyns inside the church along with the Davers and the Hervey families into which they married, not to mention the Rothschilds who have owned the estate for these last eighty years or so. One of the Rothschilds is buried to the south-east of the church beneath a simple headstone.