Over the last couple of centuries, the Borough of Lowestoft has sprawled out to encompass a number of historic parishes, including Gunton, Pakefield, Oulton, Carlton Colville and here, the former village of Kirkley. Consequently, seven of the churches in the Borough are medieval - or, more strictly, of medieval foundation, for there was a spectacular 19th Century restoration here at St Peter and St John, which has left very little of medieval origin. The church, then simply dedicated to St Peter, had been abandoned in 1680, after more than a century of neglect. The parishioners moved in up the road at Pakefield, but when the two parishes there were combined, and the dividing wall was opened up, there was no longer room for them. So they came back, patched up this church, and suffered in it for the next 120 years. However, by the 1870s, when Lowestoft had expanded and engulfed the parish, the time had come for a total rebuilding. Another church, the now demolished St John, had been built in the parish ten years before. But this was not enough for the huge population, so St Peter was built anew between 1874 and 1876 by the architect John Clemence. All that survives of the medieval church is the heavily restored tower, which once stood at the west end, but the new building was extended southwards, putting it at the north-west corner.

From the north, it looks conventional enough. But on the south west side there is a delightful surprise; an apsidal baptistery, the work of Thomas Porter in 1893. This is quite unfamiliar on an Anglican church in Suffolk, although there is something similar at St Edmund Catholic church in Bungay.

Like all churches, Kirkley Parish church continues to reinvent itself. When St John was demolished in 1976, it took on the current dual dedication. These days, the name Kirkley is roughly synonymous with that part of coastal Lowestoft south of the river, which isn't entirely accurate. The main road to Ipswich is a ratrun through the parish, giving it an urban scale beyond its sizen. Off this parade of busy little shops you find yourself in a pleasant residential suburb. Towards the sea, the houses are enormous. This area, Kirkley Cliffs, is a rather pleasant seaside resort, usually referred to as Lowestoft South Beach. Until quite recently this was a fairly seedy area, but it has undergone a huge improvement programme in the last ten years, thanks to an injection of money from the European Social Fund for deprived areas. Before coming here, I had been chatting with the man at South Lowestoft Methodist Church about this. "A lot of local people think it was a bit of a waste of money", he observed. But now, Lowestoft's are the only beaches in Suffolk to win Blue Flag awards, and are of a beautiful golden sand. Why people would rather go ten miles up the road to seedy, be-littered Great Yarmouth beats me.

In one of the large houses, now a guesthouse but until recently a dentist's surgery, as it had been for nearly a hundred years, one of the most significant events in Kirkley's history occured. In this house, on November 22nd 1913, a baby boy was born to the wife of the dentist of the day. This boy, their youngest child, would grow up to be the most famous Suffolker of the 20th century, and it is entirely appropriate that Benjamin Britten should be born on St Cecilia's Day, for she is the patron saint of music.

Britten is more closely associated with Aldeburgh, Snape and Orford than Kirkley, not least because of the music which was first performed in the churches there. But it was at the High Church St Peter and neighbouring St John that Britten first experienced the power of the liturgy, and church music first entered his consciousness. There is a blue plaque on the house, but no other fuss seems to be made of Britten in Kirkley; you have to search to find him. At the northern end of Lowestoft is the Benjamin Britten High School, and, more surreally, the Benjamin Britten Shopping Centre is in the centre of town. But Kirkley seems rather reserved about its genius son, who was certainly the greatest English composer of the 20th century, and perhaps the finest English composer of opera ever. He was also one of the first establishment gay figures to openly win acceptance, and he is buried today beside his partner for 40 years, Peter Pears, in the graveyard at Aldeburgh.

The first time I came here, shortly after the start of the new millennium, I walked around St Peter and St John on its little hill, surrounded by pleasant little terraced houses. They must have been built at about the same time. The church looked abandoned from this aspect, but I took this to be merely another manifestation of the urban shabbiness that Lowestoft in from the coast seems to rejoice in. In fact, the noticeboard at the east end, facing Kirkley High Street, showed that this was a lively parish.

Before I go any further I must tell you that this church is now open every day, but back then it wasn't, and it wasn't for almost another ten years that I came back to Kirkley and tried to get inside again. Actually, I did not have try very hard, because I visited on the day of the Historic Churches Bike Ride 2009. It is worth adding that, having taken part in this event for more than twenty years, I have a wide experience of the kind of people who sit and welcome cyclists to churches on this day. I can tell you, in all honesty, that the people of Lowestoft are probably the friendliest people I have ever had any dealings with - and not just the church welcomers. In shops and pubs, on buses and trains, even when just walking down the street, these are the kindest, most courteous people you could possibly hope to meet. And yet... and yet. When I turned up at their churches and asked to see inside back on that Historic Churches Bike Ride day of 2009, they looked at me as if I had requested something really off the wall, like wanting to examine the contents of their wallets. I was treated with mild surprise and some raised eyebrows when I asked to step past the welcome desk into the body of the church. No one actually refused, but you could tell that they thought I was a little odd. But thankfully, those days are now gone at Kirkley.

I suppose it is a cultural thing. Most parishes see their building as an act of witness in itself, and that by keeping it open they offer a prayerful space in the middle of a busy world to passing pilgrims and strangers. Evangelical churches are, not unreasonably, less likely to take this approach - they want you to come on a Sunday and join the Club, after all - but Kirkley parish church is very much in the High Church tradition, and it was a delight to finally set foot inside St Peter and St John. It is a big, urban, late 19th Century Anglo-catholic space, with white- and red-brick banding creating a polychromatic effect, a candle flickering in front of the statue of Our Lady of Walsingham by the south doorway, and jewel-like windows piercing through the gloom. The best is in the east window, good Kempe glass depicting Christ in Majesty flanked by the four archangels Michael, Raphael, Gabriel and Jophiel.

To get to it, you step through the excellent wrought-iron rood screen of the 1890s. On the north side of the chancel is the organ where Benjamin Britten's mother came to practice on weekday afternoons, the small boy sitting beside her and turning the pages. It is apparently one of the few four manual organs in Suffolk.

There is, of course, no tower arch, and a great clock ticks steadily at the west end of the nave. The Ward & Hughes glass down the south aisle is not the workshop's best, but contributes to the feel of a faded, early 20th century Anglo-catholicism which is being lost to us by reorderings and redecorations elsewhere. One set forms the war memorial window, a touching moment. At the west end of this aisle is the church's most memorable feature, the open baptistery with its wrought iron screenwork and spectacular marble font. I did try to imagine the infant Britten being dipped into it, but I suspect that he was actually baptised at the now-demolished St John. The nice lady on duty, seeing me gazing at the font, came and apologized for its dusty state. "The whole church has got so dirty, and it needs a thorough redecoration. But it is so expensive, and there are so few of us left now." I told her that I rather liked it the way it was, but I suspect that was small comfort.

One intriguing survival is the small reredos made by Robert Anning Bell for St Peter's church in north Lowestoft, which was demolished, along with St John, in the 1970s. The rationalisation of Lowestoft's Anglican churches at that time bolstered congregations in the churches which survived, and long may they continue to do so.