The Church of St. Peter and St. Paul is an ancient edifice of early English architecture, with some remains of Norman work in the lower stage of the tower. It is of the age of Henry III and stands in the area of the ancient fortress. It consists of a nave with aisles, a south transept, and a chancel, with a tower at the west end.

Under an arch in the Hundon chapel, lies the effigy of Sir John de Hundon, who died about 1350. In the north wall is a cross-legged figure of Sir Wm. de Hundon, of the time of Edward I, and opposite it is the recumbent figure of a female. In the transept is a kneeling figure of Sir Edward Maddison, Kt. Who died in 1553. On the floor are two brasses of the Dusteby and Carrington families.

Until it was discontinued in 1847, a singular ceremony took place annually in this church, by the performance of which certain lands in the parish of Broughton, near Brigg, were held. On Palm Sunday, a person from Broughton brought a large whip, called a gad whip, the stock of which was made of wood, tapered towards the top. He came to the north porch about the commencement of the first lesson, and cracked his whip at the door three times; after which, with ceremony, he wrapped the throng round the stock of the whip, and bound the whole together with whip cord, tying up with it some twigs of mountain ash; he then tied to the top of the whip-stock a small leathern purse, containing two shillings, (originally 24 silver pennies) and took the whole upon his shoulder into the Hundon choir, or chapel, where he stood in front of the reading desk until the commencement of the second lesson; he then waved the purse over the head of the clergyman, knelt down upon a cushion, and continued in that posture, with the purse suspended over the clergyman's head, till the end of the lesson, when he retired into the choir. After the service was concluded, he carried the whip and purse to the manor house of Hundon, where they were left.

The chancel was restored in the early 1800's and a new clock was fixed in the tower in 1854. The Vicarage of Caistor , with the curacies of Holton-le-Moor and Clixby annexed to it, were valued at £7. 6s. 8d., and in 1856 valued at £300, in the incumbency of the Rev. H. Maclean, B.A., and in the patronage of the Prebendary of Caistor, (Rev. Dr. Hook, vicar of Leeds) who, in right of his prebend, was appropriator of the rectory. At the enclosure of the common, the tithes of Caistor were commuted for allotments of land - viz., 91 acres to the rectory and 80 acres to the vicarage; but the hamlets of Audleby, Fonaby, and Hundon, paid a yearly modus amounting to £52. 6s. 10½d. per annum for the rectorial, and £180 for the vicarial tithes. The Duke of St. Albans was lessee of the rectory lands.