I would say to trust your instincts. Your posting gives the impression you feel he isn't ready, so I would say trust it and hold him back. I would not be at all worried that he will be bored in kindergarten -- many kids are already reading and writing when they start kindergarten and many times the activities are at various levels. Plus, so much about kindergarten is about socialization and learning the routines of school, and another year of learning to work in groups may help him. My daughter is finishing first grade now and was already reading in kindergarten (in the youngest quarter of her class), but there are lots of kids in her class who have been held back -- maybe 15-20 percent, mostly boys, and honestly they are not at the top of the class, mostly the middle. Remember that in many areas of the country the cutoff for kindergarten is way earlier -- some as early as July or August (ours is September). If you keep your son back, he will actually be in the class that he should have been if he had lived almost anywhere else in the U.S. This is important not only because when he goes to college he will be competing with these older kids, but also now with the common core standards most states are introducing, there are uniform requirements for math, reading, etc., for each grade -- this means your son is basically trying to do stuff that is a year ahead of him all along through school. Things really are getting more complex — my first grader is already doing some multiplication, when this used to be third grade! So the cut off dates may have been set in an earlier age when things were more laid back and kindergarten was for playtime and naps.
Also, I really think there is a readiness factor, where some things just click when you are ready to be there and it will be a ton of work (and frustration) otherwise. He may be frustrated with first grade math at 5 turning 6, but find it fun and interesting at 6 turning 7. I think one of the greatest gifts you can give your son is a love of learning and confidence in his ability to learn, both of which will be tough if he is not held back, as it sounds like everything may always seem "hard" to him, where it may seem "easy" or normal if he is held back. My youngest missed the cutoff by 5 days, and at first I was frustrated and thought that she would be very ready for school (like her sister would have been at the same age) but she is a different person than her sister and won't be ready ahead of schedule — if her birthday had been a week earlier, I would be considering holding her back.
So basically, if you are this strongly considering it, it probably is the right thing. I wouldn't worry about the "not going to clubs thing" (I skipped a grade and was way younger and this was really not a big issue) but I would worry a lot about academics and socialization -- not just for increased academic success, but also increased confidence, happiness and love of learning.