The distinction between organized religion and broader religious belief or "spirituality" is part of it, but even within organized religion there's a huge spectrum of different beliefs. Essentially what I'm saying is that it doesn't make sense to assume that everyone with a certain religious belief is the same as their worst representatives (e.g. young-Earth creationists), any more than it would make sense to assume that all atheists are as obnoxious and attention-hungry as Christopher Hitchens and Richard Dawkins. Lots and lots of extremely intelligent people have been thinking about the issue for millennia, and most of us have reached a comfortable balance between faith and reason.

And yes, holy books can be both creations of Man and divine works, if you believe that they were inspired by God (which can itself have a range of different interpretations), and then translated by imperfect men, reflecting the beliefs of a certain time period. Again, I think the problem is when someone takes either extreme, assuming that their beliefs take precedence over another person's: either when a self-proclaimed Christian, for example, uses the Bible as defense of something that is directly contradicted by the rest of the bible, like bigotry; or when an atheist dismisses another person's holy scripture as "Some book you like." If someone has no respect for another person's beliefs, that's his prerogative, of course. But he can't then claim he's being oppressed when he's not treated with respect in return.

Coincidentally, io9 has a blog post about "smug atheists" that surprised me, how much I agreed with it.