Never jumped. But I climb and I'm a SCUBA instructor. I agree that when something goes wrong under water, shit can snowball fast if you panic. There are physiological things going on under water that will get you quick.
One of the biggest reasons why there may be more fatalities and injuries with SCUBA divers is the sheer volume of people that have access to it. For a few bucks at an island resort, you can spend an hour in the pool then get on a boat. People get the impression that they now know WTF they're doing, and get into trouble fast. Compare it to maybe a half hour in a hanger, then a tandem jump, then the person goes off packing his own chute and thinking he's a jumper. Bad things waiting to happen.