I appreciate your concern -- I have it as well. One of the DZs I haven't talked to yet. The other one who knew my reputation well didn't have a problem with it when I wore my Canon HV-30 (which apparently doesn't work for skydiving) for the maiden flight of my new rig. The risk then was negligible as the camera is top-mounted, the jump was solo and I pulled high at 5,300 for safety and test flying.
I chose to not use the camera for the subsequent jumps given their nature - a 4-way, a hybrid, and a 2-way.
From what I've read the presence of a camera isn't unlike using a cell phone in a car. Fortunately, I'm often accurately accused (if I'm caught) of not paying attention to the other caller while I'm driving and there are plenty of occasions when I'm on the phone where I end the call due to conditions. And there are plenty of times where I refuse to use a cell phone at all.
When I use it will depend completely on my comfort with the jump, the jumpers involved, and the jumpers' comfort with me and/or my camera. If anyone says no - no questions asked.