What you fail to realize is that different people have different levels of risk aversion and tolerance.
It's not that they don't understand. They do understand.
They perceive BASE as dangerous, and rightly so. There are very few sports out there that have the same fatality rate. (even though no scientific figures exist for BASE) Your golf and hunting analogy is inaccurate in that respect.
As such, they think that this level of risk is not worth it for a 'pastime/hobby'. Too much to lose.
BTW, I'm not a base jumper. I'm not even a jumper. This winter I hope to start AFF. Maybe, one day far into the future I will try base.
But as a sportbike rider (road and track) I get the same attitude, for the same reasons. I've watched my buddy die in front of me, and had other friends die or get seriously hurt. I've never been to a trackday where someone hasn't used the ambulance or was medivaced out.
For some people, the benefits out weight the risks. For others, they do not.