I can totaly relate to your feelings. I started jumping at the age of 43 and went through everything you are talking about. My husband never jumped and percieved me or anybody in the sport as abnormal. Nobody in my surronuding ever jumped and I became a black sheep overnight. Skydiving became my passion. When leaving for the DZ I was "kissing my kids for the last time". Awful feeling, but simply couldn't stay away. The more I was learning about the sport, the more afraid I was. I realized how little attention I was given, how nobody really cared, how dangerous everything and everybody around me was. I still couldn't stay away. When I realized I was either going to kill myself with unadequate training, gear being packed by just about anybody (including myself long before I was ever cleared for it), plane that could hardly take off on most of the days... I decided, despite the problems at home to leave for a week, go south and get the best coaching, tunnel included (I highly recommend it!) and bought myself my first set of gear. Then and there I got my A licence and my confidence. Of course, the thought that something can go wrong is always there, therefore I try to be ready to save myself. I educate myself, I am aware who I jump with and expect others to make mistakes as well, I make sure the jump is properly briefed, I pack my own gear (if I let anybody pack I make sure I know who that is and I check everything twice), I do everything in my power to play safe, I am a conservative canopy pilot, I am never embarraced to ask questions, I take good care of my gear and most of all - I jump regularly. Being current builds skills and they bring the feeling of safety and calm.
I hope this will help you understand your, perfectly normal, emotions. You are not alone, but you have to learn how to control the fear by facing it and then focus on your jump.
PS. There is nothing more beautiful than being in the door, letting go into a "big blue" and knowing what you are doing. That is happiness!
I have one regret - not doing it on my 18th birthday! Never thought of it...