Swimming in open water can feel very different than a pool. I grew up swimming in lakes and the ocean, so it is generally very comfortable for me, but every once in a while if I'm doing the crawl and the water is choppy I get nervous if I go to take a breath and get a mouthful of water.
I would follow your own instincts on this. Rather than jumping right into choppier water, maybe try going back to the calm bay. And then if going over your head makes you nervous, try swimming around where you can touch, and every once in while turn over onto your back and float. Floating is key to my feeling comfortable in the water - if I'm swimming and the waves start smacking into me or if I get disoriented and I'm not sure where I am, I'll just flip over until I've gotten my bearings. I also alternate the crawl with the backstroke, because I can keep track of what's going on around me a little better when I'm doing the backstroke. My final suggestion is to start out by swimming parallel to the shore - it helps to have visual cues that tell you where you are and where you're going.
Once you're very comfortable swimming where you can touch, then move a little tiny way further out, and repeat, with the reassurance that you can always float for a bit or skedaddle back to where you can touch if you need do. I think by moving further out gradually, you'll gradually feel more comfortable. And then when the bay is good, move on to waves!
I don't know if there is a Danskin women's triathlon near you, but there were special open water training sessions available for the one near me. So you could check to see if there were any similar sessions in your area.
I'm envious, because I was training to do a triathlon this summer, but had to stop due to a back injury. I did one as a relay many years ago, where I did the running and biking and my sister did the swim, and I've been a spectator twice. They're really fun, so if you're inspired to do one, don't let the open water part get in the way!