I think if you want to go spend time with your family and you think you can enjoy the park without feeling too bad about the rides, you should go. When my family visits parks, my mom never rides the rides because she doesn't enjoy them but she still has fun.
But if you're uncomfortable or you feel like it will not be a fun experience, it's okay to pass on the trip. Like you said, not everyone enjoys theme parks. Maybe you can meet up with your family before or after the park instead