I second Mel that it's time to transition to free weights for at least some of your exercises.
Like Mel said, free weights will protect you from injury both in and out of the gym because you move in a more natural range of motion and strengthen lots of secondary and stabilizing muscles. There are good machines that I use often (lat pulldowns, assisted pullups, rear delts, Hammer strength, for example), but probably 90% of what I do in the gym uses free weights.
About the rep range - right now you're working in a rep range designed to maximize muscle
size (hypertrophy). If size is your goal, you'll want to stay in that 5 - 8 rep range. Being female, you don't have the hormones to get really big, bulky muscles, but this rep range will certainly get you the largest muscles possible.
If you decide that strength is your goal, you might want to consider moving to an 8 - 12 rep range. Moving up to a 12 - 15 rep range would focus more on strength
and endurance. The right range for you just depends on what your goals are.
But regardless of the rep range you choose, you're right to push yourself until you give out, as you said.
However, while you're transitioning to free weights, I definitely recommend that you drop the weight until you've mastered the form. You'll be surprised at the difference between machines and free weights and the amount of stabilization it takes to do a dumbbell chest or shoulder press. You may be strong in one plane of the motion - the one used in a machine - but quite weak in the others that are needed in a free weight exercise. It would be easy to hurt yourself by trying to use too much weight too fast, and we don't want that to happen!
Pick a light weight and watch your form very carefully to be sure you're doing the exercise correctly. It's helpful to use a mirror to check and see if you're wobbly or shaky or moving off track. If you work out with a friend, it's really helpful to have a spotter when you first start with free weights!
Good luck and let us know how you do!