I have been doing a program by Ralph Hennig called Four Way Burn and it utilizes a 4lb medicine ball.
I have noticed positive changes in my shoulders and I really like the workout.
It's very good for flexibility.
You do alot of pushing the ball away from you (from you chest) and lifting overhead.
I've been doing the workouts at least once usually twice per week for over a month and I still feel a little sore in my shoulders the next day, so I do feel that it's working my muscles, even if it does only weigh 4 lbs.
So, I'd say go for it
It is fun too!