There's nothing wrong with pulling back as needed. Clearly, you are taking care of yourself just by being aware of your needs. That's huge. Give yourself a pat on the back for that!
And, I'm sure you've heard it before, but it bears repeating...you can't control anyone but yourself (and even that is subject to interpretation). What matters is self-compassion. Do what you need to do to take care of yourself in a compassionate way. Taking care of yourself does not deplete another. If that "offends" the other person for whatever reason, then they are completely misinterpreting you through their own misguided filters, which you are not responsible for fixing, nor do you have the time, money, or expertise to do so.