Welcome to the life of an admin. You'd have to put up with that any single time you did an admin action.
Well, admin action I can do. I've dealt with it a lot of times when I used to be an admin for minecraft servers. I got it a few other times on Death Run when I was telling someone that they were breaking rules. So, I either started a vote mute or vote ban, depending on how bad it was, and that was that. If they continued to break rules, I reported them for it.