While it's clear he was calling several votes against you, there is nothing in chat to suggest he was abusing it.
For all we know he may have been in the right (not saying you were doing anything wrong, but we can't assume he was either based on just a few votes and nothing else.)