I can’t say I never downvote anyone but I get into these comment chains from time to time where really no matter what I post whoever I’m talking to downvotes every response but I generally shy away from downvoting even things I dislike. I don’t want to fall into an echo chamber where all I see are only the things I agree with.
Up and down votes are intended as indicators of quality that people abuse as markers of taste. If you downvote because the post/comment is unhelpful/useless/etc. that’s just voting as intended. If you are about to downvote because something is hateful, just report it to the mods instead. If you use certain clients, like Voyager, you will also see cumulative totals for your personal votes on users, which can be a great way to spot people you shouldn’t interact with, and probably just block. If you can’t tell the difference between people who are willfully nasty vs ignorant, maybe focus on learning to tell that rather than participate too much.