Maybe they never really did anything seriously wrong, but oh my god they are annoying af.
Have you ever been watching your favorite show, all comfortable and relaxed, and then that one character that you just can't stand pops up on the screen and ruins your mood? I know I certainly have. These characters aren't necessarily evil or horrible people, but they do have personalities that make them incredibly irritating. They're the kind of characters that make you want to yell at the writer, "what the hell were you thinking with this one?" Perhaps they have never hurt a fly, but it doesn't matter since their character just straight-up sucks.
By the way, some of my arguments contain spoilers, so fair warning.