Kenshi, Ermac, and Scorpion give me the most trouble.
Kenshi and Ermac have very similar moves, so if you hate one, then you typically hate the other too. I have so much trouble with them mostly because the psychic moves are very hidden. I just don't see what specials they are pulling off very easily.
Scorpion wouldn't irritate me at all if it wasn't for hellfire. Hellfire forces me to change my game against a teleporter. I don't consider Smoke a problem, but hellfire makes Scorpion a pain in the ass.