I have come across questions that became debates and users got evidently irritated in their answers or comments (I am not the one to judge if on good grounds or not). Is this enough reason to flag them as unfriendly? Or is this too harsh?
If you read a comment, and you feel that it could be perceived as unfriendly, you should flag it. The primary goal of the "unfriendly" flag isn't to punish the person leaving the comment, but to remove the sort of comments that could make people not want to participate on the site. If a discussion has gone off the rails, just flag one of the comments for moderator attention instead of flagging a bunch of comments as unfriendly.
I believe that only repeated unfriendly flags have an effect on someone's account, so if someone is just having a bad day and isn't usually unfriendly, your flag doesn't hurt them. It does get the unfriendly comment looked at by moderators, and acted on if they find it necessary. If someone is consistently being unfriendly in comments, the moderators should be made aware. The least disruptive way to do that is for people to flag comments they think are unfriendly and let the moderators sort it out.
The guideline in A guide to moderating comments from the network Meta site is if it is more likely to upset readers than it is to inform them, it qualifies.
Even comments between two users who know each other and are OK with the tone of the discussion can be candidates for an unfriendly flag. The site's content should be for everyone, and snarky comments can leave a bad impression on users who aren't familiar with the community. The discussion can always be moved to chat where it doesn't detract from the answer or question the comments were under.