Just like you can upvote a question or answer and then undo that, can this be possible for comments as well? It would be nice to be able to retract a marked comment.
This was asked on Meta.SO and has a score of 233, and the five answers all support it (with a combined score of 342) along with all the comments, but still the feature request was declined for unknown reasons.
As noted in meta.stackoverflow (in an answer to the question Hugo mentioned) an un-upvote-comment feature (to retract or revert an upvote on a comment) was added in April, 2012.
Some restrictions apply: Your upvote is locked in if you navigate away from the page with the comment, or if 60 seconds go by. You can upvote a comment at most once.