I upvoted an answer to a question, and after a few seconds realized I had upvoted the wrong answer, so I hit the downvote button to put it back the way it was. After moving on to another question I noticed I had lost a rep point.
Why would changing an upvote be regarded as a downvote? This amounts to a returning of the status quo ante, and since the system clearly knows it was me who upvoted it, it should surely be able to tell that it is I who downvotes, and thus treat it as a neutral act, not an actual downvote.