It would be helpful if you gave the link to the question you described above. If you wish to keep your reputation points, I would suggest you leave the answer, unless eventually it gains more downvotes than upvotes. You can either edit the answer, or add another answer (I have done this in the past and it only incurred 1 downvote compared to 3 downvotes for the initial answer). I had a similar question about deleting a post and this is what Andrew Leach had to say:
"An answer which has a single downvote (score −1) will have incurred a penalty of 2 points. Deleting that answer will restore those points. It also restores the −1 penalty incurred by the downvoter.
An answer which has a net score of −1 reached by an upvote and two downvotes will have gained +10 for the upvote and −4 for the two downvotes. Removing that answer will restore the rep lost from the downvotes but also the rep gained from the upvote. That will result in a net loss of six points, the same as the net gain it had from votes. Again, the downvoters' penalty is also restored.
A similar mechanism operates for questions, albeit with different numbers (an upvote is worth +5, a downvote −2, with no penalty for the downvoter).
You can see your reputation moves on your profile page, and the public view shows the movement resulting from posts which still exist (1 + 10 − 2 − 2 −2 = 5). You may be able to see more detail there than other users."
If you are planning on deleting posts, don't delete too many, otherwise you may receive a dreaded "post ban" from Stack Exchange! Hope this helps :)