I currently have 6 rep, and just got notifications for ELU saying, "Congrats, you've gained the privilege - ..." for things like remove new user restrictions (requires 10), vote up (15), and talk in chat (20). I tested up-voting an answer (on the main site) and couldn't (as expected). What reason might there be for the notifications?
My guess is that your answer got accepted, which added 15 points to your rep, but then the O.P. opted to change that, and accept a different answer. If that was the case, you may have lost the points, but I don't believe SE revokes the privileges.
A 15-point swing would explain how you jumped from 6 to over 20 and then back to 6, especially since your answer doesn't have any downvotes.
Moreover, I noticed that you answered a question posed by a newer user; I've also observed that newer users are often prone to quickly accept the first answer given, and then change that when an answer they like better gets posted later. (In my mind, there's nothing wrong with that practice, although I recommend not accepting an answer for at least 12 hours, and maybe even waiting at least a full day or so, to let a good number of users get a look at the question before deeming the matter "settled.")
The notification message is created when you reach the reputation required for a privilege, but the notification message is not removed when the reputation becomes lower than the required reputation.
A reputation could decrease because down-votes, the user offering a bounty, or another user changing idea on the accepted answer and accepting another one. In all those cases, if the reputation is decreased before the user has seen the notification about the new privilege, the user will read something that is not anymore true, until the user doesn't get more reputation. The notification about a new privilege is shown only once. This means that, if your reputation oscillates around the value required for a privilege, you will not keep to being notified.