You did research then come back to find new shitty answer
To take a narrow look at your premise that you basically dug up some research and then came back to the question to find a shitty answer I would very likely answer myself. There's two simple reasons for that.
Firstly, when the research wasn't that extensive, meaning the answers are likely posted relatively close to each other time-wise. So a minute or two in case of a SWR or maybe fifteen for something else is a time where no one can really complain. Sure I'll incorporate that someone else went down a similar avenue but that's it, mostly after reloading the page when I hit post. No need to retract the answering intention just because someone else started the same time and quit half-way.
Secondly, when the research was extensive the quality difference is very likely too significant to reasonably edit someone else's post.
Basically, if one can reasonably assume that I haven't read the post or the quality difference is too significant I self-answer.
There's a shitty answer, then you did some research and it turns out it hit the target in the first place without showing why
This is a lot more interesting and it's a very clear depends. You've pointed out the options, so let's look at them.
Commenting
Plus: On it's own this achieves two goals at the time you do it. It notifies the post owner of the posts deficiencies and anyone reading it that this is not really considered a good answer in the current state. Especially if the comment is up voted (like I just did with yours).
Meh: You've got to check back later to see if the poster reacted and retract the comment or escalate to editing or self-posting. In the mean-time you have to bench your research and the post stays in a low quality state. Later means at least a day considering that ELU users are all over the world and well do stuff besides ELU. Which also means your research is not fresh anymore when you do escalate to the other options later. You are not done after commenting.
There are two outcomes. Best case gives you the least effort, worst case it's the most effort of all options.
Conclusion: Use it if you think there is a likely reaction of the original poster which basically means consulting the crystal ball of your discretion. Can also be used to help new users. The profile might help. Don't bother with repeat offenders.
Editing
Plus: The research is incorporated into the answer. The overall quality of ELU rises. Yipee. The post is immediately more helpful than before. You should leave an additional comment or an edit comment explaining why you did it. You're done after editing.
Meh: As medica put it, you are spoon-feeding the user. The reputation goes to the original poster. Worst case it encourages people to just provide the basics and hope for guardian editors. Oh and of course some people really don't like people editing their posts. Retaliation may range from unnoticed rollback to angry comment to well, it's the internet. Trolls do what they gotta do.
Conclusion: Don't do it several times for the same user. Try to get the message to spent the effort when initially answering across. Don't bother with repeat offenders.
Answer separately
Plus: The research is incorporated into a helpful answer. The overall quality of ELU rises. Yipee. The Q&A page is immediately more helpful than before. You definitely have to mention the other user to share credit since the answer was there already when you started. The reputation goes to the one who did the research. (Of course, you could go CW.) You are done after posting.
Meh: Since a lot of the answers you talk about come from users who could really use more reputation you are taking the chance for them to do that with their idea. This might turn potential future contributors away. If the research difference is small, it can be perceived badly. If you do it a lot, it can be perceived as reputation stealing. If you explain that the other answer is shitty, you can be perceived as condescending (and thieving on top). Well again, it's the internet, you might very well piss someone off ;)
Conclusion: The less the other poster will miss that reputation the less he or she likely cares about reputation they missed out on. They also more likely know where meta is to rant about it. Furthermore users with a certain amount of reputation should know better than to post such garbage in the first place. The greater the research difference the less problems. Certainly do with repeat offenders. In any case, do credit the other user.
TL;DR: There is no hard and fast rule (that I know of). Generally speaking, take your effort, the original poster's effort, the post age and the poster's behavior in general into consideration when picking one of the options.
