I think it is arguably valid to answer a question and vote to close it under at least two scenarios:
The question is fundamentally good but has some correctable flaw (such as no showing of prior research) that the answerer hopes the questioner will provide at a later date.
The question isn't very good and is closable on that or some other grounds, but (a) the answerer doesn't want the questioner to go away empty handed, and either (b) the explanation or other response is too long to fit in a comment box or (c) the answerer is categorically opposed to answering in comment boxes even when the answer would easily fit there.
I believe that I have sometimes answered a question and voted to close it under scenario 1. In any event, I think that the decision to do so is both reasonable and good for the site.
I may once or twice have answered a question and voted to close it under scenario 2(a & b). Doing so isn't especially good for the site, but it may be of some value to the questioner. That's the point of it, anyway.
But I haven't ever answered a question and voted to close it under scenario 2(a & c). That's because I don't agree with the rule against answering in comments—especially when it comes to answering an excessively simple or otherwise off-topic question. In such situations, answering in as evanescent a form as possible seems to me to be a good thing, and comments are notoriously evanescent at English Language & Usage. That doesn't mean that answering and voting to close under scenario 2(a & c) is indefensible, but it does (in my opinion) make usefulness to the questioner a more critical criterion for justifying it.
I suspect that your answer in this particular instance drew a more-than-usually hostile reaction from downvoters for two reasons: the answer could easily have taken the form of a comment; and it was not very satisfactory as a full-fledged answer. That is, it was so brief that it amounted to a simple assertion rather than a reasoned explanation—and as such, it was unlikely to be very helpful to the questioner.
This brings us round to my original rationale for answering flawed questions in the first place under scenario 2(a): to avoid having the questioner go away empty handed. If your answer had explained why there is no way independent of context to tell which meaning of "sex" a speaker or writer has in mind, I would have considered it reasonable enough. But an answer of "When is 'sex' sex, and when is it 'gender'? When the context indicates that meaning." seems to me to be more dismissive than explanatory. I don't think that your answer deserves multiple downvotes, but I also don't think that presenting it as a formal answer was the best way to convey the point that it makes.
You are a valuable contributor to this site, and I appreciate the quality of the vast majority of answers you have provided here. Nevertheless, I think that your answer to this particular question was not a well-conceived use of an answer box—and certainly was of no more use to the questioner than a comment in the same words would have been.