Follow up to AI to detect ELL migratable questions

Question

NOTE: MY USE OF MIGRATION AS A TERM IS INACCURATE IN THE QUESTION, SEE MY ANSWER OR COLLEEN’S ANSWER FOR MORE ACCURATE APPLICATION OF THE ALGORITHM.

This is a followup post to my last post (Artificial Intelligence to Detect ELL worthy questions) where I asked about the feasibility of making a machine learning algorithm to offer suggestions to migrate certain questions to our sister site, ELL, given that there are plenty of answers that should have gone, but remain. (in the future this can be extended to an algorithm that detects hw questions, lazy questions etc, but this is beyond the scope of this post)

Since the subject generated debate, I have taken liberty to use Colleen's generously provided query as a basis to train an ML pseudo-Bayesian model as an attempt to provide a proof of concept for you guys.

First, note that I am not a mathematician, nor am I a statistician or a data scientist, I'm a bloke with too much spare time on his hands. As a result, I used the most rudimentary of techniques, and modified a Bayesian classifier algorithm to work with the limited data set that I have, and my own technical limitations.

In a nutshell, this is how this algorithm works:

1. I took a practice sample of questions that were positively received on ELU, and counted word frequencies. (sample size approx 3k)
2. I took a practice sample of questions that were migrated to ELL and positively received there, and counted word frequencies. (sample size approx 3k)
3. I selected a random test sample of questions (the ones you see below) from ELU, and the algorithm, using a modified version of Bayes Classification, compared the words in each question in the test sample, to both practice samples. The score was determined based on which practice set it resembled more. A higher score for a question meant my algorithm deemed it resembled migrated questions more.
4. Finally, the algorithm sorted the questions based on score. Therefore, the ones that appear first have low "ELL migratability scores" and are more suited to this website. As you go further in the questions, you will notice they get more suitable for ELL, or even closure.
5. I chose the 25% as a potential arbitrary cutoff point, where the ones above above are ELU standard, and those below are more likely ELL. This is just a guess, and you will find questions of both types on both ends of the fence, as mentioned in the comments.

I hope you agree that the algorithm did a good job in finding a general gradient pattern for the eligibility of migration, albeit a few odd choices.

Without further ado, the rankings of my test sample (from most ELU to most ELL):

314221 310252 308130 309133 499455 305003 310769 499265 300221 310832 307706 309252 300478 309577 302802 304714 305169 304639 307852 301554 307789 302209 499522 308004 307748 307459 499714 303864 307555 499348 301579 302328 302507 309586 499164 310768 310803 302117 499608 304679 499598 301556 304316 302478 304397 309749 499588 300596 309484 309070 302908 305074 499435 303861 307493 304551 301446 309373 300015 310853 312053 309752 311955 499591 499663 499179 499557 300510 300056 309543 499301 307534 309825 303854 304886 304948 302531 499097 499278 499613 307806 309518 499515 499644 499465 499698 310811 499271 499587 499239 308371 301853 300305 300419 304125 499601 499527 499676 304983 312102 499554 499643 499500 314249 499567 304080 302832 310234 304013 499253 499211 499339 300498 308476 307837 310619 499657 307436 304567 499518 304109 499316 499540 499548 309069 499571 499020 499129 499293 302091 309649 499501 499449 499308 499106 305123 309197 309123 499503 499038 309516 499541 312131 302161 305290 304624 300518 305273 300474 308549 499479 310843 302579 499243 305180 499295 499349 499647 301816 499180 499344 314308 302451 499040 499605 305318 499195 302793 499255 499496 305277 310596 499098 305143 499109 499304 307407 499018 499227 305233 499566 499037 499603 499409 310257 307507 499569 304990 307403 310780 304867 304259 499258 300547 307719 303939 499506 305015 301761 300050 499036 499384 301786 499419 499224 499356 499199 499066 300095 499351 307938 499648 499590 499623 499519 499189 309318 499402 499421 499262 499429 499706 300317 499460 499679 499085 310636 499147 499568 499219 310631 499382 304529 311933 309693 499256 306279 302748 499373 306157 499578 499251 499041 310627 499297 304341 499016 499190 300516 499551 499456 499222 499314 499035 499193 304248 499439 499383 499099 310298 300120 302610 300324 302250 499669 307865 499090 304824 308138 309718 499666 314316 499662 310313 499220 312081 499536 499472 499427 302095 308046 311867 499615 499555 499417 499675 306331 314213 300417 499284 303920 304697 307730 499504 499664 499238 499116 499559 499244 499057 300210 499207 499102 499653 499690 499641 499318 499089 499625 499616 304274 499046 309281 499660 499539 499650 304751 499392 304281 499100 499010 499570 499646 499300 499115 499537 499358 499394 499404 499261 499561 499487 499523 499137 499408 499194 499079 499445 499649 25th PERCENTILE POSSIBLE REJECTION POINT300092 499218 499697 499560 499491 499149 302628 499260 499312 300334 499120 499014 499076 499467 499709 499545 303804 300187 499299 499604 308668 499032 499634 304387 499520 499175 499257 499173 499213 499345 303928 499201 499618 499067 499438 499680 499371 499324 309434 499672 499547 499051 499483 499529 499186 307526 499379 499400 499678 499132 310217 499549 499336 499619 499677 499127 499639 499168 499661 499287 499342 499056 499283 499694 499015 499635 499626 499340 499229 499236 499575 499444 499452 499044 499338 499655 499163 499624 499528 309813 499424 499171 499288 499328 499565 499092 499185 499167 499172 307553 499152 499387 499033 499322 499329 499031 499162 308567 499370 499461 499482 499027 499309 499048 499148 499538 310614 499535 499249 499508 499055 499713 499259 499183 499001 499691 499248 499117 499692

Cheers

++ Please note that I do not have a defined plan with what to do with questions detected by the tool, my only claim is that most of the questions below the 25% mark are not suitable for ELU. While I see they are better on ELL, some decided they should be closed. We agree on one thing: they do not belong on this site. And it is alarming that 25% of randomly selected questions do not belong here. That is why I decided to launch this idea in the first place

Answer 1

This is a very cool way to detect questions that should be put on hold, but not a great way to detect questions that should be migrated. The problem is a good measure of “well received”. Both sites’ communities love to answer word or idiom requests, and those often hit the HNQ, so the amount of engagement with them tends to not correlate with how valuable they are to the site. Questions that make it into the close queue, and therefore become candidates for migration tend to be low quality. I’m not sure we have the data for whether the migration was controversial or not, but that might be another data point.

Maybe it would make sense to select exemplary questions from each site for the training data instead of migrated questions. We have a post with some examples of questions with good detail on ELL’s meta. Any question similar to those would be welcomed. Maybe it would make sense to look at how many times a question was added to someone’s favorites?

What we don’t want migrated are questions like Listening is to hearing as learning is to? There’s no explanation of what the author has thought of as a possible answer. The author isn’t new to Stack Exchange and should know how to ask by now. They appear to be very fluent in English based on their other questions. We don’t want those sort of questions migrated, especially when the author isn’t in our target audience (people learning English as a foreign language).

There is an older discussion of migrations on ELL’s meta where I left some examples of good and bad (in my opinion) migration candidates. At the time J.R. had a more lenient view than I did on low quality migrations if the user was new to Stack Exchange. I am less opposed to migration of low quality questions for new users, but I think directing them to ELL instead of migrating their question goes more smoothly. Migration can be confusing for new users.

Answer 2

It has been a privilege to have the opportunity to engage in debate with you guys, and I thank you for giving me your time. After fruitful discussion, I have learned more about what exactly it means to migrate a question, and this might need us to re-examine the scope, and training set, of such an intelligence.

As one of you pointed out, I do not have much reputation here, but I still think I can make a few observations that most would agree with.

The unsuitable questions for the ELU site come from 3 distinct classes:

1. Fluent English speakers who ask poorly-researched questions.
2. English Language Learners who ask well-researched questions.
3. English language Learners who ask poorly-researched questions.

We have a 4th class of Fluent English speakers asking relevant, well-researched questions, but that is obviously within the scope of the site.

The methods of dealing with the three classes?

• Class 1: Closure/Hold
• Class 2: Migration
• Class 3: This one is tricky. As many of you stated, we cannot just move the question; it does not meet ELL standards. At the same time, the user is a language learner, and we do him/her no favor by not redirecting him to the site that he seeks. Even if his question is poor, the user's content is in line with ELL. If a question asked a bad programming question here, we wouldn't migrate his question to stack overflow, we would redirect him there.

The problems?

• We have a large quantity questions of all classes 1,2, and 3. A shocking amount are not being closed/migrated/put on hold, and are receiving several answers.

• We have a large amount of new users (1 reppies) coming in and asking questions of class 3. Closing their questions does not solve their problems; rather they must, as people, be redirected to ELL because that's what they are seeking. Over there, they will learn how to meet their question standards.

What the algorithm ended up doing:

The algorithm was trained using two data sets, one with questions from from class 2, the other with questions from and class 4. It cannot distinguish between class 1 and class 4 questions, nor can it distinguish between class 3 and class 2 questions . In essence, the algorithm, in its current state, judges strictly how likely the speaker is to be a native or a learner. It cannot tell whether the question is formulated for either site, as you may have noticed. And I think it did a good job at sorting these questions based on where the user more likely belongs, barring a few odd choices.

Conclusion and proposed solutions, based on what we discussed:

1. A modified algorithm has potential to be put to work in forming review queues, easing the burden from the community (Colleen and Mitch).
2. As I mentioned before, and as you can see in the results, the algorithm can currently separate the questions into two pools: classes 1,4 and classes 2,3. The questions from class 2 belong on ELL, and so do their askers. The questions fromm class 3 do not belong on ELL, but their askers are learners, and thus do belong to ELL. Therefore, the algorithm effectively distinguishes not between good and bad questions, but between user English proficiency. So I propose that when a user is preparing to send a question, the algorithm checks it. If it deems that the English of the question is more in line with a learner's level, it simply informs him of the ELL site, leaving the choice to the user. I think this could be a huge solution to the flood of new users who are actually seeking the ELL. We do them no favor by closing their questions; a wizard to help them reach the ELL before they even send their question would do everyone a great favor. Of course, a native English speaker who the AI erroneously perceives to be a learner can just dismiss the message.

Both solutions keep the HITL, and take into consideration the level of the user, and the research of the question. What is to be implemented, I leave to the community to decide. I'm not aware how implementation happens here, but when/if you're ready, I'll happily release the algorithms and do my part in the process.

Cheers

Answer 3

I'm fairly new to this site, but I thought you had an interesting approach and wanted to give my feedback. As you already learned, there are many aspects to look at. I may answer in more ways than was thought as on-topic. Especially the "other solutions" might be interesting as the "feature-request". (Note: Since the actual goal of the project is not fully known, I decided to say "moderation" rather than "migration". It is still meant as a placeholder without limitations.)

Machine learning

I much liked your approach to it; trying instead of accepting the status quo. You also seem to have an incremental, fast paced development cycle and willing demo early, with things not being perfectly ready. The idea of using existing data, already processed and judged by humans, is clever. And I think the accuracy can become good, but it may take a lot of tweaking to get it near human level. BTW, I found this post that may be useful to show those trying to understand more of how machine learning works.

If you did this to educate yourself and to spend time, I definitely encourage you to go ahead. Understand more about the selected method. Look into other methods (perhaps RNN and LSTM) that are also already available as implementations. Possibility to classify for more types?

More parameters that should be useful as input (number of words in post, age of account, number of posts on ELU and ELL)? More data available for training? Pre-train with other corpus of "first language" and "second language" English? Split data set into Train, Validation and Test sets. Use the known Validation set to get accuracy benchmark instead of relying on the "after the fact" test-set you used. How is your cut-off-point working when you have only "good" test samples?

Would you train and act on "first version" posts or the (community) edited "final state"?

Collaborate with others to learn more. If you have a wider interest, go on to make it into a tool chain.

Bots

The forum is built for people and volunteers made themselves an identity here. There will naturally be hesitation or opposition when a bot is about to interfere. "How can bots do a better job? Do I need to find another hobby?" But if a bot can be shown to improve the forum, it will eventually be accepted. Looking at Wikipedia, the bots writing articles on their own are not as well received as the bots merely assisting moderators.

You should most likely learn from the experience of other before you go for it. I googled "stackexchange bot" and found a dedicated external site with several bot projects running on SO. They didn't seem to want marketing links, so find it yourself. Also check posts like this.

Policies

Review the available post and moderation policies. Are they changing a lot? How old data can you use as training samples? Review a selection of training samples, do they adhere to the current policy or will you learn an unwanted bias?

Most likely every moderator/user has different standards of what is accepted. The fuzziness of the group forms consensus. A bot can certainly act fuzzy, but perhaps better to set a threshold that acts only on posts that everyone agrees on. Set acceptable levels of false positives to use in evaluation and discussions.

A problem with different moderation standards is that it will be difficult to make everyone happy. Some will think it judges too hard, others that it's too nice. As an experiment it would be interesting to ask 10 moderators to "annotate" a number of samples to see how much agreement there is. Even if there is no bot in the end, this may help the community improve moderation.

Volunteering

Be careful when you find yourself using words as "shocking amounts". You may have a too narrow view of the purpose of the site and the moderation practices. If you are curious about the moderation process, you should probably have a go and learn. That's the best way to get to know the community, to learn what tools already exist and to find the most critical problems to solve.

Before making a live bot, I'd suggest you to try it manually. Take ten examples as pointed out by the bot and evaluate if you agree. Then tag it manually for moderation. See if the moderator outcome agrees with your assumption. Or do the action yourself and see what the discussion and result is. Learn from the experience. See this as your learning curve, rather than "introducing the bot".

Re-scope

Is there a slightly different task that moderators would like to get help with? Training to detect low quality questions is probably an easier target.

Was the idea to use the proof-of-concept to pitch it as an official feature from SE, or just to highlight a problem you see? What is the usual outcome of a "feature request"?

Other solutions

If there is a problem with misplaced questions, maybe there are more fundamental changes that would more easily prevent it "before it happens"?

I must say that I'm personally not all for having so many different sites. As a user, I'm confused. I find the different sites have a fairly unclear scope. The "home" view are just questions, straight up, nothing about guidelines.

The ELU have a hint next to the question form: "If your question is about learning English, ask it on ELL instead." It doesn't say "if you learn" or "if you don't know", it's: "about learning". Yet, "practical problems you encounter" is only a small part of ELL and there's even LL for that.

The "ELU Tour" says "for linguists, etymologists, and serious English language enthusiasts" which says more about the users than the questions. The "on-topic" post is more specific, but I don't remember that post being promoted very much. Most of the categories fit both sites. Most questions are about learning some thing about English language. Nowhere did it say ELU requires "first language" or "better grammar". So what are the difference: On ELU you ask for linguist sources, while ELL gives simpler answers?

Some more or less wild suggestions on what could help: * Create an admittance test, verifying proper skills when creating account on ELU. * Require higher reputation score to post questions on ELU, to make users more familiar with the site before posting. * Better clarify the separation of the sites in FAQ. * Have the "on-topic post" linked beside the question form or add a check-box "Read and understood" with it. * Add a tag "linguist-only" to be used by those who really care about the distinction of ELU towards ELL. The most serious users can ignore all other questions.