9

Stack Exchange recently started a collaboration project with the University of Melbourne, in an attempt to improve the automatic detection of duplicate questions.

More information on the project and a call for help from the community can be found here: Project Reduplication of Deduplication Has Begun!

In it, Tim Post mentions that Doris Hoogeveen (me) will post a message on the different meta sites to let you know that we're ready for help. This is it!

As a brief summary, the project is about finding ways to automatically determine if two questions are duplicates, and also to detect the opposite: falsely tagged duplicates. To validate our methods, we need manual judgements from people who really know the data. This is where we need your help. It is impossible to manually annotate all the data from Stack Exchange, and so we have applied several filters to make sure we only show you questions pairs with a high likelihood of being duplicates.

Participation is completely optional, and there's no minimum time requirement. There's a reputation requirement however: only people who have duplicate question tagging rights on the site can participate. That is people with at least 3000 reputation points, or a gold badge. This is to ensure the annotations are compatible with StackExchange's quality standard.

If you are interested in participating in the project, please head over to the annotation interface, which can be found here: http://hum.csse.unimelb.edu.au/se-annotate/

You can log in with your Stack Exchange account, so no sign up is required. The site also contains some more information on the project, which might answer some questions you may have. And I'm also happy to answer them here.

Thank you!

19
  • 3
    Could we have a bit more information on the methodology? I've looked at the demo and the pairs I was presented with were completely unrelated, with only one pair having even one tag in common.
    – Andrew Leach Mod
    Nov 10, 2016 at 22:45
  • Hi Andrew, absolutely. You can read about the methodology in this paper: people.eng.unimelb.edu.au/tbaldwin/pubs/sigir2016-webqa.pdf You're absolutely right, some questions are completely unrelated. Finding duplicate questions is a difficult problem, which is exactly why we are doing this project. The selection methods we've used are simple and fast. The idea is that we will use the annotations on this data to build a much better system.
    – Monozygotic
    Nov 10, 2016 at 23:03
  • A question: how related should questions be to be marked as "related, not duplicates"? Sometimes there is some overlap in general subject matter, but it doesn't seem like readers of one question would get any particular benefit from reading the other question. In those cases, I've been marking as "No duplicates." Is that right?
    – herisson
    Nov 10, 2016 at 23:48
  • Hi suməlic, thanks for the question. I would like you to use the same criteria that you would use for tagging duplicate questions on the site. Ultimately the question to keep in mind is: would you like an automatic system to tag these two questions as duplicates or not? From your description it sounds like you are doing the right thing.
    – Monozygotic
    Nov 11, 2016 at 0:35
  • I looked at about four questions in the demo, and they were all about computer applications, not about English.
    – ab2
    Nov 11, 2016 at 1:31
  • @ab2: Bh default, the selected site is for Android. You need to change use the drop-down menu after "StackExchange site" to select "English."
    – herisson
    Nov 11, 2016 at 1:47
  • 2
    I stopped after about six pairs, because he process was agonizingly slow. Moreover, the pairs I looked at were obviously not duplicates...not even close. I'll try to get through more when I have more time.
    – ab2
    Nov 11, 2016 at 4:07
  • 1
    @Monozygotic Thanks for the reference but I'm not a mathematician. Is there any chance of putting it into layman's terms? In particular, why are you ignoring tag correlation?
    – Andrew Leach Mod
    Nov 11, 2016 at 8:54
  • 5
    Considering the five question pairs I got there is an option missing. I cannot even think of any correlation at all. Honestly, I think this is a great project but those five pairs looked completely random. That does not seem to be a good use of our time.
    – Helmar
    Nov 11, 2016 at 13:20
  • 1
    It isn't enough to just upvote @Helmar's comment "I cannot think of any correlation at all." I have to repeat it! It is what I thought of five of the six pairs I looked at. Would it be helpful if we sent you some real duplicates....would it help your computer to learn if it had samples of real duplicates?
    – ab2
    Nov 11, 2016 at 19:12
  • 2
    I've done quiet a few, but it's just too slow.
    – curiousdannii
    Nov 12, 2016 at 11:50
  • 2
    Is there some reason why you are not using established duplicate pairs in your first round of testing? I fear you will wear out your volunteers long before you get to the stage where you really need them. (But thumbs up on the project idea.)
    – aparente001
    Nov 12, 2016 at 21:08
  • 1
    Thanks for your thought everyone. Giving us real duplicates is not necessary, because we have them already. All the existing duplicate pairs are in the StackExchange dump, so we have access to them. The problem is that in our preliminary analysis we found that there are many duplicate pairs that have note been recognised yet. Ideally we would like to annotate the full set of question pairs, but that's infeasible, it's millions of pairs, and so we have applied some simple filters to make the set as small as possible, while at the same time containing pairs with a relatively high likelihood of
    – Monozygotic
    Nov 12, 2016 at 23:51
  • 1
    being duplicates. The method works better for some forums than for others, but this will be the same for any system we build. For the forums where it works less well, it's especially important to get the annotations, because it shows that existing methods (like the standard retrieval methods we have used) are far from good enough for this task. I understand how it feels like a waste of time to label two question pairs that are clearly not duplicates to human eyes, but for a computer, both non-duplicate and duplicate labels are extremely helpful.
    – Monozygotic
    Nov 12, 2016 at 23:53
  • 2
    @Monozygotic We understand why it is difficult to identify duplicates reliably, else it would have been automated already. But I think what people were looking for is something closer to reliable, otherwise the exercise feels, from a user experience perspective, pointless. Perhaps it better to wait a bit until your algorithm is more fine-tuned and serves pages faster before asking for human input? Otherwise you might burn through goodwill which would be very useful in later stages of the project. Barring that, can you show the user why the machine thinks each pair related?
    – Dan Bron
    Nov 13, 2016 at 20:43

4 Answers 4

Reset to default
3

My immediate thoughts:

The very first pair I got were two very clearly unrelated questions that had nothing whatsoever to do with each other—same as what several others have noted.

I suspect they were matched because of an overlap in words used in the body of the questions themselves. The problem is that one of the questions is just one sentence long, while the other is ten paragraphs: weeding out filler words (“I want to know”, prepositions, copula, determiners, etc.), there are only four words in that question: grammatically, correct, long, conference.

Of these four, the term grammatically correct should be ignored on ELU, because it occurs in somewhere around 80% of the questions asked here, mostly because most people have no idea what ‘grammar’ or ‘grammatically correct’ means.

That leaves just two real words to use for comparison: long and conference. Both of these happen to occur in the ten-paragraph answer, but quite incidentally. So obviously, to a computer, that looks like a duplicate if you just go by brute-force checking words that occur in both questions.

The next pair I got was even worse: one was a one-liner (body almost entirely identical to title), the other about four paragraphs. In this pair, the only words that overlapped were is, not, the, and word. Nothing else. No tags, no similarity in title or anything else. At a quick glance through the comments (only the longer question has any) and answers (the longer question has a very long answer; the short question has seven answers ranging from very short to medium-length), nothing leaps out as being obviously repeated between the two questions, either.

This leads me to believe that the questions were chosen on the basis of a brute-force word comparison between the (questions and) answers, seemingly disregarding frequency and length, which doesn’t seem like a good starting point for identifying duplicates.

 

 

Would it be possible to show on the page itself what criteria were used as the basis upon which the question pair you’re looking at were matched as potential duplicates? That way at least users wouldn’t have to be so confused (or do as much detective work) as to why two entirely unrelated questions were marked as potential duplicates to begin with.

Or am I simply misunderstanding how this works? Are we as users presented with a mix of pairs identified as potential duplicates and pairs identified as not being potential duplicates?

2
  • Hi Janus, your examples clearly show the limitations of existing retrieval methods. Our selection methods are far from random. You can read about the selection process here: people.eng.unimelb.edu.au/tbaldwin/pubs/sigir2016-webqa.pdf
    – Monozygotic
    Nov 12, 2016 at 23:56
  • What you are seeing is not the output of a fine-tuned duplicate detection system, but question pairs selected to help us build such a system. There will be more non-duplicate pairs than duplicate pairs.
    – Monozygotic
    Nov 13, 2016 at 0:02
2

Thanks for your work on this project!

Thoughts so far (I think I've done around 30 of these):

What is the intended use of "Related, not duplicates"?

I mentioned this in a comment, but I don't think I had enough space to explain clearly. There are four options available:

  • "No duplicates"
  • "Duplicate of" (can go either direction)
  • "Related, not duplicates"
  • "Really can't tell"

I don't quite understand what criteria to use when choosing between "No duplicates" and "Related, not duplicates".

I assume "related, not duplicates" questions will not be tagged as duplicates. But what is the purpose of identifying them? Currently, I'm operating the assumption that these might trigger an automatic comment with a link, so I'm not selecting this option unless the questions are highly related, enough to make it useful for anyone interested in the answers to one question to read the answers to the other.

Minor site oddities/bugs

I don't understand why sometimes, right after I log in, the questions I see are presented above one another rather than side-by-side. (Sounds like this may be fixed now.)

One time after this happened, I got an odd diagonal layout with the next question (the second post was both below and to the right of the first one), but I accidentally navigated away from the page before getting a pic of that one.

Miscellaneous other feedback

Probably nothing can be done about this, but I find that pages load irritatingly slowly.

10
  • Hi suməlic, Related questions are questions that are not quite duplicates, but close. These will most likely be cases our automatic methods will struggle with, so it's good to have a distinction between 'clearly not duplicates' and 'not duplicates, but close'. If we have this distinction, we can fine tune our methods using the difficult cases (the related questions). Does this answer your question?
    – Monozygotic
    Nov 11, 2016 at 2:09
  • It's possible that you have seen the layout with the two questions above one another. This was the first version of the interface that I made. After feedback from users on several other sites I changed it to the side-by-side layout, to make space for the comments and answers, and to have a nicer way to present the directionality in the duplicate questions. You should not get the layout with the questions above one another anymore. If you do, please let me know!
    – Monozygotic
    Nov 11, 2016 at 2:11
  • @Monozygotic: Ah, I just got it right after logging in with my Stack Exchange account (I had taken a look at the demo to see if there were any differences).
    – herisson
    Nov 11, 2016 at 2:18
  • And the next question has a very odd diagonal layout. I'll add pictures to my post here
    – herisson
    Nov 11, 2016 at 2:19
  • I think I may have found the culprit of the vertical layout. So that should be fixed. I haven't seen a diagonal layout before. That sounds like a bug. Let me know if you manage to get a screen shot. Slowness is a difficult one. I'll see what I can do about that.
    – Monozygotic
    Nov 11, 2016 at 3:05
  • Here're some I said was related but not duplicate: imgur.com/F3KViJg imgur.com/qaQqENr These are questions that could both be made duplicates of a canonical question, but not of each other. (And possibly that canonical question would be too big for the SE format.)
    – curiousdannii
    Nov 12, 2016 at 11:35
  • Thinking a little more, that might be a more helpful way of labelling the buttons: "Both could be duplicates of another question, but not each other". A mouthful, but much more meaningful than the subjective "somehow related" - it would allow you to build up a network with siblings as well as parents and children.
    – curiousdannii
    Nov 12, 2016 at 11:43
  • Hi curiousdannii, that's an interesting idea. We would like to keep the set of possible labels manageable, but I can definitely discuss it with the other researchers to see if they think it would be a useful addition. Else, tagging them as related seems like the right thing to do.
    – Monozygotic
    Nov 13, 2016 at 0:04
  • @Monozygotic Well I don't think you will get any meaningful information out of "related" because there's no guarantee at all that anyone will use it to mean the same thing. Better to not have it than to have it IMO.
    – curiousdannii
    Nov 13, 2016 at 1:42
  • @curiousdannii, maybe. The experiments will tell us. It's always better to have the labels and find out they don't help, then to not have them and wonder if they would help.
    – Monozygotic
    Nov 13, 2016 at 23:07
1

As promised, the question pairs now load faster -- on average -- than they did last weekend. Sometimes the next pair loads quickly, and sometimes there is a long delay. If all the questions loaded as quickly as the quickest, giving Melbourne feedback would be easy and fun, but enough still load very slowly so that one needs an alternate task to turn to while waiting. Thanks for speeding it up. It is much less of a chore to give you feedback now.

There are also enough duplicates and relateds to make it interesting. Did a human being salt the Demo, or is the computer learning?

On what basis should we decide whether X is a duplicate of Y or Y a duplicate of X? (Apologies if you explained that somewhere.) I made my choices partly on the comprehensiveness and quality of the answer pairs, not solely on the c & q of the question pairs.

The majority of questions are still completely unrelated -- some are so unrelated that asking if they are duplicates made me laugh. It seems that humans are still good for something.

3
  • Thanks for the positive feedback! I'm trying to make it faster still, because having to do an alternate task because the waiting times are so long is really far from ideal. Quite demotivating. Hopefully I can fix this.
    – Monozygotic
    Nov 15, 2016 at 22:09
  • I use the demo in my own tests, that's why the numbers go up. Or is that not what you're referring to? The question pairs in it are exactly the same as before.
    – Monozygotic
    Nov 15, 2016 at 22:10
  • I added the directionality to the duplicate annotation to make it closer to the annotations on the site. So whatever considerations you use on the site can be applied in the project too. I'm logging it when someone opens the comments and answers, to get an idea of how much people use those in their decision making. And yes, humans are definitely still good for something :) I think it'll be quite a while still before computers take over. :)
    – Monozygotic
    Nov 15, 2016 at 22:13
0

The project is drawing to a close, and I am planning to take down the annotation interface this weekend.

Thank you very much to everyone who helped out, and thank you for all the valuable feedback and suggestions. Hopefully the collected annotations will lead to the development of better duplicate question detection systems, especially for StackExchange.

1
  • I have to confess that I haven't done very much if anything beyond my initial participation and comment, because the process of trying to identify duplicates from questions which had not been so linked already appeared to be not a good use of time. It's struck me that it may have been better to identify why existing duplicates had been marked as such (that is, work out a methodology based on known true positives) and then fine-tune that methodology by testing it on pairs found by using it.
    – Andrew Leach Mod
    Jan 12, 2017 at 14:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .