3

For future reference, how was this determined to be ChatGPT?

Example:

enter image description here

I think only 10K users can see:

Can one say "take an integral" instead of "calculate an integral"? What other options are there?

10K confidential (Argha) deleted answer (suspected ChatGPT)

In English, the phrase "take an integral" is not a commonly used alternative to the more standard phrase "calculate an integral." While it may be used in Russian or in certain contexts, it is not widely recognized or used in English-speaking mathematical or scientific communities.

When it comes to discussing integrals in English, the phrases "calculate an integral," "evaluate an integral," and "find an integral" are more commonly used and widely understood. These phrases convey the action of determining the value of an integral or finding its antiderivative.

Here are some alternatives for discussing integrals:

  1. Calculate an integral
  2. Evaluate an integral
  3. Find the integral
  4. Solve the integral
  5. Determine the value of the integral
  6. Compute the integral
  7. Integrate the function
  8. Antidifferentiate the function

These alternatives capture the various ways of expressing the process of working with integrals in English. It's always recommended to use the terminology that is most commonly understood and accepted within the specific context or community you are communicating with.

Were there certain markers or tells that indicated GPT...or was this just "gut feeling"... Because gut feeling is not quantifiable.

My intention is to identify ChatGPT answers so that we can flag them in the future.

7
  • 9
    "Dear moderators, Please tell us how you detected this so that the unscrupulous can defeat those methods and make it much harder for you. Thanks."
    – Andrew Leach Mod
    Commented Jun 19, 2023 at 6:19
  • 2
    First time user, on EL&U. Speaks flawless, but slightly formal English, very reminiscent of ChatGPT. If you look at the user's activity history their English is good but not that good. Is that enough to earn a suspension? Please see Meta, since may 31 there's a whole ballyhoo over there, and many moderators across the network are currently on strike.
    – Mari-Lou A
    Commented Jun 19, 2023 at 11:52
  • 1
    Perhaps seeing the lack of prior activity the user has been suspended on a different Stack Exchange site before. Guilty until proven innocent... ?
    – Mari-Lou A
    Commented Jun 19, 2023 at 11:59
  • 4
    Isn't it obvious? It's vague, overly verbose and too formal for the context. It reads like a grade school essay instead of an answer by someone who has actual expertise. If you want to take an AI generated answer and make it pass for human-written, you need to have a human who understands the topic write most of it.
    – ColleenV
    Commented Jun 19, 2023 at 13:54
  • 2
    "Grade school essays" describes many of the deleted answers going back a long time in our site history. We all know that the many of our OPs and "late answers" are closed for being sub-par i.e 8th grade level.....But that is the very level that aspiring journalists were told to write at in 101 back in my day.
    – Cascabel_StandWithUkraine_
    Commented Jun 19, 2023 at 19:32
  • 2
    @Cascabel_StandWithUkraine_ I wasn't referring to the level of writing, I was referring to the structure. It also somewhat resembles a beauty pageant interview answer... Restate the question, yammer on vaguely to try to answer the question in the most generic inoffensive way possible while trying to avoid being obviously wrong, then summarize with the "takeaway" point that hopefully reminds people what the question was since they're now lost in a fog of verbosity and vagueness.
    – ColleenV
    Commented Jun 19, 2023 at 19:57
  • 3
    "For future reference, how was this determined to be ChatGPT?" Humans excel at pattern recognition :)
    – Azor Ahai -him-
    Commented Jun 25, 2023 at 17:28

3 Answers 3

Reset to default
10

ColleenV has commented usefully:

Isn't it obvious? It's vague, overly verbose and too formal for the context. It reads like a grade school essay instead of an answer by someone who has actual expertise. If you want to take an AI generated answer and make it pass for human-written, you need to have a human who understands the topic write most of it. [link]

[It's not about] the level of writing, [what's relevant is] the structure. It also somewhat resembles a beauty pageant interview answer... Restate the question, yammer on vaguely to try to answer the question in the most generic inoffensive way possible while trying to avoid being obviously wrong, then summarize with the "takeaway" point that hopefully reminds people what the question was since they're now lost in a fog of verbosity and vagueness. [link]

There are a number of other specific “tells” in this answer as well, which I'm not going to enumerate for the reason implied in my own comment:

“Dear moderators, Please tell us how you detected this so that the unscrupulous can defeat those methods and make it much harder for you. Thanks.”

Websites do exist that claim to be able to determine whether a text is generated or not, even providing a stated degree of certainty, but while they can provide a data point in any assessment, they are not entirely reliable.

13
  • The other hallmark of an LLM answer is that it's somewhat relevant but tends to miss implied parts of the question and add related but irrelevant detail. If you look at the human answers in comparison, they focus on how "take" is used in that context, or suggest alternatives replacing just the key word of the phrase. They don't suggest things that are rarer than the phrase asked about like "Antidifferentiate the function". AI doesn't understand context cues that most humans pick up on rather easily. Fluent English that misses inferences about the question obvious to a human is my indicator.
    – ColleenV
    Commented Jun 20, 2023 at 16:39
  • @ColleenV There are also real people who have issues picking up on context clues. I am reminded of this "answer". Also, India people are often considered "verbose" by Westerners.
    – Cascabel_StandWithUkraine_
    Commented Jun 20, 2023 at 18:05
  • 4
    @Cascabel_StandWithUkraine_ I can tell the difference between a verbose human and AI. Indian English has its own flow that is quite distinct from robo-text. There is a spot on a Venn(ish) diagram where vagueness, formal fluent English, verbosity and related-but-irrelevant-in-this-context detail overlap that is only occupied by AI and a couple of humans who can be filtered out by other factors. If I could give you an exact formula, AI would be able to fool me more often than it does.
    – ColleenV
    Commented Jun 20, 2023 at 18:33
  • @ColleenV I am reviewing some questionable past posts with the idea of identifying AI Chat posts...and I can see there were many starting about 3 months ago. 3 months ago.
    – Cascabel_StandWithUkraine_
    Commented Jun 20, 2023 at 18:37
  • @Cascabel_StandWithUkraine_ Without access to some of the info mods have, I think it will be difficult to "prove" they're AI beyond a reasonable doubt.
    – ColleenV
    Commented Jun 20, 2023 at 18:43
  • 7
    @Cascabel_StandWithUkraine_ You use your judgment and ask a team of moderators to confirm. There is no algorithm that can distinguish AI text from human written in all circumstances. Fortunately, this isn't a court of law, and the consequences for accidentally mistaking a human written text for AI aren't severe or irreversible.
    – ColleenV
    Commented Jun 20, 2023 at 19:08
  • 1
    @ColleenV Perhaps we nee a new flag " i.e. suspected AI generation
    – Cascabel_StandWithUkraine_
    Commented Jun 20, 2023 at 19:11
  • 2
    @Cascabel_StandWithUkraine_ Why? Only mods have the tools to act on such a suspicion fairly and the flagger should explain why they suspect it's AI anyhow. Just use the flag for moderator attention.
    – ColleenV
    Commented Jun 20, 2023 at 19:14
  • 1
    This is where it all ends. Not with a bang but with GenAI sponsorships.
    – Mari-Lou A
    Commented Jun 21, 2023 at 10:16
  • @ColleenV This is kind of catch-22ish. "the flagger should explain why they suspect it's AI"...yet the Mods do not want want to share their thinking on identifying possible ChatGPT because as Andrew put it,"“Dear moderators, Please tell us how you detected this so that the unscrupulous can defeat those methods and make it much harder for you. Thanks." There is a major disconnect there, and I am sorry if you guys cannot see it. This feels like "mushroom management"
    – Cascabel_StandWithUkraine_
    Commented Jun 21, 2023 at 18:00
  • 1
    @Cascabel_StandWithUkraine_ I respect that the mods don't want to share explicit details of their key indicators because it will help bad actors evade detection, but frankly, if you have to very secretive about your detection methodology in general, your tools suck, you lack proper resources or you're too focused on enforcing a rule for the sake of the rule. (In this case, I think it's the first two) It's not a good idea for the entire community to be using the same indicators anyhow. There's a difference in knowing what sorts of things to be on the lookout for, and a checklist of 'proof'.
    – ColleenV
    Commented Jun 21, 2023 at 18:08
  • 2
    I have detected most of the plagiarized posts I flagged simply by noticing copy-paste errors that didn't convert into Markdown properly, which made me suspicious, which led to more investigation, which made me confident something was actually plagiarized. It's not "gut feelings". It's exploiting the fact that people plagiarizing posts or copying and pasting ChatGPT generated text aren't inclined to spend a lot of effort on their posts. We catch sock puppeteers the same way. It's a lot of work to pretend to be multiple people, and the shortcuts they take can stand out to an experienced eye.
    – ColleenV
    Commented Jun 21, 2023 at 18:12
  • Thank you for your thoughts. Gonna consider it over night. Right now I gotta feed my brood.
    – Cascabel_StandWithUkraine_
    Commented Jun 21, 2023 at 18:23
6

Background: I have more than 1,200 helpful flags on Stack Overflow specifically around GPT/AI/LLM answers (12 declines, and a handful more that were marked "Helpful" but not acted on). Since the strike, I've identified several hundred more on SO, many of which I've commented on pointing out the policy, after which some users self-deleted their post. I maintain a "Save" list of likely ChatGPT answers on each of the sites I frequent, and on SO, that list is closing in on 3,000. I also have 100 or so on Ask Ubuntu.

My intention is to identify ChatGPT answers so that we can flag them in the future.

As @AndrewLeach pointed out in the comments, disclosing how we detect AI-generated content, unfortunately, enables some unscrupulous users to try to avoid these heuristics in the future. Most of us who do have a good understanding of how to spot them (or at least think we do) try to keep our methods under wraps for the most part.

However, without disclosing any "trade secrets", as @AzorAhai-him also pointed out:

For future reference, how was this determined to be ChatGPT?

Humans excel at pattern recognition :)

I'd amend that to "most" humans, personally, but yes, this is essentially the best method for learning what ChatGPT/AI/LLM answers look like:

Simply start copying and pasting some of the questions here into ChatGPT (and/or Bing Chat and/or Google Bard) and read the answers. You'll fairly quickly start to notice the style and common patterns. Even when users attempt to edit that out of their answers (or use creative prompts), it's still possible (although often more difficult) to spot the patterns, especially as the amount of text available to analyze (e.g., multiple answers/posts) increases.

It's interesting that ChatGPT has a very different style for "code" and "technical" answers as compared to "expository" ones. So the bulk of what we see on Stack Overflow will not exhibit the same patterns that you'll see here. I'd probably be pretty bad at spotting GPT answers here, in all reality.

5

If you run this answer through Zero GPT, it will claim that the text is from ChatGPT/AI. These tools are not foolproof, but it doesn't matter, because the answer shows no research and is generally low quality.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .