What do you think about using ChatGPT and AI features to learn English? Is it trustable 100%?

Question

I'm using it all the time and I'm afraid of it. If you're against using it that much. Tell me what can I do instead (I've got no teachers in real life).

Answer 1

A rule of thumb for using ChatGPT in general is: if the answer is available online in multiple places, it generally does very well. Otherwise, it is liable to produce complete nonsense.

So I asked ChatGPT one of the more obscure questions I know about English grammar:

In English grammar when should you use "except" and when should you use "except for"?

and it produced complete nonsense:

Use "except" when it directly follows a verb, preposition, or conjunction.

• Example with a verb:
  "I invited everyone except Jane."
  (Here, "except" follows the verb "invited.")
• Example with a preposition:
  "Everyone was there except Jane."
  (Here, "except" follows the preposition "was.")
• Example with a conjunction:
  "The store is open every day except Sunday."
  (Here, "except" follows the conjunction "every day.")

Use "except for" when it introduces a phrase or clause, particularly after a noun. It is also more common when the exception is more complex or needs more emphasis.

• Example after a noun:
  "The meeting was perfect except for the technical difficulties."
  (Here, "except for" introduces the phrase "the technical difficulties.")
• Example with a clause: "He would have won the race except for his fall."
  (Here, "except for" introduces the clause "his fall.")

The details are wrong: "was" is not a preposition, "every day" is not a conjunction, "perfect" is not a noun, and "his fall" is not a clause but a noun.

And the overall rule is incorrect. (The right rule for these examples is: you can use "except" when it follows an inclusive word like everyone, every day, all, nobody, etc., although note that this rule doesn't apply to all sentences with "except"—it's more complicated.) For example, in the sentence

We would have won except for the flat tire,

except/except for directly follows a verb, so according to ChatGPT's rules, you would use "except". However, because there's no inclusive word before except/except for, you need to use except for.

Answer 2

TLDR:

• No, ChatGPT (and other LLMs like Claude and Gemini) are not 100% correct when asking "What is the rule for this grammar?" or "Is this sentence correct and why?"
• LLMs and internet search can be reasonable additions to language study, but mostly for general examples of relatively formal writing, but I would not rely on just that for learning a language (look for learning apps like Duolingo or Babbel and spoken conversation sites like iTalki or HelloTalk).

---

There are two ways to use an 'oracle' like ChatGPT for language:

• queries about language facts. For example, [ELU.SE] or Reddit questions about "What is the rule for verb agreement with 'a number of items' Is the verb plural or singular?"
• a source of examples of (written) language. For example, you're chatting away on CHatGPT and say "Please correct the text in 'there is two stroes at the corner'" and you should get the response including "There are two stores on the corner."

The first way, asking a direct factual question, is somewhat unreliable. ChatGPT and other LLMs only provide a statistically likely rehashing of its training data, which likely includes lots of blogs on grammar (but probably not professional writing style guides like CMoS). And the rehashing is not with respect to facts but about the sequence of words. So it is easy for errors to pop in statistically. They get things right a lot but also not right sometimes, enough so you can't always trust them.

But for the second way, the examples of language in English from LLMs are superficially very good. I don't think I've ever seen a grammatical or spelling typo in any of the LLMs.

I don't know of any studies that have tested LLMs competence in production of grammatical English or other languages but it matches native speakers very well, mostly because the algorithms are designed to reflect (statistically) the sequences found in the gobs of training text (a lot of Internet data) which is mostly written by native speakers.

Notes:

• re grammar and spelling - those are the two things that LLMs are good at - they follow what is in the training data. If the training data were all from the US and none from the UK, you'd never see the British 'flavour, ageing, travelling' variations. But LLMs are also good at maintaining style and word choice over several sentences, either keeping formal word choice consistently, or informal for example.
• LLMs (trained on 'everything') are heavily biased towards English, but do well in original and translation with languages that have a lot of Internet presence and are nearby to English culturally, ie European languages. They aren't so good with not as common Internet languages (eg Swahili, Indonesian). I don't know about accuracy with Chinese and Japanese, both of which have huge Internet presence, but if you're starting from those and trying to get sentences in English from them, I expect there might be more problems than with French - the range of synonyms and discourse style for those two languages can be very different from English (and that can pose more problems when using an LLM to go back and forth).

---

What can you do instead, or rather to get more reliable answers?

• Use more than one LLM and judge the answers together. Use ChatGPT, Claude, Gemini (in google mail), Llama (Facebook)... there's a new one every day. But the point is don't just try to rely on a single LLM. Ask your question to at least two LLMs. Also ask the same LLM more than once in slightly different ways.
• Ask your question to the same LLM in different ways and judge some combination of the answers. "What is the rule for..." and "What is the right grammar for saying...". Even small changes can give very different answers. It seems crazy, but like I said it's statistical, with a little non-determinism thrown in.
• Ask for examples of the construction. "Give me 5 examples of the interrogative subjunctive future perfect 2nd person plural in a Geordie accent"
• Try to elicit the grammar you want to see. "How many teams trade players each year? Answer in the form: 'A number of ...'"
• Ya know, if you have a short phrase that you want to check for grammar, put it in quotes and do an internet search to see if it exists.

None of this will be 100% reliable, but the more variations you try, the more reliable you'll get.