58

"There are three kinds of lies: lies, damned lies, and statistics." — Benjamin Disraeli, attr. by Mark Twain
"If you torture the data long enough, it will confess to anything." — Ronald Coase

For some time now, contributors to EL&U have offered NGrams in support of their arguments. Now, there is nothing wrong with this practice per se: I have done so myself, and have seen others do it in a way that acknowledges the margin for error inherent in a flawed system. When done well it is done in a spirit of inquiry, citing the NGram as possible evidence; when it is done poorly, it is trumpeted as absolute proof of someone's contention.

How flawed is the system? According to @Kosmonaut (replying to a different meta question),

[T]here is a ton of metadata error in Google Books — enough to possibly be concerned about a lot of the conclusions one might draw from it. As it happens, I am working on a project that was originally going to use some Google Books data, but in-depth analysis seems to indicate that dates are way off (as in, 25% of the pre-1800 tokens I have looked at so far seem to be off on their publication dates by an average of ~100 years).

There is also the matter of comparison terms. Sometimes one can find comparisons that will work, but very often the terms can't be compared exactly. For example, in this answer one of our high-rep users attempts to adduce an NGram to answer a question about nuances of meaning:

enter image description here

One wonders how relative frequency is germane to any discussion of the meanings of those words.

Another flaw involves book results being used to draw inferences about spoken language as well, or at least to conflate the two. In this answer [comment chain since then has been deleted for unclear reasons], another contributor uses NGrams to show that one usage is vastly more common than another, which obviously feels counter-intuitive to him because he admits in the comment that "FWIW, it surprised me too. I would probably say 'go for a swim' myself." The conclusion I draw is that Google NGrams are a hammer looking for a nail.

Here is another case where someone draws a faulty inference based on an NGram search:

enter image description here

I leave it to the eloquent @MrHen to debunk the chart:

How does that NGram support any particular usage? Isn't it just tracking uses of "a week hence"? How would it know if that hence was for the future or not?

Even when they can define their terms, users of NGrams succumb to the simpleton awe inspired by the sight of a dramatic projection of minuscule data points to show great trends and differences. They ignore the scale on the Y-axis, which reports the differences in a range that may be only a few cases in tens of millions.

And very often NGram answers garner upvotes because people don't think about the data behind the charts and simply look at the pretty line graphs and upvote the answer.

I'm not against Google NGrams. Used intelligently, they can offer some valuable information. But I believe them not to be the unimpeachable source people pretend they are on EL&U. I'm asking for some kind of standards. I don't know what kind, but I hope you all have some suggestions.

ADDENDUM

As if to prove my point, we have another NGram travesty "proving" a point with faulty data. Because of a bug (or quirk, if you will) in the Google NGram Viewer, hyphenated words invariably flatline. A hyphenated word needs to be separated into a trigram by adding spaces around the hyphen. Notice the difference between these two charts:

enter image description here

enter image description here

Sorry I switched the colors. But at least I resurrected "upper-case" from oblivion. But there is another flaw in this chart — can you spot it? How about cases where "upper case" is not used as an adjective describing a letter? Or how about cases where, even though the subject is typography, one is trying to use case as a noun: "He used the upper case quite a bit in his emails." Of course, the author's point about usage is probably in the main supportable, even without reference to these charts. But given that these charts were introduced as evidence, the answer is not really accurate, and accuracy is something we should at least strive for on this site. We aren't always 100% accurate — nobody's perfect — but if we passively give the nod to glaring errors, relying instead on the base feeling that whatever garners the most votes must be true, we become little better than the Urban Dictionary.

MORE

Finally, for your amusement and to show the relative scale of usage we're talking about, consider the following series of charts. Note especially how one single high-frequency word causes all the rest to flatline. And none of the other words may be considered obscure in the least! (Also, it is obvious from an NGram search that apples are better than oranges. Or that people like them more. Or that ... well, draw your own conclusions. The data are there for your interpretation.)

enter image description here

enter image description here

enter image description here

enter image description here

Use of "is" in the corpus peaks at 1% — one out of a hundred words. Use of those other common words is negligible by comparison. And use of obscure word combinations may not even be worth talking about from a genuine statistical point of view. If there is a statistician among you who can define the terms for such a discussion, I am open to being educated. Until then, I will look at answers involving NGrams with a jaundiced eye.

Update

See this question and its ham-fisted NGrams, some of which have already been deleted, for more evidence of NGrams Gone Wild. At one point, two answers used the same query term and arrived at opposite conclusions. You can't make this stuff up, folks.

FULL DISCLOSURE

Even I have not been immune to the lure of Google NGrams in the past. @Mechanicalsnail pointed out to me an instance of my own seduction, which was written before I had fully grokked how NGrams should and should not be used.

AND NOW THIS, from PLOS|ONE:

Overall, our findings call into question the vast majority of existing claims drawn from the Google Books corpus, and point to the need to fully characterize the dynamics of the corpus before using these data sets to draw broad conclusions about cultural and linguistic evolution.

A tip of the hat to MattEllen.

6
  • 1
    I'll admit I didn't read the whole post, but my major concern is that via some hive groupthink, something that is already popular will be chosen simply out of its popularity and no other merits.
    – user19589
    Commented Jun 20, 2012 at 0:21
  • 6
    You said it all in this statement: There is nothing wrong with this practice per se. End of story. Ngrams are what they are. This meta question can serve as a useful caveat about their use, so it is worthwhile. Otherwise, let it go - no need to police the use of ngrams or provide guidelines/caveats everywhere. We should not define a set of standards for their use. Users can call out specific ngrams that are misleading or point out specifically what a given ngram does and does not indicate/support.
    – Drew
    Commented Sep 17, 2014 at 17:59
  • 1
    @Drew: I think anyone who presents them as statistical evidence should be required, as I suggest in the title of my post, to qualify their use. That to me says it all.
    – Robusto
    Commented Sep 17, 2014 at 18:33
  • 1
    @Robusto: Just what do you mean by "presents them as statistical evidence"? Any posting of an ngram might be construed by some as presenting statistical evidence, even without any explicit argument claiming that. Better to let it be and leave it up to S.E. users to point out when something is misleading in a particular context. That, plus providing this question as general background.
    – Drew
    Commented Sep 17, 2014 at 20:29
  • 2
    It is statistical evidence, but, it can also be useless, unless one uses it wisely. Also, bear in mind that spoken language is not reflected the statistics.
    – Lambie
    Commented Jan 27, 2019 at 20:19
  • "book results being used to draw inferences about spoken language as well, or at least to conflate the two.", well said.
    – Lambie
    Commented Jan 27, 2019 at 20:52

6 Answers 6

29

I love the Google Ngram Viewer. I am a statistician, by education and profession. I agree with Robusto, regarding the tendency to misuse ngram word frequency analysis.

This is English Language and Usage SE. We are not linguists. Well, a few of us are. Yet the only one, that I know of off-hand, John Lawler, doesn't often (ever?) use ngrams to support his answers. The Google Ngram Viewer makes it very easy to bludgeon other respondents' answers into wrongness (I'm sorry, that is a terribly phrased sentence). I'm specifically referring to this thread.

Recently, I have been thinking about the hazards of ambiguity in verbal communication. When considering grammar and standard English usage, we accept variation from an absolute "right" or "wrong". But we do this logically, with additional informational guidelines, such as use of geo-localization tags (e.g. , ), by specifying diction as part of the question or with tagging, or , and in very specific contexts whether jargon (usually overflow from SO, Super User, or Sever Fault!), or even etymology.

There are grammatically and idiomatically correct versus incorrect answers to questions on EL&U SE. But unstructured text analysis and applied quantitative methods (increasingly described as "digital humanities") have so much of an aura of authority in an age of reason that they tend to overwhelm everything else.

Barry often cites OED in his answers. That is a very good reference source. It isn't as flashy or colorful as a festive multi-colored line graph embedded in an answer. I don't like to see incorrect answers receive more votes, or be designated as the accepted answer, when there are concise, correct answers submitted too.

Exposition over. Here's an answer to the question.

IF (an Ngram is used to answer a question on this site) 
    THEN ( the Ngram must be accompanied by a paragraph of prose explanation 
           AND the Ngram must comply with validity criteria )

Validity criteria should include this:

  1. Only use data from 1800 to 2000. Before 1800, there weren’t enough books published to get reliable results; after 2000, the English corpus is inconsistently impacted by the start of the Google Books project.
  2. Use for books only as Google excluded periodicals, newspapers, pamphlets, and everything else.

Final thoughts

I don't think we should try to systematically limit Ngram usage. As others observed, doing so would be authoritarian thus not a good fit for this site.

Maybe we should include a few words about using Google Ngram Viewer in the Help section of the site, especially a link to the so-called "validity criteria". I'm making this edit because today, 11 years later, I just noticed a good answer that would have been strengthened if the user had realized that it was okay for instances of a certain expression NOT to appear in Google Ngram Viewer results. That's because the expression was sourced only from books published after 2010, AND there were likely far fewer than the minimum of 40 occurrences even during the 1800 to 2010 time frame.

6
  • Excellent way to say it, which would apply to anything; using any tool should come with an explanation. Your 'final thought' however is a bit too authoritarian (and I would suppose difficult to implement).
    – Mitch
    Commented Mar 4, 2012 at 18:33
  • 2
    This is exactly the sort of clear analysis I was looking for. Thank you. Can you elaborate a bit on what criteria we might use to assess validity? That's what we really need. In other words, going beyond a minimum?
    – Robusto
    Commented Mar 4, 2012 at 18:51
  • @Robusto You are welcome! Regarding additional criteria, errr, yes. The first that came to mind was medial s, or rather, avoiding situations where it could cause invalid results. That is too technical, to say the least. I wrote (casually!) about ngrams on a different site. I'll try to find that, and will post here if I find anything useful. Commented Mar 9, 2012 at 22:27
  • 1
    I would like to know what you are referring to as the "Google ngram website warning." Commented May 27, 2012 at 15:18
  • 1
    I hate statistics with a passion, mostly because after careful calculation they're almost always misinterpreted so horribly, so I'm glad we have people like you who can suffer through the worst of at least one part of the process for us.
    – user19589
    Commented Jun 20, 2012 at 0:23
  • @shinyspoongod You are very sweet! As you said, it is the misinterpretation that is the problem. Commented Jul 16, 2012 at 3:58
15

I think you made the point when you wrote that they can be "presented as statistical evidence without qualification". Google NGrams can provide important information, but they lack authority.

Why? They are not 100% reliable and if we add this to the fact that many people present it as something "absolute", then it could lead to misinformation, at best.

My suggestion is to accept them but with reservation. They can be used, as they still can provide some usefulness, but preferably along with something else more reliable to back up the information.

I think I addressed everything, but if I forgot something, let me know.

Edit: As suggested from the comments, here are the NGrams.

The first one shows that my conclusion was not that convincing:

enter image description here

While any counter argument, except for a moment, were far away from mine:

enter image description here

11
  • 5
    You forgot the NGram supporting your conclusion.
    – Kit Z. Fox Mod
    Commented Feb 27, 2012 at 17:38
  • 3
    @KitFox Done! :D
    – Alenanno
    Commented Feb 27, 2012 at 17:49
  • Small point: I did not say "Google NGrams are useful, but up to a certain point" in my question title or anywhere else.
    – Robusto
    Commented Feb 27, 2012 at 19:06
  • @Robusto It wasn't meant to be a verbatim quote, but rather a re-wording of what I think you meant. You meant that, no?
    – Alenanno
    Commented Feb 27, 2012 at 19:08
  • 2
    @Alenanno: Well, words either mean something or they don't. I'm not channeling any negative energy in your general direction, but I thought I did say what I meant in the title. It would be more accurate to characterize the general sense of my question as that, but not to attribute those exact words to me.
    – Robusto
    Commented Feb 27, 2012 at 20:11
  • @Robusto I can re-word it if you want, but I have no idea how...
    – Alenanno
    Commented Feb 27, 2012 at 20:12
  • @Alenanno: I edited your first graf to be what I would call an accurate representation of something you could say. If this works for you, good. If not, you are of course free to say whatever you like.
    – Robusto
    Commented Feb 27, 2012 at 20:19
  • @Robusto I re-re-word it. :) Now I really quoted your words. :P
    – Alenanno
    Commented Feb 27, 2012 at 20:35
  • 1
    OK, now that that's over with, let me address your point. How do we "accept them, but with reserve"? (And did you mean "with reservation"?) Because unless there is some warning given about the inherent flaws, the average visitor will not know any of that and will fall under the spell of shoddy statistical mumbo-jumbo just the same.
    – Robusto
    Commented Feb 27, 2012 at 21:02
  • Ah yes, I meant that. I didn't think about that, as you didn't ask to propose something (no?), but since the major problem is that the words are often searched like that without even considering the context where they were used, I think we could address this in the FAQ or in a FAQ-Meta-Question. The usage of NGrams here are basically "bad corpora searches"; "bad" because in the actual corpora search, you see the sentence where a certain word/expression appears.
    – Alenanno
    Commented Feb 27, 2012 at 21:35
  • If Robusto is right, then counter-argument in your lower NGram needs to be entered as counter - argument (i.e. with a space surrounding the hyphen on each side) for the NGram algorithm to work correctly.
    – Erik Kowal
    Commented Jul 31, 2014 at 8:56
11

I agree that Ngrams can be used inappropriately. The resources that people use to support their answers all have to varying degrees their difficulties. But I think the existing methods of justifying them will work:

  • if there is a problem with the word choice, the results, whatever, the answerer should either point them out if they know
  • someone can comment on the failures of the usage
  • someone can edit the answer graph/link

But that's in general for any data (from OED, from quotes, whatever). Specifically for Ngrams, the problems that should be watched out for are:

  • dealing with punctuation and caps (as you pointed out in your example with the hyphen)
  • selecting the right pairs to compare (make sure they are comparable)
  • making sure the context is right, both semantically and in surrounding strings

Looking at the links to sources in the linked Ngram is really the only way to judge.

7
  • 4
    All sounds good to me. NGrams can be misleading, but they're not that complicated - if someone posts up a grossly misleading graph, someone else will probably be falling over themselves to point out the flaws (Robusto himself, for example! :) Commented Feb 28, 2012 at 2:32
  • 8
    @FumbleFingers: I think Robusto's point is that it's too easy (and has been done too often) to thoughtlessly pop over to NGram, create a graph and insert it in an answer without thinking of all the issues. The author of the answer should go to the trouble first rather than rely on others to fix things up.
    – Mitch
    Commented Feb 28, 2012 at 3:52
  • 4
    Particularly over recent months, a huge percentage of questions asked here are really basic - oftentimes the questioner neither seeks, nor would understand, a full discussion of why some particular usage is to be preferred over another. And in any case, the reason is often complex, disputed, or simply nets down to "It's idiomatic". It's too easy (and has been done far more often) for people who aren't even native speakers to invoke "inappropriate logic" and post grossly misleading answers as to what constitutes "standard English as she is spoke". Commented Feb 29, 2012 at 17:10
  • @FumbleFingers Well said, indeed. Now, wouldn't that be "standard English what we speak 'ere"? [desperately seeking humor]
    – Lambie
    Commented Jan 27, 2019 at 20:48
  • 2
    @Lambie It's a thing: English As She Is Spoke
    – Mitch
    Commented Jan 28, 2019 at 1:10
  • 1
    @Mitch: I've used the expression English as she is spoke facetiously since I was a schoolboy (probably picked it up from my English teacher, I dunno). But until I just followed your link I'd always assumed that the original was itself facetious (from someone like Mark Twain, GB Shaw, Churchill, or whoever). So many thanks for your interestingly enlightening link! :) Commented Jan 28, 2019 at 13:29
  • 2
    The[ Carolino book is long out of copyright](gutenberg.org/ebooks/30411), and Mark Twain's examples are just cherry-picking. The whole book is like that, and it's a scream on every page. Commented Dec 16, 2022 at 21:39
7

Should we allow Google NGrams to be presented as statistical evidence without qualification?

Should we define a set of standards for their usage?

We could define a set of standards, but what would those standards be? Your question contained a whole bunch of instances where the data could be mistaken or mis-construed, and I agree with them. But how do you translate those problems into a set of standard for their usage?

Disallowing them seems a bit extreme. As with any answer, if you feel that it's citing dubious sources or providing incorrect information, comment and down-vote. It seems better to combat the problem through the voting system and education rather than regulation.

4

Selection bias and Semantics

Statistics over published matter, whether books, newspapers, internet should always be taken with an iceberg of salt.

Selection bias: have a go at "bull" vs "cow" and "cherry" vs "plum" and try to explain the crazy oscillations before year 1800. What you'll see is an example of either insufficiency of printed material or periods when the earlier in each pair bore sexual connotation.

Semantics: consider a query "speakeasy" vs "bar" for a drinking establishment. The earlier became commonplace word since prohibition, that is 1920's, yet curiously same word was used in 1880's. The latter can be used in dozens of senses, from law bar to crowbar, assigning statistical prevalence of "bar" over "speakeasy" to "bar" in a sense of a drinking establishment is completely wrong. Same goes for any query where one option is shorter than other or was ever used to mean anything else.

2

I have pointed this out in a number of places on the main site and in Meta, but it bears repeating here, since the thrust of this post is about misuse of Google Ngrams: You can't use Ngrams to produce reliable frequency data on hyphenated words (such as "upper-case" in one of the Ngram charts reproduced in the posted question).

The Ngram coding doesn't handle hyphens as hard hyphens, perhaps because it doesn't know how to distinguish end-of-line word-break hyphens from compound-attaching hyphens. Instead, it recasts them as "X - Y" constructions (e.g., "upper - case"). I have no idea what that construction attempts to find matches for—because Google never provides any link to examples of the "X - Y" matches in the series of links that appear beneath the Ngram graph itself. Consider the Ngram chart for "uppercase" (blue line) versus "upper case" (red line) versus "upper-case" (green line) for the period 1750–2019:

The results for "upper-case" may look plausible, but if you examine the underlying Ngram graph, you'll see a couple of problematic things.

First, the Ngram graph includes an automated note that reads as follows:

Replaced upper-case with [upper - case] to match how we processed the books.

Accordingly, the label that appears to the right of the green line in the Ngram graph (but not in the Ngram chart) reads not "upper-case" but "upper - case".

Second there are no links to matches for "upper - case" in the "Search in Google Books" section of the Ngram graph presentation (which appears beneath the graph); in contrast, there are numerous linked matches for "uppercase" and "upper case". Meanwhile, adding a further bias to the results, the matches for "upper case" that Ngram/Google Books provides in the "Search in Google Books" links include multiple matches for "upper - case", which turn out to be misreads of instances of "upper-case".

The bottom line here is that Ngram results for hyphenated words and phrases are completely unreliable, and frequency data for adjacent closed-up and open forms are highly suspect (because they may include numerous instances of the hyphenated form).


I regularly cite Google Ngrams in my answers, but I try not to ask them to perform tasks that they are ill equipped to handle. They are most useful (to me) as a tool for finding early print instances of a word or phrase and as a simple way to illustrate changes in frequency of print occurrence of a word or phrase over time. They can also be useful for showing changes in relative frequency of print occurrence of two or more words or phrases over time—but only if they avoid the hyphenation pitfall and (in the case of phrases) the false positive pitfall.

The false positive pitfall involves matches that aren't really matches. The most obvious false positives are optical character recognition errors, which cause Ngram to do things like find matches and plot a graph for "Facebook" from the period 1750–1950 (either through misreading a similar word such as "factbook" or through misidentifying the date of publication of the cited text).

But a more insidious type of false positives involves word-string matches that aren't phrase matches. For example, if you were to generate an Ngram chart showing matches for "a week hence", you might suppose that the results would be built entirely on instances of the phrase "a week hence"—as in "We shall meet again a week hence." But Ngram doesn't find phrases per se; it finds word strings. And among the matches that it will use in generating its Ngram chart for "a week hence" are ones like this (from Henry Mayhew, London Labour and the London Poor, volume 1 (1861):

The number of water-carriers are sixty, and their average earnings through the year 5s. a week; hence the sum annually expended in water thus obtained amounts to . . . . . . . . £780

If you are just using Ngram to find early matches of the actual phrase you want to investigate, these word-string false positives are not much of a hazard, since you can easily recognize their irrelevance to your purpose. But if you are using Ngrams to show the frequency of occurrence of the phrase itself, they can be a serious problem. The important thing to be aware of here is that word strings aren't the same as phrases, and Ngram (like Google Books) deals in word strings.

Having said all this, I reiterate my view that Ngram is an extremely valuable tool for both research and illustration if you know how to use it properly.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .