Should we allow Google NGrams to be presented as statistical evidence without qualification? Should we define a set of standards for their usage?

Question

"There are three kinds of lies: lies, damned lies, and statistics." — Benjamin Disraeli, attr. by Mark Twain
"If you torture the data long enough, it will confess to anything." — Ronald Coase

For some time now, contributors to EL&U have offered NGrams in support of their arguments. Now, there is nothing wrong with this practice per se: I have done so myself, and have seen others do it in a way that acknowledges the margin for error inherent in a flawed system. When done well it is done in a spirit of inquiry, citing the NGram as possible evidence; when it is done poorly, it is trumpeted as absolute proof of someone's contention.

How flawed is the system? According to @Kosmonaut (replying to a different meta question),

[T]here is a ton of metadata error in Google Books — enough to possibly be concerned about a lot of the conclusions one might draw from it. As it happens, I am working on a project that was originally going to use some Google Books data, but in-depth analysis seems to indicate that dates are way off (as in, 25% of the pre-1800 tokens I have looked at so far seem to be off on their publication dates by an average of ~100 years).

There is also the matter of comparison terms. Sometimes one can find comparisons that will work, but very often the terms can't be compared exactly. For example, in this answer one of our high-rep users attempts to adduce an NGram to answer a question about nuances of meaning:

One wonders how relative frequency is germane to any discussion of the meanings of those words.

Another flaw involves book results being used to draw inferences about spoken language as well, or at least to conflate the two. In this answer [comment chain since then has been deleted for unclear reasons], another contributor uses NGrams to show that one usage is vastly more common than another, which obviously feels counter-intuitive to him because he admits in the comment that "FWIW, it surprised me too. I would probably say 'go for a swim' myself." The conclusion I draw is that Google NGrams are a hammer looking for a nail.

Here is another case where someone draws a faulty inference based on an NGram search:

I leave it to the eloquent @MrHen to debunk the chart:

How does that NGram support any particular usage? Isn't it just tracking uses of "a week hence"? How would it know if that hence was for the future or not?

Even when they can define their terms, users of NGrams succumb to the simpleton awe inspired by the sight of a dramatic projection of minuscule data points to show great trends and differences. They ignore the scale on the Y-axis, which reports the differences in a range that may be only a few cases in tens of millions.

And very often NGram answers garner upvotes because people don't think about the data behind the charts and simply look at the pretty line graphs and upvote the answer.

I'm not against Google NGrams. Used intelligently, they can offer some valuable information. But I believe them not to be the unimpeachable source people pretend they are on EL&U. I'm asking for some kind of standards. I don't know what kind, but I hope you all have some suggestions.

ADDENDUM

As if to prove my point, we have another NGram travesty "proving" a point with faulty data. Because of a bug (or quirk, if you will) in the Google NGram Viewer, hyphenated words invariably flatline. A hyphenated word needs to be separated into a trigram by adding spaces around the hyphen. Notice the difference between these two charts:

Sorry I switched the colors. But at least I resurrected "upper-case" from oblivion. But there is another flaw in this chart — can you spot it? How about cases where "upper case" is not used as an adjective describing a letter? Or how about cases where, even though the subject is typography, one is trying to use case as a noun: "He used the upper case quite a bit in his emails." Of course, the author's point about usage is probably in the main supportable, even without reference to these charts. But given that these charts were introduced as evidence, the answer is not really accurate, and accuracy is something we should at least strive for on this site. We aren't always 100% accurate — nobody's perfect — but if we passively give the nod to glaring errors, relying instead on the base feeling that whatever garners the most votes must be true, we become little better than the Urban Dictionary.

MORE

Finally, for your amusement and to show the relative scale of usage we're talking about, consider the following series of charts. Note especially how one single high-frequency word causes all the rest to flatline. And none of the other words may be considered obscure in the least! (Also, it is obvious from an NGram search that apples are better than oranges. Or that people like them more. Or that ... well, draw your own conclusions. The data are there for your interpretation.)

Use of "is" in the corpus peaks at 1% — one out of a hundred words. Use of those other common words is negligible by comparison. And use of obscure word combinations may not even be worth talking about from a genuine statistical point of view. If there is a statistician among you who can define the terms for such a discussion, I am open to being educated. Until then, I will look at answers involving NGrams with a jaundiced eye.

Update

See this question and its ham-fisted NGrams, some of which have already been deleted, for more evidence of NGrams Gone Wild. At one point, two answers used the same query term and arrived at opposite conclusions. You can't make this stuff up, folks.

FULL DISCLOSURE

Even I have not been immune to the lure of Google NGrams in the past. @Mechanicalsnail pointed out to me an instance of my own seduction, which was written before I had fully grokked how NGrams should and should not be used.

AND NOW THIS, from PLOS|ONE:

Overall, our findings call into question the vast majority of existing claims drawn from the Google Books corpus, and point to the need to fully characterize the dynamics of the corpus before using these data sets to draw broad conclusions about cultural and linguistic evolution.

A tip of the hat to MattEllen.

Answer 1

I love the Google Ngram Viewer. I am a statistician, by education and profession. I agree with Robusto, regarding the tendency to misuse ngram word frequency analysis.

This is English Language and Usage SE. We are not linguists. Well, a few of us are. Yet the only one, that I know of off-hand, John Lawler, doesn't often (ever?) use ngrams to support his answers. The Google Ngram Viewer makes it very easy to bludgeon other respondents' answers into wrongness (I'm sorry, that is a terribly phrased sentence). I'm specifically referring to this thread.

Recently, I have been thinking about the hazards of ambiguity in verbal communication. When considering grammar and standard English usage, we accept variation from an absolute "right" or "wrong". But we do this logically, with additional informational guidelines, such as use of geo-localization tags (e.g. britishenglish, americanenglish), by specifying diction as part of the question or with tagging, formal or slang, and in very specific contexts whether jargon (usually overflow from SO, Super User, or Sever Fault!), or even etymology.

There are grammatically and idiomatically correct versus incorrect answers to questions on EL&U SE. But unstructured text analysis and applied quantitative methods (increasingly described as "digital humanities") have so much of an aura of authority in an age of reason that they tend to overwhelm everything else.

Barry often cites OED in his answers. That is a very good reference source. It isn't as flashy or colorful as a festive multi-colored line graph embedded in an answer. I don't like to see incorrect answers receive more votes, or be designated as the accepted answer, when there are concise, correct answers submitted too.

Exposition over. Here's an answer to the question.

IF (an Ngram is used to answer a question on this site) 
    THEN ( the Ngram must be accompanied by a paragraph of prose explanation 
           AND the Ngram must comply with validity criteria )

Validity criteria should include this:

1. Only use data from 1800 to 2000. Before 1800, there weren’t enough books published to get reliable results; after 2000, the English corpus is inconsistently impacted by the start of the Google Books project.
2. Use for books only as Google excluded periodicals, newspapers, pamphlets, and everything else.

Final thoughts

I don't think we should try to systematically limit Ngram usage. As others observed, doing so would be authoritarian thus not a good fit for this site.

Maybe we should include a few words about using Google Ngram Viewer in the Help section of the site, especially a link to the so-called "validity criteria". I'm making this edit because today, 11 years later, I just noticed a good answer that would have been strengthened if the user had realized that it was okay for instances of a certain expression NOT to appear in Google Ngram Viewer results. That's because the expression was sourced only from books published after 2010, AND there were likely far fewer than the minimum of 40 occurrences even during the 1800 to 2010 time frame.

Answer 2

I think you made the point when you wrote that they can be "presented as statistical evidence without qualification". Google NGrams can provide important information, but they lack authority.

Why? They are not 100% reliable and if we add this to the fact that many people present it as something "absolute", then it could lead to misinformation, at best.

My suggestion is to accept them but with reservation. They can be used, as they still can provide some usefulness, but preferably along with something else more reliable to back up the information.

I think I addressed everything, but if I forgot something, let me know.

Edit: As suggested from the comments, here are the NGrams.

The first one shows that my conclusion was not that convincing:

While any counter argument, except for a moment, were far away from mine:

Answer 3

I agree that Ngrams can be used inappropriately. The resources that people use to support their answers all have to varying degrees their difficulties. But I think the existing methods of justifying them will work:

• if there is a problem with the word choice, the results, whatever, the answerer should either point them out if they know
• someone can comment on the failures of the usage
• someone can edit the answer graph/link

But that's in general for any data (from OED, from quotes, whatever). Specifically for Ngrams, the problems that should be watched out for are:

• dealing with punctuation and caps (as you pointed out in your example with the hyphen)
• selecting the right pairs to compare (make sure they are comparable)
• making sure the context is right, both semantically and in surrounding strings

Looking at the links to sources in the linked Ngram is really the only way to judge.

Answer 4

Should we allow Google NGrams to be presented as statistical evidence without qualification?

Should we define a set of standards for their usage?

We could define a set of standards, but what would those standards be? Your question contained a whole bunch of instances where the data could be mistaken or mis-construed, and I agree with them. But how do you translate those problems into a set of standard for their usage?

Disallowing them seems a bit extreme. As with any answer, if you feel that it's citing dubious sources or providing incorrect information, comment and down-vote. It seems better to combat the problem through the voting system and education rather than regulation.

Answer 5

Selection bias and Semantics

Statistics over published matter, whether books, newspapers, internet should always be taken with an iceberg of salt.

Selection bias: have a go at "bull" vs "cow" and "cherry" vs "plum" and try to explain the crazy oscillations before year 1800. What you'll see is an example of either insufficiency of printed material or periods when the earlier in each pair bore sexual connotation.

Semantics: consider a query "speakeasy" vs "bar" for a drinking establishment. The earlier became commonplace word since prohibition, that is 1920's, yet curiously same word was used in 1880's. The latter can be used in dozens of senses, from law bar to crowbar, assigning statistical prevalence of "bar" over "speakeasy" to "bar" in a sense of a drinking establishment is completely wrong. Same goes for any query where one option is shorter than other or was ever used to mean anything else.

Answer 6

I have pointed this out in a number of places on the main site and in Meta, but it bears repeating here, since the thrust of this post is about misuse of Google Ngrams: You can't use Ngrams to produce reliable frequency data on hyphenated words (such as "upper-case" in one of the Ngram charts reproduced in the posted question).

The Ngram coding doesn't handle hyphens as hard hyphens, perhaps because it doesn't know how to distinguish end-of-line word-break hyphens from compound-attaching hyphens. Instead, it recasts them as "X - Y" constructions (e.g., "upper - case"). I have no idea what that construction attempts to find matches for—because Google never provides any link to examples of the "X - Y" matches in the series of links that appear beneath the Ngram graph itself. Consider the Ngram chart for "uppercase" (blue line) versus "upper case" (red line) versus "upper-case" (green line) for the period 1750–2019:

The results for "upper-case" may look plausible, but if you examine the underlying Ngram graph, you'll see a couple of problematic things.

First, the Ngram graph includes an automated note that reads as follows:

Replaced upper-case with [upper - case] to match how we processed the books.

Accordingly, the label that appears to the right of the green line in the Ngram graph (but not in the Ngram chart) reads not "upper-case" but "upper - case".

Second there are no links to matches for "upper - case" in the "Search in Google Books" section of the Ngram graph presentation (which appears beneath the graph); in contrast, there are numerous linked matches for "uppercase" and "upper case". Meanwhile, adding a further bias to the results, the matches for "upper case" that Ngram/Google Books provides in the "Search in Google Books" links include multiple matches for "upper - case", which turn out to be misreads of instances of "upper-case".

The bottom line here is that Ngram results for hyphenated words and phrases are completely unreliable, and frequency data for adjacent closed-up and open forms are highly suspect (because they may include numerous instances of the hyphenated form).

---

I regularly cite Google Ngrams in my answers, but I try not to ask them to perform tasks that they are ill equipped to handle. They are most useful (to me) as a tool for finding early print instances of a word or phrase and as a simple way to illustrate changes in frequency of print occurrence of a word or phrase over time. They can also be useful for showing changes in relative frequency of print occurrence of two or more words or phrases over time—but only if they avoid the hyphenation pitfall and (in the case of phrases) the false positive pitfall.

The false positive pitfall involves matches that aren't really matches. The most obvious false positives are optical character recognition errors, which cause Ngram to do things like find matches and plot a graph for "Facebook" from the period 1750–1950 (either through misreading a similar word such as "factbook" or through misidentifying the date of publication of the cited text).

But a more insidious type of false positives involves word-string matches that aren't phrase matches. For example, if you were to generate an Ngram chart showing matches for "a week hence", you might suppose that the results would be built entirely on instances of the phrase "a week hence"—as in "We shall meet again a week hence." But Ngram doesn't find phrases per se; it finds word strings. And among the matches that it will use in generating its Ngram chart for "a week hence" are ones like this (from Henry Mayhew, London Labour and the London Poor, volume 1 (1861):

The number of water-carriers are sixty, and their average earnings through the year 5s. a week; hence the sum annually expended in water thus obtained amounts to . . . . . . . . £780

If you are just using Ngram to find early matches of the actual phrase you want to investigate, these word-string false positives are not much of a hazard, since you can easily recognize their irrelevance to your purpose. But if you are using Ngrams to show the frequency of occurrence of the phrase itself, they can be a serious problem. The important thing to be aware of here is that word strings aren't the same as phrases, and Ngram (like Google Books) deals in word strings.

Having said all this, I reiterate my view that Ngram is an extremely valuable tool for both research and illustration if you know how to use it properly.