How reliable is Google Ngram?

Question

Ngram seems to be more authoritative than the Periodic Table here on EL&U. As someone with more than a passing interest in the language, I wanted to know how good Ngram is. And on Wikipedia, of all authorities to cite when seeking reliability, I found these relevant facts:

Point 1:

The Google Ngram Viewer or Google Books Ngram Viewer is an online search engine that charts frequencies of any set of comma-delimited search strings using a yearly count of n-grams found in sources printed between 1500 and 2008... in Google's text corpora in American English, British English, French, German, Spanish, Russian, Hebrew, or Chinese. ... generated in either 2008 or 2012

Point 2:

Google populated the database from over 5 million books published up to 2008. Accordingly, as of January 2016, no data will match beyond the year 2008, no matter if the corpora was generated in 2009 or 2012. Due to limitations on the size of the Ngram database, only matches found in over 40 books are indexed in the database; otherwise the database could not have stored all possible combinations.

Point 3:

The data set has been criticized for its reliance upon inaccurate OCR, an overabundance of Scientific Literature, and for including large numbers of incorrectly dated and categorized texts

From points 1 and 2 we learn that the English corpus contains 5,000,000 digitized books dating from 1500 to 2008. From point 3 we learn that there are criticisms. The overabundance of Scientific Literature is a particularly telling point in tracking words in common usage. This is important because this site itself shows us that usage in scientific literature is skewed and stilted: users often ask for a more polished word for something that seems to an educated person (shameless self-promotion there) to be perfectly normal and acceptable even within the ivied porticoes of academe. So the 5-cent word the questioner would have used at the dinner table could be lost to us, and the five-dollar substitute remains.

English is changing very rapidly, and I am sure that there has been as much change since 2008 as in the 30 years before that, in both UK and US usage. These changes will obviously not show up in the linguistic fossil record used by Ngram.

In Arabic, Chinese and Brazilian Portuguese there are corpora based on newspapers. These reflect current speech, are easy to update, and are accurately dated. Since much of the material is already digitized, there are no OCR problems. Now that there are so many publications online, it should be easy to construct one for any variety of English you choose, but I haven't found one so far. I reckon I need to look harder.

The article cited mentions that Ngram has been found useful in machine translation. I reckon that accounts for much of machine translation!

In view of all this, I have three related questions:

1. In view of the fact that it is a fossil record, should we rely on Ngram as an authority for current usage?

2. If Ngram is found to be wanting, what do we use instead?

3. Is there an up-to-date currently-maintained corpus based on newspapers and Web sites for English?

Answer 1

Google Ngrams has problems for ELU questions:

• it is not speech, which is often the unspoken context of questions, even though we only research things through print.
• it is not the web, which has more colloquial usage (again more often the question context here)
• it includes a lot of strange publications
• it has a lot of OCR errors for older books (with messy printing, messy fonts)
• it has a lot of metadata errors (especially date of publication)
• it is not curated (it was a one-off in 2008)

Corpus of Contemporary American English (COCA) and British National Corpus (BNC) are somewhat comparable, curated corpora of newspaper English. They do not do websites. So the 'speech' (written language) is expected to be formal but up to date.

Many of the problems with searching Ngrams is not necessarily with the data but with the user's expectations of the search they attempt and therefore true of any corpus search. If you do a search on comparing the color terms pale vs light, there will be all sorts of FPs for multiple other meanings that will wash out entirely any kind of hypothesis support. There is some way around this (part of speech tags and phrases with surrounding words) but it can be difficult for any corpus and question. If you assess these results as a non-native speaker, you may very well get a blunt, misleading view of things, whereas a native speaker might understand the nuances of each instance better. So even with a (non-existent) perfect corpus, it may be difficult to justify definitive universal hypotheses (eg "'bucket' is always used in preference to 'pail' in a nautical context" (I made that up, I don't know)).

On the other hand, Ngrams is quick and easy, and gives links to the source immediately, so you can check your hypothesis with respect to context.

As to using an Ngram comparison on ELU as justification, you'd have to account for all the Ngram specific caveats plus the general word search problems.

So my suggestion is to, along with other online resources, use Ngrams but with great caution keeping in mind all the caveats. Don't rely on a single source. Corroborate with other corpora, a plain google search, talking to native speakers, comments here, chat here, etc, etc, etc. If you are a native speaker, you can be motivated by introspection of your own speech patterns but beware, simply asking a question or reading someone else's idea can bias you in a chaotically emotional direction.

Answer 2

One more problem with ngram that (I think) has not been mentioned (from Wired):

One of the traps in using ngrams to divine the popularity of people, ideas, or concepts is that a book only appears once—whether it’s been read once or millions of times. The Lord of the Rings is in there once, and so is some random paper on mechanics. The two texts are weighted equally. It doesn’t reflect what is people are talking about [or reading, or being influenced by] so much as what people are publishing about—and until very recently, most people didn’t have access to publishing. Like, what does this really tell you about language?

Answer 3

This was originally intended as an answer to the practical question of how current use of a term could best be gauged. As such, the answer lists common resources available for that purpose. Google Ngrams/Google Books do not satisfy the demand: being raw statistical data (the n-grams from the Google Books corpus) based on raw textual data (Google Books), inferences drawn from the mess are prima facie invalid and, if sound, only sound as a result of aleatoric serendipity; outside of tightly controlled experimental environments, the textual data as presented via the Ngrams interface is only useful as a biased and poorly focused selection of instances of term use during the stipulated time period.

Other than the painstaking, longwinded traditional lexicographical techniques of gathering and comparing dated uses of terms in context and over time, no technique or set of techniques (that I know of) suffices to address the desired distillation of 'contemporary use' into 'contemporary senses'.

I should stress that 'contemporary use', that is, what you call "current use", although I'm understanding it in the spirit you seem to intend (which is notably parochial), includes all historical use. That is to say, for example, that to the extent that Shakespeare is still read and discussed, Shakespearean usage remains contemporary to that limited extent, however rarely a Shakespearean sense of a still-common term may occur in popular contemporary use.

1. In view of the fact that it is a fossil record, should we rely on Google Ngram Viewer as an authority for current usage?

See 2.

2. If Google Ngram Viewer is found to be wanting, what do we use instead?

No (to question 1). Any "fossil record" should inform and condition conclusions regarding "current usage", but the notion that a fossil record is all that is presented when the beast (the literature) still walks the earth is just plain wrong. Google Ngram Viewer/Google Books has limited utility as a collection of corroborative evidence of term use to 2008. Current, that is, contemporary use, is best gauged with a wide-ranging combination of resources. For links to some of those resources, see the following.

3. Is there an up-to-date currently-maintained corpus based on newspapers and Web sites for English?

Yes, if "up-to-date" and "corpus" are somewhat loosely used. The "corpus" is the sum total of World Wide Web resources available through general and specialized internet search facilities. "Up-to-date" for these resources is not "up-to-the-minute".

Aside from the obvious general internet search of contemporary journals and newspapers using any of a variety of search engines, free searchable newspaper corpora include the compilation from some US, Australian, New Zealand and Singapore newspapers at Elephind, and the more specialized compilation of US papers at Chronicling America. These each have their strengths and weaknesses, in terms of the results interface, etc. Elephind includes Chronicling America results, but the interface is weaker in some important areas (highlighting search terms, or not, being the most prominent area of disparity). Note also that Elephind includes some papers from as early as 1787 and as late as 2015 (the Elephind date range is a moving target because the Veridian enterprise is ongoing), while the Chronicling America collection is limited to papers published between 1836 and 1922.

Other useful sources of newspapers and other works include these:

• The Hathi Trust Digital Library (contains many works also included in Google Books, and sometimes offers alternative routes of text access where access is limited via Google Books);
• The Internet Archive and Open Library ("10,000,000 eBooks and texts");
• Google Books (this is the base collection for Google Ngram Viewer; I find it returns different results than the Viewer, and so usually check both);
• Veridian (many of these are included in Elephind results);
• Newspapers+ Publisher Extra (some time in the past I gave up on the next paywalled source in favor of this one; I believe the problem with the next source was signal-to-noise ratio, but I don't remember, although I assume my reasons were sound);
• Newspaper Archive (this bills itself as the "World's Largest Collection", but you'll have to pay to use it);
• The Internet Library of Early Journals (limited collection of 18th and 19th Century English journals).

The usual specialized online corpora, such as COCA/COHA (Corpus of Contemporary/Historical American English), Micase (Michigan Corpus of Academic Spoken English), Corpus Concordance English, British National Corpus, etc. (see especially the listing shown at english-corpora.org), should be mentioned. Some of these are listed in other questions and answers at the ELU meta site. (See the related questions and answers listed in a sidebar box associated with the linked question on the ELU meta site. The most comprehensive lists seem to be in What good reference works on English are available? on meta and What are your favorite English language tools? on main.)

For your specific question, there doesn't seem to be an organized and comprehensive answer; the questions and answers on ELU and ELU Meta referenced in my preceding paragraph are more general, very roughly organized, and don't include the newspaper or some of the other corpora I mentioned.

Raw Google Books n-gram data and more sophisticated interface

If you have the chops, the raw Google Books n-gram data is available for use. Unless whatever application you devise includes Google Books lookup and link collection facilities, of course, you will find the Google Ngram Viewer more convenient for many uses.

A much more sophisticated interface than the Google Ngram Viewer for the Google Books n-gram data is available via English-corpora.org.

Up-To-Date Usage

For your question 3, two very useful corpora only recently (May, 2016) made available are the NOW Corpus of contemporary online newspapers and CORE: Corpus of Online Registers of English. For details see their respective sites, but here are brief descriptions from those sites.

For NOW:

The NOW corpus (Newspapers on the Web) contains about three billion words of data from web-based newspapers from 2010 to the present time. More importantly, the corpus grows by about 4 million words of data each day (from about 10,000 new articles), or about 120 million words each month.

With this corpus, you can see what is happening with the language this week — not just 10 or 20 years ago. For example, see the frequency of words since 2010, as well as new words and phrases from the last few years.

For CORE:

This corpus contains more than 50 million words of text from the web. Unlike other corpora from the web, which are just big "blobs" of data, this is the first large web-based corpus that is carefully categorized into many different registers.

... You might pay special attention to the comparisons between registers and the (new) virtual corpora, which allow you to create personalized collections of texts related to a particular area of interest.

In addition to these recently released corpora, one of particular interest for researchers looking for information about contemporary informal usage is the Corpus of American Soap Operas:

The SOAP corpus contains 100 million words of data from 22,000 transcripts from American soap operas from the early 2000s [ed.: 2001-2012], and it serves as a great resource to look at very informal language.

Answer 4

Since the header asks broadly "How reliable is Ngram?" I thought it might be useful to point out a significant aspect of using Ngram as a search tool that casual users may not be aware of: the results change within the same period of years depending on what date range you set.

Ngram's default range is 1800–2000, but the tool permits you to set the endpoints to as early as 1500 and to as late as 2008. In my experience, however, the broader the date range you set beyond 1800–2000, the greater the number of matches from the default search will drop out of the Google Books search results available through the links beneath the Ngram graph.

For example, a search for "Jesus H Christ" for the period 1800–2000 yields (false positive) matches from 1804, 1805, 1817, and 1831, but the same search extended to the period 1500–2008 returns only the 1805 match from this group. On the other hand, shortening the date range for the search to 1800–1900 yields not only the original four matches but five additional (false positive) matches—from 1840, 1842, 1843, 1847, and 1853—not included in the the results returned for either of the other two searches.

From these examples, you might suppose that the more narrowly you define the time frame, the more numerous the results will be within that time frame. But that is not the case. A search limited to the period 1800–1870 provides no new matches compared with the 1800–1900 search, and a search limited to the period 1800–1865 yields matches only for the 1804, 1805, and 1817 false positives; inexplicably, all of the 1831–1853 matches vanish.

What this means in practical terms is that, if you want to use Ngram as a search tool to find texts published within a particular time frame of interest, you have to adjust the date parameters (sometimes repeatedly) to maximize the matches that Google Books returns. In the case of "Jesus H. Christ," the additional matches from the period 1840–1853 turned out not to be helpful, which is not surprising considering that the earliest publication year for any confirmed match for the phrase that I've been able to find is 1880. But in the course of this particular research effort, running multiple searches with different end years led me to focus eventually on the period 1860–1925, and Ngram's Google Books searches from that period yielded a number of highly relevant matches that did not appear in the search results for the default 1800–2000 time period.

Answer 5

The fact is, NGrams can be a good tool for examining the language in published books. In fact, the OED uses it as part of calculating word frequency! However, the results may not be accurate unless you go and verify they are.

OCR

The fact is, Google Books relies a lot on OCR. Unfortunately, there are a lot more OCR errors than just long s (which also confuses people who are not familiar with old typography). As this search indicates, it's also pretty common in older books to have a become o (and apparently, , become t). You'll also find that some books are very garbled, especially as you look further back in time. Even this book from 1880:

ran an 'm'ia 9w twz'mz n'n'rxz “'8 "BL E'BI'“! '8831.133 '9'61'66 'HJBHOBJ Yl'sz'vv az 'oz'v: n'n Lr'rlmz ru 51; "xx: we sn'uu Ht Lrn 13H fla-... “1 . 3 ;._:nm .ag_5 9's M'Isz wot 111st ar'st 1: nu rv'awo

This obviously obscures useful matches and may create false positives.

Words spelled the same

This is another pitfall I've seen people fall into. They'll try to compare two terms for frequency, and forget to consider matches that have other meanings (such as homographs) or matches that fall into another part of speech.

The opposite (when one word has many spellings) can become a problem in older texts (although you've usually run into other problems, mainly OCR, before this is an issue).

Korpus Vokabel

Yes, I may have specifically searched for English results here, but did you notice some (all?) of the actual results are in German? (Korpus and Vokabel are the German words for corpus and vocabulary.)

And if you can't trust "English" to only show results in English, I wouldn't trust "British English" to only return results in British English. For example, (looking at modern usage) this graph for color vs. colour shows one spelling is "preferred" in AmE and the other in BrE, but it also shows a number of results for the non-preferred spelling in each dialect (which doesn't seem accurate to me).

More specifically, books written in British English that were published in the US are included in the "American English" corpora. And the reverse is true too. (This is mentioned on the info page under "corpora".)

In general, there are some problems with metadata, which also affects the year published.

Part-of-Speech tagging

The logical solution to categorizing PoS is tagging (not that I've seen anyone else use this). According to the NGrams info page, part-of-speech tagging is usually accurate, but not so much for older English or uncommon usages:

The part-of-speech tags and dependency relations are predicted automatically. Assessing the accuracy of these predictions is difficult, but for modern English we expect the accuracy of the part-of-speech tags to be around 95% and the accuracy of dependency relations around 85%. On older English text and for other languages the accuracies are lower, but likely above 90% for part-of-speech tags and above 75% for dependencies. This implies a significant number of errors, which should be taken into account when drawing conclusions.

The part-of-speech tags are constructed from a small training set (a mere million words for English). This will sometimes underrepresent uncommon usages, such as green or dog or book as verbs, or ask as a noun.

???

There are some queries that inexplicably turn up the wrong results. One such query is this one, which shows a noticeable amount of usage for "bark up the" between 1805 and 1830. When you actually click on the search results, it is clear that there were 0 occurrences of the phrase during that period.

Statistics

This isn't actually a problem with NGrams itself, but nevertheless I feel it is relevant (as I've seen it become a source of error too). Many people seem to lack the statistics knowledge needed to understand some of what's involved with making and understanding NGrams. Luckily, their info page explains some of what you need to know:

Why do I see more spikes and plateaus in early years?
Publishing was a relatively rare event in the 16th and 17th centuries. (There are only about 500,000 books published in English before the 19th century.) So if a phrase occurs in one book in one year but not in the preceding or following years, that creates a taller spike than it would in later years.

Plateaus are usually simply smoothed spikes. Change the smoothing to 0.

What does "smoothing" mean?
Often trends become more apparent when data is viewed as a moving average. A smoothing of 1 means that the data shown for 1950 will be an average of the raw count for 1950 plus 1 value on either side: ("count for 1949" + "count for 1950" + "count for 1951"), divided by 3. So a smoothing of 10 means that 21 values will be averaged: 10 on either side, plus the target value in the center of them.

At the left and right edges of the graph, fewer values are averaged. With a smoothing of 3, the leftmost value (pretend it's the year 1950) will be calculated as ("count for 1950" + "count for 1951" + "count for 1952" + "count for 1953"), divided by 4.

A smoothing of 0 means no smoothing at all: just raw data.

Many more books are published in modern years. Doesn't this skew the results?
It would if we didn't normalize by the number of books published in each year.

However, I don't have all the answers. I'm still not sure what's the math behind comparing NGrams of different sizes, such as this (previously mentioned) one which compares a 3gram and 4gram.

Answer 6

If you use Ngram and just look at the graph you are looking at garbage. If you carefully examine a representative number of the actual references you get a much better idea as to whether the graph is meaningful or not.

Once you become a billionaire you may certainly create your own database (being sure to observe copyright limitations!) and set up your own site to search it. Until then I'm thankful that Ngram works as well as it does as often as it does.