1

I'm looking for a resource where I can see the frequencies of letters in with respect to their position inside the word and the length of the word. I'd also like to see the frequencies of different combinations of letters, as in "how often does a word start with an e and end with an a".

To be extra clear: the most common letter in English is e, but the most common third letter is...? And what about the most common fourth letter in five-letters-long words?

I'm specifically interested in the frequencies within dictionaries.

6
  • 4
    You should be able to download a word list and write a simple program to gather such statistics (or get someone to do it for you). It will depend on the precise nature of the words you want. how long the list is, whether you want plurals, contractions, abbreviations, etc.
    – Stuart F
    Feb 14 at 23:52
  • @StuartF I was thinking about that, but I'm very new at programming, so I thought it best to check if such a resource was already available.
    – user110391
    Feb 15 at 0:04
  • 3
    Search for "regular expression" on Stack Overflow or the web. They are used for exactly the kind of pattern matching you are looking for. Put 170k English words in a list (or dictionary for speed) and count the number of occurrences matching your expression. There are in-built methods in many languages and probably some packages you can use off-the-shelf.
    – dubious
    Feb 15 at 9:35
  • 2
    You can do everything with Unix utilities - sed, wc, cat, etc. as long as you stick to ASCII text. Since you're dealing with English, no problem. But you'll hafta learn regular expressions. Feb 15 at 15:51
  • 1
    Pretty sure some of the tools developed around generating words with Markov chains might help. Basically, the generator analyzes a corpus or word list, then generates random text or words with similar characteristics. The tools to analyze a word list (like maybe part of github.com/unaimillan/markov-word-generator) might give you a starting point.
    – ColleenV
    Feb 16 at 15:33
  • Can you clarify the context? Might it not be important to know whether this is about code-breaking or philology, for instance? Apr 3 at 21:50

2 Answers 2

1

For specific combinations of letters within words of a specific length, use a crossword solver.

For a bit more flexibility, you can use a regex dictionary. (Of course, this means you have to know or learn regex... not trivial.) Here's the regex for "how often does a word start with an e and end with an a":

^e[^\s]*a$

For your other use case, it's a bit more complicated but you could string together several tools. First, you get (for example) the fourth character of every five letter word, which could be done using Regex101 with a wordlist and the regex ^...(.).$ (must have multi line matching and maybe case insensitive modifiers, also maybe use [a-z] in the regex instead of ., since the latter matches spaces and hyphens), then export only the matches. Then, feed the results to something that tells you the most common occurrence in the list.

1

English character frequencies are amenable to online search. See for example The frequency of the letters of the alphabet in English.

Your second problem of frequencies of position in a word of specified length has to be coded. As a coding question it belongs on another site. Nevertheless, it is impossible to fit this into the format of a comment, so I have answered it here only so as to give a pointer to how it may be done.

One of the many ways is to code in Mathematica (see the sister Stack Exchange site) and to use the built-in dictionary of almost a hundred thousand words.

Here is the answer to your specific 5 word & 4th character query but it is simple to make a routine to examine any length (n) of word and any character (m) within the words:

enter image description here

From which you may see that the most common fourth letter in five letter words is "e".

(My coding is pretty basic and could be briefer in the hands of a more competent Mathematica user)

Here is the text version of the code:

(*Look up all words in the Mathematica Dictionary and convert to  lowercase*)  
allwords = ToLowerCase[DictionaryLookup[]];  
(*Find all words with length n*)  
words[n_] := Select[allwords, StringLength[#] == n &];  
(*Find the mth character in the list of n letter words*)  
wordNcharacterM[n_, m_] := Map[Take[#, {m}] &, Characters[words[n]]];  
(*Total number of characters found*)  
totalChars[n_, m_] := 
  Total[Tally[wordNcharacterM[n, m]]\[Transpose][[2]]];  
(*Tabulate the results of a tally of the numbers of characters found*)  
table[n_, m_] := 
  Map[{#[[1]], #[[2]], N[100 #[[2]]/totalChars[n, m], 2]} &, 
   Tally[wordNcharacterM[n, m]]];  
(*Construct a title table*)  
tableOut[n_, m_] := 
  Join[{{{"word", n}, {"character", m}, " "}}, {{{"letter"}, "number",
      "%"}}, table[n, m]];  
(*Present the table*)  
outputWordNMCharacterM[n_, m_] := 
  Join[{{{"word", n}, {"character", m}, " "}}, {{{"letter"}, "number",
      "%"}}, ReverseSort[table[n, m], #1[[2]] < #2[[2]] &]];
(*Present the table*)
outputWordNMCharacterM[n_, m_] := 
  Join[{{{"word", n}, {"character", m}, " "}}, {{{"letter"}, "number",
      "%"}}, ReverseSort[table[n, m], #1[[2]] < #2[[2]] &]];
(*Present the table*)
outputWordNMCharacterM[n_, m_] := 
  Join[{{{"word", n}, {"character", m}, " "}}, {{{"letter"}, "number",
      "%"}}, ReverseSort[table[n, m], #1[[2]] < #2[[2]] &]];
(*Present the table for five letter words position 4*)
outputWordNMCharacterM[5, 4] // TableForm
5
  • @Laurel Thanks for a competent job. I have corrected my last line.
    – Anton
    Feb 16 at 13:08
  • I am a bit confused by you saying the answer is "f". In that list, "t" has a frequency of 7.4%.
    – user110391
    Feb 16 at 14:59
  • @user110391 I apologize. I made a copy of the wrong table and have now corrected the output. I have also made the last part of the code more concise.
    – Anton
    Feb 16 at 15:17
  • 1
    Thanks :) . However, I'm still confused. You now say the answer is "n", but "e" is higher up.
    – user110391
    Feb 16 at 16:15
  • 1
    @user110391 Oops! A careless typo because my attention was on the code. You are correct. I have altered the answer again.
    – Anton
    Feb 16 at 16:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .