These are frequency-sorted lists of words seen in Project Gutenberg Australia books that were available for download in plain text format. The approach was as follows:
cut -d, -f1-3 catalogue.txt | grep ',txt$' | sed 's.,./.; s/,/./; s,^,http://gutenberg.net.au/,' > urls
wget -i urls
rm -- `fgrep -l pagenotfound [0-9]*.txt`
cat [0-9]*.txt | sed -r 's/--+/ /g' | tr -cs "A-Za-z'-" '\n' | egrep "^([A-Za-z]|[A-Za-z][A-Za-z'-]{0,99}[^-])$" | fgrep -v "''" | sort -S20G | uniq -c | sort -S20G -rn > gutenberg-all-words-with-counts.txt &
cat `grep -l '[^0-9]19[0-9][0-9][^0-9]' [0-9]*.txt` | sed -r 's/--+/ /g' | tr -cs "A-Za-z'-" '\n' | egrep "^([A-Za-z]|[A-Za-z][A-Za-z'-]{0,99}[^-])$" | fgrep -v "''" | sort -S20G | uniq -c | sort -S20G -rn > gutenberg-19xx-words-with-counts.txt
wait
grep -v "[A-Z'-]" gutenberg-all-words-with-counts.txt > gutenberg-all-lowercase-words-with-counts.txt
grep -v "[A-Z'-]" gutenberg-19xx-words-with-counts.txt > gutenberg-19xx-lowercase-words-with-counts.txt
for f in gutenberg-*-words-with-counts.txt; do cut -c9- < $f > ${f/-with-counts}; done
for f in gutenberg-*-words-with-counts.txt; do sed -n '/ 9 /q; p' $f | cut -c9- > ${f/-with-counts/-10plus}; done
for f in gutenberg-*-words-with-counts.txt; do sed -n '/ 99 /q; p' $f | cut -c9- > ${f/-with-counts/-100plus}; done
for f in gutenberg-*-words-with-counts.txt; do sed -n '/ 999 /q; p' $f | cut -c9- > ${f/-with-counts/-1000plus}; done
A total of 1962 books were processed resulting in the gutenberg-all-* files.
Since Project Gutenberg only includes books where copyright has expired, the
words are biased towards older English. To partially mitigate this, processed
separately was a subset consisting of 1172 books that include the numbers 19xx
(hopefully were written in those years or an author lived into those years).
This resulted in the gutenberg-19xx-* files.
The *lowercase* files include only all-lowercase words, and are of pretty
high quality. They do not include any words with capital letters, nor any
words with dashes nor apostrophes.
The more complete lists (without *lowercase* in the filenames) do include
those other categories of words, but as a result are of much lower quality.
For example, there's no reliable way to distinguish a word that is capitalized
because it's the first word in a sentence (in which case we'd ideally convert
it to lowercase and count as such) or also because it's a given name (in which
case we should preserve the capitalization). The current approach doesn't even
try to distinguish these cases (always keeping the capitalization intact, but
also always excluding such words from the *lowercase* files).
Please do not link directly to individual files in this repository as they're likely to be moved around in future revisions. Link to the repository instead.