[do not merge] add support for the columnar index

.gitignore

@@ -3,3 +3,4 @@ dist/
 __pycache__
 cdx_toolkit.egg-info
 .coverage
+.eggs/

ATHENA.md

@@ -0,0 +1,44 @@
+# Using cdx_toolkit's columnar index with Athena
+
+## Installing
+
+```
+$ pip install cdx_toolkit[athena]
+```
+
+## Credentials and Configuration
+
+In addition to having AWS credentials, a few more configuration items are needed.
+
+credentials: can be done multiple ways, here is one: ~/.aws/config and [profile cdx_athena]
+
+aws_access_key_id
+aws_secret_access_key
+
+s3_staging_dir=, needs to be writeable, need to explain how to clear this bucket
+schema_name= will default to 'ccindex', this is the database name, not the table name
+
+region=us-east-1  # this is the default, and this is where CC's data is stored
+# "When specifying a Region inline during client initialization, this property is named region_name."
+s3_staging_dir=s3://dshfjhfkjhdfshjgdghj/staging
+
+
+## Initializing the database
+
+asetup
+asummary
+get_all_crawls
+
+## Arbitrary queries
+
+asql
+explain the partitions
+explain how to override the safety-belt LIMIT 
+
+## Iterating similar to the CDX index
+
+## Generating subset WARCs from an sql query or iteration
+
+## Clearing the staging directory
+
+configure rclone

CHANGELOG.md

@@ -1,3 +1,6 @@
+- 0.9.34 (not yet tagged)
+	+ experimental support for CC's columnar index
+
 - 0.9.33
 	+ rename master to main
 	+ drop python 3.5 testing because of setuptools-scm

README.md

@@ -12,6 +12,10 @@ hides these differences as best it can. cdx_toolkit also knits
 together the monthly Common Crawl CDX indices into a single, virtual
 index.
 
+CommonCrawl also has a non-CDX "columnar index" hosted on AWS,
+accessible via the (paid) Amazon Athena service. This index can be
+queried using SQL, and has a few columns not present in the CDX index.
+
 Finally, cdx_toolkit allows extracting archived pages from CC and IA
 into WARC files.  If you're looking to create subsets of CC or IA data
 and then process them into WET or WAT files, this is a feature you'll
@@ -218,7 +222,7 @@ cdx_toolkit has reached the beta-testing stage of development.
 
 ## License
 
-Copyright 2018-2020 Greg Lindahl and others
+Copyright 2018-2022 Greg Lindahl and others
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this software except in compliance with the License.

azure-pipelines.yml

@@ -37,7 +37,7 @@ jobs:
           echo "Note: using 3.6 because packages are deprecating 3.5 support"
           pip install -r requirements.txt
         fi
-        pip --use-feature=in-tree-build install . .[test]
+        pip --use-feature=in-tree-build install . .[test] .[athena]
       displayName: 'Install dependencies'
 
     - script: |
@@ -74,7 +74,7 @@ jobs:
 
     - script: |
         python -m pip install --upgrade pip
-        pip --use-feature=in-tree-build install . .[test]
+        pip --use-feature=in-tree-build install . .[test] .[athena]
       displayName: 'Install dependencies'
 
     - script: |