Unified "API v3" design across functional areas

[davenquinn]

With design-focused updates like #3 and #5, and the evolution of our API v3 systems in other areas (e.g., a PostgREST API system) we should start a conversation on how to unify and carry forward Macrostrat's API v3 for future functionality. A few elements that we should establish:

• Delineation what should be handled by PostgREST vs. Python-based API code (e.g., prefer basic table access/paginated results via PostgREST and geological filtering more akin to API v1/2 via Python)
• Patterns of separation between domains of functionality (e.g., paleogeography, column editing, map ingestion, etc.)
• Consistent handling of authentication cookies/tokens
• Caching of results for highly-used queries (e.g., Redis/Varnish)
• Unified documentation across both API systems (and tileserver?)
• Request logging for certain routes (establish data on usage, related to Analytics macrostrat#67)
• Some affordance for "internal" routes that are not publicly documented/stabilized

[davenquinn]

Output of our first conversation with @CannonLock, @amyfromandi, and @brianaydemir:

Handling different API patterns

We want the API to avoid "reinventing the wheel" while allowing flexibility

• We will prefer PostgREST for table operations, filtering, and basic CRUD routes where possible
• We will use Python and this repository for routes requiring custom logic (e.g., filesystem, security/tokens, filtering via complex geological criteria).
• Other services and containers can be maintained where usage patterns and infrastructure needs are different (e.g., tileserver, ml-based extensions)

We should develop libraries and well-documented patterns for API secrets, token/API key handling, etc. that can be pulled in via libraries.

• The current capabilities of Macrostrat's v2 API depend on a lot of semantic filtering via geologic criteria, so they will eventually mostly be brought into the api-v3 framework.
• Some functional areas might require several 'simple' routes that can be implemented as PostgREST views and a few more complex ones that require separate handling (e.g., validation routes). We hope to allow multiple approaches to be used simultaneously.

Documentation and public-facing capabilites

Despite being composed of a handful of separate services with different API query styles, the goal will be to present Macrostrat's API as a unified /api/v3 cluster of routes.

API docs will prioritize OpenAPI/Swagger since it is autogenerated by many tools. We will explore patterns to merge documentation between services and separate it between functional areas.

Open questions

• Caching: redis, varnish, etc. ? How to mark routes for caching and expire things when appropriate? Needs are different between tileserver, columns, etc. Don't want to get bogged down in cache management
• Best way to do overarching documentation
• Should internal/pre-production APIs (e.g., proxy servers etc.) be placed under a single umbrella somehow? It would be nice to have a dev2.macrostrat.org/api/v3/_private gateway that would automatically apply some security, etc.