Roadmap (more likely a TODO list)

This is a non-extensive roadmap for Norman DB

Commander

• Setup API skeleton
• Job system for ingestion
• State Machine Replication for recovery
• Schema creation/update/deletion
• Table creation/update/deletion
• Setup Cluster concept
• Setup Tenant concept
• Setup the retention management for Segments
• Leader election for multiple Commander services
• Segment replication on multiple Storage servers
• Rebalancing coordination of segments
• Coordinate and update the list of segments to be "used" for querying
• System to "drop" the segments
• Scheduler of compaction process

Broker

• Setup API skeleton
• Query endpoints
• Query system for retrieving data from each Storage service
• Aggregate results and present it to client
• Query planning for optimization

Storage

• Setup gRPC skeleton for communication with Commander and Broker
• Setup segment creation (immutability)
• Use Apache Parquet to store new segment to save space
• Run ingestion jobs
• Realtime ingestion
• Offline ingestion
• Setup deep-storage for segments
• Tiered storage
  • hot segments are in memory cache - Apache Arrow is sufficient perhaps?
  • warm segments are in the local storage
  • cold segments are stored in a blob storage like (S3, GCP Storage, Azure Blob Storage)
• Indexing system (inverted index, range index, etc)
• Bloom/Cuckoo filter implementation
• Segment Compaction system
• Segment caching

Common

• UI for Query
• UI for Admin work