EdgeFS - a multi-cloud scalable distributed storage system

EdgeFS is high-performance and low-latency object storage system released under Apache License v2.0 developed in C/Go. It provides Kubernetes integrated Multi-Head Scale-Out NFS (POSIX compliant, Distributed RW access to files), Amazon S3 compatible API with AI/ML S3X enhancements, iSCSI and NBD block interfaces, advanced global versioning with file-level granularity unlimited snapshots, global data deduplication and geo-transparent access to data from on-prem, private/public clouds or small footprint edge (IoT) devices.

EdgeFS is capable of spanning unlimited number of geographically distributed sites (Geo-site), connected with each other as one global name space data fabric running on top of Kubernetes platform, providing persistent, fault-tolerant and high-performance fully compatible S3 Object API and CSI volumes for stateful Kubernetes Applications.

At each Geo-site, EdgeFS segment nodes deployed as containers (Kubernetes StatefulSet or Docker Compose) on physical or virtual nodes, pooling available storage capacity and presenting it via compatible S3/NFS/iSCSI/etc storage emulated protocols for cloud-native applications running on the same or dedicated servers.

How it works, in a Nutshell?

If you familiar with "git", where all modifications are fully versioned and globally immutable, it is highly likely you already know how it works at its core. Think of it as a world-scale copy-on-write technique. Now, if we can make a parallel for you to understand it better - what EdgeFS does, it expands "git" paradigm to object storage and making Kubernetes Persistent Volumes accessible via emulated storage standard protocols e.g. S3, NFS and even block devices such as iSCSI, in a high-performance and low-latency ways. With fully versioned modifications, fully immutable metadata and data, users data can be transparently replicated, distributed and dynamically pre-fetched across many Geo-sites.

Developer Guides, Services and APIs

• Development Guide - details on build environment
• Harness Tests - details on how to run Harness Test package
• Minimalistic CLI tool - CLI tool implemented in Go
• LIBCCOW API - generic client library and its direct access API
• NFS and FSIO API - POSIX file system layer API
• Block and LIBCCOWVOL API - Block device API and iSCSI/NBD integration
• Edge-X S3 API - details on advanced S3-compabile features, e.g. Versioned Edits, Snapshots, etc
• FIO Engines - details on how to run FIO benchmarking tool at various directly conntected I/O layers
• LIBPMU userspace networking - details on libpmu, DPDK-like library

Build it (Production Image)

Make sure Docker package is installed on your build server with version >= 17.05 with support for staged builds. Execute the following command:

git clone [email protected]:Nexenta/edgefs.git
cd edgefs
docker build -t edgefs .

Quick Starts

Deployments:

• Docker
• Kubernetes RookIO

Configurations:

• Initialization - generic initialization procedure, applicable
• Kubernetes RookIO Deployment - segment deployment procedure
• Kubernetes RookIO CSI Integration - detailed instructions on how to get CSI configured with EdgeFS RookIO
• Kubernetes RookIO Monitoring - Prometheus and Graphana integrations
• Kubernetes RookIO VDEV Management - disk/VDEV health checking, replacement, addition, etc
• Kubernetes RookIO Upgrade - detailed instructions on how to execute rolling upgrade
• Kubernetes RookIO GUI - segment dashboard, CRD wizard
• Kubernetes RookIO NFS - setting up Scale-Out NFS (File)
• Kubernetes RookIO S3 - setting up AWS S3 compatible interface (Object)
• Kubernetes RookIO S3X AI/ML - setting up S3X interface for AI/ML, NoSQL and other intensive low latency workloads
• Kubernetes RookIO iSCSI - setting up Scale-Out iSCSI (Block)
• Kubernetes RookIO ISGW (Global Namespaces) - setting up geo-transparent capable global name space
• Kubernetes RookIO OpenStack/SWIFT) - setting up OpenStack/SWIFT inteface (Object)

Join our growing community!

• Users group: edgefs-users
• Slack channel: slack channel
• Twitter: @edgefs_io
• Telegram: edgefs

Community Meeting

A regular community meeting takes place monthly on every first Tuesday at 10:00 AM PT (Pacific Time). Convert to your local timezone.

Any changes to the meeting schedule will be added to the agenda doc and posted to Slack #users and the edgefs-users mailing list.

Anyone who wants to discuss the direction of the project, design and implementation reviews, or general questions with the broader community is welcome and encouraged to join.

• Meeting link: https://zoom.us/j/404796463
• Current agenda and past meeting notes
• Past meeting recordings - coming