🎢 DHRW ~ Visual Data Pipelines

DHRW is a proof-of-concept for a low-code, IaaS platform to let you visually create data processing pipelines. See it in action!

✅ Features

⌨️ Write your Python functions (or use functions written by others and imported in the app)

🔗 Add them to a GitHub repository (example repo)

📥 Import them in the app

💫 Connect them to create execution graphs - a sequence of functions that you can pipe data through

🚀 Deploy in the cloud

With just the press of one button, you can auto-magically turn a series of functions into a network of Docker containers that are provisioned with your code and wait to receive and process your data.

📑 Process data

You can then upload CSV files with your data and receive the processing results right in your browser. Supports both text as well as image outputs (for plots).

👨🏻‍💻 Live edit & redeploy

🌐 Pairs really well with GitHub Codespaces to completely develop right in the browser. Edit your function code using VSCode in the browser, reupload it in the graph, and redeploy in two clicks. 🙌

🐳 Get it running

Setup is a bit tricky since this is a proof-of-concept. Here are the steps to get it running locally:

Main setup

1. Clone repository
2. docker compose up
3. Manually install python dependencies (found in worker/requirements.txt on the meteor container) - known issue, they need to be synchronised 😅
4. Get an access token for your function repository and add it in settings.json, replacing the one that is there (expired, used to point to this repo).

RabbitMQ setup

1. enter localhost:15672 and add workers exchange of type topic
2. also add a routing key named worker_reply.* to server_responses queue on this exchange. - known issue, they need to be added manually for now.

Get in the app

Enter the app via localhost:3000.

🤖 Tech stack

📜 Docs

Read the docs here (in Romanian, this was my master's thesis project).

🎬 Demo video