Feature Request: Implement CID allowlist HTTP Middleware

[mmuenker]

I'm reaching out to discuss an idea I have for enhancing security and access control in the Rainbow IPFS Gateway. I'd like to gauge interest and gather feedback before deciding on implementation.

My Idea: Implementing CID allowlist HTTP Middleware

The goal of this feature would be to prevent flooding self-hosted instances with requests or compromised CIDs by restricting access to certain resources based on an allowlist of allowed CIDs. This would reduce the risk of Distributed Denial-of-Service (DDoS) attacks, floods of requests, and unauthorized access attempts.

Before deciding on implementation, I'd like to get your thoughts on the following:

• Is my idea aligned with the project's goals and values?
• Would this feature significantly impact performance or scalability?

---

Assuming interest in implementing this feature, here are some potential ways it could be done:

To implement CID allowlist Middleware, we would need to add a middleware layer that intercepts incoming HTTP requests, extracts the CID from the URL path, and checks it against a predefined allowlist.

I'd be happy to contribute to this feature if you're interested. Please let me know if this is something that could be explored further, or if there are any concerns or issues with my idea.

[aschmahmann]

@mmuenker have you tried using the denylist feature (https://github.com/ipfs/rainbow?tab=readme-ov-file#denylists)? The spec https://specs.ipfs.tech/compact-denylist-format/ seems to indicate you can block everything and then allowlist (using negation) the data you need. If not then the denylist format can be used as allowlist as mentioned in the spec.

cc @hsanjuan we may want to add an example of using .deny for allowlists into the spec if that's an option rather than pushing people to .allow.

[mmuenker]

@aschmahmann How must denylist be defined to deny everything by default?

In the spec I could not find a hint.

version: 1
name: Allowlist Example
description: A exmaple for using the Compact Denylist Format as allowlist
author: [email protected]
---
/ipfs/*
/ipns/*
!/ipfs/Qmd8NDeJhzf614FSBxZwu4QD2Az14tQtJhQXJf8h4fqiSx

would something like this work?

[lidel]

@mmuenker yes, this should already be possible with existing impl.:

• if list is in a file named .deny then we allow everything and only block things from the file
• if list is in a file named .allow then we deny everything and only allow entries from the file

Did you test it, and it did not work?

[mmuenker]

@lidel I wasn't aware that using the file name suffixes .deny and .allow was required.

I tested this with a demo allow/deny file, and while the deny file works as expected, the allow file doesn't seem to function correctly.

When the allow file is added, it allows all CIDs instead of just the ones specified in the file.

You can find the files I've used here: https://gitlab.com/-/snippets/4797690 .

The CIDs point to the same file in both v0 and v1.

I have used the environment variable to specify the allow/deny list:

RAINBOW_DENYLISTS=https://gitlab.com/-/snippets/4797690/raw/main/demo.allow

[hsanjuan]

@mmuenker yes, this should already be possible with existing impl.:

* if list is in a file named `.deny` then we allow everything and only block things from the file

* if list is in a file named `.allow` then we deny everything and only allow entries from the file

I don't think this is the case. Only .deny are supported.

There is a workaround which is pointing Rainbow to a Kubo gateway with NoFetch: true, so what is stored there is your allowlist.

Otherwise, I was hoping an allowlist.deny as follows to work:

# Block all subpaths
/* 
# Allow specific cids and subpaths
/ipfs/<cid>/*

It doesn't however, but it would be easy to fix in NoPFS, at least for anything that is not accessing Blockstore directly (the /* subpath blocking rule` is not considered when checking CID-only blocks). But I think rainbow hits the happy path on that regard.

Changes would happen here: https://github.com/ipfs-shipyard/nopfs/blob/349ea17adc28c25a2711f0c431ce02b33c9ae24b/denylist.go#L885-L926

• Move subpath-blocking check to the bottom so that negated rules match first.
• Check also empty subpaths, so that /* matches when path is empty.
• Add a tests for an allowlist in the form above that checks that all paths are blocked except the negated ones.

[hsanjuan]

(direct subpath-blocking exists in nopfs but doesn't seem to be part of the spec, I don't remember if there is a reason or it was just a fly-by feature during implementation).