feat(sessions): irn client scaffolding #681

geekbrother · 2024-06-21T14:13:07Z

Description

This PR implements the IRN client abstraction into the blockchain-api. The following changes are made:

CI/CD and build

WalletConnect/irn-node repository was added as a git submodule.
irn_api and network packages were added as dependencies from the irn-node.
BSL-1.0 license was added to the allowed list.

The following IRN client configuration parameters were added:

RPC_PROXY_IRN_NODE environment variable and irn_node terraform variable for the IRN node address to connect to in IP:Socket format.
RPC_PROXY_IRN_KEY environment variable and irn_key terraform variable for the IRN client private key.
RPC_PROXY_IRN_NAMESPACE environment variable and irn_namespace terraform variable for the IRN storage namespace.
RPC_PROXY_IRN_NAMESPACE_SECRET environment variable and irn_namespace_secret terraform variable for the IRN storage namespace secret.

The following IRN client storage commands were added:

Key-value storage set,get, del.
Hashmap storage hset, hget, hdel, hfields, hvalues.

IRN client abstraction is bound to the irn global state to be accessible from handlers.

The record TTL is 30 days. (Which should be ok for sessions).

How Has This Been Tested?

Unit test for the helper function was implemented and run in CI.
Test to run get, set, del were implemented. The test is ignored by default, because we don't want to spin up a cluster for each CI (and will be covered in the handler integration test).
- It can be run locally by starting the local cluster and run cargo test -- storage::irn::tests::test_irn_client_set_get_del --exact --show-output --ignored. The test were successfully passed in the local environment.
Test to run hashmap operations were implemented. The test is ignored by default, because we don't want to spin up a cluster for each CI (and will be covered in the handler integration test).
- It can be run locally by starting the local cluster and run cargo test -- storage::irn::tests::test_irn_client_hashmap --exact --show-output --ignored. The test were successfully passed in the local environment.

Due Diligence

Breaking change
Requires a documentation update
Requires a e2e/integration test update

chris13524 · 2024-06-25T00:06:07Z

src/storage/irn/mod.rs

+            key_base64.as_bytes(),
+            irn_api::auth::Encoding::Base64,
+        )
+        .map_err(|_| StorageError::WrongCredentialsFormat("wrong key is provided".to_string()))?;


suggestion: Separate the WrongCredentialsFormat variant into 2 variants instead of passing a string to it

Makes sense, thanks! Added in f44679c

chris13524 · 2024-06-25T00:06:52Z

src/storage/irn/mod.rs

+        .map_err(|_| StorageError::WrongCredentialsFormat("wrong key is provided".to_string()))?;
+        let peer_id = irn_api::auth::peer_id(&key.verifying_key());
+        let node_addr = node_addr.parse::<SocketAddr>().map_err(|_| {
+            StorageError::WrongArgument("node_addr is not in socket address format".to_string())


Similar here: just make a variant for this specific error and pass the original error

Makes sense, thanks! Added in f44679c

geekbrother · 2024-06-25T09:43:35Z

src/storage/irn/mod.rs

+            irn_api::auth::Encoding::Base64,
+        )
+        .map_err(|_| StorageError::WrongKey(key_base64))?;
+        // IRN connection error Network(ConnectionHandler(NoAvailablePeers)) when using


@heilhead I have NoAvailablePeers when generating peer_id from the existing key. It works when generating a new one. Any idea where I am wrong? 🤔

As discussed on slack, you're trying to use the client peer ID for server address, and this isn't supported by our network machinery. The proper server peer IDs should be parsed as part of the node address, see the following relay code on how to parse them from a {PEER_ID}-{MULTIADDR} string.

Created a follow-up for this #684

xDarksome · 2024-06-25T13:28:40Z

.env.example

@@ -31,3 +31,9 @@ export RPC_PROXY_POSTGRES_URI="postgres://postgres@localhost/postgres"
 # export RPC_PROXY_RATE_LIMITING_MAX_TOKENS=100
 # export RPC_PROXY_RATE_LIMITING_REFILL_INTERVAL_SEC=1
 # export RPC_PROXY_RATE_LIMITING_REFILL_RATE=2
+
+# Uncomment for using the IRN client
+# export RPC_PROXY_IRN_NODE=127.0.0.1:3011


I suggest using multiple IRN nodes, as a fall-back mechanism. One node can have a downtime even under normal conditions

Makes sense. Good follow-up. Created it as #685

xDarksome · 2024-06-25T13:31:57Z

The record TTL is 90 days. (Which should be ok for sessions).

does it need to be 90 days? our current longest living data is 30 days

I'm talking from the perspective of data migrations, if our highest value is going to be 90 days, then next time we decide to do full migration we would need to shadow the data for 90 days to be sure

And it probably will happen in the next 3 month, as we will be transitioning to the permissioned network

geekbrother · 2024-06-25T13:45:29Z

The record TTL is 90 days. (Which should be ok for sessions).

does it need to be 90 days? our current longest living data is 30 days

I'm talking from the perspective of data migrations, if our highest value is going to be 90 days, then next time we decide to do full migration we would need to shadow the data for 90 days to be sure

And it probably will happen in the next 3 month, as we will be transitioning to the permissioned network

It decreased to 30 days. The context: slack.

xDarksome · 2024-06-25T13:49:10Z

src/env/mod.rs

@@ -48,6 +49,7 @@ pub struct Config {
    pub profiler: ProfilerConfig,
    pub providers: ProvidersConfig,
    pub rate_limiting: RateLimitingConfig,
+    pub irn: IRNConfig,


https://doc.rust-lang.org/1.0.0/style/style/naming/README.html

In CamelCase, acronyms count as one word: use Uuid rather than UUID. In snake_case, acronyms are lower-cased: is_xid_start.

Hmm, Clippy missed this. Don't fully rely on clippy 😅 Fixed in 87fb41f. Thanks!

xDarksome · 2024-06-25T13:55:18Z

src/storage/irn/mod.rs

+        let result = self.client.get(self.key(key.as_bytes().into())).await;
+
+        match result {
+            Ok(Some(data)) => match String::from_utf8(data) {


Why are you returning a String instead of a Vec<u8>? I'm guessing that it's going to be deserialized later anyway

geekbrother self-assigned this Jun 21, 2024

geekbrother force-pushed the feat/sessions/1-client-scaffolding branch 3 times, most recently from 6c9fea2 to c0479d6 Compare June 21, 2024 21:41

This was referenced Jun 24, 2024

feat: adding support for checkout private submodules WalletConnect/ci_workflows#9

Merged

feat(terraform): IRN VPC Peering #682

Merged

feat(ci): bumping shared ci_workflows version to 0.2.11 #683

Merged

geekbrother force-pushed the feat/sessions/1-client-scaffolding branch from c0479d6 to 0b67934 Compare June 24, 2024 20:13

geekbrother changed the base branch from master to feat/update_ci_version June 24, 2024 20:14

geekbrother force-pushed the feat/sessions/1-client-scaffolding branch from 0b67934 to d6bc453 Compare June 24, 2024 20:17

chris13524 reviewed Jun 25, 2024

View reviewed changes

Base automatically changed from feat/update_ci_version to master June 25, 2024 07:17

feat(sessions): irn client scaffolding

b578d0f

geekbrother force-pushed the feat/sessions/1-client-scaffolding branch from d6bc453 to b578d0f Compare June 25, 2024 07:18

geekbrother added 2 commits June 25, 2024 09:25

fix: adding error variants

f44679c

feat: implementing get, set and test

332bff5

geekbrother commented Jun 25, 2024

View reviewed changes

geekbrother added 2 commits June 25, 2024 13:14

feat: implement hashmap functions

e20680e

feat: adding terraform variables

0450365

geekbrother force-pushed the feat/sessions/1-client-scaffolding branch from 5717a3f to 0450365 Compare June 25, 2024 11:39

geekbrother had a problem deploying to prod June 25, 2024 11:40 — with GitHub Actions Failure

geekbrother had a problem deploying to staging June 25, 2024 11:40 — with GitHub Actions Failure

geekbrother had a problem deploying to prod June 25, 2024 12:05 — with GitHub Actions Failure

geekbrother temporarily deployed to staging June 25, 2024 12:05 — with GitHub Actions Inactive

geekbrother temporarily deployed to prod June 25, 2024 12:10 — with GitHub Actions Inactive

geekbrother temporarily deployed to staging June 25, 2024 12:10 — with GitHub Actions Inactive

geekbrother temporarily deployed to prod June 25, 2024 12:41 — with GitHub Actions Inactive

geekbrother temporarily deployed to staging June 25, 2024 12:41 — with GitHub Actions Inactive

geekbrother mentioned this pull request Jun 25, 2024

feat(irn): using the peer_id instead of generating it. #684

Open

geekbrother requested review from heilhead, nopestack, xDarksome and chris13524 June 25, 2024 12:46

geekbrother marked this pull request as ready for review June 25, 2024 12:47

geekbrother temporarily deployed to prod June 25, 2024 12:51 — with GitHub Actions Inactive

geekbrother temporarily deployed to staging June 25, 2024 12:51 — with GitHub Actions Inactive

xDarksome reviewed Jun 25, 2024

View reviewed changes

geekbrother requested a review from xDarksome June 25, 2024 13:30

fix: decreasing ttl to 30 days

30c2649

geekbrother had a problem deploying to prod June 25, 2024 13:44 — with GitHub Actions Error

geekbrother had a problem deploying to staging June 25, 2024 13:44 — with GitHub Actions Error

geekbrother temporarily deployed to prod June 25, 2024 13:45 — with GitHub Actions Inactive

geekbrother temporarily deployed to staging June 25, 2024 13:45 — with GitHub Actions Inactive

xDarksome reviewed Jun 25, 2024

View reviewed changes

fix: irn config structure name to convention

87fb41f

geekbrother temporarily deployed to prod June 25, 2024 13:54 — with GitHub Actions Inactive

geekbrother temporarily deployed to staging June 25, 2024 13:54 — with GitHub Actions Inactive

xDarksome reviewed Jun 25, 2024

View reviewed changes

xDarksome approved these changes Jun 25, 2024

View reviewed changes

geekbrother merged commit f5626e8 into master Jun 25, 2024
16 checks passed

geekbrother deleted the feat/sessions/1-client-scaffolding branch June 25, 2024 20:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sessions): irn client scaffolding #681

feat(sessions): irn client scaffolding #681

geekbrother commented Jun 21, 2024 •

edited

Loading

chris13524 Jun 25, 2024 •

edited

Loading

geekbrother Jun 25, 2024

chris13524 Jun 25, 2024

geekbrother Jun 25, 2024

geekbrother Jun 25, 2024

heilhead Jun 25, 2024

geekbrother Jun 25, 2024

xDarksome Jun 25, 2024

geekbrother Jun 25, 2024

xDarksome commented Jun 25, 2024 •

edited

Loading

geekbrother commented Jun 25, 2024

xDarksome Jun 25, 2024

geekbrother Jun 25, 2024

xDarksome Jun 25, 2024 •

edited

Loading

feat(sessions): irn client scaffolding #681

feat(sessions): irn client scaffolding #681

Conversation

geekbrother commented Jun 21, 2024 • edited Loading

Description

How Has This Been Tested?

Due Diligence

chris13524 Jun 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xDarksome commented Jun 25, 2024 • edited Loading

geekbrother commented Jun 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xDarksome Jun 25, 2024 • edited Loading

Choose a reason for hiding this comment

geekbrother commented Jun 21, 2024 •

edited

Loading

chris13524 Jun 25, 2024 •

edited

Loading

xDarksome commented Jun 25, 2024 •

edited

Loading

xDarksome Jun 25, 2024 •

edited

Loading