Peers won't discover neighbors topics subscriptions when relayed

[d-roak]

We are currently having an issue where peers are not aware of their neighbors topics when they use a relay. Everything works perfectly when both peers are in the same network (either localhost or LAN), we assume it would also work if both peers weren't behind NATs and had a direct connection (we didn't tested this).

Things we tested:

• Both peers create a connection between them (circuit-relay multiaddress)
• Both peers exchange the identify initial message and fetch the relevant meshsub in the available protocols

Things to check:

• Base pubsub abstract class exchanging topics (not being extended by chainsafe's gossipsub impl)
  https://github.com/libp2p/js-libp2p/blob/0b55625d146940994a306101650a55ee58e32f6c/packages/pubsub/src/index.ts#L253

• Chainsafe's gossipsub impl on connected workflow
  https://github.com/ChainSafe/js-libp2p-gossipsub/blob/bd5460c6c0fba63b27d8671b438cb0495d232773/src/index.ts#L770

Things to test around:

• Check libp2p's webrtc guide for clues (https://docs.libp2p.io/guides/getting-started/webrtc/)
• Check the universal connectivity repo if it has the same problems (https://github.com/libp2p/universal-connectivity/)

[achingbrain]

By default the Circuit Relay protocol only allows a small amount of data to be transferred over a relayed connection - if the limit is breached the relay will close the connection - this is to ensure relays can be run to help other nodes create direct connections but not be an open relay which is ripe for abuse.

By default there are only a few protocols that will run over relayed connections - Identify is one of them but Gossipsub is not.

Your peers should use the relay to establish a direct connection, over which Gossipsub should run as expected.

[d-roak]

@achingbrain thanks for helping out! It seems that enabling runOnTransientConnection on both gossipsub configs and stream options fixed the issues of no communication at all.
With what u said, I am afraid we will still have limitations regarding the amount of data transmitted on those connections.

[achingbrain]

runOnTransientConnection should be left as the default value of false, otherwise you're likely to have your connection closed unexpectedly by the relay once the data/time limits are reached.

Since it works on a LAN and it works via a relay it sounds like the two nodes are failing to make a direct connection.

Have you configured any STUN servers?

If not can you please try updating the config of the two NATed nodes to something like:

const node = await createLibp2p({
  //... other config
  transports: [
    //... other transports
    webRTC({
      rtcConfiguration: {
        iceServers: [{
          urls: ['stun:stun.l.google.com:19302']
        }, {
          urls: ['stun:global.stun.twilio.com:3478']
        }]
      }
    })
  ]
})

There will be something like this as default config in the next release so it shouldn't be necessary forever.

[d-roak]

Yep. The initial tests we made didn't have the STUN servers configured and the ones that followed had both STUN and relaying enabled, so the results were unclear.

Currently, it's working, we still need to debug some scenarios where there is a connection, the messaging is working, but the handler is failing (high probability is on our side).

We still found some edge cases where STUN is insufficient to make a direct connection (w/ VPNs in the mix). This is out-of-scope for now.

Thank you so much for taking the time to help out fixing our issues!

[achingbrain]

Great stuff, glad you got it sorted out.

There will be something like this as default config in the next release so it shouldn't be necessary forever.

This shipped in @libp2p/[email protected] yesterday so you shouldn't need to specify STUN servers manually any more.

Also if you like, please feel free to PR your logo into the js-libp2p README - https://github.com/libp2p/js-libp2p#used-by