Slow API calls

[DeanDonkov]

We, a customer of Breez LSP, are having issues on the following endpoints:

• nodeInfo
• receiveOnChain
• onchainPaymentLimits()

Those endpoints result in Breez SDK connection taking upwards of 30 seconds. After communication with their team, it seems the issue might be in Greenlight's nodes.

LOG initializeBreezSDK: 38105ms

[cdecker]

Not quite sure where the slow.startuo in the SDK comes.from. generally speaking using gl-client directly returns getinfo calls in <es. The SDK does a lot of caching, maybe the busy startup sync is delaying user issued commands?

[DeanDonkov]

@cdecker Are you aware of other LSPs using your product? Can you point us to alternatives to Breez LSP? Thanks.

[roeierez]

@DeanDonkov it seems very weird nideInfo (which only use cached data) to take mote than a few milliseconds. Can you share logs of the call so we can drill down here?

[martinivv]

Hi, @roeierez! We're encountering persistent issues with node synchronization and method latency throughout most of the day. What began as sporadic issues 2-3 weeks ago has now escalated into more frequent and disruptive occurrences.

---

I. Preconditions:

1. Breez SDK Version: 0.5.2;
2. Internet Speed: not a limiting factor in this case.

 

II. Issue Details:

1. SDK Initialization Delays:
On the first initialization of the SDK, the connection process is significantly delayed and often results in errors.

Error Examples:

[Error: Generic: status: Unavailable, nic:: transport:: Error (Transport, hyper:: Error (Connect, Custom { kind: UnexpectedEof, error: message: were trying to connect t handsheede metadata: Metadata "tls handshake eof" ...]

LOG:: initializeBreezSDK: 300_484ms

[Error: Generic: status: Internal, message: "Too many open files (os error 24)", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Fri, 09 Aug 2024 09:23:56 GMT", "content-length": "0"} }, source: None]

 

2. Excessive Method Latency:
Some methods, particularly those related to the “node state”, are taking an excessive amount of time to complete.

Example:
The receiveOnChain method within the Breez SDK sometimes takes 3-4 minutes or even longer to return a response.

[cdecker]

Please do not ping developers willy nilly, it is rather disruptive, and in this case the Greenlight dev team can't even help as this appears to be entirely handled inside the Breez SDK.

[cdecker]

The Too many open files is indicative of #494 by the way, and may appear spuriously until we fix colliding state updates.