switch out `grpc` for `@grpc/grpc-js` + ts types #423

antonilol · 2022-09-08T14:26:06Z

fixes #295
fixes #425

i managed to get @grpc/grpc-js working with camel_case fields, ~~but still some tests are failing~~

also added generated typescript types for the grpc calls

all files in src/grpc/types/ are generated with the ./grpc_gen_types.sh bash script, which also runs in gh workflows after updating the proto files

antonilol · 2022-09-08T15:10:14Z

what does failed to getContactAndCheckKeyExchange actually mean? i see it in most of the failed tests. a more descriptive error would help

Evanfeenstra · 2022-09-13T17:01:08Z

When a contact is added, you send a keysend with your contact_key (end-to-end encryption key)... then they send a keysend back with their contact_key

antonilol · 2022-09-13T17:28:46Z

When a contact is added, you send a keysend with your contact_key (end-to-end encryption key)... then they send a keysend back with their contact_key

ok got it, i think it didnt got a key back from the other party then...

Evanfeenstra · 2022-09-14T20:29:04Z

@antonilol #425

antonilol · 2022-09-15T08:20:42Z

@Evanfeenstra can you fix package-lock.json on master? it contains merge conflict markers...

Evanfeenstra · 2022-09-15T17:45:13Z

@Evanfeenstra can you fix package-lock.json on master? it contains merge conflict markers...

ok @antonilol its fixed

Evanfeenstra · 2022-09-15T18:44:48Z

https://github.com/stakwork/sphinx-relay/pull/423/files#diff-acc59c19f30ca8e2dd83cb49828c32894fce27500a1fb8bdc6320d148cb4257bL68

Does that need to be return await loadProxyLightning(ownerPubkey) @antonilol ?

antonilol · 2022-09-15T18:47:46Z

lets try

Evanfeenstra · 2022-09-15T18:49:34Z

lets try

that will not fix the errors tho, the stack tests are not build around proxy. I would recommend cloning sphinx-stack and building sphinx-relay locally, then using your local image in sphinx-stack. Then you can see all the logs and what is going on (with Docker Desktop is easiest) @antonilol

antonilol · 2022-09-15T18:56:26Z

lets try

that will not fix the errors tho, the stack tests are not build around proxy. I would recommend cloning sphinx-stack and building sphinx-relay locally, then using your local image in sphinx-stack. Then you can see all the logs and what is going on (with Docker Desktop is easiest) @antonilol

yeah i wanted to do this but stack comes with all types of random errors, like the one i described here stakwork/sphinx-stack#45 (i managed to fix this by running docker-compose up -d en then docker-compose down, and after that the no alice config seemed to work)
now i am getting an endless spam of => try http://host.docker.internal:3001/is_setup in docker logs -f sphinx-stack-relaysetup-1 (i believe port 3001 is from alice.sphinx but there is no alice...)
and bob and carol report this:

(node:1) DeprecationWarning: grpc.load: Use the @grpc/proto-loader module with grpc.loadPackageDefinition instead
[info] 09-15-22T18:49:53 [MISC] Node listening on 3002.
=> initGrpcSubscriptions error Error: 12 UNIMPLEMENTED: unknown service lnrpc.Lightning
    at Object.exports.createStatusError (/relay/node_modules/grpc/src/common.js:91:15)
    at Object.onReceiveStatus (/relay/node_modules/grpc/src/client_interceptors.js:1209:28)
    at InterceptingListener._callNext (/relay/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/relay/node_modules/grpc/src/client_interceptors.js:618:8)
    at callback (/relay/node_modules/grpc/src/client_interceptors.js:847:24) {
  code: 12,
  metadata: Metadata { _internal_repr: {}, flags: 0 },
  details: 'unknown service lnrpc.Lightning'
}

antonilol · 2022-09-15T19:23:35Z

i got stack working by running just docker-compose up -d, waiting till everything is running and then stopping alice.sphinx, then running sphinx relay and at least i got one test to pass on my machine (queryroutes)

antonilol · 2022-09-15T20:10:34Z

i got some more tests working now locally with only the bare minimum to get grpcjs working, without some suggested type checks

antonilol · 2022-09-15T22:16:10Z

i got all tests passing locally!!!
i squashed all commits and now waiting for the gh ci tests to pass

Evanfeenstra · 2022-09-15T22:41:27Z

i got all tests passing locally!!! i squashed all commits and now waiting for the gh ci tests to pass

what was the issue that was stopping key exchange?

antonilol · 2022-09-15T22:48:34Z

i got all tests passing locally!!! i squashed all commits and now waiting for the gh ci tests to pass

what was the issue that was stopping key exchange?

there were 3 issues:

keysend didnt work because grpcjs thought a bytebuffer always had a length of 0, i replaced it with a nodejs native buffer
map<uint64, something> in proto will cause grpcjs to hash the uint64 before giving it to the client, resulting in the keys of the dest_custom_records of an InvoiceHTLC being hashed, then no key 133773310 existed anymore (subscribeInvoices line 27)
lightning[cmd] did not have any arguments (the first one is undefined then and it gave an error). (subscribeInvoices line 24)

Evanfeenstra · 2022-09-26T15:52:46Z

@antonilol so this file https://github.com/stakwork/sphinx-relay/blob/97ea3a3a34e42d5102c9396760ba706a0206d176/protobuf_long.patch is needed in order to make the map<uint64, something> work? Do we need to do anything specific to make it work? I'm not familiar with npm patches

kevkevinpal · 2022-10-18T20:05:32Z

Hey @antonilol were you able to manually test this change I pulled in the changes and I was only able to send out messages, receive wasn't working

do you know if this would fail if you're running nodejs 12 or is it required it be updated?

antonilol · 2022-10-18T20:46:23Z

Hey @antonilol were you able to manually test this change I pulled in the changes and I was only able to send out messages, receive wasn't working

do you know if this would fail if you're running nodejs 12 or is it required it be updated?

i use node v16, and it works. with node v12.22.12 i cant npm install because this protobuf patch doesnt apply. if i just start it on node v12 i get an error that mcrypt was compiled for a newer version of nodejs

antonilol · 2022-10-18T21:22:25Z

rebased

kevkevinpal · 2022-10-18T22:27:42Z

Ok was able to get this running on nodejs 16 and it seems to be working fine with manual testing, just need to remember to update things from node 12 to 16 now

kevkevinpal

Looks like linter is failing but otherwise I am good with these changes

antonilol · 2022-10-19T14:39:39Z

Looks like linter is failing

this is fixed in #445, eslint misses some files and previous changes added warnings, can you merge it before this one?

kevkevinpal · 2022-10-19T20:23:26Z

Looks like linter is failing

this is fixed in #445, eslint misses some files and previous changes added warnings, can you merge it before this one?

merged

antonilol · 2022-10-20T10:13:58Z

rebased, got the warning count right and lint check passed now

antonilol · 2022-10-20T12:00:02Z

all tests passed except one port in use error (stakwork/sphinx-stack#47)

kevkevinpal · 2022-10-21T03:35:20Z

all tests passed except one port in use error (stakwork/sphinx-stack#47)

test are passing now 🥳

antonilol · 2022-11-01T22:50:11Z

rebased and ready to merge. did you manage to upgrade the lite nodes to nodejs v16?

kevkevinpal · 2022-12-09T16:38:52Z

rebased and ready to merge. did you manage to upgrade the lite nodes to nodejs v16?

Sorry seeing this now, I'm trying to upgrade the nodes on our end just had to add more storage since the clusters we're getting full after trying to reinstall the node_modules. I have a script to auto update it just might take a long time update, I'm getting stuck on this node_modules when installing [##################] / reify:macaroon: timing reifyNode:node_modules/typescript not sure if you know why this in one in particular takes so long

antonilol · 2022-12-09T17:12:16Z

I'm getting stuck on this node_modules when installing [##################] / reify:macaroon: timing reifyNode:node_modules/typescript not sure if you know why this in one in particular takes so long

On my devices it also takes more time than other things done during npm install, on my bitcoin node it was significantly slower than on my pc and laptop, i think it is some cpu bound task, like compiling

antonilol · 2022-12-28T21:42:51Z

rebased this and #362

kevkevinpal · 2022-12-29T16:21:15Z

rebased this and #362

thanks for rebasing I'm still in the process updating the clusters just ran into some issues and it takes a while to update each one so I'm doing them at night

Thanks for your patience

notes: * grpcjs doesnt like ByteBuffer, it has been replaced with the nodejs native Buffer * grpcjs doesnt like calls without an argument, add {} instead see src/grpc/subscribe.ts:24 * grpcjs converts uint64 in a map key (map<uint64, ...>) to some hash, this is fixed with protobuf_long.patch which is automatically applied after running `npm install` this pr would fix the bug upstream protobufjs/protobuf.js#1669

antonilol force-pushed the grpcjs branch from af7af42 to d0ca7b2 Compare September 15, 2022 08:18

antonilol changed the title ~~switch out grpc for @grpc/grpc-js + generated ts types~~ switch out grpc for @grpc/grpc-js ~+ generated ts types~ Sep 15, 2022

antonilol changed the title ~~switch out grpc for @grpc/grpc-js ~+ generated ts types~~~ switch out grpc for @grpc/grpc-js + ~generated ts types~ Sep 15, 2022

antonilol changed the title ~~switch out grpc for @grpc/grpc-js + ~generated ts types~~~ switch out grpc for @grpc/grpc-js Sep 15, 2022

antonilol force-pushed the grpcjs branch from d0ca7b2 to d91cc3f Compare September 15, 2022 18:08

antonilol force-pushed the grpcjs branch 2 times, most recently from 4399888 to d6aeedd Compare September 15, 2022 20:06

antonilol force-pushed the grpcjs branch from 7b370fb to 6460369 Compare September 15, 2022 22:14

antonilol force-pushed the grpcjs branch from 6460369 to b68fdcc Compare September 15, 2022 22:17

antonilol changed the title ~~switch out grpc for @grpc/grpc-js~~ switch out grpc for @grpc/grpc-js + ts types Sep 15, 2022

antonilol marked this pull request as ready for review September 15, 2022 22:53

antonilol force-pushed the grpcjs branch 2 times, most recently from 79d85b6 to 97ea3a3 Compare September 16, 2022 14:21

antonilol mentioned this pull request Sep 16, 2022

run prettier in github actions #435

Merged

antonilol force-pushed the grpcjs branch 2 times, most recently from 71c7dd8 to 691cd60 Compare October 4, 2022 21:14

antonilol force-pushed the grpcjs branch from 691cd60 to 8bd8f38 Compare October 18, 2022 21:22

kevkevinpal approved these changes Oct 19, 2022

View reviewed changes

antonilol force-pushed the grpcjs branch 2 times, most recently from 791cae2 to bce6caa Compare October 20, 2022 10:11

antonilol force-pushed the grpcjs branch from bce6caa to 36c4244 Compare November 1, 2022 22:49

antonilol force-pushed the grpcjs branch from 36c4244 to eb39acf Compare November 6, 2022 16:23

antonilol force-pushed the grpcjs branch 2 times, most recently from 2ecd889 to 09e8dba Compare December 28, 2022 21:23

antonilol force-pushed the grpcjs branch from 09e8dba to 5604109 Compare February 5, 2023 15:51

antonilol force-pushed the grpcjs branch from 5604109 to a8658b8 Compare February 17, 2023 20:38

retry tests

b11d1c4

Evanfeenstra merged commit 3b35d23 into stakwork:master Feb 18, 2023

antonilol deleted the grpcjs branch April 6, 2023 15:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

switch out `grpc` for `@grpc/grpc-js` + ts types #423

switch out `grpc` for `@grpc/grpc-js` + ts types #423

antonilol commented Sep 8, 2022 •

edited

Loading

antonilol commented Sep 8, 2022

Evanfeenstra commented Sep 13, 2022

antonilol commented Sep 13, 2022

Evanfeenstra commented Sep 14, 2022

antonilol commented Sep 15, 2022

Evanfeenstra commented Sep 15, 2022

Evanfeenstra commented Sep 15, 2022

antonilol commented Sep 15, 2022

Evanfeenstra commented Sep 15, 2022

antonilol commented Sep 15, 2022

antonilol commented Sep 15, 2022

antonilol commented Sep 15, 2022 •

edited

Loading

antonilol commented Sep 15, 2022

Evanfeenstra commented Sep 15, 2022

antonilol commented Sep 15, 2022 •

edited

Loading

Evanfeenstra commented Sep 26, 2022

kevkevinpal commented Oct 18, 2022

antonilol commented Oct 18, 2022

antonilol commented Oct 18, 2022

kevkevinpal commented Oct 18, 2022

kevkevinpal left a comment

antonilol commented Oct 19, 2022 •

edited

Loading

kevkevinpal commented Oct 19, 2022

antonilol commented Oct 20, 2022

antonilol commented Oct 20, 2022

kevkevinpal commented Oct 21, 2022

antonilol commented Nov 1, 2022

kevkevinpal commented Dec 9, 2022

antonilol commented Dec 9, 2022

antonilol commented Dec 28, 2022

kevkevinpal commented Dec 29, 2022

switch out grpc for @grpc/grpc-js + ts types #423

switch out grpc for @grpc/grpc-js + ts types #423

Conversation

antonilol commented Sep 8, 2022 • edited Loading

antonilol commented Sep 8, 2022

Evanfeenstra commented Sep 13, 2022

antonilol commented Sep 13, 2022

Evanfeenstra commented Sep 14, 2022

antonilol commented Sep 15, 2022

Evanfeenstra commented Sep 15, 2022

Evanfeenstra commented Sep 15, 2022

antonilol commented Sep 15, 2022

Evanfeenstra commented Sep 15, 2022

antonilol commented Sep 15, 2022

antonilol commented Sep 15, 2022

antonilol commented Sep 15, 2022 • edited Loading

antonilol commented Sep 15, 2022

Evanfeenstra commented Sep 15, 2022

antonilol commented Sep 15, 2022 • edited Loading

Evanfeenstra commented Sep 26, 2022

kevkevinpal commented Oct 18, 2022

antonilol commented Oct 18, 2022

antonilol commented Oct 18, 2022

kevkevinpal commented Oct 18, 2022

kevkevinpal left a comment

Choose a reason for hiding this comment

antonilol commented Oct 19, 2022 • edited Loading

kevkevinpal commented Oct 19, 2022

antonilol commented Oct 20, 2022

antonilol commented Oct 20, 2022

kevkevinpal commented Oct 21, 2022

antonilol commented Nov 1, 2022

kevkevinpal commented Dec 9, 2022

antonilol commented Dec 9, 2022

antonilol commented Dec 28, 2022

kevkevinpal commented Dec 29, 2022

switch out `grpc` for `@grpc/grpc-js` + ts types #423

switch out `grpc` for `@grpc/grpc-js` + ts types #423

antonilol commented Sep 8, 2022 •

edited

Loading

antonilol commented Sep 15, 2022 •

edited

Loading

antonilol commented Sep 15, 2022 •

edited

Loading

antonilol commented Oct 19, 2022 •

edited

Loading