Blob verifications fails for a cache

[Inkvi]

After a blob is successfully dispersed and saved to s3 cache, op-node tries to get it but cache retrieval fails due to a verification error

| t=2024-10-04T01:17:47+0000 lvl=info msg=request role=eigenda_proxy method=POST url=/put/                                                                                                │
│ t=2024-10-04T01:17:47+0000 lvl=info msg="Attempting to disperse blob to EigenDA" role=eigenda_proxy                                                                                     │
│ t=2024-10-04T01:17:47+0000 lvl=info msg="Blob dispersed to EigenDA, now waiting for confirmation" role=eigenda_proxy requestID=ZjZiYzliODVhMmM5ZDIzOWY5NjU2NDJlNjdhMTgxNzUzMmZkNWFjNTI1 │
│ MmZkOWRhNWQ1MWUyM2QwYTMzODVlYS0zMTM3MzIzODMwMzAzNDM2MzYzNzM1MzYzMTMxMzEzNDM3MzczNjJmMzAyZjMzMzMyZjMxMmYzMzMzMmZlM2IwYzQ0Mjk4ZmMxYzE0OWFmYmY0Yzg5OTZmYjkyNDI3YWU0MWU0NjQ5YjkzNGNhNDk1OTk │
│ xYjc4NTJiODU1                                                                                                                                                                           │
│ t=2024-10-04T01:18:17+0000 lvl=info msg="Blob submitted, waiting for dispersal from EigenDA" role=eigenda_proxy requestID=ZjZiYzliODVhMmM5ZDIzOWY5NjU2NDJlNjdhMTgxNzUzMmZkNWFjNTI1MmZkO │
│ WRhNWQ1MWUyM2QwYTMzODVlYS0zMTM3MzIzODMwMzAzNDM2MzYzNzM1MzYzMTMxMzEzNDM3MzczNjJmMzAyZjMzMzMyZjMxMmYzMzMzMmZlM2IwYzQ0Mjk4ZmMxYzE0OWFmYmY0Yzg5OTZmYjkyNDI3YWU0MWU0NjQ5YjkzNGNhNDk1OTkxYjc4 │
│ NTJiODU1                                                                                                                                                                                │
│ t=2024-10-04T01:18:34+0000 lvl=info msg=request role=eigenda_proxy method=GET url=/get/0x010000f901f4f850f842a013281d650fd20267e886a54b22d67f984b7b012a7f9eadb11643bbcc46ae2d61a0143bfa │
│ be5cfed35b2fcfad1de87f823a5fb31cbc6ec71eaee725be55a4de762808cac480213701c401213701f9019f82a0f738f873eba03c94ac6c65ac7278fdf88621b721be0b2e660543ff1c251936d03d9381b571e3820001826464832 │
│ 59366a0d60abdfa10a64d7f37ef563ddaf1560fb0c89f33c87755d9ac2f48211e5671fb00832593c4a0c63598cdfaae30ad331fe3054a4989c4a7b2598c4e4bc1a143828f26bb03abeeb90120232fd9f9e5f845086256a7c2c51e78 │
│ 76de855dbacd8002d32de576fa3b869e2a5b5e43a570bcea696d86a12514658d72f8f3cde6505616657b8d2df046830f8b93cf5092e1df13f0c3b47f81b4e6a722e74986937aca3f3f85c7c3d1c087d3bfe03f7da7d2800ae37b970 │
│ 1fc2856f3d80f576d9ac221c9c970caea42fb80d5cf9564bf0ed4d25fd06097d45703b971afc8629c606bb1041cdabb6d785cec7bf487dc992405dbeae1f9318b53202db659e65ee5b8b5330e23087b4176a24fef010cb50e3cd2a6 │
│ f8c4b387eb37ffbc8203dade7f8ae3ec7c8fb6453363eca6f5e3c061d70f874e538ad9458f77cd62ad84cadc4bf81dac186371658b67896d8ef1759d68832f377fbcc2c8649ac04f6020137e7e5bef8234880e6ad5b3a204a437820 │
│ 001                                                                                                                                                                                     │
│ t=2024-10-04T01:18:34+0000 lvl=warn msg="Failed to verify blob" role=eigenda_proxy err="field elements do not match, x actual commit: 0ae93eb4b4e2c32919318a4d5a625c6f7f9f567da57904ba8 │
│ 598b47e0801ebb9, x expected commit: 13281d650fd20267e886a54b22d67f984b7b012a7f9eadb11643bbcc46ae2d61, y actual commit: 02b680752a311733d45da555a7c73c0b4c738b756a79a3af1e0f08f18a0cb053 │
│ , y expected commit: 143bfabe5cfed35b2fcfad1de87f823a5fb31cbc6ec71eaee725be55a4de7628" backend=S3                                                                                       │
│ t=2024-10-04T01:18:34+0000 lvl=warn msg="Failed to read from cache targets" role=eigenda_proxy err="no data found in any redundant backend"

I tried to use Redis instead of S3 and got the same problem. It exists for fallback as well since they share athe same codepath to retrieve the data.

ghcr.io/layr-labs/eigenda-proxy:v1.4.1 is used.

[Inkvi]

I believe it's caused by #167 since I use Google Storage too.

[epociask]

@Inkvi thank you for filing this issue. Have you verified that the issue is due to blob contents being written to redis differing from the ones written to EigenDA? In the case of #167, there was signature metadata wrapped around the initial blob contents that the aws client couldn't reason about since it was unique to gcp

[Inkvi]

@Inkvi thank you for filing this issue. Have you verified that the issue is due to blob contents being written to redis differing from the ones written to EigenDA? In the case of #167, there was signature metadata wrapped around the initial blob contents that the aws client couldn't reason about since it was unique to gcp

I haven't had a chance to verify this but based on the code paths for uploading data to EigenDA and to S3/Redis, they shouldn't differ.

Do you know an easy way to download an EigenDA blob for a testnet without writing code?

[Inkvi]

An update from my side. I tested the latest commit with Redis and there is no issue with verification. The verification only fails for S3 cache when Google Storage is used.

[jcortejoso]

@Inkvi One easy way to check for GCP encoding issues is to download a blob that fails from GCS and inspect the first ~20 bytes of the file to see if there is a chunk-signature= string.

[Inkvi]

@Inkvi One easy way to check for GCP encoding issues is to download a blob that fails from GCS and inspect the first ~20 bytes of the file to see if there is a chunk-signature= string.

Did exactly that and chunk-sugnature is there.

6c;chunk-signature=50d1653b4ceb9845d388dd0c4b270e17dea1e9fc2c15b64547aedd544ac51baa
�Ä∆<y∏lí�¯ç∂â◊ëG������Tx⁄⁄qä±≠Éi¡ÚÖÃØCÚü-9qáœ#˝�fl˘€ÀŸmÏü�º∂H7$ƒµFB˛¬ö◊y��ƒ˚ø?kgdapd��#�Ö��Ö&�Ü¡�����ˇˇ}#�t�
0;chunk-signature=c8ba154eb017d442d2e755a2bf7c9cb13e572eeee7e24be9b50709514f3cd2ac

[Inkvi]

I cherry-picked #167 in my fork and it didn't fix the issue for me. chunk-signature is still present in the uploaded data. I will try again whenever you merge it into main.

[epociask]

@Inkvi #167 is a fix for S3 backend type but not redis which uses the go-redis client.

[epociask]

Hi @Inkvi - is this still something affecting you?

[Inkvi]

It was still affecting us the last time I checked but I am out till the end of October so can't confirm the most recent changes before that

…

On Wed, Oct 16, 2024, 8:09 PM Ethen ***@***.***> wrote: Hi @Inkvi <https://github.com/Inkvi> - is this still something affecting you? — Reply to this email directly, view it on GitHub <#166 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC3LO3GXV5PRE7IVXVMZR3Z32TVBAVCNFSM6AAAAABPK5AEM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJXGU3DGMJRHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>