Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement and use larger bao block/chunk group size #741

Closed
rklaehn opened this issue Jan 27, 2023 · 1 comment
Closed

Implement and use larger bao block/chunk group size #741

rklaehn opened this issue Jan 27, 2023 · 1 comment
Assignees

Comments

@rklaehn
Copy link
Contributor

rklaehn commented Jan 27, 2023

The current bao block size is = the chunk size = 1024 bytes.

That the size of the outboard will be 2 32 byte hashes per chunk, so 64 bytes per chunk, so 1/16 of the data size.

Unfortunately that means that it is not possible to keep the outboard in memory when sending large files from small devices. E.g. sending a 32 gb disk image would require a 2gb outboard, which is a bit much for e.g. a raspberry pi with 4 and impossible for a rpi with 1 gb of ram.

One option would be to persist the outboards on disk, but that is a can of worms. Also, 1024 byte chunks are less than optimal also for transfer speed.

The easiest thing to do would be to have a larger block size. E.g. 64k -> outboard = 1/1024 of data => no problem for the above use case.

There is an issue on the bao repo about larger chunk groups. Either get this merged (preferably), or fork bao and add it ourselves...

oconnor663/bao#34

@rklaehn rklaehn self-assigned this Jan 27, 2023
@dignifiedquire dignifiedquire transferred this issue from n0-computer/sendme-legacy Feb 20, 2023
@ramfox ramfox added this to iroh Feb 21, 2023
@github-project-automation github-project-automation bot moved this to 📋 Backlog - unassigned issues in iroh Feb 21, 2023
@ramfox ramfox moved this from 📋 Backlog - unassigned issues to 🟢 Ready - assigned issues in iroh Feb 21, 2023
@rklaehn
Copy link
Contributor Author

rklaehn commented Mar 29, 2023

We are currently using a chunk group log of 4 (chunk group size 16kb) via the abao crate. We will continue to do so in the bao-tree crate #894 where this is runtime adjustable.

@rklaehn rklaehn closed this as completed Mar 29, 2023
@github-project-automation github-project-automation bot moved this from 🔖 Ready - assigned issues to ✅ Done in iroh Mar 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

1 participant