Improve json decoding #1402
Merged
Improve json decoding #1402
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cmaglie
force-pushed
the
improve_json_decoding
branch
3 times, most recently
from
21db43c to
ee5eeca
Compare
per1234
added
type: enhancement
cmaglie
force-pushed
the
improve_json_decoding
branch
from
ee5eeca to
470e2b9
Compare
cmaglie
force-pushed
the
improve_json_decoding
branch
from
470e2b9 to
fdfb53b
Compare
Results: $ go test -v -benchmem -bench BenchmarkIndexParsing github.com/arduino/arduino-cli/arduino/libraries/librariesindex === RUN TestIndexer --- PASS: TestIndexer (0.16s) goos: linux goarch: amd64 pkg: github.com/arduino/arduino-cli/arduino/libraries/librariesindex cpu: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz BenchmarkIndexParsingStdJSON BenchmarkIndexParsingStdJSON-8 5 214872730 ns/op 94.52 MB/s 58956539 B/op 418973 allocs/op BenchmarkIndexParsingEasyJSON BenchmarkIndexParsingEasyJSON-8 16 69215472 ns/op 293.42 MB/s 56162664 B/op 418966 allocs/op PASS ok github.com/arduino/arduino-cli/arduino/libraries/librariesindex 4.442s easyjson is 3x faster.
cmaglie
force-pushed
the
improve_json_decoding
branch
from
fdfb53b to
878168e
Compare
per1234
requested changes
May 4, 2022
There was a problem hiding this comment.
Would it be possible to set up some infrastructure for the code generation?
- Task that runs the generation command
- GitHub Actions workflow that runs the task and then checks for a diff (which would indicate that the generated code has gone out of sync)
I did that for the generated code in Arduino Lint and found it to be very useful. The benefits should be even more here where there are more contributors.
Probably should also be documented in the contributor guide:
https://github.com/arduino/arduino-cli/blob/master/docs/CONTRIBUTING.md
ubidefeo
approved these changes
May 5, 2022
cmaglie
force-pushed
the
improve_json_decoding
branch
2 times, most recently
from
f4d814a to
ab68a8e
Compare
cmaglie
force-pushed
the
improve_json_decoding
branch
from
ab68a8e to
7f393b1
Compare
per1234
requested changes
May 5, 2022
Co-authored-by: per1234 <[email protected]>
per1234
approved these changes
May 5, 2022
umbynos
approved these changes
May 5, 2022
|
For the record: we are using a patched version of |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I found that the standard golang JSON decoder is very slow. This is particularly noticeable because the library_index.json is constantly growing (currently we are at ~20Mb) and the parsing time adds up at every invocation of the cli.
This PR:
easyjsonlibrary for parsing the library_index.jsonHere the result: