Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: ingest csaf documents in a single transaction #95

Merged
merged 1 commit into from
Mar 19, 2024
Merged

fix: ingest csaf documents in a single transaction #95

merged 1 commit into from
Mar 19, 2024

Conversation

ctron
Copy link
Contributor

@ctron ctron commented Mar 19, 2024

It not only seems to make sense to import a single document in a single transaction, but it also does have a performance impact:

Before

[2024-03-19T13:40:59.775Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-24537 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-24537.json: took 25s 575ms 8us 931ns
[2024-03-19T13:41:02.789Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-2650 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-2650.json: took 2s 410ms 886us 980ns
[2024-03-19T13:41:03.273Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-2157 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-2157.json: took 22ms 169us 612ns
[2024-03-19T13:41:03.847Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-3128 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-3128.json: took 148ms 144us 231ns
[2024-03-19T13:41:06.610Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-1981 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-1981.json: took 2s 254ms 190us 902ns
[2024-03-19T13:41:07.802Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-29404 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-29404.json: took 553ms 623us 716ns
[2024-03-19T13:41:08.808Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-29405 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-29405.json: took 422ms 719us 626ns
[2024-03-19T13:41:09.616Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-29402 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-29402.json: took 382ms 116us 373ns
[2024-03-19T13:41:10.005Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-52477 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-52477.json: took 25ms 878us 241ns
[2024-03-19T13:41:11.705Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-50868 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-50868.json: took 1s 78ms 149us 482ns
[2024-03-19T13:41:13.010Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-50387 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-50387.json: took 835ms 77us 861ns
[2024-03-19T13:41:13.419Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-52605 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-52605.json: took 27ms 516us 351ns
[2024-03-19T13:41:31.461Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-40267 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-40267.json: took 17s 272ms 282us 730ns
[2024-03-19T13:41:31.883Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-4408 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-4408.json: took 29ms 559us 709ns
[2024-03-19T13:41:32.867Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-2283 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-2283.json: took 524ms 342us 123ns
[2024-03-19T13:41:50.068Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-24536 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-24536.json: took 16s 372ms 728us 689ns
[2024-03-19T13:41:59.196Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-29401 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-29401.json: took 8s 614ms 482us 701ns
[2024-03-19T13:42:59.373Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-3978 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-3978.json: took 58s 953ms 538us 691ns
[2024-03-19T13:42:59.734Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-36191 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-36191.json: took 23ms 205us 520ns

After

[2024-03-19T13:43:44.668Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-24537 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-24537.json: took 5s 465ms 620us 159ns
[2024-03-19T13:43:45.508Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-2650 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-2650.json: took 477ms 219us 545ns
[2024-03-19T13:43:45.848Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-2157 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-2157.json: took 15ms 577us 800ns
[2024-03-19T13:43:46.202Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-3128 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-3128.json: took 36ms 626us 40ns
[2024-03-19T13:43:47.537Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-1981 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-1981.json: took 807ms 279us 80ns
[2024-03-19T13:43:48.117Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-29404 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-29404.json: took 171ms 790us 438ns
[2024-03-19T13:43:48.620Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-29405 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-29405.json: took 176ms 268us 511ns
[2024-03-19T13:43:49.139Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-29402 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-29402.json: took 167ms 417us 216ns
[2024-03-19T13:43:49.462Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-52477 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-52477.json: took 10ms 373us 472ns
[2024-03-19T13:43:50.238Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-50868 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-50868.json: took 365ms 88us 934ns
[2024-03-19T13:43:51.034Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-50387 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-50387.json: took 405ms 572us 135ns
[2024-03-19T13:43:51.360Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-52605 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-52605.json: took 13ms 269us 216ns
[2024-03-19T13:43:55.568Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-40267 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-40267.json: took 3s 675ms 267us 27ns
[2024-03-19T13:43:55.884Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-4408 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-4408.json: took 8ms 45us 949ns
[2024-03-19T13:43:56.402Z INFO  trustify_ingestors::advisory::csaf] Ingested: CVE-2023-2283 from https://access.redhat.com/security/data/csaf/beta/vex/2023/cve-2023-2283.json: took 182ms 221us 882ns

@ctron ctron requested a review from bobmcwhirter March 19, 2024 13:47
@ctron ctron mentioned this pull request Mar 19, 2024
Merged
bobmcwhirter
bobmcwhirter approved these changes Mar 19, 2024
View reviewed changes
Copy link
Contributor

@bobmcwhirter bobmcwhirter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in a subsequent PR move out of graph?

@bobmcwhirter bobmcwhirter merged commit ad9196a into trustification:main Mar 19, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bobmcwhirter bobmcwhirter bobmcwhirter approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ctron @bobmcwhirter