enhance: record memory size (uncompressed) item for index #38770

chyezh · 2024-12-26T03:57:15Z

Current milvus use a serialized index size(compressed) for estimate resource for loading.
Add a new field MemSize (before compressing) for index to estimate resource.

sre-ci-robot · 2024-12-26T03:57:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign yanliang567 after the PR has been reviewed.
You can assign the PR to them by writing /assign @yanliang567 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2024-12-26T04:48:36Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T04:50:52Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2024-12-26T05:47:17Z

rerun cpp-unit-test

mergify · 2024-12-26T06:10:10Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T08:33:46Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-26T09:07:20Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T12:32:42Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T12:40:59Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2024-12-26T13:30:49Z

rerun cpp-unit-test

mergify · 2024-12-26T14:19:25Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

codecov · 2024-12-26T15:28:38Z

Codecov Report

Attention: Patch coverage is 71.79487% with 44 lines in your changes missing coverage. Please review.

Project coverage is 81.15%. Comparing base (a00ba86) to head (fd22729).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/core/src/index/IndexStats.cpp	74.35%	10 Missing ⚠️
internal/util/indexcgowrapper/index.go	37.50%	9 Missing and 1 partial ⚠️
internal/core/src/index/TextMatchIndex.cpp	0.00%	7 Missing ⚠️
internal/core/src/common/protobuf_utils.cpp	50.00%	6 Missing ⚠️
internal/core/src/indexbuilder/index_c.cpp	0.00%	6 Missing ⚠️
internal/core/src/index/StringIndexMarisa.cpp	0.00%	2 Missing ⚠️
internal/core/src/common/protobuf_utils.h	83.33%	1 Missing ⚠️
internal/core/src/index/ScalarIndexSort.cpp	0.00%	1 Missing ⚠️
internal/core/src/storage/DiskFileManagerImpl.cpp	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #38770       +/-   ##
===========================================
+ Coverage   69.66%   81.15%   +11.48%     
===========================================
  Files         296     1399     +1103     
  Lines       26676   197549   +170873     
===========================================
+ Hits        18585   160315   +141730     
- Misses       8091    31602    +23511     
- Partials        0     5632     +5632

Components	Coverage Δ
Client	`79.53% <ø> (∅)`
Core	`69.66% <63.44%> (-0.01%)`	⬇️
Go	`83.09% <84.12%> (∅)`

Files with missing lines	Coverage Δ
internal/core/src/index/BitmapIndex.cpp	`80.94% <100.00%> (-0.06%)`	⬇️
internal/core/src/index/BitmapIndex.h	`44.44% <ø> (ø)`
internal/core/src/index/HybridScalarIndex.cpp	`79.12% <100.00%> (-0.33%)`	⬇️
internal/core/src/index/HybridScalarIndex.h	`39.02% <ø> (ø)`
internal/core/src/index/Index.h	`100.00% <ø> (ø)`
internal/core/src/index/IndexStats.h	`100.00% <100.00%> (ø)`
internal/core/src/index/InvertedIndexTantivy.cpp	`92.47% <100.00%> (-0.26%)`	⬇️
internal/core/src/index/InvertedIndexTantivy.h	`53.84% <ø> (ø)`
internal/core/src/index/ScalarIndexSort.h	`38.46% <ø> (ø)`
internal/core/src/index/StringIndexMarisa.h	`71.42% <ø> (ø)`
... and 29 more

... and 1089 files with indirect coverage changes

chyezh · 2024-12-27T06:39:30Z

rerun cpp-unit-test

mergify · 2024-12-27T07:27:22Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-27T07:30:26Z

rerun cpp-unit-test

chyezh · 2024-12-27T07:30:45Z

rerun go-sdk

mergify · 2024-12-27T08:13:53Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-27T08:24:07Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2024-12-27T16:13:07Z

rerun go-sdk

chyezh · 2024-12-27T16:13:12Z

rerun cpp-unit-test

mergify · 2024-12-27T17:01:06Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-28T04:57:27Z

rerun cpp-unit-test

mergify · 2024-12-28T05:37:14Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-02T06:48:39Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2025-01-02T07:05:24Z

rerun go-sdk

mergify · 2025-01-02T08:00:19Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-05T04:18:33Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2025-01-05T05:31:15Z

/run-cpu-e2e

mergify · 2025-01-10T05:07:03Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-10T06:34:43Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2025-01-10T07:02:04Z

rerun go-sdk

chyezh · 2025-01-10T07:02:12Z

rerun ut

chyezh · 2025-01-10T07:20:53Z

rerun ut

- Current milvus use a serialized index size(compressed) for estimate resource for loading. - Add a new field MemSize (before compressing) for index to estimate resource. Signed-off-by: chyezh <[email protected]>

mergify · 2025-01-13T10:01:20Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

Signed-off-by: chyezh <[email protected]>

mergify · 2025-01-13T12:16:57Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2025-01-13T12:31:35Z

rerun go-sdk

mergify · 2025-01-13T13:07:04Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2025-01-13T14:04:20Z

rerun go-sdk

sre-ci-robot added area/internal-api size/XXL Denotes a PR that changes 1000+ lines. labels Dec 26, 2024

sre-ci-robot requested review from congqixia and czs007 December 26, 2024 03:57

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Dec 26, 2024

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from 4cb8035 to 1ebba3e Compare December 26, 2024 11:44

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from dbf4b36 to 1831a35 Compare January 5, 2025 03:03

mergify bot added the ci-passed label Jan 5, 2025

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from 1831a35 to 0db8290 Compare January 10, 2025 03:33

mergify bot removed the ci-passed label Jan 10, 2025

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch 2 times, most recently from 438db1d to 8f01a59 Compare January 10, 2025 04:20

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from 8f01a59 to d51cd6b Compare January 10, 2025 05:40

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from d51cd6b to 54f939c Compare January 10, 2025 07:36

mergify bot added the ci-passed label Jan 10, 2025

enhance: record memory size (uncompressed) item for index

44007aa

- Current milvus use a serialized index size(compressed) for estimate resource for loading. - Add a new field MemSize (before compressing) for index to estimate resource. Signed-off-by: chyezh <[email protected]>

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from 54f939c to 6cc2ced Compare January 13, 2025 08:58

mergify bot removed the ci-passed label Jan 13, 2025

fix: some comment

fd22729

Signed-off-by: chyezh <[email protected]>

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from 6cc2ced to fd22729 Compare January 13, 2025 11:50

mergify bot added the ci-passed label Jan 13, 2025

enhance: record memory size (uncompressed) item for index #38770

Are you sure you want to change the base?

enhance: record memory size (uncompressed) item for index #38770

Conversation

chyezh commented Dec 26, 2024

sre-ci-robot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

chyezh commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

chyezh commented Dec 26, 2024

mergify bot commented Dec 26, 2024

codecov bot commented Dec 26, 2024 • edited Loading

Codecov Report

chyezh commented Dec 27, 2024

mergify bot commented Dec 27, 2024

chyezh commented Dec 27, 2024

chyezh commented Dec 27, 2024

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

chyezh commented Dec 27, 2024

chyezh commented Dec 27, 2024

mergify bot commented Dec 27, 2024

chyezh commented Dec 28, 2024

mergify bot commented Dec 28, 2024

mergify bot commented Jan 2, 2025

chyezh commented Jan 2, 2025

mergify bot commented Jan 2, 2025

mergify bot commented Jan 5, 2025

chyezh commented Jan 5, 2025

mergify bot commented Jan 10, 2025

mergify bot commented Jan 10, 2025

chyezh commented Jan 10, 2025

chyezh commented Jan 10, 2025

chyezh commented Jan 10, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

chyezh commented Jan 13, 2025

mergify bot commented Jan 13, 2025

chyezh commented Jan 13, 2025

codecov bot commented Dec 26, 2024 •

edited

Loading