Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash every day after compacting TSM and TSI #24432

Closed
astrxlegaulois opened this issue Oct 28, 2023 · 1 comment
Closed

Crash every day after compacting TSM and TSI #24432

astrxlegaulois opened this issue Oct 28, 2023 · 1 comment

Comments

@astrxlegaulois
Copy link

Hello,

Ever since I installed influxdb the daemon crashes right after automatically compacting the TSI.
The log shows that tsdb.ReadSeriesKeyMeasurement() fails to access the 3rd item of a list than only contains 2.
There is only a few thousands of floats stored in the database.

Environment info:

I am running influxdb-2.7.1 on OpenBSD. http is the only non-default pluggin enabled.

Logs:

> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.080793Z lvl=info msg="Cache snapshot (start)" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.098220Z lvl=info msg="Cache snapshot (start)" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.115050Z lvl=info msg="Snapshot for path written" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9 duration=34.257ms
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.115136Z lvl=info msg="Cache snapshot (end)" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=34.362ms
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.131786Z lvl=info msg="Snapshot for path written" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10 duration=33.566ms
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.131842Z lvl=info msg="Cache snapshot (end)" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=33.635ms
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081461Z lvl=info msg="TSM compaction (start)" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=start
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081529Z lvl=info msg="Beginning compaction" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=8
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081548Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000001-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081562Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000002-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081574Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000003-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081588Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000004-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081600Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000005-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081612Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000006-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081623Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000007-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081636Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000008-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101320Z lvl=info msg="TSM compaction (start)" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=start
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101366Z lvl=info msg="Beginning compaction" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=8
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101380Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000001-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101392Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000002-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101404Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000003-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101416Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000004-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101428Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000005-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101440Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000006-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101451Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000007-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101462Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000008-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.155109Z lvl=info msg="Compacted file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000008-000000002.tsm.tmp
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.155168Z lvl=info msg="Finished compacting files" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=1
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.155184Z lvl=info msg="TSM compaction (end)" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=end op_elapsed=73.960ms
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.169039Z lvl=info msg="Compacted file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000008-000000002.tsm.tmp
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.169091Z lvl=info msg="Finished compacting files" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=1
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.169106Z lvl=info msg="TSM compaction (end)" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=end op_elapsed=67.859ms
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: ts=2023-10-24T11:34:37.379351Z lvl=info msg="TSI log compaction (start)" log_id=0l487GTG000 service=storage-engine index=tsi tsi1_partition=8 op_name=tsi1_compact_log_file tsi1_log_file_id=2 op_event=start
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: ts=2023-10-24T11:34:37.385303Z lvl=info msg="TSI log compaction (end)" log_id=0l487GTG000 service=storage-engine index=tsi tsi1_partition=8 op_name=tsi1_compact_log_file tsi1_log_file_id=2 op_event=end op_elapsed=6.005ms
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: panic: runtime error: index out of range [1] with length 0
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: goroutine 249206 [running]:
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: encoding/binary.bigEndian.Uint16(...)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	encoding/binary/binary.go:139
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.ReadSeriesKeyMeasurement(...)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/series_file.go:380
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.parseSeriesKey({0x4757243cf?, 0x32c14cc?, 0x5c1aa0?}, {0x0, 0x36f2236?, 0x0})
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/series_file.go:417 +0x300
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.ParseSeriesKey(...)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/series_file.go:400
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.(*SeriesFile).Series(0x400783bea0?, 0x40078cb7c0?)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/series_file.go:242 +0x30
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).seriesSketches.func1(0x40029a1488?)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/index/tsi1/log_file.go:1055 +0x34
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.(*SeriesIDSet).ForEach(0x40029a1510?, 0x4007cc1510)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/series_set.go:181 +0xe0
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).seriesSketches(0x4001bcfc20)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/index/tsi1/log_file.go:1054 +0x78
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).CompactTo(0x4001bcfc20, {0xcf83a0?, 0x40082a7a10?}, 0x3?, 0x3?, 0x4001ba1920)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/index/tsi1/log_file.go:869 +0x350
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).compactLogFile(0x4001a25440, 0x4001bcfc20)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/index/tsi1/partition.go:1285 +0x3ec
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).compact.func1()
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/index/tsi1/partition.go:1030 +0x28
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: created by github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).compact in goroutine 249187
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: 	github.com/influxdata/influxdb/v2/tsdb/index/tsi1/partition.go:1029 +0x2e0
@jdockerty jdockerty mentioned this issue Jan 22, 2024
Merged
2 tasks
@davidby-influx
Copy link
Contributor

Closed by #24565
Available in OSS 2.7.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@astrxlegaulois @davidby-influx