Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Interval type can't flush to parquet #3235

Closed
QuenKar opened this issue Jan 25, 2024 · 8 comments · Fixed by #5422
Closed

Interval type can't flush to parquet #3235

QuenKar opened this issue Jan 25, 2024 · 8 comments · Fixed by #5422
Assignees
@evenyag
Labels
C-bug Category Bugs

Comments

@QuenKar
Copy link
Contributor

QuenKar commented Jan 25, 2024 •
edited
Loading

What type of bug is this?

Unexpected error

What subsystems are affected?

Storage Engine

Minimal reproduce step

  1. start database, use mysql client to connect it.
  2. execute the following sql statements.
create table test2(val interval, ts timestamp time index);

insert into test2 values('1 year', 1000);

alter table test2 add column label varchar;

it will show Failed to encode record batch: NYI: Attempting to write an Arrow interval type MonthDayNano to parquet that is not yet implemented

What did you expect to see?

None

What did you see instead?

None

What operating system did you use?

Mac OS X Ventura 13.2 ARM

What version of GreptimeDB did you use?

0.6

Relevant log output and stack trace

2024-01-24T14:59:49.930290Z ERROR servers::mysql::writer: Failed to handle mysql query err=0: Failed to execute query, query: alter table test2 add column label string, at /Users/xxx/rust-project/greptimedb/src/servers/src/query_handler/sql.rs:89:22
1: Table operation error, at src/frontend/src/instance.rs:273:14
2: Failed to execute ddl, at /Users/xxx/rust-project/greptimedb/src/operator/src/statement/ddl.rs:313:14
3: Failed to wait procedure done, at src/common/meta/src/ddl_manager.rs:264:14
4: Failed to execute procedure, at /Users/xxx/rust-project/greptimedb/src/common/procedure/src/watcher.rs:35:43
5: Failed to execute procedure due to external error
6: Failed to operate on datanode: peer-0(), at src/common/meta/src/ddl/utils.rs:30:27
7: External error, at src/frontend/src/instance/standalone.rs:78:14
8: Operation to region server failed, at src/frontend/src/instance/standalone.rs:60:14
9: Execute gRPC request error, at src/datanode/src/region_server.rs:242:14
10: Failed to handle request for region 4440996184064(1034, 0), at src/datanode/src/region_server.rs:479:14
11: Failed to flush region 4440996184064(1034, 0), at src/mito2/src/flush.rs:672:46
12: Failed to write to buffer, at src/mito2/src/sst/parquet/writer.rs:178:10
13: Failed to encode record batch, at src/common/datasource/src/file_format/parquet.rs:152:27
14: NYI("Attempting to write an Arrow interval type MonthDayNano to parquet that is not yet implemented")
@QuenKar QuenKar added the C-bug Category Bugs label Jan 25, 2024
@QuenKar QuenKar self-assigned this Jan 25, 2024
@QuenKar QuenKar changed the title interval type can't flush to parquet Interval type can't flush to parquet Jan 25, 2024
@yihong0618 yihong0618 mentioned this issue Jan 10, 2025
Closed
@yihong0618
Copy link
Contributor

more detail for this:

apache/arrow-rs#5875
apache/arrow-rs#5849
https://github.com/apache/arrow/blob/65974672a356f34889ed7b9bfb8b76230c27c7ee/java/dataset/src/test/java/org/apache/arrow/dataset/TestAllTypes.java#L94-L96
apache/arrow-rs@5c4a65a

@evenyag
Copy link
Contributor

evenyag commented Jan 20, 2025

We should not allow users to create a table with an interval type. We can validate this while creating a table.

@yihong0618
Copy link
Contributor

We should not allow users to create a table with an interval type. We can validate this while creating a table.

it will be a break change?

@yihong0618 yihong0618 mentioned this issue Jan 20, 2025
Open
@yihong0618
Copy link
Contributor

We should not allow users to create a table with an interval type. We can validate this while creating a table.

I think if not allow users to create a table with an interval type, will break a lot of user tables.. maybe we can not allow it flush to parquet maybe?

yihong0618 added a commit to yihong0618/greptimedb that referenced this issue Jan 22, 2025
@yihong0618
fix: flash table panic when table has interval column close GreptimeT…
ffc63ef 
…eam#3235

Signed-off-by: yihong0618 <[email protected]>
@yihong0618 yihong0618 mentioned this issue Jan 22, 2025
Merged
5 tasks
@evenyag
Copy link
Contributor

evenyag commented Jan 22, 2025

will break a lot of user tables..

This breaking should be fine since most users won't create a table with interval types. A table with interval type actually won't work now because the database can't persist it.

@yihong0618
Copy link
Contributor

will break a lot of user tables..

This breaking should be fine since most users won't create a table with interval types. A table with interval type actually won't work now because the database can't persist it.

sounds fine, will update my pull request in create side, is that OK?

FYI: for now, there are a few tables with interval field in the test

@evenyag
Copy link
Contributor

evenyag commented Jan 22, 2025

will break a lot of user tables..

This breaking should be fine since most users won't create a table with interval types. A table with interval type actually won't work now because the database can't persist it.

sounds fine, will update my pull request in create side, is that OK?

FYI: for now, there are a few tables with interval field in the test

Can we update these tests? I'll recommend users not store an interval type. e.g. It's hard to answer the maximum interval of 1 month and 31 days......

@yihong0618
Copy link
Contributor

will break a lot of user tables..

This breaking should be fine since most users won't create a table with interval types. A table with interval type actually won't work now because the database can't persist it.

sounds fine, will update my pull request in create side, is that OK?
FYI: for now, there are a few tables with interval field in the test

Can we update these tests? I'll recommend users not store an interval type. e.g. It's hard to answer the maximum interval of 1 month and 31 days......

yes, will try

yihong0618 added a commit to yihong0618/greptimedb that referenced this issue Jan 22, 2025
@yihong0618
Revert "fix: flash table panic when table has interval column close G…
b0c3487 
…reptimeTeam#3235"

This reverts commit ffc63ef.
yihong0618 added a commit to yihong0618/greptimedb that referenced this issue Jan 22, 2025
@yihong0618
fix: create table do not support interval type for now close Greptime…
01fcf66 
…Team#3235

Signed-off-by: yihong0618 <[email protected]>
yihong0618 added a commit to yihong0618/greptimedb that referenced this issue Jan 22, 2025
@yihong0618
fix: flash table panic when table has interval column close GreptimeT…
7a91355 
…eam#3235

Signed-off-by: yihong0618 <[email protected]>
yihong0618 added a commit to yihong0618/greptimedb that referenced this issue Jan 22, 2025
@yihong0618
Revert "fix: flash table panic when table has interval column close G…
3f262ce 
…reptimeTeam#3235"

This reverts commit ffc63ef.
yihong0618 added a commit to yihong0618/greptimedb that referenced this issue Jan 22, 2025
@yihong0618
fix: create table do not support interval type for now close Greptime…
c5ecdc0 
…Team#3235

Signed-off-by: yihong0618 <[email protected]>
github-merge-queue bot pushed a commit that referenced this issue Jan 25, 2025
@yihong0618
fix: flush table panic when table has interval column close #3235 (#5422
adb5c37 
)

* fix: flash table panic when table has interval column close #3235

Signed-off-by: yihong0618 <[email protected]>

* Revert "fix: flash table panic when table has interval column close #3235"

This reverts commit ffc63ef.

* fix: create table do not support interval type for now close #3235

Signed-off-by: yihong0618 <[email protected]>

* fix: sqlness

Signed-off-by: yihong0618 <[email protected]>

* fix: address comments

Signed-off-by: yihong0618 <[email protected]>

* fix: address comments fix conflict and more tests

Signed-off-by: yihong0618 <[email protected]>

* fix: address final comments drop useless sqlness tests

Signed-off-by: yihong0618 <[email protected]>

---------

Signed-off-by: yihong0618 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@evenyag evenyag

Labels
C-bug Category Bugs
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: flush table panic when table has interval column close #3235 yihong0618/greptimedb
3 participants
@evenyag @yihong0618 @QuenKar