chore: [comet-parquet-exec] Unit test fixes, default scan impl to native_comet

[parthchandra]

Notable changes:

1. There are three scan implementations:

Name	Description	Operator	Underlying implementation
native_comet	original, supports primitive types (default)	CometScan	Comet
native_datafusion	POC1, supports (or will support) complex types	CometNativeScan	DataFusion
native_iceberg_compat	POC2, supports (or will support) complex types, exposes API for Iceberg	Comet Scan	DataFusion

The scan implementation can be selected by setting the conf spark.comet.scan.impl or by setting the environment variable COMET_PARQUET_SCAN_IMPL

2. Plan compatibility suites generate a different plan based on the implementation. As a result, we now have three sets of expected plans based on the scan implementation chosen

3. We now use the Spark Session timezone instead of UTC while reading timestamp fields. This is so that we can compare them with literal timestamps (Spark apparently automatically applies the session timezone to those)

[parthchandra]

@andygrove @mbutrovich

[andygrove]

@parthchandra could you run cargo fmt and cargo clippy

[andygrove]

@parthchandra could you run cargo fmt and cargo clippy

and make format .. that is if we want to see tests passing on this PR, which I think they should now? Or do we want to wait until we PR comet-parquet-exec against main before trying to get CI green?

[parthchandra]

@parthchandra could you run cargo fmt and cargo clippy

and make format .. that is if we want to see tests passing on this PR, which I think they should now? Or do we want to wait until we PR comet-parquet-exec against main before trying to get CI green?

Fixed the build. Yes, we should get the CI to turn green with this PR before we attempt any more changes to the feature branch

[parthchandra]

Updated the plns for Spark 3.5 and Spark 4.0. However plan generation for the native_datafusion impl is failing which will not affect the ci, but which needs to be addressed at some point.

[andygrove]

LGTM. Thanks @parthchandra

[parthchandra]

Added the plans for Spark 4.0 for the NATIVE_DATAFUSION and NATIVE_ICEBERG_COMPAT scans