From 478672bd81cfdc8e109a36353149c6aeb09a1291 Mon Sep 17 00:00:00 2001 From: KAZUYUKI TANIMURA Date: Thu, 30 Jan 2025 16:56:12 -0800 Subject: [PATCH] minor: commit compatibility doc (#1358) Leftover from #1349 --- docs/source/user-guide/compatibility.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/source/user-guide/compatibility.md b/docs/source/user-guide/compatibility.md index 94fd9ea7e..893326e19 100644 --- a/docs/source/user-guide/compatibility.md +++ b/docs/source/user-guide/compatibility.md @@ -32,12 +32,6 @@ be used in production. There is an [epic](https://github.com/apache/datafusion-comet/issues/313) where we are tracking the work to fully implement ANSI support. -## Regular Expressions - -Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's -regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but -this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`. - ## Floating number comparison Spark normalizes NaN and zero for floating point numbers for several cases. See `NormalizeFloatingNumbers` optimization rule in Spark. @@ -46,6 +40,22 @@ because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)). So Comet will add additional normalization expression of NaN and zero for comparison. +## Incompatible Expressions + +Some Comet native expressions are not 100% compatible with Spark and are disabled by default. These expressions +will fall back to Spark but can be enabled by setting `spark.comet.expression.allowIncompatible=true`. + +## Array Expressions + +Comet has experimental support for a number of array expressions. These are experimental and currently marked +as incompatible and can be enabled by setting `spark.comet.expression.allowIncompatible=true`. + +## Regular Expressions + +Comet uses the Rust regexp crate for evaluating regular expressions, and this has different behavior from Java's +regular expression engine. Comet will fall back to Spark for patterns that are known to produce different results, but +this can be overridden by setting `spark.comet.regexp.allowIncompatible=true`. + ## Cast Cast operations in Comet fall into three levels of support: