You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When validating a pyspark dataframe using DataFrameModel with Pysaprk SQL, if there is a regular expression that is not met by any column of the dataframe, pandera can't validate the df, due to a bug handling the list of columns that match the reg_ex, it seems.
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandera.
(optional) I have confirmed this bug exists on the main branch of pandera.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
frompyspark.sqlimportSparkSessionimportpyspark.sql.typesasTimportpandera.pysparkaspa# Create a example dataframespark=SparkSession.builder.master("local").appName("pyspark-pandera").getOrCreate()
schema=T.StructType([
T.StructField("user_name", T.StringType(), True),
T.StructField("email", T.StringType(), True)
])
data= [
("Alice", "[email protected]"),
("Bob", "[email protected]"),
]
df=spark.createDataFrame(data, schema)
# Validate dataframe with Pandera DataFrameModelclassExampleModel(pa.DataFrameModel):
user_name: T.StringType() =pa.Field(nullable=False)
# There is no field that matches this reg_exage_field: T.IntegerType() =pa.Field(alias="(.+)age(.+)", regex=True)
df_out=ExampleModel.validate(df)
Expected behavior
I expect df to be validated, and df_out to contain all the errors with the dataframe, according to the dataframe model.
In contrast, ExampleModel.validate(df) leads to an error handling the reg_ex columns:
AttributeError: 'list' object has no attribute 'tolist'
Desktop (please complete the following information):
OS: Windows 11
Browser: Chrome
Version: Pandera version 0.20.3
Screenshots
The text was updated successfully, but these errors were encountered:
Describe the bug
When validating a pyspark dataframe using DataFrameModel with Pysaprk SQL, if there is a regular expression that is not met by any column of the dataframe, pandera can't validate the df, due to a bug handling the list of columns that match the reg_ex, it seems.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
Expected behavior
I expect df to be validated, and df_out to contain all the errors with the dataframe, according to the dataframe model.
In contrast, ExampleModel.validate(df) leads to an error handling the reg_ex columns:
AttributeError: 'list' object has no attribute 'tolist'
Desktop (please complete the following information):
Screenshots
The text was updated successfully, but these errors were encountered: