Need help setting up.

[gmnmedeiros]

The TL;DR of my problem is: I'd like to run a fuzzy match search on a public dataset containing companies registered in the UK. I'm following this AWS Glue article, but some information is missing.

I have some questions to get started (sorry if they are dumb):

1- I should train a new model, right? Since my data schema is not the same as any of the Zingg training data.
2—When preparing training data, I grouped positive and negative pairs in a single dataset. Then, I created z_isMatch and z_cluster columns and assigned them accordingly. Are there any other key columns that Zingg expects to see in the schema?
3- If I train the model locally, can I upload it to S3 and use it on a Glue Notebook?
4- What's the recommended way to do local development? I'm trying to pip install zingg, but it just keeps downloading past PySpark versions...

Thanks so much for the attention :)

[Nitish1814]

1- Yes, you need to train new model(even though if schema matches with some existing model, domain of data is not yet learned)
2- After preparing data, you need to make config file like example. You just need additional columns as z_ismatch and z_cluster in your training data file. Also make sure your trainingData file is aligned to this example. Basically all the matching records should have same z_cluster and z_ismatch == 1 and un-mtached records should have z_ismatch == 0
3- yes, you can do that as well by providing "zinggDir": "s3://your bucket name/models" in config file and having your localy trained model inside s3://your bucket name/models. Make sure "modelId" in config file is same as modelId of model you trained.
4- Please refer to this: https://docs.zingg.ai/zingg/stepbystep/installation

Note: Instead of making training data by yourself, it's recommended to do findTrainingData and label phase and then train on zingg created training data. In this way Zingg would be able to learn domain and distribution of data more precisely

Thanks!

[gmnmedeiros]

Thanks so much for the reply!

[gmnmedeiros]

@Nitish1814 I'm facing this error here: do you have any idea, on top of your mind, of what may it mean?

Py4JJavaError: An error occurred while calling o115.execute.
: java.lang.NoSuchMethodError: org.apache.spark.sql.Encoders.row(Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Encoder;
	at zingg.spark.core.util.SparkBlockingTreeUtil.getBlockHashes(SparkBlockingTreeUtil.java:47)
	at zingg.common.core.executor.Matcher.getBlocked(Matcher.java:45)
	at zingg.common.core.executor.Matcher.execute(Matcher.java:115)
	at zingg.common.core.executor.TrainMatcher.execute(TrainMatcher.java:35)
	at zingg.common.client.Client.execute(Client.java:251)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:750)

I'm executing this piece of code with version 0.4.0:

sc._jsc.hadoopConfiguration().set('fs.defaultFS', f's3://{bucket_name}/')
sc._jsc.hadoopConfiguration().set('mapred.output.committer.class', "org.apache.hadoop.mapred.FileOutputCommitter")

from zingg.client import Arguments, ClientOptions, FieldDefinition, Zingg
zopts = ClientOptions(["--phase", "match",  "--conf", "/tmp/config.json"])
zargs = Arguments.createArgumentsFromJSON(zopts.getConf(), zopts.getPhase())
zingg = Zingg(zargs, zopts)
zingg.init()
zingg.execute()

[sonalgoyal]

Spark version needs to be 3.5. - please check