Symbolic Transformer

[jmrichardson]

Hi,

Newbie question that I can't find reference in documentation. Is it possible to generate new features instead of predicting similar to GPlearn Symbolic Transfomer?

[MilesCranmer]

You could write down an arbitrary forward function (https://ai.damtp.cam.ac.uk/pysr/examples/#11-expression-specifications) or loss function (https://ai.damtp.cam.ac.uk/pysr/examples/#9-custom-objectives) and could set up a problem where you optimize an unsupervised objective (you could write down a fake value for y like just a vector of zeros, and then avoid using it inside the loss function). Does that help? But yeah it would be cool to add an example of this to the PySR docs to show how to do it.

[jmrichardson]

Thank you for the fast reply @MilesCranmer. I am still a bit confused about how you would extract the features from a fitted model? With GPLearn the features generated are created to improve the prediction of target y in future downstream models. How would a fake values for y help here unless you have y embedded in your loss function? For clarity, I am trying to create more predictive features generated from GP that I can use for modeling. Thanks for your help.

[MilesCranmer]

the features generated are created to improve the prediction of target y in future downstream models

Could you be a bit more specific what this means? What exactly is the loss function being optimised?

[jmrichardson]

Currently, I am using GPlearn to create additional features from a dataset that I can then use in a model like lightgbm. Essentially, it creates expressions that generate new features based on spearmen correlation to the target. I essentially provide the dataset to GPlearn, it creates these these independent features which in theory have predictive value for the target. I can then tansform the dataset to create these features for new data. For example here is the class for reference:

import joblib
from gplearn.genetic import SymbolicTransformer
from loguru import logger
from fml.utils import get_name
import pandas as pd
from joblib import Memory


class GPlearnSymbolic:

    def __init__(self, dimension=10, generations=5, n_jobs=joblib.cpu_count()-1, cols=None):
        self.dimension = dimension
        self.generations = generations
        self.n_jobs = n_jobs
        self.cols = cols

        memory = Memory(f"store/{get_name()}", verbose=0)
        self.fit = memory.cache(self.fit)
        self.transform = memory.cache(self.transform)

    def fit(self, data):
        logger.info("Features symbolic fit")
        if self.cols is None:
            self.cols = [c for c in data.columns if c != 'label']
        X = data[self.cols]
        function_sets = ('add', 'sub', 'mul', 'div', 'sqrt', 'log', 'abs', 'neg', 'inv', 'max', 'min', 'sin', 'cos', 'tan')
        self.transformer = SymbolicTransformer(generations=self.generations, n_components=self.dimension,
                                               parsimony_coefficient='auto',
                                               random_state=42, n_jobs=self.n_jobs, metric='spearman',
                                               function_set=function_sets,
                                               feature_names=X.columns)
        self.transformer.fit(X, data['label'])
        return self

    def transform(self, data):
        logger.info("Feature symbolic transform")
        features = self.transformer.transform(data[self.cols])
        # cols = [str(bp) for bp in self.transformer._best_programs]  # Column names too long
        cols = [f"gplearn_{i}" for i, bp in enumerate(self.transformer._best_programs)]
        features = pd.DataFrame(features, columns=cols, index=data.index)
        return features

As you can see, the transform generates the new features from the fitted GPlearn transformer. Hope this provides context in what I am looking for. I would prefer to use pysr if this is possible. Thanks.

[MilesCranmer]

Thanks. I’m trying to understand the actual math here.

Essentially, it creates expressions that generate new features based on spearmen correlation to the target

Could you be more specific about this and what you mean? Isn’t this the same thing that PySRRegressor is doing with the default L2DistLoss? What is the difference?

[MilesCranmer]

Here’s a Spearman Correlation loss for PySR in case this is what you were after?

loss_function = """
function my_spearman_loss(tree, dataset::Dataset{T,L}, options) where {T,L}
    prediction, complete = eval_tree_array(tree, dataset.X, options)
    if !complete
        return L(Inf)
    end
    
    # Get ranks
    rank_pred = sortperm(sortperm(prediction))
    rank_y = sortperm(sortperm(dataset.y))
    
    # Calculate Spearman
    d = rank_pred .- rank_y
    rho = 1 - (6 * sum(d .^ 2)) / (dataset.n * (dataset.n^2 - 1))
    
    # Return 1-rho so it's a loss to minimize (rho ranges from -1 to 1)
    return L(1 - rho)
end
"""

model = PySRRegressor(loss_function=loss_function)

model.fit(X, y)

features = [model.predict(X, index=i) for i in range(len(model.equations_))]

the index just enumerates simplest to most complex model: https://ai.damtp.cam.ac.uk/pysr/api/#pysr.PySRRegressor.predict

[jmrichardson]

Thank you. I was missing the fact you can predict on all the equations to produce the features. Awesome, will give it a try :)