Reproducing results from version SymbolicRegression.jl 0.24.5 in version 1

[ArnoStrouwen]

The following script produces different results in

using SymbolicRegression
using Symbolics
using Distributions
using Optimization, OptimizationBBO
using Plots
using Random; Random.seed!(12345)
y(x) = exp(-x)*sin(2π*x) + cos(π/2*x)
n_obs = 10
design_region = Uniform(0.0,10.0)
X = rand(design_region,n_obs)
Y = y.(X)
options = SymbolicRegression.Options(
    unary_operators = (exp, sin, cos),
    binary_operators=(+, *, /, -),
    seed=123,
    deterministic=true,
    save_to_file=false
)
hall_of_fame = EquationSearch(X', Y, options=options, niterations=100, runtests=false, parallelism=:serial)
n_best_max = 10
#incase < 10 model structures were returned
n_best = min(length(hall_of_fame.members),n_best_max)
best_models = sort(hall_of_fame.members,by=member->member.loss)[1:n_best]
@syms x
eqn = node_to_symbolic(best_models[1].tree, options,varMap=["x"])
f = build_function(eqn, x, expression=Val{false})
f.(X)
plot(0.0:0.1:10.0,y.(0.0:0.1:10.0),lw=5,label="true model",ls=:dash);
model_structures = Function[]
for i = 1:n_best
    eqn = node_to_symbolic(best_models[i].tree, options,varMap=["x"])
    fi = build_function(eqn, x, expression=Val{false})
    x_plot = Float64[]
    y_plot = Float64[]
    for x_try in 0.0:0.1:10.0
        try
            y_try = fi(x_try)
            append!(x_plot,x_try)
            append!(y_plot,y_try)
        catch
        end
    end
    plot!(x_plot, y_plot,label="model $i");
    push!(model_structures,fi)
end
scatter!(X,Y,ms=5,label="data",ls=:dash);
plot!(xlabel="x",ylabel="y", ylims=(-1.2,1.6));
plot!(tickfontsize=12, guidefontsize=14, legendfontsize=8, grid=false, dpi=600)

0.24.5

latest release

For the projects @SebastianM-C and me are working on, the older version gives better results, but this version is rapidly becoming incompatible with other packages in the project, and it is a bummer not being able to upgrade.
Is there a way to reproduce the old results in the new version?

[MilesCranmer]

You can use defaults=v"0.24.5" to get the old default hyperparameters - pass it to Options(). Does that help? The new version is tuned for longer jobs. But I've had a few people tell me similar things, so I'm definitely open to re-tuning things again if it's a universal feeling!

Also I noticed various calls in your script are deprecated. If you start Julia with --depwarn=yes it will show the warnings for those.

[MilesCranmer]

P.S., the full list of hyperparameter changes is defined in this function: https://github.com/MilesCranmer/SymbolicRegression.jl/blob/e5afc911e059662ab10581910417473a3b671d77/src/Options.jl#L953-L1049

[ArnoStrouwen]

Passing the default keyword still does not reproduce the exact same result.

[MilesCranmer]

It might not be identical due very slight changes in the internal algorithm. But generally to get reproducible results you can run with deterministic=true, seed=0 and use serial for parallelism.

[gm89uk]

I had a similar issue. The biggest changes in parameters I found on performance were changing the regularisation parameters, adaptive_parsimony_scaling back to 20 from 1040 and increasing parsimony from 0 again.

Setting adaptive_parsimony_scaling=20 made it behave somewhat similarly, but it did reduce diversification.

I did eventually get the new settings to work by retuning population_size which I had optimised (had to reduce to account for the higher default population).

[MilesCranmer]

Good to know this, thanks!

Some background: the reason I disabled parsimony (and tried to tune around the lack of it) is that it had a scale dependence to it which made the defaults kind of less generic (like if someone had an MSE 10^5 as a “good” loss, parsimony would be negligible, while if someone had an MSE 10^-8 as a “good” loss, parsimony would completely dominate the fitness of expressions). But maybe from a practical perspective there’s a better way to do this.

[gm89uk]

Thanks that makes sense.

Perhaps start with parsimony at 0 then at each iteration set parsimony to (min loss across all equations) / 10, so that parsimony scales with the evolution?

For sure the newly optimised parameters let adaptive_parsimony_scaling shine much more than the old ones.