What is the idea of Track

[ngabrek]

Hi everyone,

I'm currently playing around with the package and would like to compare the performance of a model on different datasets.
While I was skimming through the documentation I stumbled over the Track class, which should evaluate a model's performance. In order to initialize a Track object one has to set the name, datasets and metrics that it should evaluate on. However, in order to start the evaluation by running run(), it requires to handover a dataset, such that only this dataset is evaluated, instead of using the datasets already given to the track. So it is just the same as running iter_progressive_val_score(), which is what the run() function actually does.

So my question now is, what is exactly the point of a track? Am I missing out on something?

[MaxHalford]

Hey there. Excellent question. We haven't put a lot of effort in documenting tracks, yet.

The Track class is a base class: you're not really expected to use it directly. It is indeed comparable to iter_progressive_val_score. There are several classes you can use, depending on your learning task:

• BinaryClassificationTrack
• MultiClassClassificationTrack
• RegressionTrack

Does that clarify things?

[ngabrek]

Thanks for the quick reply! It's a bit clearer now.
I was trying to use the BinaryClassificationTrack, which according to the source code initially has some datasets assigned to it, like the Elec2. However, when I execute run() it asks for a dataset to run the evaluation on, instead of using the preassigned datasets. So, when I understand it correctly, the only advantage over using iter_progressive_val_score is that the metrics are preset, right?

[MaxHalford]

I understand what you mean. Basically tracks are meant to be used like this:

track = BinaryClassificationTrack()
for dataset in track.datasets:
	for checkpoint in track.run(model, dataset):
		print(checkpoint)

So yeah, track are not much help. I did this initial design because I think it's quite flexible. We could of course imagine a higher level interface.

[ngabrek]

Okay, perfect now I 100% get it. Thank you!