How to mark a score for logging?

[EricJohnson86]

I am using EpochScoring() to log a score.
Yet, it doesn't show on the TensorBoard graphs.

How can I mark a score for logging?

I saw a question on SO: https://stackoverflow.com/questions/75781328/pytorch-skorch-make-the-tensorboard-callbakc-keep-track-on-manual-scores-defi.
Yet the order of the callback didn't make any change.

[BenjaminBossan]

Could you please post a bit of code for context? Ideally a fully working minimal example, but if not available, as much as possible.

[EricJohnson86]

I will try to make something.
But the question remains, is the policy for new scoring in TensorBoard is opt-in?
Or basically all scoring should be incorporated into it beside those who match the names of the filtered data?

Because if the policy is that everything is logged, then my MWE will be a bug not just a question.

[BenjaminBossan]

Scores from the net's history are logged automatically, except for a few things that are filtered out (i.e. logging is opt out). Here is a complete example using a custom "roc_auc" score with EpochScoring, maybe it helps:

import numpy as np
from sklearn.datasets import make_classification
from torch import nn
import torch.nn.functional as F
from torch.utils.tensorboard import SummaryWriter
from skorch import NeuralNetClassifier
from skorch.callbacks import EpochScoring, TensorBoard

X, y = make_classification(1000, 20, n_informative=10, random_state=0)
X, y = X.astype(np.float32), y.astype(np.int64)

class ClassifierModule(nn.Module):
    def __init__(
            self,
            num_units=10,
            nonlin=F.relu,
            dropout=0.5,
    ):
        super().__init__()
        self.dense0 = nn.Linear(20, num_units)
        self.nonlin = nonlin
        self.dropout = nn.Dropout(dropout)
        self.dense1 = nn.Linear(num_units, 10)
        self.output = nn.Linear(10, 2)

    def forward(self, X, **kwargs):
        X = self.nonlin(self.dense0(X))
        X = self.dropout(X)
        X = F.relu(self.dense1(X))
        X = F.softmax(self.output(X), dim=-1)
        return X

writer = SummaryWriter()
net = NeuralNetClassifier(
    ClassifierModule,
    max_epochs=20,
    lr=0.1,
    callbacks=[
        EpochScoring(scoring='roc_auc', lower_is_better=False),
        TensorBoard(writer),
    ],
)
net.fit(X, y)

Below is a screenshot of what is logged for me. Note that the custom score "roc_auc" was logged automatically:

[BenjaminBossan]

@EricJohnson86 Any updates?

[EricJohnson86]

@BenjaminBossan , I still haven't had the chance to get back to it. I will though.

[BenjaminBossan]

Since I haven't heard back, I assume the problem is solved. If not, please let me know.