Create a Model Moniter [FEA]

[vikashg]

Is your feature request related to a problem? Please describe.
Often we would like to know how the model is performing on the data over a period of time and maintain overall statistics of the model performance. These statistics can be further granularized to specific centers and demographics. If several models are deployed we can use this performance to track the performance of each separate model.

Describe the solution you'd like
A web-based dashboard where the model performance can be displayed. Something like Tensorboard.

Describe alternatives you've considered
None so far.

Additional context
These statistics can be further used for design model updating rules in a continuous fashion.

[Leengit]

I use torch.utils.tensorboard.SummaryWriter and TensorBoard. Would that do what you want? If not, what are you looking for?

[MMelQin]

This is more than a feature request, and in fact this is an area that requires a suite of software.

• Model themselves, specifically the one App SDK support, do not change over time
• Sample models from Model Zoo, and any other source, need to be retrained with local data, and could be biased or specialized for certain demographics
• How a model performs in poduction, can not be from the inference application alone. The results have to be analyzed/evaluated either with expert and/or set of corresponding ground truth data, in a way, like in a double blind read, and the AI results compared with clinicians's result
• Is there any benefit for a realtime/rolling stats monitoring? Questionable, but
  • technically, the application and/or orchestrator needs to track the clinical data of the input images, e.g. demographics and more important symptom and lab data
  • technically, the corresponding ground truth data, i,e, a manual liver tumor seg of a given CT exam, can be extracted the application and associated with the AI results, for comparison
  • all collected data then need to go through data analytics, AI based? Big data? Who'd be the users of the dashboard, and how sophisticated and interactive will the analytics be?
• More realistic and more feasible, is to have periodic retro study
  • Rely on the PACS or image archive/data lake
  • collect the AI results and the corresponding ground truth data
  • for clinical use, both the AI and ground truth data are most likely DICOM
  • Utilize/enhance the App SDK DICOM parsing capabilities, combined with image processing algorithms in MONAI and others, do batch evaluation

It is a area of active research, and I have had in discussion with some with regard to the production approach. Would be good that we continue the discussion here. As more and more models get into clinical (IRB'ed) use, we'd be able to access data and do some PoC work.