reproduce-research-paper-ipol.md

[ Back to index ]

Tutorial: Reproduce research paper (IPOL'22 example)

This tutorial was prepared during a public Collective Knowledge challenge to reproduce results from the IPOL'22 439 journal article using the MLCommons CM automation language.

Please check the CM introduction to understand CM motivation and concepts.

Organizers

• Jose Hernandez
• Miguel Colom
• Grigori Fursin
• cTuning foundation
• cKnowledge Ltd

Initial discussion and materials

• mlcommons#617
• https://access.cknowledge.org/playground/?action=challenges&name=reproduce-and-automate-ipol-paper

Implementation

We have implemented two portable CM scripts to automate reproducibility on any platform:

• Download IPOL paper sources and cache them in CM
• Run IPOL 2022 439 paper demo using above script and PyTorch

Reproducibility study

1. Install MLCommons CM(CK2) automation framework as described here.

2. Install the latest MLCommons repository with reusable CM scripts:

cm pull repo mlcommons@ck

3. Install author sources from IPOL 2022 439 paper using CM:

cm run script "get ipol src" --year=2022 --number=439

cm show cache --tags=ipol,src

4. Run script to install dependencies and reproduce results

cm run script "reproduce ipol 2022-439"

This script will use these sample images and should produce a diff.png file in the current directory.

Using other data

You can use any other 2 images by specifying their full path as follows:

cm run script "reproduce ipol 2022-439" \
       --image1={full path to png image 1} \
       --image2={full path to png image 2}