Incorporate feedback from alpha test (#2)

* [REP] Ignore build files of doc

* [DOC] Add pip install cmd to example

Highlight the need for the utils file

* [REP] Ignore logfiles (from running examples)

* [FIX] Move examples utilites out of subdir

* [DOC] Add install instruction for DeepOBS

* [DOC] Make pip install very prominent

* [DOC] Remove sphinx copybutton due to issue

Highlighted lines have an extra line break, probably related to this: executablebooks/sphinx-copybutton#118

If there is a fix, we can enable it again.

* [DOC] Remove sphinx copybutton due to issue

    Highlighted lines have an extra line break, probably related to this: executablebooks/sphinx-copybutton#118

    If there is a fix, we can enable it again.

* [FIX] Output to terminal when plotting

* [FIX] Ignore irrelevant torch warning for examples

* [DOC] Document terminal output in examples

* [TST] Define MPLBACKEND for tests

* [FIX] More meaningful plotting msg

* [DOC] Add a blocking plot after training

* [FIX] Move Curvature plots slighty right

* [FIX] Round histograms ticks

* [FIX] Format ticks

* [DOC] Update preview example 01

* [DOC] Make example 02 longer

* [DOC] Run DeepOBS Example for 50 epochs

* [DOC] Add quickstart page [WIP]

* [FIX] Use correct shift between start and end in UpdateSize (#3)

The old value was correctly documented to be 1, but wrongly set to 0.
This lead to the UpdateSize being zero, because parameters from the same
iteration would be used as start/end points in the computation.

* [DOC] Replace personal output dir

* [DOC] Update Backobs install command

* [DOC] Install DeepOBS from development

* [DOC] Put utility file reminder in note block

* [DOC] Add download button for script

* [DOC] Add download button to script

* [DOC] Remove quickstart page (for now)

Co-authored-by: Felix Dangel <[email protected]>

.conda_env.yml

@@ -24,7 +24,7 @@ dependencies:
     - memory-profiler
     - pre-commit
     - graphviz
-    - sphinx-copybutton
+    # - sphinx-copybutton
     - sphinx-notfound-page
     - git+https://[email protected]/fsschneider/DeepOBS.git@develop#egg=deepobs
     - git+https://[email protected]/f-dangel/backobs.git@development#egg=backobs

.gitignore

@@ -307,17 +307,12 @@ TSWLatexianTemp*
 # Tensorboard
 **/*events.out.tfevents.*
 
-# Paper
-tex/CVPR2021/cockpit_paper.pdf
-tex/CVPR2021/main.pdf
-tex/CVPR2021/suppl.pdf
-
-tex/CVPR2021/rebuttal/cvpr_rebuttal.pdf
-
-tex/CVPR2021/arXiv/
-
 # Test
 .coverage
+**/logfiles
+
+# Doc
+docs/build/
 
 # local emacs variables
 !**/.dir-locals.el

README.md

@@ -26,6 +26,12 @@
 
 ---
 
+```bash
+pip install 'git+https://github.com/f-dangel/cockpit.git@development'
+```
+
+---
+
 **Cockpit is a visual and statistical debugger specifically designed for deep learning.** Training a deep neural network is often a pain! Successfully training such a network usually requires either years of intuition or expensive parameter searches involving lots of trial and error. Traditional debuggers provide only limited help: They can find *syntactical errors* but not *training bugs* such as ill-chosen learning rates. **Cockpit** offers a closer, more meaningful look into the training process with multiple well-chosen *instruments*.
 
 ---

cockpit/instruments/gradient_tests_gauge.py

@@ -4,6 +4,7 @@
 
 import matplotlib as mpl
 import matplotlib.pyplot as plt
+from matplotlib import ticker
 
 from cockpit.instruments.utils_instruments import check_data
 
@@ -88,26 +89,29 @@ def _format(self, ax_all, ax_norm, ax_inner, ax_ortho):
 
     ax_norm.set_ylabel("Norm")
     ax_norm.set_yscale("log")
-
     ax_norm.xaxis.tick_top()
     ax_norm.set_facecolor(self.bg_color_instruments)
     ax_norm.set_xscale(iter_scale)
+    ax_norm.yaxis.set_minor_locator(ticker.MaxNLocator(3))
+    ax_norm.yaxis.set_minor_formatter(ticker.FormatStrFormatter("%.2g"))
 
     ax_inner.set_ylabel("Inner")
     ax_inner.set_yscale("log")
-
     ax_inner.invert_yaxis()
     ax_inner.set_facecolor(self.bg_color_instruments)
     ax_inner.set_xscale(iter_scale)
+    ax_inner.yaxis.set_minor_locator(ticker.MaxNLocator(3))
+    ax_inner.yaxis.set_minor_formatter(ticker.FormatStrFormatter("%.2g"))
 
     ax_ortho.set_title("Ortho")
     ax_ortho.xaxis.tick_top()
     ax_ortho.yaxis.tick_right()
     ax_ortho.set_xscale("log")
-
     ax_ortho.invert_yaxis()
     ax_ortho.set_yscale(iter_scale)
     ax_ortho.set_facecolor(self.bg_color_instruments)
+    ax_ortho.xaxis.set_minor_locator(ticker.MaxNLocator(2))
+    ax_ortho.xaxis.set_minor_formatter(ticker.FormatStrFormatter("%.2g"))
 
 
 def _plot(self, ax_all, ax_norm, ax_inner, ax_ortho):

cockpit/instruments/histogram_2d_gauge.py

@@ -5,8 +5,13 @@
 import numpy as np
 import pandas as pd
 import seaborn as sns
+from matplotlib import ticker
 
-from cockpit.instruments.utils_instruments import _beautify_plot, check_data
+from cockpit.instruments.utils_instruments import (
+    _beautify_plot,
+    _ticks_formatter,
+    check_data,
+)
 
 
 def histogram_2d_gauge(
@@ -76,7 +81,7 @@ def histogram_2d_gauge(
     joint_plot_args = {
         "facecolor": self.bg_color_instruments,
         "xlabel": "Parameter Element Value",
-        "ylabel": "Gradient Element Value",
+        "ylabel": "Gradient Element\nValue",
     }
 
     df = _get_2d_histogram_data(
@@ -89,6 +94,9 @@ def histogram_2d_gauge(
 
     _beautify_plot(ax=ax_joint, **joint_plot_args)
 
+    ax_joint.set_xticklabels(_ticks_formatter(ax_joint.get_xticklabels()))
+    ax_joint.set_yticklabels(_ticks_formatter(ax_joint.get_yticklabels()))
+
     # "Zero lines
     # TODO This assumes that the bins are symmetrical!
     ax_joint.axvline(df.shape[1] / 2, ls="-", color="#ababba", linewidth=1.5, zorder=0)
@@ -109,6 +117,7 @@ def histogram_2d_gauge(
         ax_xmargin.barh(
             mid_points, vals, height=bin_size, color=self.primary_color, linewidth=0.1
         )
+        ax_xmargin.xaxis.set_minor_locator(ticker.MaxNLocator(3))
 
         ax_ymargin = fig.add_subplot(gs[0, :2])
         ax_ymargin.set_yscale("log")
@@ -127,6 +136,8 @@ def histogram_2d_gauge(
             color=self.primary_color,
             linewidth=0.2,
         )
+        ax_ymargin.yaxis.set_minor_locator(ticker.MaxNLocator(3))
+        ax_ymargin.yaxis.set_minor_formatter(ticker.FormatStrFormatter("%.3g"))
 
 
 def _default_trafo(array):

cockpit/instruments/max_ev_gauge.py

@@ -2,6 +2,8 @@
 
 import warnings
 
+from matplotlib import ticker
+
 from cockpit.instruments.utils_instruments import check_data, create_basic_plot
 
 
@@ -64,3 +66,5 @@ def max_ev_gauge(self, fig, gridspec):
     # part that should be plotted
     ax = fig.add_subplot(gridspec)
     create_basic_plot(**plot_args, ax=ax)
+
+    ax.yaxis.set_minor_formatter(ticker.FormatStrFormatter("%.2g"))

cockpit/instruments/utils_instruments.py

@@ -2,6 +2,7 @@
 
 import warnings
 
+import matplotlib as mpl
 import seaborn as sns
 
 
@@ -294,3 +295,22 @@ def check_data(data, requires, min_elements=1):
                 return False
 
     return True
+
+
+def _ticks_formatter(ticklabels, format_str="{:.2f}"):
+    """Format the ticklabels.
+
+    Args:
+        ticklabels ([mpl.text.Text]): List of ticklabels.
+        format_str (str, optional): Formatting string for the labels.
+            Defaults to "{:.2f}".
+
+    Returns:
+        [mpl.text.Text]: Reformatted list of ticklabels.
+    """
+    new_ticks = []
+    for tick in ticklabels:
+        rounded_label = format_str.format(float(tick.get_text()))
+        new_tick = mpl.text.Text(*tick.get_position(), rounded_label)
+        new_ticks.append(new_tick)
+    return new_ticks

cockpit/plotter.py

@@ -152,6 +152,9 @@ def plot(
 
         # Show or Save plots
         if show_plot:
+            msg = "[cockpit|plot] Showing current Cockpit."
+            msg += " Blocking. Close plot to continue." if block else ""
+            print(msg)
             plt.show(block=block)
             plt.pause(0.001)
         if save_plot:
@@ -236,10 +239,13 @@ def _plot_curvature(self, grid_spec):
 
         # Build inner structure of this plotting group
         # We use additional "dummy" gridspecs to position the instruments
+        inner_width_ratios_curvature = self.inner_width_ratios[:]
+        inner_width_ratios_curvature[2] = 0.0
+        inner_width_ratios_curvature[0] = 1.5 * inner_width_ratios_curvature[0]
         self.gs_curvature = grid_spec.subgridspec(
             self.inner_num_rows,
             self.inner_num_cols,
-            width_ratios=self.inner_width_ratios,
+            width_ratios=inner_width_ratios_curvature,
             height_ratios=self.inner_height_ratios,
             hspace=self.inner_hspace,
         )

cockpit/quantities/update_size.py

@@ -10,10 +10,10 @@ class UpdateSize(TwoStepQuantity):
     """str: String under which the parameters are cached for computation.
        Default: ``'params'``.
     """
-    SAVE_SHIFT = 0
-    """int: Difference between iteration at which information is computed versus 
-       iteration under which it is stored. For instance, if set to ``1``, the 
-       information computed at iteration ``n + 1`` is saved under iteration ``n``. 
+    SAVE_SHIFT = 1
+    """int: Difference between iteration at which information is computed versus
+       iteration under which it is stored. For instance, if set to ``1``, the
+       information computed at iteration ``n + 1`` is saved under iteration ``n``.
        Defaults to ``1``.
     """
 

docs/requirements_doc.txt

@@ -4,7 +4,7 @@ sphinx
 # Sphinx extensions
 sphinx-rtd-theme
 sphinx-automodapi
-sphinx-copybutton
+#sphinx-copybutton
 sphinx-notfound-page
 
 # Markdown conversion

docs/source/conf.py

@@ -55,7 +55,7 @@
     "sphinx.ext.napoleon",
     "sphinx.ext.viewcode",
     "sphinx_automodapi.automodapi",
-    "sphinx_copybutton",
+    # "sphinx_copybutton",
     "sphinx_rtd_theme",
 ]
 

docs/source/examples/01_basic_fmnist.rst

@@ -9,26 +9,59 @@ training loop.
 Lines that are highlighted in yellow highlight **Cockpit**-specific code, but 
 don't worry, most of these changes are simple plug-and-play solutions.
 
+You can try out this basic example yourself.
+Simply install **Cockpit** via
+
+.. code:: bash
+
+  pip install 'git+https://github.com/f-dangel/cockpit.git@development'
+
+and then copy the `example files <https://github.com/f-dangel/cockpit/tree/development/examples>`_
+from the repository or from the code block below. 
+
+.. note::
+
+  Don't forget the :download:`utility file <../../../examples/_utils_examples.py>` 
+  which provides the data for this example.
+
 .. literalinclude:: ../../../examples/01_basic_fmnist.py
    :language: python
-   :emphasize-lines: 4-5,7-8,12-14,20-21,31, 34-43, 51
+   :emphasize-lines: 4-5, 7-8, 12-14, 20-21, 31, 34-43, 51, 56
    :linenos:
 
-You can try out this basic example yourself. Simply install **Cockpit** and then 
-copy the `example files <https://github.com/f-dangel/cockpit/tree/development/examples>`_
-from the repository.
-
-In this first example, we will run this 
-:download:`example script <../../../examples/01_basic_fmnist.py>`  
-(and the :download:`utility file <../../../examples/utils/utils_examples.py>` 
-for defining the data of the example) via
+Try running this :download:`example script <../../../examples/01_basic_fmnist.py>`  
+via
 
 .. code:: bash
 
   python 01_basic_fmnist.py
 
-In the following, we will break-down and explain each step required to include 
-**Cockpit** to a training loop.
+During every iteration of the training process, **Cockpit** will show you a 
+status screen of the training.
+
+.. code-block:: console
+
+  $ python 01_basic_fmnist.py
+  
+  Step:     1 | Loss: 2.5365
+  [cockpit|plot] Showing current Cockpit.
+  Step:     2 | Loss: 2.1643
+  [cockpit|plot] Showing current Cockpit.
+  Step:     3 | Loss: 1.9929
+  [cockpit|plot] Showing current Cockpit.
+  Step:     4 | Loss: 1.9733
+  [cockpit|plot] Showing current Cockpit.
+  Step:     5 | Loss: 1.6479
+  [cockpit|plot] Showing current Cockpit.
+  [cockpit|plot] Showing current Cockpit. Blocking. Close plot to continue.
+
+which will look something like this
+
+.. image:: ../_static/01_basic_fmnist.png
+        :alt: Preview Cockpit Basic Example
+
+In the following, we will break-down and explain each step of this exampe, which
+also explains what is required to include **Cockpit** to a training loop.
 
 Imports
 =======
@@ -44,7 +77,7 @@ be installed when installing **Cockpit**. We also import the :class:`~cockpit.Co
 and :class:`~cockpit.CockpitPlotter` class which will let us track and then 
 visualize insightful quantities.
 
-To simplify the code snippet, in line 5, we import from a utils file which will 
+To simplify the code snippet, in line 4, we import from a utils file which will 
 provide us with the Fashion-MNIST data.
 
 Defining the Problem
@@ -118,17 +151,28 @@ Plotting the Cockpit
 
 .. literalinclude:: ../../../examples/01_basic_fmnist.py
    :language: python
-   :emphasize-lines: 1
+   :emphasize-lines: 3
    :linenos:
-   :lines: 51
-   :lineno-start: 51
+   :lines: 49-51
+   :lineno-start: 49
 
 At any point during the training, here we do it in every single iteration, the 
 computed metrics can be visualized by calling the plotting functionality of the 
 :class:`~cockpit.CockpitPlotter` via :func:`~cockpit.CockpitPlotter.plot()`.
 
-This will result in a status screen similar to this:
+.. literalinclude:: ../../../examples/01_basic_fmnist.py
+   :language: python
+   :emphasize-lines: 4
+   :linenos:
+   :lines: 53-56
+   :lineno-start: 53
+
+After the final iteration, we will again show the full **Cockpit** view. The option
+``block=True`` allows us to pause our program and inspect the plot for as long as
+we want.
+The final **Cockpit** status screen will look similar to this:
 
 .. image:: ../_static/01_basic_fmnist.png
         :alt: Preview Cockpit Basic Example
 
+Simply closing the plotting window ends the program and this example.