Analysis

General notes

The following paragraphs explain the functionalities of analyze.py and some useful examples of how to use it to run analysis on networks in practice. The functionality can mainly be divided into the following three categories:

1. Analysis using white noise inputs (RF mapping)
2. Analysis using input generated by the agent-environment loop (Environment responses)
3. Analysis using CIFAR and MNIST entries as input (Dataset responses)

Since some analyses are not extremely computationally expensive, namely mapping RFs and simulating short 1000 step experiences, these are always run by default when using analyze.py. These are also the most convenient to track the learning progress while a model is being trained, so can be routinely used.

For checking progress while training the following example command can be used:

python -m analyze --env=retinal_cifar_gathering_01 --algo=APPO --encoder_custom=lindsey --experiment=lindsey109

This should output a gif like this one:

And a plot of receptive fields like this:

For more than just simple analysis see the 'Environment responses' and 'Dataset responses' below.

Before going into the details of the analysis, this is the general Lindsey architecture that the analysis is optimised for, so its useful to refer back to this image to know where the 'conv', 'FC' and 'RNN' are within the architecture:

RF mapping

As mentioned in the paragraph above, this is always run by default. It plots the receptive fields of the convolutional layers (see figure above). It works very well, the only issues might be that the GPU runs out of memory when running a lot of processes in parallel (in this case the btchsz in spike_triggered_average()should be changed to a smaller number, but this of course affects the noisiness of the RFs).

Environment responses

During behaviour responses on all levels of the network are analyzed. Simulating 1000 step experiences (mentioned above) is just the activation of the 'Pixels' layer (which is done by default). The rest of the analysis requires the following command line input: --analyze_acts=environment. By default the analysis will run for 1000 environment steps, unless a command line argument is input, for example --analyze_max_num_frames=100000. This will however only affect the visualizations of fc and rnn layers.

Conv layers

Currently the activations of the convolutional layers are visualized by plotting gifs of activations, which is computationally quite slow. Because of this, the limit on the number of steps is hard-coded to be limited to 1000 even if analyze_max_num_frames is set to be higher. The output of this script should look something like this, for each different conv layer:

FC layer

Dimensionality reduction is used for visualizing the activations of the fully connected layer (output of the encoder) during simulations. PCA is performed by an older script that scatters PC1 against PC2 of the FC activations color-coded by the value predicted by the critic. It also plots a scree plot for pixels and FC activations as well as FC PC1 in time, plotted together with some other variables such as value predicted by the critic, PC1 of pixels and health of the player. The time plot makes sense for shorter analyze_max_num_frames. Example output (notice for example that PC1 is predictive of positive health changes):

Additionally the activations are also visualized using tSNE, which is recommended for longer analyze_max_num_frames, ideally 100k steps. If shorter ones are used, embedding just tend to cluster based on time (and of course in this case the value for adjacent states is also expected to be similar). This is plotted side by side with an embedding of RNN activations. Example of nice-looking output:

Another visualisation I added recently is plotting the activation of all neurons over time as a 'raster plot' sorted by their 1-D tSNE embedding. This is just nice to visualise some patterns in the raw activations. I also added the times at which the agent pics up a stimulus (dashed grey lines). For example in trained networks we can see that the output of the encoder is usually silent/the activity is sparse. Neurons mostly become active only before picking up a stimulus. Example visualisation:

This same analysis can also be done for RNN (see below) and flattened conv layers (not shown), but FC seems most informative, conv is probably more similar to pixels, especially earlier layers.

RNN layer

See paragraph above. An additional issue with the RNN is that at the beginning of running analyze.py the activation of its hidden state is initialized. We thus crop the first 500 steps of the environment to avoid artefacts that arise due to this.

The script also outputs the same analysis/visualisation of activations as above sorted by tSNE. Here the data usually looks more symmetric across half neurons, for example:

Attribution analysis (simulation)

Attribution during behaviour is implemented in the rl_encoder_analyze_sim.ipynb notebook. For now plot attribution of an output neuron wrt the input image (changes in which pixels of the image will cause the biggest change in the activation of an output neuron). This is computed using guided backprop (same goes for attribution on dataset entries, see below). Here the output neurons represent either the scalar value (if critic) or the probability of action(s) (6 neurons). The notebook outputs some example FOVs with attribution, looking like this (example for neuron index 5 (walking backward)):

Note: Here the color scaling is independent on each frame.

It also generates a .gif of attribution during episodes of experience, for example (also for neuron index 5 (walking backward)):

Note: Here the color scaling is fixed to the max/min across the whole movie.

We could construct such a .gif for each one of the actions, as well as for the value predicted by the critic.

Dataset responses

This analysis is performed by adding cfg.analyze_acts == 'dataset' to the call of analyze.py.

Visualisation of encoder output

This is a different line of analysis, where the network will be probed for responses to stimuli taken from a particular dataset. Currently these are only MNIST and CIFAR(10), since these are the ones that have corresponding .wad files. To run this analysis one should set: --analyze_acts=dataset, MNIST can be chosen by setting: --analyze_ds_name=MNIST (note: this parameter is set to CIFAR by default).
What this analysis does is it performs a forward pass of the whole dataset and then looks at activations using similar methods as described above. At this point only the actviation of the FC layer is implemented, but in principle the same analysis could easily be extended to different conv layers. Adapting it for the RNN would be difficult, since it requires a hidden state that would normally come from a previous time-step, and initialisation causes artefacts.
Since MNIST and CIFAR images are of different shape than the input to the agent during training, they are padded with an example background from the simulations:

Giving the final stimulus:

Running the script should give embeddings similar to the following:

Additional notes on dataset responses:

1. Color-coding represents the stimulus class sorted by its contribution to the agent's health. Note that this is currently hard-coded for the two specific assignments that are specified in the accompanying .wad files.
2. Responses to dataset images in such a way might not be the best strategy for probing responses, especially for negative stimuli, since (judging by behaviour) the agents rarely walk up close to a negative stimulus, so such images would appear rarely in the agent's actual observations.

Classification (decoding) analysis

To get a more quantitative assessment, I also implemented a classification analysis on this representation. This is equivalent as retraining the network and keeping all convolutional and FC layer fixed. I implemented this both as a 10-class classification, as well as binary (positive vs. negative). The accuracy is output to the terminal and also saved in a text file analyze_class_score.txt, a chance level is also output, for example:

multi classification scores:
  -Train: 0.3198
  -Test: 0.3096

permuted multi classification scores:
  -Train: 0.1181
  -Test: 0.1033

bin classification scores:
  -Train: 0.7573
  -Test: 0.7572

permuted bin classification scores:
  -Train: 0.5119
  -Test: 0.5026

It would be interesting to look into this a bit further and more systematically, for now I can only say that (obviously) the accuracy increases during training of the RL agent and that in the 'distractors' task, the network where MNIST are distractors performs much worse than the one where apples are the distractors (which also makes sense). I think there is still a lot that can be done along those lines though.

Attribution analysis (dataset)

This is implemented both for the networks trained on classification (class_encoder_analyze.ipynb) and RL (rl_encoder_analyze_ds.ipynb). Both of these scripts for now plot attribution of an output neuron wrt the input image (changes in which pixels of the image will cause the biggest change in the activation of an output neuron). For the classification the output neurons represent each class and in the actor-critic they represent either the scalar value (if critic) or the probability of action(s) (6 neurons). Example output of this plot for actor neuron index 5 (walking backward) should look like this:

Action analysis

This is also done by rl_encoder_analyze_ds.ipynb. Essentially its just a forward pass of the whole dataset and then visualising the last layer. This is either done before passing through softmax non-linearity for making PCA embedding visualisations looking like this for example:

Or a better visualisation is showing the probability of taking an action given a stimulus class. These are just the activations from above passed through a softmax non-linearity to get a distribution across actions. For example this is a visualisation for the probability of taking an action for the stimulus being close up in front of the agent.

We can see that for positive stimuli the agent will walk forward and for negative backward. The notebook also allows the user to change the positioning and the scaling of the stimulus within the FOV.