October 11-18, 2024
+October 14-18, 2024
Date: Deadline is Friday October 18 at midnight
The aim of the exercises this week is to get started with implementing a neural network. There are a lot of technical and finicky parts of implementing a neutal network, so take your time.
-This week, you will implement only the feed-forward pass and updating the network parameters with simple gradient descent, the gradient will be computed using autograd using code we provide. Next week, you will implement backpropagation. We recommend that you do the exercises this week by editing and running this notebook file, as it includes some checks along the way that you have implemented the pieces of the feed-forward pass correctly, and running small parts of the code at a time will be important for understanding the methods.
-If you have trouble running a notebook, you can run this notebook in google colab instead (https://colab.research.google.com/drive/1OCQm1tlTWB6hZSf9I7gGUgW9M8SbVeQu#offline=true&sandboxMode=true), an updated link will be provided on the course discord (you can also send an email to k.h.fredly@fys.uio.no if you encounter any trouble), though we recommend that you set up VSCode and your python environment to run code like this locally.
+This week, you will implement the entire feed-forward pass of a neural network! Next week you will compute the gradient of the network by implementing back-propagation manually, and by using autograd which does back-propagation for you (much easier!). Next week, you will also use the gradient to optimize the network with a gradient method! However, there is an optional exercise this week to get started on training the network and getting good results!
+We recommend that you do the exercises this week by editing and running this notebook file, as it includes some checks along the way that you have implemented the pieces of the feed-forward pass correctly, and running small parts of the code at a time will be important for understanding the methods.
+If you have trouble running a notebook, you can run this notebook in google colab instead (https://colab.research.google.com/drive/1zKibVQf-iAYaAn2-GlKfgRjHtLnPlBX4#offline=true&sandboxMode=true), an updated link will be provided on the course discord (you can also send an email to k.h.fredly@fys.uio.no if you encounter any trouble), though we recommend that you set up VSCode and your python environment to run code like this locally.
+First, here are some functions you are going to need, don’t change this cell. If you are unable to import autograd, just swap in normal numpy until you want to do the final optional exercise.
import autograd.numpy as np # We need to use this numpy wrapper to make automatic differentiation work later
@@ -762,9 +765,9 @@ Exercise 4 - Custom activation for each layerfeed_forward function which accepts a list of activation functions as an argument, and which evaluates these activation functions at each layer.
-def feed_forward(input, layers, activations):
+def feed_forward(input, layers, activation_funcs):
a = input
- for (W, b), activation in zip(layers, activations):
+ for (W, b), activation_func in zip(layers, activation_funcs):
z = ...
a = ...
return a
@@ -772,21 +775,22 @@ Exercise 4 - Custom activation for each layer
network_input_size = ...
layer_output_sizes = [...]
-activations = [...]
+activation_funcs = [ReLU, ReLU, sigmoid]
layers = ...
x = np.random.randn(network_input_size)
-feed_forward(x, layers, activations)
+feed_forward(x, layers, activation_funcs)
+c) How does the output of the network change if you use sigmoid in the hidden layers and ReLU in the output layer?
Exercise 5 - Processing multiple inputs at once¶
@@ -816,9 +820,9 @@ Exercise 5 - Processing multiple inputs at onceinputs = np.random.rand(1000, 4)
-def feed_forward_batch(inputs, layers, activations):
+def feed_forward_batch(inputs, layers, activation_funcs):
a = inputs
- for (W, b), activation in zip(layers, activations):
+ for (W, b), activation_func in zip(layers, activation_funcs):
z = ...
a = ...
return a
@@ -831,11 +835,11 @@ Exercise 5 - Processing multiple inputs at once
network_input_size = ...
layer_output_sizes = [...]
-activations = [...]
+activation_funcs = [...]
layers = create_layers_batch(network_input_size, layer_output_sizes)
x = np.random.randn(network_input_size)
-feed_forward_batch(inputs, layers, activations)
+feed_forward_batch(inputs, layers, activation_funcs)
@@ -881,7 +885,7 @@ Exercise 6 - Predicting on real data
@@ -894,7 +898,7 @@ Exercise 6 - Predicting on real data
-predictions = feed_forward_batch(inputs, layers, activations)
+predictions = feed_forward_batch(inputs, layers, activation_funcs)
@@ -908,27 +912,37 @@ Exercise 6 - Predicting on real data
-Exercise 6 - Training on real data¶
+
+Exercise 7 - Training on real data (Optional)¶
To be able to actually do anything useful with your neural network, you need to train it. For this, we need a cost function and a way to take the gradient of the cost function wrt. the network parameters. The following exercises guide you through taking the gradient using autograd, and updating the network parameters using the gradient. Feel free to implement gradient methods like ADAM if you finish everything.
-The cross-entropy loss function can evaluate performance on classification tasks. It sees if your prediction is “most certain” on the correct target.
+Since we are doing a classification task with multiple output classes, we use the cross-entropy loss function, which can evaluate performance on classification tasks. It sees if your prediction is “most certain” on the correct target.
-from autograd import grad
+def cross_entropy(predict, target):
+ return np.sum(-target * np.log(predict))
-def cost(input, layers, activations, target):
- predict = feed_forward_batch(input, layers, activations)
+def cost(input, layers, activation_funcs, target):
+ predict = feed_forward_batch(input, layers, activation_funcs)
return cross_entropy(predict, target)
-
-
-def cross_entropy(predict, target):
- return np.sum(-target * np.log(predict))
+
+
+
+
+To improve our network on whatever prediction task we have given it, we need to use a sensible cost function, take the gradient of that cost function with respect to our network parameters, the weights and biases, and then update the weights and biases using these gradients. To clarify, we need to find and use these
+
+\[
+\frac{\partial C}{\partial W}, \frac{\partial C}{\partial b}
+\]
+Now we need to compute these gradients. This is pretty hard to do for a neural network, we will use most of next week to do this, but we can also use autograd to just do it for us, which is what we always do in practice. With the code cell below, we create a function which takes all of these gradients for us.
+
+
+from autograd import grad
gradient_func = grad(
- cross_entropy, 1
-) # Taking the gradient wrt. the second input to the cost function
+ cost, 1
+) # Taking the gradient wrt. the second input to the cost function, i.e. the layers
@@ -937,7 +951,9 @@ Exercise 6 - Training on real datagradient_func function to take the gradient of the cross entropy wrt. the weights and biases of the network. Check the shapes of what’s inside. What does the grad func from autograd actually do?