This repository presents a unique approach towards classifying the classic MNIST dataset using a custom neural network implemented in PyTorch. This network incorporates custom weight initialization, custom layers, custom activation function, and custom loss function to improve upon the vanilla implementation.
-
Custom Weight Initialization : A method that samples random values from Beta Distribution and uses them as initial weights.
The weight initialization function,
beta_init
, will follow the equation:where
w_i
are the weights,n
is the total number of weights, andBeta
represents the Beta Distribution with shape parameters\alpha
and\beta
. -
Custom Neural Network Layer : A layer performing a unique operation as shown in the class
CustomLayer
. The operation this layer performs can be represented as:where
O_i
is the output of the ith neuron,w_{j,i}
represents the weight from the jth neuron of the previous layer to the ith neuron of the current layer,x_j
is the jth input,b_i
is the bias for the ith neuron, andn
is the total number of inputs. -
Custom Activation Function : A unique activation function
CustomActivation
combining three common activation functions (tanh
,sigmoid
,relu
) controlled by trainable parameters. The operation this function performs can be represented as:where
y
is the output,x
is the input,\alpha_1
,\alpha_2
, and\alpha_3
are trainable parameters, andtanh
,sigmoid
, andrelu
are the respective activation functions. -
Custom Loss Function : A custom loss function
CustomLoss
designed to work well with the unique aspects of the neural network. The loss function can be represented as:where
L
is the loss,yhat_i
is the true probability andy_i
is the predicted probability for the ith instance, andn
is the total number of instances. -
Dense Neural Network : The neural network is built using the custom objects defined above in conjunction with dense layers. It is trained on the MNIST dataset.
After training the custom neural network for num_epochs = 10
, batch_size = 100
, and learning_rate = 0.001
, the model achieved an accuracy of 96.91%
.
Please refer to the Jupyter notebook for more information on how to use these custom classes and the specific implementation details.