IN3063 Mathematics and Programming of AI

Task:

Task 1. Implementing linear and ReLu layers

For this task, you should implement the forward and backward passes for linear layers and ReLU activations.

Task 2. Implementing dropout

For this task, you should implement inverted dropout. Forward and backward passes should be implemented here.

Note: Since the test performance is critical, it is also preferable to leaving the forward pass unchanged at test time. Therefore, in most implementations inverted dropout is employed to overcome the undesirable property of the original dropout.

Task 3. Implementing softmax classifier

For this task, you should implement softmax loss and gradients. Explain numerical issues when calculating the softmax function.

Task 4. Implementing fully-connected NN

For this task, you should implement a fully-connected NN with arbitrary number of hidden layers, ReLu activation, softmax classification and optional dropout. This task is about reusing your implementations from Task 1 to Task 3. In addition, you will add a L2 regularizer. Report the parameters used (update rule, learning rate, decay, epochs, batch size) and include the plots in your report.

Task 5. Optimization of hyper-parameters

For this task, you should optimize the hyper-parameters of a fully-connected NN with your chosen emotion classification dataset. Here you should implement SGD with momentum. You need to select a performance measure which will be used to compare the performance of the different parameters on the validation set.

Hyper-parameter optimization steps:

1. Choose a proper network architecture as starting point. Define a momentum and a learning rate.

2. Try a different stopping criterion or a different learning rate update schedule.

3. Optimize the learning rate (disable regularization). Report how you found a good initial value for learning rate. Include the plot for training loss and report training and validation classification error.

4. Apply dropout and see if there is any improvement in the validation performance.

5. Compare L2 performance with dropout.

6. Finally, optimize the topology of the network (the number of hidden layers and thenumber of neurons in each hidden layer).

Report

Your final report should present the research questions that you investigate, and the experiments you designed and carried out. You should provide clear presentation of the methods (network architectures, learning schedules, etc.) that were used, and an outline of how they were implemented. You should present the results clearly and concisely and provide a discussion of the results, with conclusions related to the research questions. The conclusions section might propose some further work based on the results of this coursework.