# Extending the Forward Forward Algorithm

07/09/2023

∙

by [Saumya Gandhi, et al.](/content/profile/saumya-gandhi/index.html)

∙

The Forward Forward algorithm, proposed by Geoffrey Hinton in November 2022, is a novel method for training [neural networks](/content/machine-learning-glossary-and-terms/neural-network/index.html) as an alternative to [backpropagation](/content/machine-learning-glossary-and-terms/backpropagation/index.html). In this project, we replicate Hinton's experiments on the MNIST dataset, and subsequently extend the scope of the method with two significant contributions.

First, we establish a baseline performance for the Forward Forward network on the IMDb movie reviews dataset. As far as we know, our results on this [sentiment analysis](/content/machine-learning-model/sentiment-analysis/index.html) task marks the first instance of the algorithm's extension beyond [computer vision](/content/machine-learning-glossary-and-terms/computer-vision/index.html).

Second, we introduce a novel pyramidal optimization strategy for the loss threshold - a [hyperparameter](/content/machine-learning-glossary-and-terms/hyperparameter/index.html) specific to the Forward Forward method. Our pyramidal approach shows that a good thresholding strategy causes a difference of up to 8. Lastly, we perform visualizations of the trained parameters and derived several significant insights, such as a notably larger (10-20x) mean and [variance](/content/machine-learning-glossary-and-terms/variance/index.html) in the weights acquired by the Forward Forward network.

[READ FULL TEXT](http://arxiv.org/pdf/2307.04205v1)