digantamisra98/Mish

What it solves

Mish is a self-regularized non-monotonic activation function designed to improve the performance of deep neural networks. It aims to overcome the limitations of traditional activation functions like ReLU by providing a smoother profile and a better-conditioned loss landscape, which helps the model converge faster and achieve higher accuracy.

How it works

Mish is defined by the formula $f(x) = x \tanh(\ln(1 + e^x))$. It is a non-monotonic function, meaning it doesn't just increase; it has a small negative dip. This property, combined with its $C^\infty$ continuity, allows for smoother gradients during training. The project hypothesizes that this smoothness acts as a pre-conditioner, making the gradient descent process more stable and easier to optimize.

Who it’s for

This project is for machine learning researchers and engineers who are building deep learning models (particularly in computer vision and language tasks) and want to replace standard activation functions to boost accuracy and reduce training loss.

Highlights

• Improved Accuracy: Demonstrated higher Top-1 and Top-5 accuracy across various architectures like ResNet, DarkNet, and DenseNet on ImageNet.
• Smoother Loss Landscapes: Visualizations show that Mish provides a more well-conditioned and easier-to-optimize loss landscape compared to ReLU and Swish.
• Broad Integration: Integrated into major frameworks including PyTorch, TensorFlow JS, MXNet, and OpenVino.
• SOTA Performance: Contributed to state-of-the-art results in object detection on the MS-COCO dataset using CSP-p7 + Mish.