digantamisra98/Mish

What it solves

Mish is a self-regularized non-monotonic activation function designed to improve the performance of deep neural networks. It aims to provide better accuracy and lower loss by creating a smoother, more well-conditioned loss landscape that is easier for models to optimize compared to traditional functions like ReLU or Swish.

How it works

Mish is defined by the mathematical formula $f(x) = x \tanh(\ln(1 + e^x))$. It is a smooth, continuous function with a parametric order of continuity of $C^\infty$. The project hypothesizes that its derivative acts as a pre-conditioner, making gradients smoother and improving the flow of information through the network.

Who it’s for

This project is for machine learning researchers and practitioners who are building deep neural networks (particularly in computer vision and language tasks) and looking for alternative activation functions to improve model convergence and accuracy.

Highlights

• Improved Accuracy: Demonstrated gains in Top-1 and Top-5 accuracy across various architectures like ResNet, DarkNet, and DenseNet on the ImageNet dataset.
• Smoother Loss Landscapes: Visualizations show a more stable and easier-to-optimize loss landscape compared to ReLU and Swish.
• Broad Integration: Integrated into major frameworks including PyTorch, MXNet, TensorFlow JS, and OpenVino.
• SOTA Performance: Contributed to state-of-the-art results in object detection on the MS-COCO test-dev dataset using CSP-p7 + Mish.