digantamisra98/Mish
Official Repository for "Mish: A Self Regularized Non-Monotonic Neural Activation Function" [BMVC 2020]
What it solves
Mish 是一种自我正则化的非单调激活函数，旨在提升深度神经网络的性能。它旨在克服传统激活函数（如 ReLU）的局限性，提供更平滑的特性和更好条件的损失曲面，从而帮助模型更快收敛并实现更高的准确率。
How it works
Mish 通过公式 $f(x) = x \tanh(\ln(1 + e^x))$ 定义。它是一个非单调函数，意味着它不仅仅是单向递增，还会出现小幅负向凹陷。这一特性结合其 $C^\infty$ 连续性，使得训练过程中的梯度更加平滑。项目假设这种平滑性充当预条件器，使梯度下降过程更稳定且更易优化。
Who it’s for
该项目面向机器学习研究人员和工程师，尤其是构建深度学习模型（特别是计算机视觉和语言任务）的用户，旨在通过替换标准激活函数来提升准确率并降低训练损失。
Highlights
- Improved Accuracy: Demonstrated higher Top-1 and Top-5 accuracy across various architectures like ResNet, DarkNet, and DenseNet on ImageNet.
- Smoother Loss Landscapes: Visualizations show that Mish provides a more well-conditioned and easier-to-optimize loss landscape compared to ReLU and Swish.
- Broad Integration: Integrated into major frameworks including PyTorch, TensorFlow JS, MXNet, and OpenVino.
- SOTA Performance: Contributed to state-of-the-art results in object detection on the MS-COCO dataset using CSP-p7 + Mish.