digantamisra98/Mish
Official Repository for "Mish: A Self Regularized Non-Monotonic Neural Activation Function" [BMVC 2020]
What it solves
Mish 是一種自我正則化的非單調激活函數，旨在提升深度神經網路的效能。它旨在克服傳統激活函數（如 ReLU）的限制，提供更平滑的特性與更佳條件的損失曲面，從而協助模型更快收斂並達到更高的準確度。
How it works
Mish 由公式 $f(x) = x \tanh(\ln(1 + e^x))$ 定義。它是一個非單調函數，意味著它不僅僅是單向遞增，還會出現小幅負向凹陷。此特性結合其 $C^\infty$ 連續性，使得訓練過程中的梯度更為平滑。該專案假設此平滑性可作為預條件，使梯度下降過程更穩定且更易於優化。
Who it’s for
此專案適用於從事機器學習研究與工程的使用者，特別是構建深度學習模型（尤其是電腦視覺與語言任務）的研究者與工程師，並希望以取代標準激活函數的方式提升準確度與降低訓練損失。
Highlights
- Improved Accuracy: Demonstrated higher Top-1 and Top-5 accuracy across various architectures like ResNet, DarkNet, and DenseNet on ImageNet.
- Smoother Loss Landscapes: Visualizations show that Mish provides a more well-conditioned and easier-to-optimize loss landscape compared to ReLU and Swish.
- Broad Integration: Integrated into major frameworks including PyTorch, TensorFlow JS, MXNet, and OpenVino.
- SOTA Performance: Contributed to state-of-the-art results in object detection on the MS-COCO dataset using CSP-p7 + Mish.