vc1492a/PyNomaly

What it solves

PyNomaly is a Python implementation of Local Outlier Probabilities (LoOP), a method for detecting outliers in datasets. Unlike traditional methods that provide a raw score, LoOP provides normalized scores between 0 and 1, which can be directly interpreted as the probability that a specific data point is an outlier.

How it works

The library calculates outlier probabilities through a multi-step pipeline:

1. Nearest Neighbor Distances: It finds the k nearest neighbors for each observation.
2. Standard Distance: It computes the root mean square of these distances to capture average neighbor distance.
3. Probabilistic Distance: The standard distance is scaled by an "extent" parameter (lambda) to control sensitivity.
4. Probabilistic Local Outlier Factor (PLOF): It calculates the ratio of an observation's probabilistic distance to the mean probabilistic distance of its neighbors.
5. Normalized PLOF (nPLOF): A normalization constant is applied to ensure scores are comparable across different clusters.
6. Local Outlier Probability (LoOP): A Gaussian error function is applied to the final ratio to produce a probability between 0 and 1.

Who it’s for

It is designed for data scientists and practitioners who need a density-based anomaly detection tool that provides interpretable, probabilistic results rather than arbitrary scores.

Highlights

• Interpretable Scores: Outputs are normalized probabilities [0, 1].
• Cluster-Aware: Supports optional cluster labels to calculate outlier probabilities relative to specific groups rather than the entire dataset.
• Performance Optimizations: Integrates with Numba for JIT compilation and multi-core parallelism, as well as Scipy for optimized distance computations.
• Flexible Distance Metrics: Allows users to provide their own distance and neighbor matrices to use metrics other than Euclidean distance.
• Streaming Support: Includes a modified approach for streaming data to avoid expensive full-dataset refitting.