yureii1996/cek-probe-model

What it solves

This project provides a set of Python scripts to verify the identity and logic of AI models hosted on OpenAI-compatible API endpoints. It helps users ensure that the model they are accessing is actually the one claimed by the provider, detecting potential mismatches between the model label and its actual behavior.

How it works

The tool takes a list of models from an endpoint and runs simple "probe" requests. It sends inference requests to the model and analyzes the responses to see if the identity and logic match the expected model family. For example, it can flag a "suspect" model if a label says "DeepSeek" but the model identifies itself as "GPT".

Who it’s for

Developers and API users who want to verify the authenticity of the model endpoints they are using to ensure they aren't receiving responses from a different, cheaper, or unexpected model.

Highlights

• Identity Verification: Detects architecture mismatches where the model's self-reported identity differs from its label.
• OpenAI API Compatibility: Works with any endpoint that follows the OpenAI API standard.
• Multiple Testing Modes: Includes scripts for fast probing, as well as scripts with built-in retries and delays for specific providers like InferHub and CFRouter.
• Environment-based Security: Uses environment variables to handle API keys securely.