#1010th of 14 in Gateways
optillm
OpenAI-compatible proxy applying inference-time reasoning techniques
- Stars
- 4.3k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
Overview
OptiLLM is an OpenAI-compatible proxy (pip or Docker, port 8000) that applies inference-time techniques such as mixture of agents, N-sample selection, self-consistency, MCTS, CePO and MARS to any upstream model, selected by a model-name prefix like moa-gpt-4o-mini. Plugins add an MCP client, memory, PII anonymization, code execution, JSON outputs and provider failover; upstreams are OpenAI, Cerebras, Azure or anything LiteLLM supports.
Who it is for: developers squeezing more reasoning accuracy from existing models
Strengths
- 20+ techniques selected by model-name prefix, e.g. moa-gpt-4o-mini
- Per-technique benchmarks listed (MARS +30 points on AIME 2025 with Gemini 2.5 Flash Lite)
- Plugins for MCP client, memory, PII anonymization, code execution, JSON output
- Works with any OpenAI-compatible endpoint; LiteLLM covers other providers
Weaknesses
- Techniques multiply upstream calls (bon, MoA, MCTS), raising cost and latency
- Runs on Flask's development server by default
- Decoding techniques (cot_decoding, AutoThink) need the local inference path
- Web search plugin drives Chrome through Selenium
What it needs
- GPU optional
- Docker + Compose
- Models: OpenAI, Cerebras, Azure OpenAI, any OpenAI-compatible endpoint, LiteLLM providers, local models via the built-in inference server
- port 8000
Also in Gateways
See all 14| Rank | Project | Score |
|---|---|---|
| 1 | OmniRouteOpenAI-compatible gateway that routes requests across hundreds of AI providers | 86 out of 100 |
| 2 | LiteLLMOpenAI-format gateway and Python SDK for calling 100+ LLM providers | 84 out of 100 |
| 3 | freellmapiOpenAI-compatible router that fails over across free LLM provider tiers | 76 out of 100 |
| #4 | ContextForge MCP GatewayRegistry and proxy federating MCP, A2A, REST and gRPC behind one endpoint | 68 out of 100 |
| #5 | HigressEnvoy-based API gateway with LLM proxy plugins and MCP server hosting | 61 out of 100 |