Design Principles

One API for All LLMs

Provide a unified interface representing multiple LLMs to reduce users' development costs and minimize service risks due to model upgrades. This includes complementing for providers that don't support Batch or Stream functionalities, ensuring a consistent One API experience for all users.
Cost Efficiency

Through agreements with LLM Providers, we offer services at a more affordable rate than direct model usage. We also strive to route model traffic efficiently to further reduce model usage costs.
Service Reliability

If one of the providers' API faces quota issues or experiences an outage, the system will route to another provider's model using a fallback logic. This rerouting occurs automatically and will persist until the original model stabilizes. As a result, the LLM service is prepared for sudden disruptions and holds an advantage in terms of high availability compared to other services.
Unified Observability and Safety

Addressing monitoring challenges with LLM calls becomes complex when integrating various providers. Each LLM Provider has its unique metrics, increasing the monitoring overhead and associated costs. Multi-LLM GW simplifies this by offering model-specific pricing policies, integrated billing, and cost-optimized LLM selections. Moreover, the platform emphasizes safety by incorporating protective mechanisms, safeguards against potential issues, and ensuring data privacy and integrity for users.
Integrated User Authentication and Access Management

Multi-LLM GW supports unified authentication and organizational management for LLM usage. Managing different user and organizational authentications across providers can be challenging. With Multi-LLM GW, you can manage LLM models for multiple organizations or users through integrated user authentication and organizational management.
Telecom Specific LLM and LLM Agent

Support for LLMs tailored to Telecom services is provided. Domain-specific models offer more natural interactions and reduced hallucination compared to foundation models. Additionally, the Telco-specific LLM Agent facilitates easy integration with Telecom services (OSS/BSS), simplifying the development process of Telco-specific LLM services.

Functional Components

Features - Phase 0

LLM Inference API
An Application Programming Interface that provides a unified gateway to access and manage various LLMs across different providers. This ensures a standardized way for users to interact with different LLMs without having to change their codebase. For providers that do not have Batch and Stream functionalities, the Gateway implements these features, allowing users to utilize them in a provider-agnostic manner. This ensures a seamless experience for users regardless of the underlying LLM provider's capabilities.
This addition emphasizes the adaptability and user-centric design of the Multi-LLM Gateway.
LLM Providers
Different companies or entities that offer LLM services. The Multi-LLM Gateway integrates with various providers to give users a wide array of LLM choices.
Authn / authz
Stands for Authentication and Authorization. This component ensures that only authorized users can access and make requests to the LLMs. It verifies the identity of users and checks their permissions before granting access.

Features - Phase 1

Quota Management
Manages the allocation and usage of resources. This ensures that users do not exceed their allocated usage limits for a particular LLM and helps distribute resources efficiently.
Monitoring and Safeguard
This subsystem continuously tracks and analyzes the operational metrics and health indicators of the LLMs, ensuring optimal performance and responsiveness. Utilizing a suite of advanced diagnostic tools and real-time analytics, it provides a comprehensive view of the system's status. The integrated safeguard mechanism is built upon a robust set of protocols designed to act preemptively. It establishes a defense-in-depth strategy, identifying potential weak points and reinforcing them to prevent any system vulnerabilities or threats that might compromise the stability or security of the LLMs.
Web console
A user-friendly graphical interface where users can manage, monitor, and interact with various functionalities of the Multi-LLM Gateway.

Features - Phase 2

Logging Pipeline for LLM invocation
A systematic process that captures, stores, and manages logs related to LLM requests and responses. This helps in debugging, monitoring, and analyzing the performance of the LLMs.
Prompt Management & Experiment
Central to efficient LLM interaction is the ability to craft and refine prompts that elicit the desired outputs. Our Prompt Management tool empowers users to store, edit, and organize their prompts in a user-friendly interface, ensuring optimal interactions with the LLMs. But it doesn't stop there. With the experimentation feature, users can test these prompts across various models, gauging the effectiveness and relevance of each response. To streamline the decision-making process, our platform offers a side-by-side comparison tool. This allows users to visualize and contrast results from different LLMs in real-time. By analyzing these juxtaposed outcomes, users can make informed decisions about which model aligns best with their specific objectives and requirements.

Future Additions

Fallback & Smart LLM Router
In case of an LLM failure or disruption, the system will automatically reroute the requests to another available and functional LLM, ensuring high availability. The smart router also optimizes routing based on various parameters to improve efficiency.
Drift Detection
In the ever-evolving landscape of machine learning models, ensuring that predictions remain consistent and accurate over time is paramount. The Drift Detection system is specifically engineered to monitor the outputs of the LLMs continuously. It watches for signs that the model's predictions are starting to 'drift' away from what's expected or from previously established baselines. By employing statistical tests and anomaly detection algorithms, this feature can identify subtle shifts in model behavior, often before they become noticeable issues. When such deviations are detected, alerts are triggered, allowing timely interventions and recalibrations to ensure the LLMs maintain their high standards of accuracy and reliability.