LLMGW application

LLMGW is built as docker containers which can be deployed on Azure or AWS providing a cloud-native, scalable foundation with enterprise-grade security and operational capabilities. The platform consists of four primary container applications that work together to deliver comprehensive AI gateway functionality.

LLMGW proxy

Function: Core API gateway handling all AI provider communications

Features: Request routing, load balancing, authentication, and response processing

Scaling: Available auto-scaling based on request volume and latency requirements

Access: Internal application only allowed to be accessed from selected networks with all requests being authenticated

Admin Portal

Function: Web-based management interface for platform configuration

Features: User management, project configuration, budget controls, and system monitoring

Access: Secure web interface secured by SSO access with RBA

Grafana Dashboards

Function: Advanced analytics and visualization platform

Features: Cost monitoring, alerting, and performance metrics visualization

Integration: Connected to all telemetry and metrics data sources

Access: Secure web interface secured by SSO access supporting RBAC

Open Telemetry Metrics Emitter

Function: Standardized observability and metrics collection

Features: Distributed tracing, custom metrics, and performance monitoring

Standards: Full Open Telemetry compliance