9 papers
A Gateway Architecture for Enterprise MCP Authentication: Unifying Heterogeneous Auth, Identity Delegation, and the User / Non-User Persona Problem
Suraj Kumar, Amy Wang, Srinivasan Manoharan
The Model Context Protocol (MCP) has become the de-facto interface for connecting LLM agents to enterprise tools, and adoption has been explosive: within a year, large organization…
Task-to-Model Optimization for Enterprise LLM Coding Assistants: A Data-Driven Framework for Cost-Optimal Routing
Srinivasan Manoharan, Junhua Zhao, Fangbo Tu +6
Enterprise AI coding assistants incur substantial inference spend, and naive token-cost minimization often fails to reduce end-to-end cost once retries, escalations, and developer…
RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
Haifeng Wu, Srinivasan Manoharan, Fangbo Tu +2
We present RLM-Cascade, a proxy-layer system that applies speculative decoding at the response level to reduce LLM API costs without requiring model architecture access or a shared…
Domain-Adapted Small Language Models with Hybrid Post-Processing: Achieving Cost-Efficient, Low-Latency Multi-Label Structured Prediction via LoRA Fine-Tuning on Scarce Data
Srinivasan Manoharan, Dilipkumar Nallusamy, Sachin Kumar +1
Deploying frontier large language models (LLMs) for domain-specific structured evaluation tasks incurs prohibitive latency, cost, and data-privacy overhead. We present a hybrid fra…
Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation
Fangbo Tu, Junhua Zhao, Chi Liu +4
Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production enviro…
Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting
Chi Liu, Xin Chen, Xu Zhou +2
Large Language Models (LLMs) have achieved remarkable success, underpinning diverse AI applications. However, they often suffer from performance degradation due to factors such as…