13 papers
Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models
Yuchuan Weng, Chi Liu, Xin Chen +1
A typed decision maps evidence, a question and a fixed set of options to a probability for every option, computed in one forward pass, by reading a language model's logits or a sma…
Student-Guided Teacher Distillation for Efficient LLM Task Routing: Positioning Against Jev-Style System-1 Classifiers
Haifeng Wu, Srinivasan Manoharan, Jian Wan +3
Zero-shot classifiers are useful for routing user requests to specialized LLM tasks, but scoring every request against a large candidate set is expensive: a zero-shot NLI classifie…
Zero-Trust Authorization and Discovery for Enterprise MCP
Huan Li, Yuwei Wang, Srinivasan Manoharan
LLM agents translate natural-language context, which may include attacker-controlled text, into privileged tool calls, so authorization must remain effective even when an agent is…
Task-to-Model Optimization for Enterprise LLM Coding Assistants: A Data-Driven Framework for Cost-Optimal Routing
Srinivasan Manoharan, Junhua Zhao, Fangbo Tu +6
Enterprise AI coding assistants incur substantial inference spend, and naive token-cost minimization often fails to reduce end-to-end cost once retries, escalations, and developer…
Hybrid Semantic Tool Discovery for Enterprise MCP Gateway: Architecture and Implementation
Olympia Saha, Amy Wang, Srinivasan Manoharan
Large language model (LLM) agents invoke external tools to retrieve and reason over information beyond pretrained knowledge. The Model Context Protocol (MCP) standardizes how such…
A Gateway Architecture for Enterprise MCP Authentication: Unifying Heterogeneous Auth, Identity Delegation, and the User / Non-User Persona Problem
Suraj Kumar, Amy Wang, Srinivasan Manoharan
The Model Context Protocol (MCP) has become the de-facto interface for connecting LLM agents to enterprise tools, and adoption has been explosive: within a year, large organization…
RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
Haifeng Wu, Srinivasan Manoharan, Fangbo Tu +2
We present RLM-Cascade, a proxy-layer system that applies speculative decoding at the response level to reduce LLM API costs without requiring model architecture access or a shared…
Domain-Adapted Small Language Models with Hybrid Post-Processing: Achieving Cost-Efficient, Low-Latency Multi-Label Structured Prediction via LoRA Fine-Tuning on Scarce Data
Srinivasan Manoharan, Dilipkumar Nallusamy, Sachin Kumar +1
Deploying frontier large language models (LLMs) for domain-specific structured evaluation tasks incurs prohibitive latency, cost, and data-privacy overhead. We present a hybrid fra…
Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation
Fangbo Tu, Junhua Zhao, Chi Liu +4
Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production enviro…
Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting
Chi Liu, Xin Chen, Xu Zhou +2
Large Language Models (LLMs) have achieved remarkable success, underpinning diverse AI applications. However, they often suffer from performance degradation due to factors such as…
Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models
Ally Qin, Jian Wan, Sarat Mudunuri +1
We evaluate speculative decoding with EAGLE3 as an inference-time optimization for PayPal's Commerce Agent, powered by a fine-tuned llama3.1-nemotron-nano-8B-v1 model. Building on…
Semantic Tool Discovery for Large Language Models: A Vector-Based Approach to MCP Tool Selection
Sarat Mudunuri, Jian Wan, Ally Qin +1
Large Language Models (LLMs) with tool-calling capabilities have demonstrated remarkable potential in executing complex tasks through external tool integration. The Model Context P…
NEMO-4-PAYPAL: Leveraging NVIDIA's Nemo Framework for empowering PayPal's Commerce Agent
Sudhanshu Garg, Andrew Wang, Chaitanya Kulkarni +11
We present the development and optimization of PayPal's Commerce Agent, powered by NEMO-4-PAYPAL, a multi-agent system designed to revolutionize agentic commerce on the PayPal plat…