4 papers
Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Required
Egor Shibaev, Vera Kudrevskaia, Timur Galimzyanov +9
Post-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks (e.g., SWE-bench and LiveCodeBench) as evidence of broad coding capability,…
Attention Mechanism and Heuristic Approach: Context-Aware File Ranking Using Multi-Head Self-Attention
Pradeep Kumar Sharma, Shantanu Godbole, Sarada Prasad Jena +1
The identification and ranking of impacted files within software reposi-tories is a key challenge in change impact analysis. Existing deterministic approaches that combine heuristi…
Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS
Varun Rajesh, Om Jodhpurkar, Pooja Anbuselvan +5
We present a systematic, empirical evaluation of five local large language model (LLM) runtimes on Apple Silicon: MLX, MLC-LLM, llama.cpp, Ollama, and PyTorch MPS. Experiments were…
From Multiple-Choice to Extractive QA: A Case Study for English and Arabic
Teresa Lynn, Malik H. Altakrori, Samar Mohamed Magdy +11
The rapid evolution of Natural Language Processing (NLP) has favoured major languages such as English, leaving a significant gap for many others due to limited resources. This is e…