EvalAI: Towards Better Evaluation Systems for AI Agents
arXiv:1902.03570
Abstract
We introduce EvalAI, an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence algorithms (AI) at scale. EvalAI is built to provide a scalable solution to the research community to fulfill the critical need of evaluating machine learning models and agents acting in an environment against annotations or with a human-in-the-loop. This will help researchers, students, and data scientists to create, collaborate, and participate in AI challenges organized around the globe. By simplifying and standardizing the process of benchmarking these models, EvalAI seeks to lower the barrier to entry for participating in the global scientific effort to push the frontiers of machine learning and artificial intelligence, thereby increasing the rate of measurable progress in this domain.
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Residual Learning for Image Recognition
- OpenML: networked science in machine learning
- Benchmarking Deep Reinforcement Learning for Continuous Control
- Fully Convolutional Networks for Semantic Segmentation
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- fastMRI: An Open Dataset and Benchmarks for Accelerated MRI
- AI2-THOR: An Interactive 3D Environment for Visual AI
- Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
- Towards Diverse and Natural Image Descriptions via a Conditional GAN
- Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
Cited by in corpus (10)
- DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
- The Information Retrieval Experiment Platform
- Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity
- The VoxCeleb Speaker Recognition Challenge: A Retrospective
- ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents
- Soft Expert Reward Learning for Vision-and-Language Navigation
- ECOL-R: Encouraging Copying in Novel Object Captioning with Reinforcement Learning
- Bridging Scene Understanding and Task Execution with Flexible Simulation Environments
- NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands
- A Novel Methodology For Crowdsourcing AI Models in an Enterprise