232 citations · 400 across the 19 of their papers we have counts for
21 papers
PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming
Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson +13
We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based test…
Constrained Decoding with Speculative Lookaheads
Nishanth Nakshatri, Shamik Roy, Rajarshi Das +3
Constrained decoding with lookahead heuristics (CDLH) is a highly effective method for aligning LLM generations to human preferences. However, the extensive lookahead roll-out oper…
KazQAD: Kazakh Open-Domain Question Answering Dataset
Rustem Yeshpanov, Pavel Efimov, Leonid Boytsov +2
We introduce KazQAD -- a Kazakh open-domain question answering (ODQA) dataset -- that can be used in both reading comprehension and full ODQA settings, as well as for information r…
A Curious Case of Remarkable Resilience to Gradient Attacks via Fully Convolutional and Differentiable Front End with a Skip Connection
Leonid Boytsov, Ameya Joshi, Filipe Condessa
We experimented with front-end enhanced neural models where a differentiable and fully convolutional model with a skip connection is added before a frozen backbone classifier. By t…
InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers
Leonid Boytsov, Preksha Patel, Vivek Sourabh +4
We carried out a reproducibility study of InPars, which is a method for unsupervised training of neural rankers (Bonifacio et al., 2022). As a by-product, we developed InPars-light…
Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigation
Leonid Boytsov, David Akinpelu, Nipun Katyal +5
We tested over 20 Transformer models for ranking long documents (including recent LongP models trained with FlashAttention and RankGPT models "powered" by OpenAI and Anthropic clou…
Smooth-Reduce: Leveraging Patches for Improved Certified Robustness
Ameya Joshi, Minh Pham, Minsu Cho +4
Randomized smoothing (RS) has been shown to be a fast, scalable technique for certifying the robustness of deep neural network classifiers. However, methods based on RS require aug…
The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
Pavel Efimov, Leonid Boytsov, Elena Arslanova +1
Large multilingual language models such as mBERT or XLM-R enable zero-shot cross-lingual transfer in various IR and NLP tasks. Cao et al. (2020) proposed a data- and compute-effici…
Exploring Classic and Neural Lexical Translation Models for Information Retrieval: Interpretability, Effectiveness, and Efficiency Benefits
Leonid Boytsov, Zico Kolter
We study the utility of the lexical translation model (IBM Model 1) for English text retrieval, in particular, its neural variants that are trained end-to-end. We use the neural Mo…
A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models
Iurii Mokrii, Leonid Boytsov, Pavel Braslavski
Due to high annotation costs making the best use of existing human-created training data is an important research direction. We, therefore, carry out a systematic evaluation of tra…
Traditional IR rivals neural models on the MS MARCO Document Ranking Leaderboard
Leonid Boytsov
This short document describes a traditional IR system that achieved MRR@100 equal to 0.298 on the MS MARCO Document Ranking leaderboard (on 2020-12-06). Although inferior to most B…
Flexible retrieval with NMSLIB and FlexNeuART
Leonid Boytsov, Eric Nyberg
Our objective is to introduce to the NLP community an existing k-NN search library NMSLIB, a new retrieval toolkit FlexNeuART, as well as their integration capabilities. NMSLIB, wh…
SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis
Pavel Efimov, Andrey Chertok, Leonid Boytsov +1
SberQuAD -- a large scale analog of Stanford SQuAD in the Russian language - is a valuable resource that has not been properly presented to the scientific community. We fill this g…
Pruning Algorithms for Low-Dimensional Non-metric k-NN Search: A Case Study
Leonid Boytsov, Eric Nyberg
We focus on low-dimensional non-metric search, where tree-based approaches permit efficient and accurate retrieval while having short indexing time. These methods rely on space par…
Accurate and Fast Retrieval for Complex Non-metric Data via Neighborhood Graphs
Leonid Boytsov, Eric Nyberg
We demonstrate that a graph-based search algorithm-relying on the construction of an approximate neighborhood graph-can directly work with challenging non-metric and/or non-symmetr…
A Simple Derivation of the Heap's Law from the Generalized Zipf's Law
Leonid Boytsov
I reproduce a rather simple formal derivation of the Heaps' law from the generalized Zipf's law, which I previously published in Russian.
Off the Beaten Path: Let's Replace Term-Based Retrieval with k-NN Search
Leonid Boytsov, David Novak, Yury Malkov +1
Retrieval pipelines commonly rely on a term-based search to obtain candidate records, which are subsequently re-ranked. Some candidates are missed by this approach, e.g., due to a…
Non-Metric Space Library Manual
Bilegsaikhan Naidan, Leonid Boytsov, Yury Malkov +1
This document covers a library for fast similarity (k-NN)search. It describes only search methods and distances (spaces). Details about building, installing, Python bindings can be…
Permutation Search Methods are Efficient, Yet Faster Search is Possible
Bilegsaikhan Naidan, Leonid Boytsov, Eric Nyberg
We survey permutation-based methods for approximate k-nearest neighbor search. In these methods, every data point is represented by a ranked list of pivots sorted by the distance t…
SIMD Compression and the Intersection of Sorted Integers
Daniel Lemire, Leonid Boytsov, Nathan Kurz
Sorted lists of integers are commonly used in inverted indexes and database systems. They are often compressed in memory. We can use the SIMD instructions available in common proce…
Decoding billions of integers per second through vectorization
Daniel Lemire, Leonid Boytsov
In many important applications -- such as search engines and relational database systems -- data is stored in the form of arrays of integers. Encoding and, most importantly, decodi…