200 citations · 433 across the 16 of their papers we have counts for
19 papers
SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence
Zhining Liu, Rana Ali Amjad, Ravinarayana Adkathimar +2
Providing Language Models (LMs) with relevant evidence in the context (either via retrieval or user-provided) can significantly improve their ability to provide better-grounded res…
Context-Aware Clustering using Large Language Models
Sindhu Tipirneni, Ravinarayana Adkathimar, Nurendra Choudhary +5
Despite the remarkable success of Large Language Models (LLMs) in text understanding and generation, their potential for text clustering tasks remains underexplored. We observed th…
Neural Augmentation of Kalman Filter with Hypernetwork for Channel Tracking
Kumar Pratik, Rana Ali Amjad, Arash Behboodi +2
We propose Hypernetwork Kalman Filter (HKF) for tracking applications with multiple different dynamics. The HKF combines generalization power of Kalman filters with expressive powe…
A White Paper on Neural Network Quantization
Markus Nagel, Marios Fournarakis, Rana Ali Amjad +3
While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is…
Invertible Low-Divergence Coding
Patrick Schulte, Rana Ali Amjad, Thomas Wiegart +1
Several applications in communication, control, and learning require approximating target distributions to within small informational divergence (I-divergence). The additional requ…
Bayesian Bits: Unifying Quantization and Pruning
Mart van Baalen, Christos Louizos, Markus Nagel +4
We introduce Bayesian Bits, a practical method for joint mixed precision quantization and pruning through gradient based optimization. Bayesian Bits employs a novel decomposition o…
Up or Down? Adaptive Rounding for Post-Training Quantization
Markus Nagel, Rana Ali Amjad, Mart van Baalen +2
When quantizing neural networks, assigning each floating-point weight to its nearest fixed-point value is the predominant approach. We find that, perhaps surprisingly, this is not…
Class-Conditional Compression and Disentanglement: Bridging the Gap between Neural Networks and Naive Bayes Classifiers
Rana Ali Amjad, Bernhard C. Geiger
In this draft, which reports on work in progress, we 1) adapt the information bottleneck functional by replacing the compression term by class-conditional compression, 2) relax thi…
Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
Rana Ali Amjad, Kairen Liu, Bernhard C. Geiger
In this work, we investigate the use of three information-theoretic quantities -- entropy, mutual information with the class variable, and a class selectivity measure based on Kull…
Extended Affinity Propagation: Global Discovery and Local Insights
Rayyan Ahmad Khan, Rana Ali Amjad, Martin Kleinsteuber
We propose a new clustering algorithm, Extended Affinity Propagation, based on pairwise similarities. Extended Affinity Propagation is developed by modifying Affinity Propagation s…
Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
Rana Ali Amjad, Bernhard C. Geiger
In this theory paper, we investigate training deep neural networks (DNNs) for classification via minimizing the information bottleneck (IB) functional. We show that the resulting o…
Information Rates and Error Exponents for Probabilistic Amplitude Shaping
Rana Ali Amjad
Probabilistic Amplitude Shaping (PAS) is a coded-modulation scheme in which the encoder is a concatenation of a distribution matcher with a systematic Forward Error Correction (FEC…
Co-Clustering via Information-Theoretic Markov Aggregation
Clemens Bloechl, Rana Ali Amjad, Bernhard C. Geiger
We present an information-theoretic cost function for co-clustering, i.e., for simultaneous clustering of two sets based on similarities between their elements. By constructing a s…
A Generalized Framework for Kullback-Leibler Markov Aggregation
Rana Ali Amjad, Clemens Blöchl, Bernhard C. Geiger
This paper proposes an information-theoretic cost function for aggregating a Markov chain via a (possibly stochastic) mapping. The cost function is motivated by two objectives: 1)…
Hard Clusters Maximize Mutual Information
Bernhard C. Geiger, Rana Ali Amjad
In this paper, we investigate mutual information as a cost function for clustering, and show in which cases hard, i.e., deterministic, clusters are optimal. Using convexity propert…
Informational Divergence and Entropy Rate on Rooted Trees with Probabilities
Georg Böcherer, Rana Ali Amjad
Rooted trees with probabilities are used to analyze properties of a variable length code. A bound is derived on the difference between the entropy rates of the code and a memoryles…
Fixed-to-Variable Length Resolution Coding for Target Distributions
Georg Böcherer, Rana Ali Amjad
The number of random bits required to approximate a target distribution in terms of un-normalized informational divergence is considered. It is shown that for a variable-to-variabl…
Block-to-Block Distribution Matching
Georg Böcherer, Rana Ali Amjad
In this work, binary block-to-block distribution matching is considered. m independent and uniformly distributed bits are mapped to n output bits resembling a target product distri…
Fixed-to-Variable Length Distribution Matching
Rana Ali Amjad, Georg Böcherer
Fixed-to-variable length (f2v) matchers are used to reversibly transform an input sequence of independent and uniformly distributed bits into an output sequence of bits that are (a…