Fundamental Inequalities and Identities Involving Mutual and Directed Informations in Closed-Loop Systems

Abstract

We present several novel identities and inequalities relating the mutual information and the directed information in systems with feedback. The internal blocks within such systems are restricted only to be causal mappings, but are allowed to be non-linear, stochastic and time varying. Moreover, the involved signals can be arbitrarily distributed. We bound the directed information between signals inside the feedback loop by the mutual information between signals inside and outside the feedback loop. This fundamental result has an interesting interpretation as a law of conservation of information flow. Building upon it, we derive several novel identities and inequalities, which allow us to prove some existing information inequalities under less restrictive assumptions. Finally, we establish new relationships between nested directed informations inside a feedback loop. This yields a new and general data-processing inequality for systems with feedback.

arXiv commentSubmitted to IEEE Trans. on Information Theory

References in corpus (2)

• A New Data Processing Inequality and Its Applications in Distributed Source and Channel Coding
• The Information Flow and Capacity of Channels with Noisy Feedback

Cited by in corpus (3)

• A Characterization of the Minimal Average Data Rate that Guarantees a Given Closed-Loop Performance Level
• LQG Control with Minimum Directed Information: Semidefinite Programming Approach
• Interplay Between Transmission Delay, Average Data Rate, and Performance in Output Feedback Control over Digital Communication Channels