Probabilistic Examination of Least Squares Error in Low-bitwidth Cholesky Decomposition

Alexander Osinsky, Roman Bychkov, Mikhail Trefilov +2

In this paper, we propose a new approach to justify a round-off error impact on the accuracy of the linear least squares (LS) solution using Cholesky decomposition. This decomposit…