Shuffling is Not Enough: Breaking Permutation-Based Model Confidentiality in Hybrid FHE Inference

Jiseung Kim, Hyung Tae Lee

Hybrid fully homomorphic encryption~(FHE) inference improves the practicality of private inference by letting the server evaluate linear layers homomorphically while the client dec…