Privacy-Preserving Vision Transformer Using Images Encrypted with Restricted Random Permutation Matrices

Kouki Horio, Kiyoshi Nishikawa, Hitoshi Kiya

We propose a novel method for privacy-preserving fine-tuning vision transformers (ViTs) with encrypted images. Conventional methods using encrypted images degrade model performance…