Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks

Nan Wu, Stanisław Jastrzębski, Kyunghyun Cho +1

We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks, these models tend to rely on just one modality while under-fitting the other modalitie…