Training Language Models via Neural Cellular Automata

Dan Lee, Seungwook Han, Akarsh Kumar +1

Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: hig…