Creating an AI Observer: Generative Semantic Workspaces

Abstract

An experienced human Observer reading a document -- such as a crime report -- creates a succinct plot-like $\text{“Working Memory”}$ comprising different actors, their prototypical roles and states at any point, their evolution over time based on their interactions, and even a map of missing Semantic parts anticipating them in the future. $\text{An equivalent AI Observer currently does not exist}$. We introduce the $\text{[G]}$enerative $\text{[S]}$emantic $\text{[W]}$orkspace (GSW) -- comprising an $\text{“Operator”}$ and a $\text{“Reconciler”}$ -- that leverages advancements in LLMs to create a generative-style Semantic framework, as opposed to a traditionally predefined set of lexicon labels. Given a text segment $C_n$ that describes an ongoing situation, the $\text{Operator}$ instantiates actor-centric Semantic maps (termed ``Workspace instance'' ${\mathcal{W}}_n$). The $\text{Reconciler}$ resolves differences between ${\mathcal{W}}_n$ and a ``Working memory'' ${\mathcal{M}}_n^{\ast }$ to generate the updated ${\mathcal{M}}_{n+1}^{\ast }$. GSW outperforms well-known baselines on several tasks ($\sim 94\%$ vs. FST, GLEN, BertSRL - multi-sentence Semantics extraction, $\sim 15\%$ vs. NLI-BERT, $\sim 35\%$ vs. QA). By mirroring the real Observer, GSW provides the first step towards Spatial Computing assistants capable of understanding individual intentions and predicting future behavior.

37 pages with appendix, 28 figures