A Grounded and Naturalistic Approach to Language Invention
A summary of the contributions of my recent work at CogSci 2025
In this piece I briefly summarize the contributions of my recent work, Visual Theory of Mind Enables the Invention of Proto-Writing, coauthored with Lucas Gelfond, which I will be presenting as an oral at CogSci 2025 — and how it connects to my broader research agenda.
🔗 Watch my talk at CogSci in the Computational modeling 1 session starting at 10:30am PST on July 31st, 2025. See the original tweet thread or the Perplexity snippet on this work.
The Symbol Grounding Problem, Revisited
Most computational approaches to language — including LLMs and models of emergent communication — make numerous faulty implicit philosophical assumptions about its structure, purpose, and overall function signature. As a consequence of treating language in such a contrived manner, these approaches are barely relevant for making claims about the human capacity for language use and its origins. In my recent paper we eliminated two major assumptions of these works that led to significantly more naturalism:
Instead of simulating communication in isolation from the world (like in a referential game) we situate communication in the context of an environment via a new tool called a Signification Game (it’s a kind of MDP). This paves the way for enabling agents to form languages not only about entities, but about any facet of the environment.
Instead of supplying agents with a set of discrete, meaningless tokens that gradually acquire meaning through use, we endow agents with a continuous signaling space and find that they make their own symbols with it. Not only that, but their signals mirror the trajectories of ancient human scripts: they start as iconic pictographs and shed visual detail until they become abstract symbols.
By shedding these assumptions, we were able to start addressing questions about where symbols come from, and the specific form of cognition necessary to leap from primitive animalistic signaling to full-blown pictographic signification.
The Signification Gap
Our simulations start when agents begin learning stimulus-response (S-R) behaviors (see snake → run away) via reinforcement learning in their environment. These S-R behaviors form an early basis of communication: if you can replicate a stimulus, you can communicate a referent response action.
While this works on MNIST, where agents can actually make convincing digits using splines on a canvas, it fails on CIFAR-100, which contains more natural images. You just can’t draw a convincing-looking tree with splines.
We label the gap between what senders can produce and what receivers recognize as a signification gap — and suggest it explains why animals never invented proto-writing despite sophisticated signaling abilities. In short, they can't replicate complex stimuli with simple marks, and they lack the cognition to reinterpret crude drawings as something other than their prima facie impression of them.
Bridging Signification Gaps with Visual Theory of Mind
But humans don't need photorealistic drawings to recognize pictographs. We can recognize when drawings depict something, even when we aren’t genuinely convinced we are seeing the referent.
We do this by leveraging visual theory of mind — reasoning about what others can draw and how they perceive. Our model formalizes this as a Bayesian inference procedure. It’s like Rational Speech Acts and Inverse-Planning but with naturalistic assumptions and a new referent sensitivity term — a learned measure of how hard each referent is to communicate.
The results are dramatic. Behaviorist agents struggle to communicate for thousands of epochs but inferential agents achieve high communication accuracy within 300 epochs. The referent sensitivity term is crucial — agents learn to "look harder" for difficult-to-draw concepts like rockets while easily recognizing simpler referents like spiders.
From Icons to Symbols
What emerges from these simulations is remarkable. Agents independently recapitulate the evolution of human writing systems. Early signals are recognizably iconic — crude but identifiable pictographs of their referents. Over thousands of iterations, they grow more visually abstract while maintaining communicative function until they become unrecognizable symbols. By playing with the signaling reward function, we can also encourage or discourage certain visual features, simulating how various writing implements visually impact symbols.
Even more interesting is that referent sensitivity values converge to 1 over time, signifying that agents eventually don’t struggle to communicate. When this happens, our model converges to Rational Speech Acts, and it does this without ever being given a denotational semantics! The shared Theory of Mind assumption provides just enough common ground to bootstrap a full denotational semantics from basic stimulus-response relationships.
In Summary
Current methods treat language as merely another data source, and as a result do little to advance theories on its origins in humans. This work demonstrates an alternative approach to processing language that 1) lays a foundation for future work on learning useful, grounded languages and 2) maintains enough naturalism to be relevant to discussions on human language and proto-language. Altogether this work aligns with my broader research agenda on letting modern modalities — like symbols — emerge from more general cognitive representations.
🎉 Very excited to present this as an oral at CogSci 2025!! 🎉
🔗 Watch my talk in the Computational modeling 1 session starting at 10:30am PST on July 31st, 2025.
A huge thanks to my co-author, Lucas Gelfond, who is now doing great things in SF!
Very cool, will have to read the paper and comment further. Your point about symbols starting as pictographs and then becoming more abstract is true in math as well as normal language, for example the equal sign starting as parallel line sand then becoming a more abstract, general symbol.
I wonder if an analogous model can be used to simulate the emergence of prices. Maybe prices start as pictographs in the sense of being very particular exchange ratios—my two fish for your one coconut—and then become more general, more usable and abstract over time.