Models are maps

A model is not the thing. It's a compressed representation of the thing. This idea echoes through Latent spaces as maps — we don't see the territory by studying the map.

In machine learning we pretend the model is the phenomenon. We celebrate when it predicts well. But prediction is not understanding. Attention is all you need (in life too) reminds us that real cognition is about choosing what to focus on.

A trained language model can write convincingly about topics it has no contact with. It has learned the shape of discourse, but not the territory. The shape of a thought explores this problem from the human side.

The map is useful. Maps are incredibly useful. But you learn geography by traveling, not by studying maps. Why I write is my way of traveling through my own thoughts.

Every model I build is a map. Every model I trust too much becomes a cage. Notes on attention captures the danger of misplaced focus.

The real understanding is in knowing which parts of the map are useful, and which parts lie.