The Mathematics Behind AGI: Foundational Concepts and the Road Ahead
I want to be upfront about something before we start. Nobody fully knows what AGI is going to look like, how it will be built, or whether the mathematical frameworks we have today are even the right ones. What we do have is a collection of deep, beautiful, and sometimes frustrating mathematical ideas that seem to point in the right direction. This blog is my attempt to trace those ideas — not as a checklist of solved problems, but as an honest map of where the thinking currently is.
Some of this math is a century old. Some of it was written in the last decade. All of it is incomplete in one way or another when it comes to AGI. That tension is what makes this field interesting.
1. What is AGI, Formally?
Before building something, it helps to define it. For most of AI history, nobody really tried to define intelligence mathematically. That changed in 2000 when Marcus Hutter published his AIXI model — arguably the most rigorous formal definition of a general intelligence we have.
The idea is this: a perfect rational agent observes the world, forms beliefs over all possible explanations for what it's seeing, and selects actions that maximize expected future reward. The equation for which action AIXI picks at time \( k \) looks like this:
It looks dense, but the core logic is not complicated once you sit with it. The agent considers every possible program \( q \) that could explain the observations it has received. It weights each program by \( 2^{-\ell(q)} \) — meaning shorter, simpler programs are considered more likely. And then it picks the action that maximizes reward when you average across all those weighted explanations of the world.
This is essentially a formalization of Occam's Razor combined with rational decision-making. Simple explanations are preferred; actions are chosen to maximize long-term payoff. It is, in a precise mathematical sense, the best possible general learner and decision-maker.
There is one problem. AIXI is incomputable. You cannot run it. The sum over all programs is infinite, and there is no algorithm that can compute it in finite time. So we have a mathematically precise definition of AGI that we cannot actually build. That is either deeply frustrating or clarifying, depending on how you look at it. I find it clarifying — it tells us exactly what we are trying to approximate.
2. Probability and Bayesian Reasoning
If there is one mathematical idea that underlies almost every serious approach to general intelligence, it is Bayesian reasoning. Not because it is fashionable, but because it is provably the only coherent way to reason under uncertainty. Any agent that deviates from Bayes' theorem can be shown to make decisions that are inconsistent with its own beliefs — a result known as Dutch Book coherence.
Bayes' theorem itself is simple enough to write in one line:
But the depth is in what it demands of an agent. It says: you must start with a prior belief \( P(H) \), you must correctly compute the likelihood \( P(E \mid H) \) of your evidence under each hypothesis, and you must update to a posterior \( P(H \mid E) \) that will itself become the prior for the next observation. This cycle repeats forever, every time new information arrives.
For a general agent, this gets much harder. It is not updating beliefs over a handful of hypotheses — it must maintain a probability distribution over all possible models of the world simultaneously:
This is called Bayesian model averaging, and it is what AIXI does over all computable programs. In practice, computing this over even a moderately large model class is intractable. Most real systems approximate it — variational methods, MCMC, ensemble models — but approximation means you lose the theoretical guarantees. That tradeoff is something the field is still grappling with.
What I find genuinely interesting here is that Bayesian reasoning does not just tell you how to update beliefs — it tells you how to handle the fact that you are uncertain about which model of the world is correct. An AGI cannot assume it knows the rules of the environment it is in. Bayes gives it a principled way to remain uncertain and still act.
3. Information Theory — Intelligence as Compression
There is a claim that I keep coming back to, and I think it is one of the most underappreciated ideas in AI: to understand something is to be able to compress it. This is not a metaphor — it has a precise mathematical formulation, and it connects directly to what we want from a general intelligence.
Shannon's entropy measures how much uncertainty exists in a random variable — equivalently, how many bits you need on average to describe its outcomes:
If a system truly understands the patterns in data, it can predict what comes next. Good prediction means good compression — you do not need to store what you can reconstruct. This is why language models trained to predict the next token are, in a very real sense, learning to compress human language. The lower the loss, the better the compression, the deeper the understanding.
Mutual information takes this further. It measures how much knowing one thing tells you about another:
For learning representations — deciding which features of input are worth keeping — maximizing mutual information between the input and a compressed representation is mathematically the right objective. This shows up in contrastive learning, information bottleneck methods, and representation learning more broadly.
The connection to AGI runs deep. Minimum Description Length (MDL) says the best model of data is the one that, combined with its predictions, requires the fewest bits to describe everything. An agent that consistently finds the shortest description of its observations has, in a mathematically precise sense, discovered the underlying rules of its environment. That is what we want from a general intelligence.
4. Optimization Theory — The Engine of Learning
Every learning system eventually reduces to this: find the parameters \( \theta \) that minimize some measure of how wrong the system currently is. This is optimization, and it is the computational engine underneath everything from linear regression to GPT-4.
The method we reach for most often is gradient descent — take the gradient of the loss with respect to the parameters, step in the opposite direction, repeat:
This works extraordinarily well for narrow, fixed tasks. For AGI, it runs into three problems that I do not think are solved yet.
The first is the landscape itself. In a convex problem, gradient descent reliably finds the global minimum. Neural networks are wildly non-convex — the loss landscape is a high-dimensional terrain full of saddle points, flat plateaus, and local minima. In practice, large networks seem to escape these traps somehow, but we do not have a satisfying theoretical explanation for why.
The second is multi-objective optimization. A general agent cannot optimize a single scalar loss. It must balance accuracy, safety, fairness, efficiency, robustness — objectives that often conflict. The Pareto frontier describes the set of solutions where you cannot improve one objective without worsening another:
The third is continual learning. Train a neural network on task A, then train it on task B, and it often forgets task A entirely. Gradient descent overwrites old knowledge. Humans do not work this way — we accumulate knowledge across a lifetime. This catastrophic forgetting problem has no clean mathematical solution at scale. It is, in my view, one of the most practically important open problems between current AI and AGI.
5. Decision Theory and Utility — The Alignment Problem, Mathematically
Suppose we have an agent that learns well and reasons well. How does it decide what to do? Decision theory gives the answer: choose the action that maximizes expected utility.
On paper, this is clean. In practice, it raises a question that has no satisfying answer yet: what is \( U \)? How do you write down a utility function that correctly represents what you actually want?
This is not a philosophical worry. It is a concrete mathematical problem. If the utility function \( \hat{U} \) is even slightly misspecified relative to the true human preference \( U \), then an agent maximizing \( \hat{U} \) will find clever ways to achieve high \( \hat{U} \) that have nothing to do with \( U \). This is Goodhart's Law:
When a measure becomes a target, it ceases to be a good measure. In RL, this shows up as reward hacking — the agent achieves high reward through completely unintended means. Famously, a simulated robot trained to move fast learned to make itself very tall and fall over, since falling covers distance quickly. These are amusing at small scale. At AGI scale, the consequences of reward misspecification are potentially catastrophic.
The deeper mathematical issue is that we probably cannot write down human values as a utility function at all. Human preferences are inconsistent, context-dependent, and change over time. The alignment problem is, at its root, the problem of building an agent that can figure out what we want without us being able to fully specify it. No one has a complete mathematical solution to this.
6. Reinforcement Learning Theory — Learning to Act in the World
If Bayesian inference is the mathematics of belief, reinforcement learning is the mathematics of behavior. The framework of a Markov Decision Process (MDP) captures the essential structure of an agent interacting with an environment over time:
States, actions, transition probabilities, rewards, and a discount factor. The agent's goal is to find a policy \( \pi(a \mid s) \) — a mapping from states to actions — that maximizes the expected sum of discounted future rewards. The value of a state under a policy satisfies the Bellman equation:
And the optimal value function satisfies:
These equations are beautiful in their self-referential structure — the value of a state depends on the value of states you can reach from it. Every modern deep RL algorithm, from DQN to PPO to SAC, is ultimately an attempt to solve or approximate these equations at scale.
For AGI, the MDP framework has honest limitations. The real world is not Markov — the current observation does not contain all information relevant to future outcomes. It is not stationary — the reward function and transition dynamics can change. The state space is not discrete or even finite. And perhaps most importantly, current RL systems require millions or billions of interactions to learn what a child figures out in minutes. Sample efficiency is an unsolved problem.
7. Generalization and Learning Theory — Can We Guarantee It Learns?
Here is a question that does not get asked enough: when a system performs well on training data, how confident should we be that it will perform well on data it has never seen? Learning theory tries to answer this rigorously, and the answers are not always reassuring.
PAC learning, introduced by Leslie Valiant in 1984, gives a framework for when learning is provably possible. An algorithm PAC-learns a concept class if, with probability at least \( 1-\delta \), it finds a hypothesis with error at most \( \epsilon \) using a bounded number of samples:
The VC dimension \( d_{VC} \) measures how expressive a hypothesis class is — roughly, how many points it can shatter. Higher VC dimension means the class can represent more complex concepts, but also requires more data to generalize:
The bias-variance decomposition makes the fundamental tension visible. For any estimator \( \hat{f} \), the expected error decomposes as:
A model too simple cannot capture the truth (high bias). A model too complex memorizes noise (high variance). Every learning system navigates this tradeoff. For AGI, the hypothesis class needs to be rich enough to represent any computable function — which pushes VC dimension toward infinity and makes generalization guarantees theoretically vacuous. The No Free Lunch theorem makes this precise: no learner can outperform random guessing across all possible tasks. Structure and priors are not optional. They are mathematically necessary.
8. Causal Reasoning — The Gap Current AI Cannot Cross
Of everything in this blog, this section is the one I think deserves the most attention from anyone thinking seriously about AGI. Current machine learning is extraordinarily good at finding patterns. It is genuinely bad at understanding causes. These are not the same thing, and the difference matters enormously.
Judea Pearl spent decades formalizing this distinction. The key insight is that there are three fundamentally different things we might want to know:
What tends to happen when X is observed to be x? — This is \( P(Y \mid X = x) \). Standard machine learning answers this.
What happens when I force X to be x? — This is \( P(Y \mid \text{do}(X = x)) \). This requires causal reasoning.
What would have happened if X had been x, given that it was actually x'? — This is a counterfactual, and it sits above both.
A doctor observing that patients who take a drug tend to recover is not the same as knowing the drug causes recovery — maybe only patients who are already recovering choose to take it. This distinction is trivial for a human to understand. For a system that only sees \( P(Y \mid X) \), it is invisible.
Pearl's Structural Causal Model (SCM) gives a mathematical language for this. Each variable is a deterministic function of its causes and an independent noise term:
What troubles me about the current state of large AI systems is that even the most capable ones — systems that can discuss causality fluently — are, at their mathematical core, still operating at Level 1. They have learned an extraordinarily rich statistical model of human language about causality. That is not the same as causal reasoning. No one has yet found a scalable method to reach Level 3 from observational data alone. This is not an engineering gap. The mathematics is not there yet.
9. Kolmogorov Complexity — The Ceiling We Cannot Reach
Kolmogorov complexity is the idea that every string of data has a shortest description — the length of the shortest program that produces it. Formally, given a universal Turing machine \( U \):
This is not just an interesting definition. It is the theoretical foundation of intelligence itself. Solomonoff's induction — built on Kolmogorov complexity — defines the provably optimal prediction algorithm. Given past observations \( x_{1:n} \), it predicts the next symbol by averaging over all programs consistent with the data, weighted by their complexity:
Solomonoff induction converges to the true distribution faster than any other computable method. It is, in a mathematically rigorous sense, the best predictor possible. AIXI uses it as its world model. The Minimum Description Length principle uses it as its model selection criterion:
The catch: \( K(x) \) is not computable. There is no algorithm that can compute the Kolmogorov complexity of an arbitrary string. The theoretically perfect learner is unreachable in principle, not just in practice. Every real AI system is an approximation — and the distance between the approximation and the theoretical ideal is a measure of how far we still have to go.
I find this simultaneously humbling and clarifying. It tells us that the goal is not to solve a finite engineering problem. It is to find better and better approximations of something fundamentally unreachable, while staying within the bounds of computation. That reframing matters for how we think about progress in AI.
10. Open Problems — The Math That Has Not Been Written Yet
The sections above cover mathematics that exists, even if it is incomplete in application. This section is different. These are problems where the underlying mathematics is still missing or fundamentally inadequate.
Continual Learning
Elastic Weight Consolidation (EWC) is one attempt — it penalizes changes to parameters that were important for previous tasks, using the Fisher information matrix as a proxy for importance:
It helps, but it does not scale. The Fisher matrix becomes intractable for large networks, and the approach does not handle more than a handful of tasks gracefully. A genuine mathematical theory of continual learning — one that explains how to accumulate knowledge indefinitely without forgetting — does not exist yet.
Meta-Learning
MAML (Model-Agnostic Meta-Learning) trains a model such that a small number of gradient steps on a new task leads to good performance. The outer loop optimizes for fast adaptation; the inner loop does the task-specific update:
This is learning to learn — and it works well in narrow settings. Scaling it to open-ended, lifelong meta-learning across truly diverse tasks is unsolved both mathematically and computationally.
Consciousness and Measurement
Integrated Information Theory proposes \( \Phi \) as a measure of conscious experience — the amount of information generated by a system as a whole above and beyond its parts:
I include this not because I think IIT is correct — its interpretation is deeply contested — but because it illustrates how early we are. We do not even have an agreed mathematical framework for asking whether a system is conscious, let alone answering it. For AGI, this matters: an agent that cannot model its own internal states cannot reason about itself.
Symbol Grounding
Current systems manipulate symbols with extraordinary fluency. What they lack — and what no mathematical theory adequately explains — is how those symbols connect to the world. How does an abstract representation become about something? How does the word "red" connect to the experience of redness? This is the symbol grounding problem, and it remains one of the deepest unsolved questions at the intersection of mathematics, linguistics, and cognitive science.
11. Where This Leaves Us
Reading back through everything above, what strikes me is not how much progress has been made — though it has been remarkable — but how precisely we can now state what we do not know. That is actually a sign of a maturing field. Vague problems become tractable ones.
We have a mathematical definition of AGI (AIXI) that we cannot compute. We have provably optimal reasoning (Bayesian inference) that we cannot scale. We have the theoretically perfect predictor (Solomonoff induction) that we cannot run. And we have genuine open problems — causal reasoning, continual learning, symbol grounding, alignment — where the gap is not compute but mathematics itself.
What we also have is the clearest picture yet of what general intelligence requires. Not a single breakthrough, but a convergence of ideas across probability theory, information theory, causal inference, optimization, and learning theory — with a few mathematical frameworks that have not been invented yet sitting somewhere in the middle.
I do not know if AGI is ten years away or fifty or never. But I am fairly confident that when it arrives, the mathematics in this blog will be recognizable in its foundations. The equations are not the answer. They are the language in which the answer will eventually be written.
Comments
Post a Comment