The general assumption that we can find an equation for every data sequence is proven false when we observe the evolution of Rule 30 Cellular automaton. Computational irreducibility proves that we cannot always capture a summary of data with an equation, but sometimes the most concise representation of data is the program that generated the data itself.
And there are thousands of equations like these all around us in physics from newtons laws to the crazier equations of the standard model of physics. With such sophistication we might think that we can always produce an equation that models or provides some kind of summary for any stream of data. But like we can see for simple systems like the Rule 30 cellular automaton, no equation can be produced that predicts the sequence of output the is generated by this system.
While this might seem like some kind of isolated event, a freak of nature, it is not so. The universe of such programs is so massive that isolated cases of reducibility like the ones we are used to from science are few and far apart.
What I have written above is just a regurgitation of what Stephen Wolfram founder of Wolfram Research and the author of the tome: A New Kind of Science. has been saying for years but what I am trying to bring attention to in this post is the fact that Deep learning models, as sophisticated as they are all still equational models and thus subject to the limitations of equational models.
This might come as a sharp and painful blow to many people that have put that their hopes on deeplearning systems as the roadmap to AGI but I doubt that. The belief that deeplearning systems will lead us to AGI is like the belief that we once had that we could produce an equation that summarized the functioning of our minds.
The reasoning goes like this, the universe is made of physics, many physical systems can be reduced to equations; our brains are physical and thus obey the laws of physics and thus our brains can be fully understood with physics; our minds are the result of our brains functioning under the laws of physics, thus they too are subject to the laws of physics, if we can model physical systems with equations, there should be an equational model of the mind.
The deep deep problem with equations is that they use a limited set of rules, the rules of arithmetic (+, *, -,/, etc.) or other higher functional rules (Integrals, dy/dx kinda stuff, etc.) In many cases, where we find island of reducibility, we can always find a combination of these operators, variables, and constants that somehow models the system we are dealing with. But when we encounter more general rules like the ones in Rule 30, that simply map certain types of inputs to outputs, we see that we cannot find an equation that reduces the output of the system rather we have to accept that the most concise representation of the output of the system is represented by the program that generated it.
For more clarity, when we encounter a physical system operating in nature the first point of duty is to obtain data about the system, when we have these data and start analyzing it, we can sometimes come up with an equation that summarizes the data we have obtained from the system. This is how Newton produced his laws, or Einstein or Niels Bohr. This is how science is done. But there are system that we could produce data for which no equational system can be setup and this is not to say that there is some equation we have to discover, computational irreducibility says that it is just not possible.
This comes right in the face of our belief that we can create General Intelligence by collecting increasingly massive amounts of data and coming up with models (Equations) that can effectively summarize (learn) from these data. Its like Kurt Gödel's incompleteness theorem which hit David Hilbert right in the face and destroyed the hopes of Bertrand Russel and Alfred White Northhead for Principia Mathematica.
Rather than trying to learn equations from data like we do now we should try to create generalized programs that when executed produce data similar to what we seek, then this programs, executable on a computer, would be an effective representation of the data and hence an appropriate model. This might sound confusing as we might think that this is what we are already doing by trying to create models (equations) for our data as we might imagine that our models are programs because a models can be coded up as a program. But what I am trying to say here is that our models based on the kind of equations we have now are made up of a much limited set of operations and that we need a richer set or more generalized rules rather than the fixed set we have with equations from physics or wherever.
One might think of computing that uses such a minimal set of primitives like NAND gates to create the sophistication we currently have access to in our modern computing infrastructure and assume that the simple operations we use in standard equational models are sufficient, but this is not so. The real power behind the sophistication of our computing infrastructure is not necessary the ALU that performs most of the arithmetic and logic but rather the memory and our ability to manipulate this memory in a general fashion. So even though the NAND logical gate is used to "hold" a bit of memory the magic of computing does not come from using the logic of the NAND gate itself, but through the fact that this memory allow use to manipulate stuff in a general non-standard way.
The simple primitive low-level operations of our computing hardware allows us to build much general computations that can lead to richness. You could implement a cellular automaton directly with hardware and it will still produce the kind of complexity it does. Why it does so is because it has general rules not limited to standard rules used in equational systems.
Our human minds by themselves have very limited ability to directly capture or create much complexity on its own so we cannot really build models directly that capture the kind of sophistication that an AGI system like the human mind itself is. Yes the human mind is highly complex but it cannot understand its own complexity because the user of the mind, the hardware-human itself has very little conscious access to the whole system.
So should we go out and search for a particular cellular automaton that represents the human mind? Not in a direct fashion I would say. Cellular automata show us that complexity can be produced from initially simple rules, they serve as a flagpost for a non-human engineered way of producing complex structures as complex as what we observe in nature without using any kind of equations. Some Cellular automata are subject to traditional analysis and equations can be produced that model them, but some others are so complex and rich that no traditional system of analysis can reduce them into an equation. Thus the cellular automaton program itself is the simplest representation of the data that can be produced from the cellular automaton.
So using cellular automatons as pillars of hope then we can now look into other kinds of systems that use general rules and there is not better system to look at everyday vanilla computer programming.
Programs with rules written by human beings can do complicated things that we cannot always predict but sometimes this is just a nuisance as we are not really writing programs for the sole purpose of producing complicated programs. But if we were to focus on writing programs that generate complicated behavior we as humans are not very good at producing programs that produce the kind of rich behavior that cellular automata exhibits.
Lets take for example our human attempts to write programs that generate randomness, at best we can only produce pseudo randomness and not true randomness but a system like the rule 30 cellular automaton produces very robust randomness without much engineering.
So how can we harness the power of systems like rule 30? Are cellular automatons the only systems that produce this kind of richness? Stephen Wolfram has already proven in his books that this complexity can be found in all kinds of systems and are by no means limited to cellular automata, so with this inspiration we can see that to get as something that is equivalent to a mind we might have to search for systems that allow us to encode much richer structure than what is available in traditional equational models.
Comments
Post a Comment