Skip to main content

What is Intelligence: Neural Architecture Search


In the early days of neural network research, human engineers were tasked with coming up with network architectures to achieve their goals. Nowadays there is some pioneering work on Neural architecture search where computers are used to search for useful networks that achieve better performance than architectures that were designed by human beings.

Apart from just searching for appropriate architecture that could perform better than those that were engineered by humans, the field of Neural Architecture Search (NAS) is involved in the wider automatic search for hyperparameters that enable networks to reach their goals via a more efficient path than what was engineered by humans.

When people hear these kinds of announcements they assume that AI is now doing the job of AI engineers and this is kind of like a doomsday scenario. I want to use this opportunity to talk about something many people have not understood about programming and software development in itself.

Writing code is only part of the software development process. There is also testing, profiling and a host of other things that must be done before any piece of software are deployed in the real world. The more critical a job the software has to do like control a nuclear power plant or a jet plane, the more rigorous tests the software system has to pass through. Less critical software can always be released without so much fear as to what may happen if a bug is discovered except its some kind of security hole.

Apart from this basic stuff like testing and profiling software must go through, although these in themselves are time-consuming and take a whole chunk of the time required to deploy a piece of software, the hardest part of the software cycle is the long term maintenance.

During the long term maintenance cycle, a lot of things that you cannot predict will come and the programmer must exercise their intelligence and creativity in trying to find solutions for these problems. So until we have some kind of human equivalent AGI most of these non-routine-unpredictable tasks will remain human domain.

The central question to ask is what can we write software to perform? Although there are pieces of software that work in non-repetitive highly unpredictable environments, the bulk of software we design mostly works on highly repetitive tasks.

The goal of automation is to automate away the boring tasks and what we mean by boring here are tasks that are repetitive enough and predictable enough that we are usually capable of just automating away and allowing some software-based system to handle it.

As far as domain is not too open, by open I mean capable of being interfered with at unpredictable times by unpredictable variables then it can be automated away.

Initially, in the early days of AI research, it was usually very difficult to come up with efficient architectures to accomplish goals using resources efficiently in a reasonable time.

But as the space of the solutions to network architecture problems became more defined, it was now possible to automate the search for new network architectures. Google has been a pioneer in this field but there are other research groups working hard on being able to search the space of possible networks in an efficient manner to yield networks that solve problems already solved by existing networks but in a much more efficient manner and networks to solve problems that have been hard to find architectures for.

Most of the times the architecture that has been built using automated search techniques look very weird and quite different from that which the human mind would have conceived. This has made many doomsayers imagine that this is the end of the usefulness of humans in the very job of AI engineering but this is a misconception which has no grounding in reality.

In traditional software engineering, even after deploying the software we never really stop building the software as requirements change. Although some aspects of software development like profiling have been automated away, when the profiler reports locations where there are bottlenecks in the software, it usually takes a human engineer to figure out how to fix it.

People mistake the capabilities of tools and as such when they heard about neural architecture search they simply imagined that the tool would be greater than maker and thus all the talk going on about AI engineers about to be eliminated by AI itself. This is just a ridiculous assumption.

Neural architecture search has revealed an enormous space and will require a lot of very hard and smart work from AI engineers to figure out the full extent of this space and the best strategies for exploiting the space.

Currently, engineers are using stuff like genetic algorithms to search for neural nets but I have an intuition that using something like Cellular Automata evolution to structure search over such vast spaces will yield greater results.

A slice of Inception V3 NetModel from Wolfram Mathematica. An example of a hand engineered neural network architecture.

The kinds of Architecture we are beginning to discover with neural architecture search.

It is still a mystery how the human brain comes up with the neural architectures that it does to solve the problems that it encounters. We don’t even know the full structure of human neural architecture, although we can see neurons interconnected in certain ways and via some kind of labelling we can come up with diagrams of connected neurons that do certain stuff, we do not still know in the majority how the brain does many of the things it does and at what level.

Observing neuronal interconnectivity inspired us to start creating artificial neural networks but modern neural network research has taken a completely different path and is looking more like traditional software programs and less like the idealized neurons that were their initial inspiration.

Although as pertains to science it is meaningful to study the human brain and even find out how it does certain things, and we can indeed still discover structures that inspire us to create other artificial things that are similar to our current artificial neural networks, it is not really necessary to rely too much on brain for inspiration.

Rather we should try to improve on basic research of the fundamental structures that can represent vast types of system and problems, the network. I think that the most important inspiration the brain has for us is that it makes networks very explicit by literarily connecting neurons together.

Although anything in the human body that passes information to another thing can be modelled like a network, it is the brain that makes this connection very explicit thereby revealing the path for us to follow and it is clear that modern neural network research has capitalized on this fundamental.

My goal with this work is to tell us that although we have been very successful with our modern neural network paradigm, trained with backpropagation, the field of networks is much broader and fundamental research is needed to uncover deeper capabilities of networks. I theorize that it could lead us all the way to building full on AGI from first principles.


Popular posts from this blog

Next Steps Towards Strong Artificial Intelligence

What is Intelligence? Pathways to Synthetic Intelligence If you follow current AI Research then it will be apparent to you that AI research, the deep learning type has stalled! This does not mean that new areas of application for existing techniques are not appearing but that the fundamentals have been solved and things have become pretty standardized.

How to become an AI researcher

Artificial Intelligence is all the rage these days. Everyone is getting on the bandwagon but there seems to be a shortage of AI researchers everywhere these days. Although many people are talking about doing AI not many people are actually doing AI research.