Autonomous Innovation with Neural Networks?

Earlier this year, an essay by Gary Marcus threw some cold water on the idea that humanity is on the brink of general artificial intelligence. Specifically, Marcus targeted Deep Learning, the method underlying modern neural networks. Marcus makes a series of critiques, but one prominent one (discussed earlier) is that neural networks have abysmal judgment for problems that lie outside the borders of their training set. This doesn’t mean they can’t be very good inside their training set. But give them a problem not well approximated by a mix of the features of examples they are trained on, and they fall to pieces.

On this (and other) dimensions, neural networks fare worse than human beings. But how important is this? In this post, I want to think a little about how far you can get with purely “interpolative innovation.” By interpolative innovation, I mean innovation that consists in the discovery of new things that are “mixes” of preexisting thing. A lot of innovation falls under this category.

A nice 2005 paper by economist Ola Olsson serves as a roadmap. Olsson’s “Technological Opportunity and Growth,” published in the Journal of Economic Growth (paywall, sorry), makes no mention of neural networks. It was meant to be a paper of how innovation in general happens. But it nicely illustrates how the dynamics of primarily interpolative innovation might play out in the long run, and how interpolative innovation can come to look like it evades Marcus’ critique: it (seemingly) breaks free of its training set.

The Technology Space

Olsson asks us to imagine a highly-dimensional space. Each of the dimensions in this space corresponds to some kind of attribute that an idea might have, and which might be measured by a number. Olsson suggests dimensions could correspond to things like “complexity”, “abstractness”, “mathematics”, “utility”, and so on. Scattered throughout this space are specific technologies, represented as points. Essentially, he asks us to imagine the human technological system as a cloud of points, where each point corresponds to a technology and a point’s position tells us about its features. Technologies with a lot of similar features are closely bunched together, and technologies with very different features will be distant.

Slide1Figure 1. Technology as a set of points… except with a lot more than two dimensions

This can be mapped into a neural network setting pretty easily. For a neural network to work with data, the features of the data need to be inputed to specific neurons as numbers. We can imagine those numbers correspond to positions along axes in Olsson’s technology space. Just as the set of technologies floats out there as a cloud of points in highly-dimensional space, so to does the set of training examples float out there in highly-dimensional space. Examples that have very similar features are close together, examples with very different features are far apart.

Innovation in the technological space

In Olsson’s model, all innovation is an interpolation between existing technologies. To begin, he defines incremental innovation as the discovery of a new technological point lying on a line connecting two technologies that are “close” together and which are already known. Essentially, we add new points to our cloud, but we can only add points in the spaces between existing technologies. However, as we innovate, we add new technologies, and these give us new possibilities for combination. If incremental innovation was all there was, then in the long run, we would eventually fill up all the gaps between technologies. In technical parlance, we would be left with the convex hull of technologies close enough to eventually be fully connected. The convex hull is the region such that no line drawn between points in it fall outside the region.

Slide2Figure 2. The convex hull of Figure 1. Again, pretend there are a lot more than 2 dimensions.

This is precisely Marcus’ critique of neural networks. They cannot extrapolate, and they cannot go beyond their training data. At best, they can recover the convex hull of their training set. Additionally, note that Olsson assumes it’s only possible to interpolate in between technologies that are already “close” in technological space. This is reminiscent of the way that neural networks need to be tuned to the kind of data they receive. For instance, the inputs to image classification neural nets differ dramatically from AlphaGo’s inputs, and the networks cannot transfer what they’ve learned in one domain to another (indeed, this is another of Marcus’ critiques). So we might imagine, in Olsson’s framework, that neural networks are only capable of interpolating between very similar (i.e., “close”) sets of technologies.

Olsson adds to his model the assumption that every once in awhile, purely by random chance, serendipitous discoveries are made. These are points in the technological space that simply appear on the scene. By chance, some of them will lie outside the convex hull described above. We can image these correspond to the incidents like Fleming’s lucky observation that a stray bit of penicillin mould had retarded the growth of bacteria in a petri dish. Or maybe they correspond to scientific anomalies in Thomas Kuhn’s sense.

So long as incremental innovation is feasible, researchers ignore these discoveries. However, at some point, incremental innovation begins to exhaust its possibilities. All possible combinations have been discovered. When this is the case, researchers turn their attention to these discoveries and engage in what Olsson calls radical innovation. He assumes this is riskier or more costly, and therefore the less favored choice of researchers. However, when incremental innovation is not possible, they have no choice but to turn to radical innovation.

Radical innovation is the creation of a new technology lying on a line between an existing technology and one of the serendipitous discoveries lying outside the convex hull. After much radical innovation, there are enough new technologies close enough to existing ones for incremental innovation to begin again. This time, the incremental innovation makes use of the technologies discovered by radical innovation. In this way, innovation proceeds, breaking free of its old paradigm by exploiting and developing serendipitous discoveries.

Again, this framework seems a good fit for neural networks. We can imagine radical innovation corresponds to retraining a neural network with additional examples that lie outside its previous training set. Just as Olsson assumes radical innovation to be in some sense harder than incremental innovation, retraining neural networks in this way seems harder than business as usual. In our last post, for example, it took several years before computer scientists figured out how to represent the styles of multiple painters in the same neural network. If we want to add new and distinctive examples to our training data, we may need to modify the network so that it can handle both the existing data and the new one. This kind of “radical” training set expansion is painful and time-consuming, but eventually enables neural networks to go to work interpolating new technologies between existing data and the new discoveries.

An Autopilot for Technological Progress

In Olsson’s model, incremental innovation proceeds as long as the paradigm remains productive (meaning all ideas in the convex hull have not been found). At the start of a paradigm, there are abundant opportunities for combining ideas. The returns to R&D are high. Over time, these ideas get fished out and the return on R&D falls. Throughout this period, random and serendipitous discoveries or anomalies are occasionally discovered, but they are left unexploited. As time goes on though, incremental innovation runs out (and this happens more quickly if there are more researchers). At that point, a period of difficult and groping R&D happens as firms engage in radical R&D. This requires interpolating between the known paradigm and the serendipitous discoveries. After some successes though, the convex hull is expanded and a period of incremental R&D starts anew.

Olsson meant to describe the cyclical nature of innovation by human civilization. But his model also provides an (admittedly speculative) blueprint for open-ended automated innovation by next generation neural networks. For as long as it’s valuable, the neural networks generate new discoveries by “filling in the gaps” of their training data with interpolations. Think of AlphaGo discovering better moves in Go, and style-transfer networks discovering new styles of painting, but also next generation neural networks discovering new molecules for drugs or material science, and new three-dimensional structures for buildings and vehicles (no links because I made those up).

And we could also automate non-incremental innovation. We could begin by programming the neural network to look for it’s own serendipitous discoveries.  In Olsson’s model, these come about by random luck. But in a neural network, we could program them to occasionally try something completely random (and outside the boundaries of their training set). This will almost never work. But on the rare occasions when the neural net tries something outside its training set that actually works, it can incorporate this information into it’s next training set. It can interpolate between this discovery and its existing ideas, and in this way it can escape the cage of its training set.

The barriers

For this kind of autonomous technological progress to work (at least!) two problems would need to be solved. The first we have already alluded to. Neural networks are quite domain specific. There is no guarantee they can even “see” examples that lie outside their training data, especially if different features of that data are what’s relevant. Maybe, we could build neural networks who are trained on the specific task of putting new data into a form usable for old neural networks… but we are well outside my area of expertise here.  I have no idea. In any event, maybe humans can do that task (not that it would be easy!).

The other barrier is the nature of the feedback a neural network would receive. Neural networks tune their internal structure according to well-defined goals, whether that goal is winning at Go or matching the style of a painter while preserving the content of an image. A neural network trained to deliver useful technologies would need to be trained on how valuable its discoveries are. How would that be determined? The answer is not so clear. In some cases, it’s relatively easy. If the neural network is generating new drugs, we can run clinical trials and see how they fare. But what if we’re developing polymers for material science or three dimensional structure? We can rate these discoveries on various criteria, but they may have unexpected and unanticipated uses. An alternative would be to let the market decide: after all, technologies that are profitable are ones that consumers will pay a lot for, relative to production costs, and this seems to be closely related to the value of an idea. But this solution is not without its own problems. For example, it might lead the neural networks to develop baubles for the super-rich.

I don’t intend to resolve this issue here. Indeed, how best to incentivize human innovators to focus their efforts where it is most valuable is an open question (and one I’ll explore in later posts)! But this need not distract us too much; my main point is to illustrate that it is at least possible for innovation to go a very long way, even if it’s primarily interpolative.

However, just because something is possible, doesn’t mean it’s a good idea. Might there be better ways to innovate? At the end of the day, neural networks are only one way to represent regularities in nature. In upcoming posts, we’ll discuss some of the others.


Neural Networks and Old Regularities

In my last post, I wrote about the possibility that neural networks can represent new regularities in nature. These regularities are impossible to concisely represent with the kinds of representations humans are comfortable with: chiefly rules, probabilistic statements, and metaphor. This can make their pronouncements seem eerie and magical. But neural networks (hereafter NNs) are nothing if not flexible, and can also represent old and familiar regularites. These are the type we can translate into easier-to-digest formats. And this is another avenue in which they can innovate.

To explore this, let’s talk about NNs copying famous painters.

For a variety of reasons, a lot of computer scientists are interested in teaching NNs to paint. Well, more precisely, how to generate images that apply a given artistic style to any photo. One of the early papers in this area is Gatys, Ecker, and Bethge (2015) (real “early” right?). In figure 1, they apply the style of Van Gogh’s Starry Night to a photograph of canal houses.

Figure 1 - row houses and starry nightFigure 1. Applying Van Gogh to an image via neural networks. Source: Gatys, Ecker, and Bethge (2015)

In contrast to the baffling genius of AlphaGo, this is a case where we can understand what the NN is doing. Copying the style of Van Gogh is not that mysterious. People do it all the time. Here’s a lovely painting of trees by Sylvie Burr.

Figure 2 - TreesFigure 2. Trees in the style of Van Gogh (creative commons, by Sylvie Burr)

This is not a case where regularities are mysterious and defy explanation. The regularities that characterize Van Gogh’s style include a texturing of thick swirling lines and a moody (rather than realistic) color palette. Show us examples of his style and even someone who has never seen his work will begin to pick out commonalities.

Neural Network Representations in images

NNs are also capable of representing those regularities. But, in an emerging theme, the way it represents those regularities is opaque. We can’t “tell” it what regularities characterize Van Gogh’s style. Instead, we give it lots of examples and let it rediscover these regularities on its own.

However, before we go on, we have to talk about a second set of regularities that a NN has to represent to transfer style. These are regularities in the content and perspective of images done in different styles. In Figure 1, we can tell that both images correspond to the same subject matter and view. They only differ in their styles. In contrast, figure 2 and the right-hand side of figure 1 have the same style (sort of), but clearly depict different subject matter. The NN has to represent both regularities in style and content.

It does this in different ways (I am drawing on this, and this for this section). For the computer scientists, “style” is understood as a form of diffuse texture. It is the curving lines and color palette of Van Gogh, not the composition and choice of subject (in this respect, they miss a lot). When they train a NN to match the style of a painter, the “style” of the image is converted into numbers corresponding to non-localized regularities over the whole surface of the image. For example, in figure 1, it doesn’t care much about making the left-hand side of the image dark (to match the mountain spire of Starry Night). Instead, it cares about matching the thick wavy lines and color palette of the entire image.  By assessing how much the numerical score of the NN’s style differ from that of the example, it can be evaluated. The NN’s weights, links, and thresholds (link to post 1) are tweaked in pursuit of this style target.

So much for style. To match the subject matter of a NN, the evaluation is done with respect to large-scale local regularities. In figure 1, it wants to see a recognizable sky on top, row houses in the middle, and water on the bottom.

Recognizing that two images are of the same subject and perspective, even if all their pixels are different, is closely related to the problem image classification. Image classification problems includes facial recognition (realizing two different images correspond to the same face) and labeling image content (as corresponding to, say, dogs, or cats, or “inappropriate content”). In all cases, we want to match regularities in the relative position of large chunks of visual image. For example, if we’re identifying faces, we might be comparing the relative size of the “nose” to the “mouth” and “eyes” (although actually we don’t know what the NNs are doing).

Identifying regularities in images with different styles is closely related to the image classification problem. So computer scientists actually borrow the NN representation of these image classification problems! In a literal sense, they start with the hidden layers of NNs trained on image recognition (including the nodes, links, weights, and thresholds of the NN) and simply copy them over to the style-transfer NN. And recall, we don’t really “understand” what the NN is doing to classify images. Here, the fact that we don’t “understand” means we struggle to translate what the NN is doing into metaphors and rules. But it’s not necessary for us to do that. The representation encoded by the NN still does what we want a representation to do: it conveys information about a regularity in nature. We don’t really “understand” the NN’s internal representation of the regularities in an image, but that doesn’t stop us from redeploying that representation in a new context. Does it reliably identify image content? Great, that’s all we care about!

Is this really innovation?

So, NNs are capable of representing regularities in image style and content in such a way that styles can be swapped and content retained. By my definition, this is an innovation: the NN has stepped into the unknown and exploited regularities to generate something a lot more “interesting” than a random collection of pixels. But it’s fair to say these innovations are not world-changing. Indeed, they can fairly be described as derivative. An artist who only copied other artists’ styles wouldn’t be described as innovative, even if he did apply those styles to new contexts.

This is related to a critique of NNs by NYU psychologist Gary Marcus:

In general, the neural nets I tested could learn their training examples, and interpolate to a set of test examples that were in a cloud of points around those examples in n-dimensional space (which I dubbed the training space), but they could not extrapolate beyond that training space. (p. 16)

Put another way, NNs are good at combining aspects of what they are trained on (the content of pictures, the style of painters), but they are always trapped in the cage of these examples. A NN trained in the above manner won’t ever take us beyond Van Gogh.

But this is not as much of a shortcoming as it seems. The ability to usefully combine aspects of disconnected things is, in fact, one of the fundamental creative acts. Indeed, Keith Sawyer (who wrote the book on creativity) defines innovation as “a new mental combination that is expressed in the world.” (pg. 7) I’ll briefly give examples in three different domains.

  • In art, borrowing and recombining ideas can be seen in super-literal forms. Think like Pride and Prejudice and Zombies, and mashup artists like GirlTalk. But it’s also there, just below the surface, in things that like Star Wars.
  • Earlier, I asserted all technologies are combinations of pre-existing components. The internal combustion engine is one clear example. The modern combustion engine is built from a dizzying set of components that were often pioneered elsewhere. To take two examples, crankshafts and flywheels together convert uneven back-and-forth motion of a piston into smoothly rotating energy. Crankshafts had previously been employed to transform the rotational motion in waterwheels and windmills into back-and-forth motion. And flywheels had long been used to give potter’s wheels smooth and continuous motion. (Dartnell, pg. 201-207)
  • Lastly, the product of sexual reproduction is of course a new organism that draws on a mix of genes from each of its parents. Over time, this mixing, matching, and selection generates entirely new species.

The difference between the above and what NNs are doing is a different of degree, rather than a difference of kind. Most of the innovation done by humans and nature is also bound by “the training space” of available examples.

Expanding the Training Set

The difference is that humans and nature have a vastly, vastly more diverse storehouse of training examples than NNs. It’s not possible for a NN trained to reproduce the style of Van Gogh to go beyond him, because the only examples of painting it has are those of Van Gogh. To develop a new style, it would need examples drawn from other styles of painting at a minimum. More importantly, to really generate something we’ve never seen before, the NN would need the capacity to interpolate between different styles. Is this possible?

Yes, and it’s been done. Dumoulin, Shlens, and Kudlur (2017) from Google Brain trained a single NN to transfer the styles of 32 different artists to new images. Because the same NN represented all these different styles, the NN is also capable of applying interpolations of their styles to images. Figure 3 is an example from their paper:

Figure 3 - Interpolative StyleFigure 3. Combining the styles of different painters (figure 7 from Dumoulin, Shlens, and Kudlur 2017)

In this figure, the style of Starry Night has been applied to a picture of Brad Pitt’s face in the upper left corner. Head of a Clown by Georges Rouault is the upper right style, The Scream by Edvard Munch is the lower left style, and Bicentennial Print by Roy Lichtenstein is the lower right style. In between we have interpolations between the different styles. Subsequent work by Ghiasi et al (2017) (a group that includes the same team as above) generalized these techniques to a much wider set of painting styles.

This work shows its possible for NNs to develop styles that did not previously exist. Are they any good? In the small set of examples given in Figure 3, I tend to like the interpolations between Rouault and Lichtenstein more than I like pure Rouault Pitt and pure Lichtenstein Pitt. But the main point of this post is simply to show NNs can innovate, even when they are using the kinds of regularities that humans are able to understand.

Now, what I am not claiming is that NNs can match humans in our ability to combine and interpolate between different ideas. Reading these papers, it’s clear that representing the different styles of painting in a NN was a major technical challenge that took considerable work to implement. Worse, their solution to this problem cannot be applied to problems and data different from the painting-style problem, at least without considerable modification (and maybe not even then). It is going to be a long time before a single NN can combine ideas and concepts from vastly different domains like us. But on the other hand, a lot of progress has been made in just 3 years.


Neural Networks and New Regularities

In my first post, I argued reality is full of regularities. Exploiting these regularities requires information about them that can be conveyed and communicated. I call this information representations, of which there are five categories: rules, probablistic statements, metaphors, neural networks, and instantiations. Today I want to talk about neural networks.

Neural Networks: A Very Basic Primer

If you are already familiar with neural networks, feel free to skip this section. This is a really simplified explanation, and I’m omitting a lot of detail . If you want to learn more, the chapter on neural networks in The Master Algorithm by Pedro Domingos is a recent overview. The medium series “Machine Learning is Fun!” is an excellent primer if you want to get your hands a bit dirty.

Neural networks were originally inspired by the structure of our brains. Somehow, these arrangements of biological matter are capable of thinking, computing, and learning. What is it about our brains that makes that possible?

Our brains are a network of neurons. Neurons are complicated creatures, but three of their most important features are:

  • They are connected to each other.
  • They can send signals of varying intensity (including negative or inhibitory signals) to each other.
  • They can turn more or less “on” as a function of the incoming signals

Neural networks jump off from there, replacing actual cells with simplified virtual counterparts. In a modern neural network, the “neurons” can belong to one of three layers: input, output, and hidden.

Input layers code for “features” of something. For example, if the neural network is designed to classify black and white images, each pixel in the image could be a feature associated with an input neuron. The brighter the pixel, the more “on” the input neuron. If the neural network recognizes speech, the intensity of sound waves at a certain frequency might be a feature. More intensity at that frequency might correspond to being “on” for one of the neurons. If the neural network is designed to play Go, the presence of a white stone at each point on the game board might be a feature. The neuron is “on” if a white stone is present at the input neuron’s associated point on the game board.

Output layers are how the neural network communicates with its users. If the network is an image classifier, there could be an output neuron for every class of photo. How “on” the neuron is could convey the confidence that the image belongs to that class. If the neural network is a speech recognizer, the output neurons could be text (words or letters). If the neural network is designed to play Go, the output neurons communicate the chosen move (there may be a neuron for every space, and the one the network would like to place a stone turns on).

Hidden layers lie between the inputs and output layers, and do the “thinking” of the network. In modern neural networks there can be many hidden layers of neurons. Together, they form a complex web of connections, each with potentially positive or negative weights, and with each neuron having a potentially different threshold for activation. The operation of a neural network is about the propagation of signals forward through the network. The input neurons corresponding to the data’s features turn on and send signals to connected hidden neurons. Each of those neurons adds up the strength of the incoming signals (which can each be positive or negative), and depending on how much the sum exceeds a threshold, turn more or less on. The hidden layer neurons then send signals to the neurons connected to them. This process propagates forward until some of the output neurons are activated, communicating the “thoughts” of the neural network.

That’s it. Nothing magical is happening. The propagation of signals at each step happens according to relatively simple rules. The big picture is complicated, but only because there are so many of these small steps, and because they build on each other.

A useful thing about neural networks is that you don’t actually have to set the connections, signal strengths, and thresholds. If you have enough data, there are algorithms that can do all that automatically. Give the neural network an example set of features and see what its output neurons say. If they are wrong, propagate back from the outputs adjustments to the signals and thresholds so that the network is more likely to make the correct identification the next time it sees similar features.

Do our brains really work this way? I certainly don’t sense neurons signalling to each other and adding up thresholds. To the extent our brains work this way, the hidden layers are unconscious and we are only conscious of the output. We see a face, and we experience the thought “oh, that’s Grandma” leaping into our head unbidden. But underlying this recognition is (possibly) a biological neural network trained from birth to identify and categorize faces. The light coming into our eyes gets shunted off to various input neurons, which then propagate that information through hidden layers until the correct output neuron (or set of neurons) lights up. At that point, I have the conscious thought “Grandma!”

Neural Networks and Representations

At a more abstract level, neural networks represent local regularities in the data they are trained on with the structure of the network (including its signal strengths and each neuron’s threshold). This turns out to be a very flexible way of representing regularities. Indeed, neural networks can represent regularities that are difficult to concisely express in alternative schemes such as rules, probabilistic statement, and metaphor.

There are two sides to this coin. On the one hand, neural networks are capable of representing regularities that simply can’t be concisely represented any other way. On the other hand, the very nature of these regularities is such that they defy translation into alternative forms of representation. There is no metaphor, rule, or probability that “explains” the decision-making of the neural network (except a rule that tediously describes the neural network’s complex inner-structure).

Take this excerpt from Part IV of “Machine Learning is Fun!”, which describes how to train a neural network to identify faces:

…the neural network learns to reliably generate 128 measurements for each [face]. Any ten different pictures of the same person should give roughly the same measurements.
…So what parts of the face are these 128 numbers measuring exactly? It  turns out that we have no idea. It doesn’t really matter to us. All that we care is that the network generates nearly the same numbers when looking at two different pictures of the same person.

The tradeoff is that neural networks allow us to learn regularities that can’t easily be represented in our favored modes, but at the cost of unintelligibility.

Neural networks are also capable of innovating, in the sense of exploiting these regularities to step into the unknown and make a choice. But since the regularities they exploit are foreign to us and our very way of thinking, the innovations they come up with may seem mysterious and almost magical.

This is demonstrated really well in the netflix documentary AlphaGo.

AlphaGo and Unknown Regularities

AlphaGo’s 2016 match against Lee Sedol, as captured in the documentary AlphaGo, is a great example of how neural networks can exploit local regularities we don’t understand. AlphaGo is a program designed to play the board game “Go.” The game is relatively straightforward: two players take turns laying stones on a 19×19 grid. The goal is to enclose more territory than the opponent. However, because the board is so large, the set of possible games is enormous. There are 361 possible positions for the first stone, 360 possible positions for the second, 359 for the third, and so on. There are over 5.9 trillion possible positions of the first five stones. This enormous space of possible games has long meant that brute force calculation doesn’t work very well, even for computers. Go has been played for so long, and by so many though, that a large number of local regularities related to the game have been identified.

AlphaGo does not rely completely on neural networks, but they are a prominent component of its programming. In 2016 AlphaGo faced off against Lee Sedol, a legendary Go player considered to be the greatest of the last decade. Throughout the five game match, AlphaGo made surprising moves that baffled commentators, but later paid off. Move 37 in game 2 is a wonderful illustration:

Commentator #1: Oh, wow.
Commentator #2: Oh, it’s a totally unthinkable move.
Commentator #1: Yes.
Commentator #3: The value… that’s a very… that’s a very surprising move.
Commentator #4: (chuckling) I thought it was a mistake.
Fan: When I see this move, for me, it’s just a big shock. What? Normally, humans, we never play this one because it’s bad. It’s just bad. We don’t know why, it’s bad.

Fan (a European Go champion) makes my point very well. Humans have played Go a long time, and they have internalized certain regularities (indeed, in this case, the regularity that this is just a bad move is known without being understood why!). AlphaGo is playing a move based on regularities unappreciated by human players. In fact, one of the creators later peers inside AlphaGo’s program and learns AlphaGo assigns a probability that a human would play this move at 1 in 10,000.

As the game unfolds, the brilliance of the move becomes clear. Lee Sodol later discusses this move:

Lee Sodol: I though AlphaGo was based on probability calculation and it was merely a machine. But when I saw this move, I changed my mind. Surely, AlphaGo is creative. This move was really creative and beautiful… This move made me think about Go in a new light. What does creativity mean in Go? It was a really meaningful move.

AlphaGo goes on to win the game, with move 37 eventually seen as a turning point.

I don’t the nature of the regularity that AlphaGo exploited. It might have been the kind of thing easily explained but something human players had simply missed for millenia. That strikes me as unlikely. It might have been something easy to explain (if AlphaGo had been trained to do so), but only exploitable if you have the capacities of AlphaGo (e.g., the ability to track dozens of positions in parallel). I prefer to believe it was something new: a regularity impossible to express in our favored forms.

Uncanny Genius

While neural networks are inspired by the brain, it’s not clear the extent to which our brains actually work that way. I don’t have the expertise to weigh in on this debate. But, to conclude, let’s assume the picture painted above is broadly applicable to the human brain. Doing so can provide a plausible explanation for the mysterious judgments of geniuses, when they simply intuit an answer with uncanny precision, unable to provide an explanation for their insight.

Earlier, I gave the example of how brain structures, organized like neural networks, could identify your grandma from a sea of faces. The interesting thing here is that we experience this as automatic. We just “know” that’s grandma, without access to the underlying categorization process in our own heads.

Our ability to just “know” Grandma’s face doesn’t strike us as particularly mysterious. But when well trained neural networks in our brains do less common things, it can seem mysterious and magical. An expert in a particular domain – mathematics, engineering, science – sees countless examples in their domain over a career. Each example trains their internal representation of the domain. Then, one day, facing a novel situation they just know what to do. And their inability to explain themselves leaves observers dumbstruck and in awe.

I’ll close with an example of this from Gary Klein’s study of firefighters (recounted in Superforecasting by Phillip Tetlock and Dan Gardner). An experienced firefighter commander is combatting a routine kitchen fire that is behaving a bit strange. The commander is seized with an uneasy feeling. He orders everyone out of the house. Moments later, the floor collapses. It turns out the true source of the fire had been the basement.

How did he know trouble was afoot? We can imagine the neurons and connections in the firefighter’s head were tuned by countless experiences with fire, until their structure encoded regularities in fires impossible to clearly express in rules, metaphor, or probabilities. Just like AlphaGo the commander was unable to explain how he knew to get out. He described it as ESP.


What is innovation? Fundamentals


“The thing that hath been, it is that which shall be; and that which is done is that which shall be done: and there is no new thing under the sun.” – Ecclesiastes 1:9 (King James Bible)

Ecclesiastes is wrong. New things happen all the time these days. Airplanes, iPhones, wikipedia, nuclear weapons, chemotherapy, and skyscrapers were new things. Evolution, quantum mechanics, general relativity, Marxism, and computer science were new things. Facebook, Amazon, SpaceX, Microsoft, and Walmart were new things. Star Wars, Harry Potter, Guernica, and YMCA (the song) were new things. Shakespeare would have been new to the cynical narrator of Ecclesiastes. And new things had been emerging for millions of years when Ecclesiastes was written. True, at the time it would have been hard to see novelty. But mammals, fish, insects, and multi-celled life were all new long before Ecclesiastes was new.

Did these things bear no resemblance to that which came before? Of course not. Do they resemble various antecedents? Certainly. Much of this series will explicitly explore these resemblances. But to say these are therefore not “really” new is to miss something vital.

This is a blog about the new things under the sun. It is a blog about innovation and the emergence of reproducible novelty.  This is the first in a planned series of posts answering the question “how does innovation happen?” This introduction sets the series up and lays out the way I think about innovation. With that foundation established, I hope the remaining posts in the series will stand on their own.

Defining innovation

By innovation, I mean something quite broad. For sure I mean to include new physical technologies, such as the iPhone. But I want innovation to also encompass a much wider class of human creation. By innovation, I also mean to include academic theories. And art. And I want to include new ways of organizing people – e.g., into armies, churches, and companies. And when I say innovation, I don’t even want to limit it to humans. Nature innovates in the proliferation of countless new species and life forms. And one day humans might outsource their innovation to artificial intelligence.

For the purposes of this blog, I adopt a big tent definition: innovation is the emergence of things that are novel, interesting, and reproducible. “Novel” and “interesting” are self-explanatory, albeit a bit in the eye of the beholder. In contrast, it may not be immediately obvious why reproducibility ought to matter in a definition of innovation. Reproducibility implies two main things.

First, it limits us to things that can, in principle, be classes rather than specific instances. If the thing is a complete one-off then it’s not an innovation, even if it’s new and interesting. For example:

  • The iPhone was an innovation because Apple can (and does) churn them out by the millions. Steve Jobs’ personal iPhone might be interesting (for example, a museum might want to buy it). It was new at one time. But we can’t make more of them, so it’s not an innovation. It’s an artifact.
  • The “ride-share platform firm” is an innovation with several examples (Uber and Lyft being most prominent). Uber itself isn’t an innovation because it’s a singular firm. An exact copy isn’t possible and wouldn’t be Uber… it would be a ride-sharing firm. But if Uber develops a new strategy that others can copy, at least in principle, then that new strategy would be an innovation.
  • The human species was an innovation when it emerged. Jeff Bezos is not an innovation because we can’t make more of him, even if he was new at one time and is an interesting individual.

So when I speak of innovation, I am talking about the creation of a new blueprint for a category of things. I’m not talking about a new and interesting singular instance. However, if I were to leave it at that, I would be leaving in a lot of things we usually don’t think of as innovation. Specifically, I would be leaving in anything new and interesting that happens repeatedly. Things like car crashes, volcanic eruptions, and sunsets.

This is where the second implication of reproducibility comes in. When I say reproducible, I mean the replication requires access to information embodied in the thing. It requires access to a blueprint, or, failing that, access to the thing itself. In this specific (idiosyncratic?) sense, car crashes, volcanic eruptions and sunsets are not reproducible. Instead, they are just things that happen whenevever physical conditions are right. This might be often, so that they happen repeatedly. But in each instance, there is no reference to earlier instances or blueprints. Reproducing a new thing means using information to recreate it without rediscovering it.

In this series, the primary objects of study will be organisms, technologies, organizations, and ideas. Sometimes the boundaries between these objects is fuzzy. Don’t get hung up on that. The distinctions aren’t really important, so long as they are new things under the sun.

What’s the challenge?

So innovation requires creating something new, interesting, and reproducible. Of these, the first two are the really hard part.

It’s not very hard to come up with something new. Kick a bucket of blocks, scatter paint on a wall, or string letters together in random gibberish. With enough stuff, almost any bit of disorder is an unprecedented configuration of atoms. Neither is it very hard to make things reproducible. If you make the incentives strong enough, a dedicated person can carefully place the blocks, paint the scattered pattern with a brush, or type out letters. Indeed, with enough care and description, lots of disorder can be reproduced. But if it’s not interesting, why would anyone want to?

“Interesting” things are rare in the universe. A blind leap into the unknown will frequently be new, and if you take good notes, reproducible. But a blind leap into the unknown will almost certainly fail to discover something interesting.

This intuition underlies a common (but misguided) argument against evolution by natural selection and for intelligent design ( the chances a randomly swirling cloud of atoms will settle into the shape of DNA (much less a living cell) is lower than the probability of a tornado whirling through a junkyard and assembling a Boeing 747 by chance. As a basic premise, I think that’s probably right. And it seems to be a fundamental operating principle of the universe. A bunch of monkeys banging on keyboards won’t produce Shakespeare by chance. Well, unless you have a long time to wait and a lot of monkeys. And only children think you can invent something useful by randomly connecting disparate bits of technology.

Thus the challenge of innovation is finding a way to take a leap into the unknown and to find something interesting when you land. The key is, innovation is not a random leap into the unknown. Innovation is a considered leap into the unknown.

What guides our leaps is knowledge of regularities in the world.

A regularity is a pattern in reality. The laws of physics are regularities, but most regularities do not rise to this universal level. A regularity may only hold in your local environment. It may be temporary. It may be inconsistent. To be useful to an innovator, it just needs to be exploitable. With information about regularities, innovators do far better than random chance.

This will be clearer with some examples.

Regularities in technologies, organisms, organizations, and ideas

Let’s start with physical technologies. In The Nature of Technology: What it is and How it Evolves, economist Brian Arthur makes the observation that all technologies are built from sub-components. These sub-components are themselves composed of sub-sub-components, which are in turn composed of sub-sub-sub-components and so on. The F-35 jet, for example, is composed of:

the wings and empennage; the powerplant (or engine); the avionics suite (or aircraft electronic systems); landing gear; flight control systems; hydraulic system; and so forth. If we single out the powerplant (in this case a Pratt & Whitney F135 turbofan) we can decompose it into the usual jet-engine subsystems: air inlet system, compressor system, combustion system, turbine system, nozzle system. If we follow the air inlet system it consists of two boxlike supersonic inlets mounted on either side of the fuselage, just ahead of the wings. The supersonic inlets… (Arthur 2009, pg. 40)

And on and on until we arrive at some “raw element” of technology. Arthur argues that these fundamental elements are “captured natural phenomena.” I prefer the term exploited regularities. Arthur provides a non-exhaustive list of regularities (pg. 52) that are commonly exploited in technology (I have broken them up into bullets):

  • A fluid flow alters in velocity and pressure when energy is transferred to it (used in the compressor);
  • certain carbon-based molecules release energy when combined with oxygen and raised to a high temperature (the combustion system);
  • a greater temperature difference between source and sink produces greater thermal efficiency (again the combustion system);
  • a thin film of certain molecules allows materials to slide past each other easily (the lubrication systems);
  • a fluid flow impinging on a movable surface can produce “work” (the turbine);
  • load forces deflect materials (certain pressure-measuring devices);
  • load forces can be transmitted by physical structure (load bearing and structural components);
  • increased fluid motion causes a drop in pressure (the Bernoulli effect, used in flow-measuring instruments);
  • mass expelled at a velocity produces an equal and opposite reaction (the fan and exhaust propulsion systems).

Technological elements exploit these and many other regularities to do something interesting. In Arthur’s framing, the interesting thing technologies do is “fulfill a human purpose.” These elements are then combined with others, and built up into modules and assemblies and other subcomponents, which are themselves rearranged and recombined with others, until a set of interdependent regularities are coordinated and arranged to perform some complex desired task. The exploited regularities in a technology are like the disparate voices of instruments in an orchestra, coordinated by score and conductor to produce music.

It isn’t only the “atomic elements” of technologies that exploit regularities. Frequently, a suite of components collectively allows for new regularities to be exploited. For example, the atomic bomb taps into regularities about the behavior of highly concentrated Uranium isotopes. At a certain density, naturally occuring atomic decay can trigger a chain reaction, with the release of tremendous energy as a side-effect. Accessing this regularity, however, requires an entire mini-industry of other technologies to create the uranium and push its density past the critical point.

And so technologies fundamentally rely on the exploitation of regularities to do something interesting. But this applies to organisms as well as physical technologies. After all, in a sense, organisms are nothing but very complicated machines. They even share the hierarchical nature of technologies. Our bodies are built from a set of organ systems, which are themselves composed of differentiated tissues, which are themselves built from an army of distinct cell types, which are themselves assembled from a huge collection of molecular machines.

And these tiny molecular machines run on regularities in nature, just like human technologies. Life’s Ratchet: How Molecular Machines Extract Order from Chaos by Peter Hoffman provides an overview of the regularities exploited by the smallest units in our bodies. These regularities differ from the kind exploited by human technology, because they exist only at the tiny scale within a cell. Yet, in large numbers, carefully orchestrated, they produce us. Examples include:

  • electric charges of the same sign repel, opposite ones attract (used to “lock” proteins and molecular machines into stable configurations)
  • thermal energies are random (used to jostle molecular machines out of stable configurations)
  • at the nano-scale, thermal and electrostatic forces are approximately equal (used by molecular machines to move from one stable configuration to another)
  • events that individually occur with very small probability almost certainly occur at least once, given many chances (used by cells to ensure desirable small probability events happen at least once)
  • In a positive feedback loop, a small initial cause can have a large final effect (used by cells to amplify desirable events that happen rarely)

The regularities listed above have the character of universal natural laws. But organisms also rely on mere local regularities. For example, many plants cannot grow well when moved to new latitudes. They use the length of day as a clock to coordinate their growing cycles and moving them to a new latitude, where the length of day changes differently over the year, disrupts these signals. As an aside, note again that the large-scale organism exploits regularities (e.g., length of days over a year) that are not used by the individual molecular machines that comprise it. The sum is more than its parts.

And neither are organisms the only things that rely on local regularities. Arthur, for instance, points out that a whole host of technologies break down when moved into space. They require gravity to function as intended. More broadly, there is a rich literature in the economics of growth about “local technologies.” Most technologies are invented in wealthy countries, and are often less productive when deployed in poorer ones. Tacit in their successful operation are all sorts of assumptions about regularities in the prices of inputs, the knowledge of users, the tasks they will be asked to perform, and the availability of complementary goods and services.

Next, what we have said already about technologies and organisms can also be applied to human organizations. They too organize themselves in branches, divisions, phalanxes, units, and so on. And they too exploit regularities. It’s just that these regularities pertain to human society. More so than natural phenomena, the regularities in society are local. Here are a few regularities in human society, ordered roughly from the more to less universal:

  • humans are adept at learning to copy new tasks, if shown how
  • humans can work with attention to detail for 4-8 hour blocks
  • more of a good or service will be demanded if the price declines
  • human desires for specific goods and services are relatively stable
  • people will accept fiat currency as a unit of exhange
  • money can safely be invested and earn a 2% return
  • the electricity will only rarely go out
  • most consumers know how to use the apple app store

Lasly, art too, relies on regularities. Being “interesting” in art can mean being “thought-provoking”, “beautiful”, “novel”, and other things. Art relies on regularities at several levels. Audio-visual forms exploit regularities in physical laws to transmit sound and light. They may also exploit regularities in human psychology to elicit reactions in viewers.  For example, certain techniques in horror movies can regularly elicit feelings of tension and surprise in audiences. At another level, art can exploit cultural regularities. Certain symbols, motifs, and themes may be more or less recognizable to the audience.

I’ll stop the tour at this point. We’ve seen briefly how technologies, organisms, organizations, and art all exploit regularities. Some of these regularities hold fairly universally. Others are highly local. Later, I will assert that innovations in the narrative arts and knowledge itself rely on regularities. But first, we need to introduce one more concept: representations.

Representations and Regularities

Regularities are “out there” in nature, waiting to be exploited. But to reproduce an innovation, you need a way to communicate the regularity to the reproducer. A representation is information about a regularity that can be communicated and conveyed.

There are several kinds of representation. In this series I focus on five. Their borders are fuzzy, rather than sharp. In later posts, I’ll talk about each in more detail. Here I briefly introduce them.


This includes logical statements, causal assertions and quantitative equations. At bottom rules insist things are related in a certain, definite way. The discussion of regularities in the preceding section frequently used rules as representations. Examples included:

  • “a fluid flow alters in velocity and pressure when energy is transferred to it”
  • “electric charges of the same sign repel, opposite ones attract”
  • “more of a good or service will be demanded if the price declines”

Other examples of rules include:

  • Tuesday comes after Monday.
  • All mammals are warm-blooded, have hair of some sort, and mammary glands.
  • Every reaction has an equal and opposite reaction.
  • The pythagorean theorem.
  • E = mc^2
  • Light travels at 299,792,458 meters per second.

Humans seem to prefer expressing regularities as rules whenever possible. Rules are clear and straightforward. They can be chained together to form new rules. Alas, many of the regularities in nature cannot easily or concisely be expressed as rules.

Probabilitistic statements

In a host of domains definite ironclad rules don’t exist. Things “usually” work one way, or “sometimes” work. Some notion of uncertainty, randomness, and probability is necessary to communicate many regularities. Probabilty allow us to exploit regularities in domains where we lack the information or computational power to develop rules. We also used probablistic statements in our preceding dicussion of regularities. Examples included:

  • “technologies [invented in rich countries] are often less productive when deployed in poorer ones”
  • “events that individually occur with very small probability almost certainly occur at least once, given many chances”
  • “the electricity will only rarely go out”

Other examples of probablistic statements include:

  • There is 50% chance of getting heads when you flip a fair coin.
  • Hillary Clinton has a 70% probability of winning the 2016 presidential election.
  • Thunderstorms sometimes generate tornadoes.
  • Most people are less than 7 feet tall.
  • The probability of a given quantium state is equal to its amplitude squared.

Despite their utility, humans are often resistant to probabilistic thinking, and frequently compress probabilistic statements into rules (for example, many interpreted a 70% probability that Hillary Clinton would win simply as “Hillary is going to win”). The book Superforecasting by Philip Tetlock and Dan Gardner provides a good overview of human resistance to thinking probabilistically.

Nonetheless, rules and probabilistic statements are the preferred representation in the quantitative sciences. There are a whole host of rules and probabilistic statements (the field of statistics) that can be applied to them. We can use these meta-rules to combine, transform and derive new probalistic statements from old ones. But rules and probalistic statements are relatively new in the history of the universe. We now turn to older forms of representing regularities.

Metaphors and analogies

A fuzzier but even more widespread form of representation is the metaphor or analogy: this thing you don’t know so well is like this other thing you do know. Metaphors and analogies tie together bundles of attributes and properties. They allow us to map them from one setting to another that shares some of the same attributes and properties. There are some who argue metaphors and analogies are the basic structure of thought (Surfaces and Essences: Analogy as the Fuel and Fire of Thinking by Douglas Hofstadter and Emmanuel Sander is one example I hope to write about later).

I too have used metaphor and analogy to represent regularities in this post. Examples include:

  • “the challenge of innovation is finding a way to take a leap into the unknown and to find something interesting when you land.”
  • “like the disparate voices of instruments in an orchestra, coordinated by score and conductor to produce music.”
  • “After all, in a sense, organisms are nothing but very complicated machines.”

Innovation is not literally about jumping into strange places. But that is an idea we have some intuitions about and those intuitions can be tranferred. Technologies aren’t literally orchestras. But our intuitions about the scale and complexity of how an orchestra unifies disparate sounds is a useful intuition to transfer. There are differences between organisms and machines. But not for the purposes of the discussion at hand.

Metaphors are natural to us and capable of capturing complex regularities not easily expressed as rules and probabilistic statements. And they are useful for guiding our excursions into the unknown.  For example, someone who is familiar with dogs knows they generally have a bundle of certain attributes: hair, four legs, sharp teeth, etc. Suppose that person encounters an unknown animal with many of these same attributes, perhaps a wolf. If it starts growling at them, they can use the analogy of dog behavior to infer what might happen next (they might get bit). But the analogy gets less useful the farther the new example is from the category’s archetypes. Do lions growl before they strike? What about alligators? What about a man in a gorilla suit?

Neural Networks

The previous three types of representation are familiar to us, because they can all be conveyed in language, and this is the primary way human convey information to each other. But there are other ways to encode regularities as information. Our brains, for example, gives us intuitions and “feelings” we cannot always express in language. And babies and animals can learn regularities without language too.

It turns out regularities can also be represented in the structure of special kinds of interconnected networks. The brain probably operates, at least in part, on these principles. Even if does not, recently, this method of representing regularities has been highly successful in artificial intelligence. We will come back to this form of representation, because it is not easy to describe succinctly if one is not already familiar with the concept.


Finally, we come to the oldest, simplest, and most robust way of representing regularities: as an instantiation of whatever is exploiting the regularity.

For example, suppose I discover an alien technology operating on principles completely foreign to me, and lacking any explanatory text. I may still be able to reproduce it by exactly replicating the technology (atom by atom if necessary). More prosaically, there are frequently techniques and processes that are very difficult to convey in language. For example, teaching someone how to shoot a basketball is probably much easier to do by demonstration than by any form of written or verbal communication. Complicated lab techniques may also need to be demonstrated rather than communicated. It may even be that the performer can’t explain why this technique works. They simply know “if you copy this technique, it will work.”

This is the type of representation most frequently used by nature itself. Nature stores the regularities it has discovered in the physical design of its organisms, who pass on instructions about how to replicate themselves, rather than instruction about the regularities they exploit.

Narratives and Knowledge; Representations, and Regularities

We now return to knowledge and narratives, which I earlier asserted also rely on regularities. Let’s begin with knowledge. When we think of innovation in knowledge, I am essentially referring to new explanations and theoretical constructs, rather than the documentation of new phenomena. For my purposes, “explanations” are essentially a more informal form of “theories” (the kind of thing often laid out in an academic article or book).

The unusual thing about explanations and theories is that the raw materials that they are made from are representations themselves. Explanations and theories in their simplest form essentially are representations. In more complex forms, they are large sets of ordered and interacting representations. They deploy rules, probabalistic statements and metaphor to convey information about regularities.

Inventing new explanations and theories is a matter of discovering new representations, or more likely, combining previously known representations in novel and interesting combinations. What makes an explanation or theory insteresting? Here, being “interesting” might mean the explanation “allows one to make predictions about the world” (the classic Popperian model of a good theory). But it could also mean the explanation is “plausible,” “provocative,” “beautiful,” “thought-provoking,” “widely applicable”, and so on.
Moreover, just as a random collection of technological components is unlikely to do something interesting, a random collection of representations is also unlikely to be interesting. Interesting explanations and theories exploit meta-regularities about the relationships between different representations. The rules of logical inference, for example, form a set of meta-regularities about representations. One of these meta-regularities, represented as a rule, might say “If you have a representation of the form ‘if A then B’ and a representation of the form ‘if B then C’, then the representation ‘if A then C’ corresponds to a regularity as well.” Mathematical and statistical operations of all stripes fall under a similar meta-regularity. Other examples inclue:

  • Occam’s Razor: the simplest representation is most likely to best express a regularity
  • Accordance with data: representations that generate predictions that match data are more likely to accurately represent regularities.

Finally, what of narrative art? As noted, to be interesting in art can mean a lot of things: “thought provoking”, “revelatory”, “beautiful”, “thrilling”, “surprising”, etc. Narrative arts can use a variety of techniques to elicit these responses, but a plausible match with regularities in the world forms a unique aspect of narrative art. Do the character’s motivations “make sense” in the context of the world? That is, do the character’s actions and thoughts align with the audience’s pre-existing internal representations about how people act? In some settings, a book explores unfamiliar settings, in which the audience knows few regularities to judge the plausibility of the narrative. In these kinds of works, sometimes the interest is the uncovering of new regularities in the fictional world. It is a regularity in our nature that we enjoy discovering regularities! (Indeed, I’m banking on that to find an audience for these posts!)

A Theory of Innovation

To summarize.

Interesting things tend to be interesting when they are hierarchical systems of interacting components that exploit regularities in the world. Creating new interesting things (innovation) requires leaping into the unknown and assembling a new system. If we had nothing to guide us, these leaps would rarely pay off. Innovation is possible largely because it uses regularities about the way the world works to guide its leaps. It assumes the regularities that hold in the world we know will continue to hold in the unknown (a leap of faith not always validated!). Innovators do not directly perceive regularities, but rather representations of them. Representations can take several forms, and I focus on five: rules, probabilistic statements, metaphors, neural networks, and instantiations. Finally, representations can themselves be woven together into hierarchical systems called explanations and theories. These “innovations” are themselves interesting when they exploit meta-regularities about representations.