A question of hardware capability?
When can we expect the Singularity? What kind of hardware would be required for an artificial cortex? How far out into the future of Moore’s Law is such technology?
The startling answer is that the artificial cortex, and thus the transition to a profoundly new historical era, is potentially much closer than most people realize. The problem is mainly one of asking the right questions. What is the computational power of the human brain? This is not quite the right question. With a few simple tools a human can perform generic computation – indeed computers were human long before they were digital (see the history of the word: computer). The computational speed of the human brain aided with simple tools is very very low, less than one operation per second. Most studies then of reverse engineering the human brain are really asking a different question: how much digital computation would it require to simulate the human brain? Estimates vary, but they are usually of order near 10^15 – quadrillions of operations per second or less for functional equivalence, up to around 10^18 for direct simulation, plus or minus a few orders of magnitude. The problem with this approach is its similar to asking how much digital computation would it require to simulate a typical desktop processor by physically simulating each transistor. The answer is surprising. A typical circa 2010 desktop processor has on the order of a billion transistors, which switch on the order of a few billion times per second. So simulating a current desktop processor using the same approach that we used to estimate brain capacity gives us a lower bound of a billion billion or 10^18 operations per second, realistically closer to 10^20 operations per second required to physically simulate a current desktop processor in real-time – beyond the upper ranges for typical estimates of simulating the human brain in real time. This is surprising given the conventional wisdom that the human brain is so much more complex than our current computers, so its worth restating:
If we define computational time complexity as the number of operations per second required to simulate a physical system on a generic computer, then current desktop processors circa 2010 have already exceeded the complexity of the human brain.
This space-time complexity analysis can be more accurately broken into two components: space complexity and speed. Space complexity is simply the information storage capacity of the system, measured in bits or bytes. Brains get their massive information capacity from their synapses, which can be conservatively estimated as the equivalent to a byte of digital storage each, thus giving an upper bound of around 10^15 bytes for directly storing all the brain’s synapses – a petabyte of data storage, down to around a hundred terrabytes depending on the particular neuroscience estimate we use. Personal computers now have hard drives with terrabytes of storage, and supercomputers of 2010 are just now hitting a petabyte of memory capacity, which means they have the base storage capacity required to comfortably simulate the brain completely in RAM. Clearly brains have a big advantage in the space complexity department: their storage density is several orders of magnitude greater than our 2010 electronics (although this will change in about another 10-15 years of moore’s law). However, along the speed dimension the advantage completely flips: current silicon electronics are about a million times faster than organic circuits. So your desktop processor may only have the intrinsic spatial complexity of a cockroach, but signals flow through its circuits about six orders of magnitude faster – like a hyper accelerated cockroach. Using one computational system to simulate another always implies a massive trade-off in speed. The simplest modern processor cores (much simpler than the intel CPU you are using) uses hundreds of thousands to millions of transistors, and thus even if we could simulate a synapse with just a single instruction per clock cycle, we may only just barely manage to simulate a cockroach brain in real-time. And note that the desktop processor would never be able to naively simulate something as complex as a human brain without vastly increasing its memory or storage capacity up to that of a super computer. And even then, running on supercomputers detailed brain simulations to date achieve only a small fraction of real-time performance: much less than 10%. It takes a human brain years to acquire language, so slow simulations are completely out of the question: we can’t simulate for 20 years just to see if our brain model develops to the intelligence level of a two year old! Clearly, the speed issue is critical, and detailed simulation on a generic computer is not the right approach.
Capacity vs Speed
The memory capacity of a cortex is one principle quantitative measure underlying intelligence – a larger cortex with more synaptic connections can store and hold more memory patterns, and perform more total associative computations every cycle in direct proportion. Certainly after we can match the human brain’s capacity, we will experiment with larger brains, but they will always have a proportionally higher cost in construction and power. Past some point of scaling a brain 2 or 4 or X times larger and more expensive is probably not an improvement over an equivalent number of separate brains (and the distinction further blurs if the separate brains are networked together through something like language). On this note, there are some reasons to believe that the human brain is already near a point of diminishing returns in the size department. Whales and elephants, both large advanced mammals with plenty of room for much more massive capacities, sport brains built with a similar order of neurons as humans. In numerous long separated branches of the mammalian line, brains grew to surface areas all within a narrow logarithmic factor: around 2,500 cm^2 in humans, 3,700 cm^2 in bottlenose dolphins, and around 6,000-8,000 cm^2 in elephant and whale lineages. They all compare similarly in terms of neuron and synapse counts even though the body sizes, and thus the marginal resource cost of a % increase in brain size vary vastly: a whale or elephant brain is small compared to its body size, and consumes a small portion of its total resources. The human brain definitely evolved rapidly from the hominid line, and is remarkably large given our body size, but our design’s uniqueness is really a matter of packing a full-sized large mammal brain into a small, crammed space. The wiring problem poses a dimensional scaling constraint on brain size: total computation power scales with volume, but non-local communication scales with surface area, limiting a larger brain’s ability to effectively coordinate itself. Similar dimensional scaling constraints govern body sizes, making insects insanely strong relative to their size and limiting the maximum plausible dimension of land animals to something dinosaur sized before they begin to fall apart. A larger brain developed in humans hand in hand with language and early technology, and is probably optimized to human’s age: providing enough pattern-recognition prowess and capacity to learn complex concepts continuously for decades before running into capacity limits. The other large-brained mammals have similar natural ages. Approaching the capacity limit we can expect aging brains to becoming increasingly saturated, losing flexibility and the ability to learn new information, or retaining flexibility at the expense of forgetfulness and memory loss. Its thus reasonable to conclude that the storage capacity of a human brain would be the minimum, the starting point, but increasing capacity further probably has a only a sublinear increase in effective intelligence. Its probably more useful only in combination with speed, as a much faster thinking being swould be able to soak up knowledge proportionally faster.
The Great Shortcut: Fast Algorithmic Equivalence
We can do much, much better than simulating a brain synapse by synapse. As the brain’s circuits are mapped, we can figure out what fundamental computations they are performing by recording mases of neuron data, simulating the circuits, and then fitting this data to matching functions. For much of the brain, this has already been done. The principle circuits of the cortex have been mapped fairly well, and all though there are still several competing implementation ideas at the circuit level, we have a pretty good idea of what these circuits can do at the more abstract level. More importantly, simulations built on these concepts can accurately recreate visual data processing in the associated circuits that is both close to biologically measured results and effective for the circuit’s task – which is in this case is immediate fast object recognition (for more details, see papers such as: “Robust Object Recognition with Cortex-Like Mechanisms“.) As the cortex – the great outer bulk of the brain – reuses this same circuit element throughout its surface, we can now see possible routes for performing equivalent computations but with dramatically faster algorithms. Why should this be possible in principle? Several reasons:
- Serial vs Parallel: For the brain and its extremely slow circuits, time is critical and circuits are cheap – it has so many billions of neurons (and hundreds of trillions of synapses) that it will prefer solutions that waste neuronal circuitry if they reduce the critical path length and thus are faster. From the brain’s perspective, a circuit that takes 4 steps and uses a million synapses is much better than one which takes 30 steps and uses a thousand synapses. Running on a digital computer that is a million times faster and a million times less parallel, we can choose more appropriate and complex (but equivalent) algorithms.
- Redundancy: Not all – if any – synapses store unique data. For example, the some hundred million neurons in the V1 layer of each visual cortex all compute simple gabor-like edge filters from a library of a few dozen possible orientations and scales. The synapse weights for this layer could be reused and would take up memory in the kilobytes – a difference of at least 6 orders of magnitude vs the naive full simulation (where synapse = byte). This level of redundancy is probably on the far end of the scale, but redundancy is definitely a common cortical theme.
- Time Slicing: Only a fraction of the brain’s neurons are active at any one point in time (if this fraction escalates too high the result is a seizure), and if we ignore random background firing, this fraction is quite low – in the range of 1% or possibly even as low as 0.1%. This is of course a net average and depends on the circuit – some are more active than others – but if you think of the vast accumulated knowledge in a human mind and what small fraction of it is available or relevant at any one point, its clear that only a fraction of the total cortical circuitry (and brain) is important during any one simulation step.
The Cortexture: The end result of these observations is that a smart algorithmic equivalent cortical simulation could be at least three orders of magnitude faster than a direct simulation which naively evaluates every synapse every timestep. The architecture I envision would organize cortical sheets into a spatial database that helps track data flow dependencies, storing most of the unique synaptic data (probably compressed) on a RAID disk array (possibly flash) which would feed one or more GPUs. With a few terrabytes of disk and some compression, you could store at least a primate level brain, if not a human-equivalent cortex. A couple of GPUs with a couple gigabytes of RAM each would store the active circuits (less than 1% of total synapses), which would be constantly changing and being streamed out as needed. Fast flash RAID systems can get over a gigabyte per second of bandwidth, so you could swap out the active cortical elements every second. I believe this is roughly fast enough to match human task or train of thought switching time. The actual cortical circuit evaluation would be handled by a small library of special optimized equivalent GPU programs. One would simulate the canonical circuit – and I believe I have an algorithm that is at least 10 times faster than naive evaluation for what the canonical circuit can do, but other algorithms could be even faster for some regions where the functionality is known and specialized. For example, the V1 layers which perform gabor-like filters use a very naive technique in the brain and the equivalent result could be computed perhaps 100x faster with a very smart algorithm. I’m currently exploring these techniques in more detail.
End Conclusion: If the brain was fully mapped (and that is the main task at hand – many mechanisms such as learning are still being teased out) and a sufficient group of smart engineers started working on optimizing its algorithms, we could probably implement a real-time artificial cortex in less than five years using today’s hardware on a machine costing somewhere between $10,000-$1,000,000. (I know that is a wide error range, but I believe it is thus accurate.) This cost is of course falling exponentially year by year.
Neuromorphic Computing
A sober analysis of the current weight of neuroscience data – specifically the computational complexity of the mapped cortical circuits and their potential for dramatic algorithmic optimization on faster machines – leads to the startling, remarkable conclusion that we already have the hardware capability to implement the brain’s algorithms in real-time today. In fact, it seems rather likely that by the time the brain is reverse engineered and we do figure out the software, the hardware will have already advanced enough that achieving faster than real-time performance will be quite easy. The takeoff will likely be very rapid.
The Cortexture approach I described earlier, or any AI architecture running on today’s computers, will eventually run into a scalability problem due to disk and bus bandwidth speeds. To really accelerate into the singularity, and get to 100′s or 1,000′s of times acceleration vs human thoughtspeed will likely require a fundamental redesign of our hardware along cortical lines. Cortical neural circuitry is based on mixed analog and digital processing, and combines memory and analog computation in a single elemental structure – the synapse. The data storage and processing are both built into the synapses and the equivalent total raw information flow rate is roughly the total synapses multiplied by their signaling rate. The important question really is thus what is the minimal efficient equivalent of the synapse for CMOS technology? Remarkably, the answer may be the mysterious 4th element of basic computing, the memristor. Discovered mathematically decades ago, this circuit building block was only realized recently and is already being heralded as the ‘future of artificial intelligence‘ – as it has electric properties very similar to the synapse – combining long term data storage and computation in a single element. For a more in depth design for a complete artificial cortex based on this new circuit element, take a look at “Cortical computing with memristive nanodevices“. This is a fascinating approach, and could achieve cortical complexity parity fairly soon, if the required fabrication technology was ready and developed. However, even though the memristor is quite exciting and looks likely to play a major role in future neuromorphic systems, conventional plain old CMOS circuits certainly can emulate synapses. Existing mixed digital/analog technique can represent synapses in artificial neurons effectively using around or under 10 transistors. This hybrid method has the distinct advantage of avoiding costly digital multipliers that use tens of thousands of transistors – instead using just a handful of transistors per synapse. The idea is designs in this space can directly emulate cortical circuits in highly specialized hardware, performing the equivalent of a multiplication for every synaptic connection every clock cycle. There are a wide space of possible realizations of neuromorphic architectures, and this field looks to just be coming into its own. Google: “artificial cortex” or “neuromorphic” for papers and more info. DARPA, not to be undone, has launched its own neuromoprhic computing, called SyNAPSE – which has a blog here just so you can keep tabs on skynet.
The important quantitative dimensions for the cortex are synapse density, total capacity (which is just density * surface area), and clock rate. The cortex topology is actually 2d: that of a flat, relatively thin sheet (around 6 neurons thick) which is heavily folded into the volume of the brain, a space filling fractal. If you were to unfold it, it would occupy about one square foot or 2,500 square centimeters – the area of roughly a thousand typical processor dies. It has a density of about 100-4,000 million (10^8-10^9) synapses per mm^2. Current 40nm and 32nm CMOS technology circa 2010 can pack roughly 6-10 million (10^6-10^7) transistors onto a mm^2, so semiconductor density is within about a factor of 20-500 of the biological cortex in terms of feature density (more accurate synapse density figures await more detailed brain scans). This is a critical upcoming milestone – when our CMOS technology will match the information density and miniaturization level of the cortex. This represents another 4 to 8 density doublings (currently occurring every 2 years, but expected to slow down soon), which we can expect to hit around the 11nm node or shortly thereafter in the early to mid 2020′s – the end of the semiconductor industry’s current roadmap. This is also the projected end of the road for conventional CMOS technology and where the semiconductor roadmap wanders into the more hypothetical realm of nano-electronics. When that does happen, neuromorphic designs will have some distinct advantages in terms of fault tolerance and noise resistance which could allow them to scale forward more quickly. It is also expected that moving more into the 3rd dimension will be important, and leakage and other related quantum issues will limit further speed and power efficiency improvements – all pointing towards more brain-like computer designs.
Scaling up to the total memory/feature capacity of the brain (hundreds to a thousand trillion synapses), even when semiconductor technology reaches parity in density, will still take a large number of chips (having roughly equivalent total surface area). Today’s highest density memory chips have a few billion transistors, and you would need hundreds of thousands to equal the total memory of the brain. High end servers are just starting to reach a terrabyte of memory (with hundreds of individual chips), and you would then need hundreds of these. A far more economical and cortically inspired idea is to forgo ‘chips’ completely and just turn the entire silicon wafer into a single large usable neuromorphic computing surface. The inherent fault tolerance of the cortex can be exploited by these architectures – there is no need to cut up the wafer into dies and identify defective components, they can be simply disabled or statistically ignored during learning. This fascinating contrarian approach to achieving large neuromorphic circuits is being explored by the FACETS research team in Europe. So, in the end analysis, it looks reasonable that in the near future (roughly a decade) a few hundred terrabytes of cortical equivalent neuromorphic circuitry could soon be produced on one to a couple dozen CMOS wafers (the equivalent of a few thousand cheap chips), even using conventional CMOS technology. More importantly this type of architecture can be relatively simple and highly repetitive and it can run efficiently at low clock rates and thus at low power, greatly simplifying manufacturing issues. Its hard to estimate the exact cost, but due to the combination of low voltage/clock, single uncut wafer design, perfect yield, and so on, the economics should be similar to memory – RAM chips, which arrive first at new manufacturing nodes, are cheaper to produce, and consume less power. Current 2010 RAM prices are at about $10 per GB, or very roughly 1 billion transistors per dollar.
Continuum of hardware efficiencies for cortical learning systems:
CPU,GPU Simulation: efficiency (die area, performance, power) 10^-8-10^6
FPGA, ASIC: efficiency 10^-5 to 10^-3
Neuromorphic (mixed analog/digital or memristors): 10^-2 to 1
CPU simulation is incredibly inefficient compared to the best solutions for a given problem, but CPU’s versatility and general applicability across all problems ensures they dominate the market and thus they get the most research attention, the economy of scale advantage, and are first to benefit from new foundry process improvements. Dedicated ASIC’s are certainly employed widely today in the markets that are big enough to support them, but always face competition from CPU’s scaling up faster. At the far end are hypothetical cortical systems built from memristors, which could function as direct 1:1 synapse equivalents. We can expect that as moore’s law slows down this balance will eventually break down and favor designs farther down the spectrum. Several forces will combine to bring about this shift: approaching quantum limitations which cortical designs are better adapted for, increased market potential of AI applications, and the end of the road for conventional lithography.
The Road Ahead
A human scale artificial cortex could be built today, if we had a complete connectome. In the beginning it would start out as only an infant brain – it would then take years to accumulate the pattern recognition knowledge base of a two year old and begin to speak, and then could take a few dozen additional years to achieve an education and any real economic value. This assumes, unrealistically, that the first design tested would work. It would almost certainly fail. Successive iterations would take even more time. This is of course is the real reason why we don’t have human-replacement AI yet: humans are still many orders of magnitude more economically efficient.
Yet we should not be so complacently comfortable in our economic superiority, for moore’s law ensures that the cost of such a cortical system will decrease exponentially.
Now consider another scenario, where instead of being constrained to current CPUs or GPUs, we invest billions in radical new chip technologies and even new foundries to move down the effeciency spectrum with a neuromorphic designs or at least a very powerful dedicated cortical ASIC. Armed with some form of specialized cortical chip ready for mass volume production today at the cheap end of chip prices (where a dollar buys about a billion transistors, instead of hundreds of dollars for a billion transistors as in the case of high end logic CPUs and GPUs), we would expect a full human brain sized system to cost dramatically less: closer to the cost of the equivalent number of RAM transistors – on the order of $10 million dollars for a petabyte (assuming 10 transistors = synapse, memristors are even better). Following the semiconductor industry roadmap (which factors in a slowing of moore’s law this decade), we could expect the cost of a quadrillion synapse or petabyte system to fall below $1 million by the end of the decade, and reach $100,000 in volume by the mid 2020′s – the economic tipping point of no return. But even at a million dollars a pop, a future neuromorphic computing system of human cortical capacity and complexity would be of immeasurable value, for it could possess a fundamental, simply mind-boggling advantage of speed. As you advance down the specialization spectrum from GPU’s to dedicated cortical ASICs and eventually neuromorphic chips and memristors, speed and power efficiency increases by orders of magnitude – with dedicated ASICS offering 10x to 100x speedups, and direct neuromorphic systems offerings speedups of 1000x or more. If it takes 20-30 years to train a cortex from infant to educated adult mind, power and training time are the main cost. A system that could do that in 1/10th the time would be suddenly economical, and a system that could do that in 1/100th of the time of a human would rapidly bring about the end of the world as we know it.
Thinking at the Speed of Light
Our biological brains have high information densities and are extraordinarily power efficient, but this is mainly because they are extremely slow: with cycle times in the hundreds of hertz or approaching a kilohertz. This relatively slow speed is a fundamental limitation of computing with living cells and (primarily)chemical synapses with their organic fragility. Semiconductor circuits do not have this limitation. Operating at the low frequencies and clock rates of their biological inspirations, neuromorphic systems can easily simulate biological networks in real-time and with comparable energy efficiency. The most efficient neuromorphic computer generally can access all of its memory and synapses every clock cycle, so it can still perform immense calculations per second at very low speeds, just like biological neural nets. But you can also push up the clock rate, pump more power through the system, and run the circuit at megahertz rate or even gigahertz rate, equivalent to one thousand to one million times biological speed. Current systems with mixed digital/analog synaptic circuits can already achieve 1000-10000x biological ‘real-time’ on old CMOS manufacturing nodes and at low power and heat points. This is not an order of magnitude improvement over simulation on a similar sized and tech digital computer, its more like six orders of magnitude. That being said, the wiring problem will still be a fundamental obstacle. The brain optimizes against this constraint by taking the form of a 2D sheet excessively folded into a packed 3D space – a space filling curve. The entire outer surface is occupied by connectivity wiring – the white matter. Our computer chips are currently largely 2D, but are already starting to move into the 3rd dimension. A practical full electronic speed artificial cortex may require some novel solutions for high-speed connectivity, such as directly laser optical links, or perhaps a huge mass of fiber connections. Advanced artificial cortices may end up looking much like the brain: with the 2D circuity folded up into a 3D sphere, interspersed with something resembling a vascular system for liquid cooling, en-sheathed in a mass of dense optical interconnects. Whatever the final form, we can expect that the fundamental speed advantage inherit to electronics will be fully exploited.
By the time we can build a human complexity artificial cortex, we will necessarily already be able to run it many times faster than real time, eventually accelerating by factors of thousands and then even millions.
Speed is important because of the huge amount of time a human mind takes to develop. Building practical artificial cortex hardware is only the first step. To build a practical mind, we must also unlock the meta-algorithm responsible for the brain’s emergent learning behavior. This is an active area of research, and there are some interesting emerging general theories, but testing any of them on a human sized cortex is still inordinately costly. A fresh new artificial brain will be like an infant: full of noisy, randomized synaptic connections. An infant brain does not have a mind so much as the potential space from which a mind will etch itself through the process of development. Running in real-time in a sufficiently rich virtual reality, it would take years of simulation just to test development to a childhood stage, and decades to educate a full adult mind. Thus accelerating the simulation many times beyond real-time has a huge practical advantage. Thus the need for speed.
The test of a human-level AI is rather simple and is the same qualitative intelligence tests we apply to humans: its mind develops sufficiently to learn human language, then it learns to read, and it progresses through education into a working adult. Learning human language is ultimately the fundamental aspect of becoming a modern human mind – far more than your exact brain architecture or even substrate. If the brain’s cortical capacity is sufficient and the wiring organization is correctly mapped, it should then be able to self-educate and develop rapidly.
I highly doubt that other potential short cut routes to AGI (artificial general intelligence) will bear fruit – although narrow AIs will always have their uses as will simpler, animal-like AIs (non-language capable), but it seems inevitable that a human level intelligence will require something similar to a cortex (at least at the meta-algorithmic level of some form of self-organizing deep, hierarchical probabilistic networks – doesn’t necessarily have to use ‘neurons’ ). Furthermore, even if the other routes being explored to AI do succeed, its even less likely that they will scale to the insane speeds that the cortex design should be capable of (remember, the cortex runs at < 1000hz, which means we can eventually take that same design and speed it up by a factor of at least a million.) From a systems view, its seems likely that the configuration space of our biological cortex meta-wiring is effectively close to optimal in some sense – evolution has already well explored that state space. From an engineering perspective, taking an existing, heavily optimized design for intelligence and then porting it to a substrate that can run many orders of magnitude faster is a clear winning strategy.
Clock rate control, just like on our current computers, should allow posthumans to alter their speed of thought as needed. In the shared virtual realities they will inhabit with their human teachers and observers, they will think at ‘slow’ real-time human rates, with kilohertz clock rates and low power usage. But they will also be able to venture into a vastly accelerated inner space, drawing more power and thinking many many times faster than us. Running on even today’s CMOS technology, they could theoretically attain speeds up to about a million times faster than a biological brain, although at these extreme speeds the power and heat dissipation requirements would be large – like that of current supercomputers.
Most futurist visions of the Singularity consider AIs that are more intelligent, but not necessarily faster than human minds, but its clear that the speed is the fundemental difference between the two substrates. Imagine one of these mind children growing up in a virtual environment where it could dilate time by a factor of 1-1000x at will. Like real children, it will probably require both imitation and reinforcement learning with adults to kick start the early development phases (walking, basic environment interaction, language, learning to read). Assuming everything else was identical (the hardware cortex is a very close emulation), this child could develop very rapidly – the main bottleneck being the slow-time interaction with biological humans. Once a young posthuman learns to read, it can hop on the web, download texts, and progress at a truly staggering pace – assuming a blindingly fast internet connection to keep up (although perceived internet latency would be subjectively far worse in proportion to the acceleration – can’t do much about that) . Going to college wouldn’t really be a realistic option, but reading at 1000x real-time would have some pretty staggering advantages. It could read 30 years of material in just 10 days, potentially becoming a world class expert in a field of its choosing in just a week. The implications are truly profound. Entering this hypertime acceleration would make the most sense when reading, working, or doing some intellectual work. The effect for a human observer would be that anything the posthuman was intelligent enough to do it would be able to do near instantly, from our perspective. The presence of a posthuman would be unnerving. With a constant direct internet connection and a 1000x acceleration factor, a posthuman could read a book during the time it would take a human to utter a few sentences in conversation.
Clearly existing in a different phase space than us, its only true peers would be other equivalent posthumans; if built as a lone research model, it could be lonely in the extreme. Perhaps it could be selected or engineered for monastic or ascetic qualities, but it would probably be more sensible to create small societies of posthumans which can interact and evolve together – humans are social creatures, and our software descendants would presumably inherit this feature by default. The number of posthumans and their relative intelligence will be limited by our current computing process technology: the transitor density and cost per wafer – so their potential population growth will be more predictable and follow semiconductor trends (at least initially). Posthumans with more equivalent synapses and neurons than humans could presumably become super-intelligent in other quantitative dimension, that of mental capacity – able to keep track of more concepts, learn and recall more knowledge, and so on than humans – albeit with the slow linear scaling discussed previously. But even posthumans with mere human-capacity brains could be profoundly, unimaginably super-intelligent in their speed of thought, thanks to the dramatically higher clock rates possible on their substrate – and thus in a short matter of time they would become vastly more knowledgeable. The maximum size of an artificial cortex would be limited mainly by economics for the wafers and then by bandwidth and latency constraints. There are tradeoffs between size, speed, and power for a given fabrication technology, but in general, larger cortices would be more limited in their top speed. The initial generations will probably occupy a fair amount of server floor space and operate not much faster than real-time, but then each successive generation will be smaller and faster, eventually approaching a form factor similar to the human brain, and eventually pushing the potential clock rate to the technology limits (more than a million times real-time for current CMOS tech). But even with small populations at first, it seems likely that the first successful generation of posthumans to reach upper-percentile human intelligence will make an abrupt and disruptive impact on the world. But fortunately for us, physics does impose some costs to thinking at hyperspeed.
Fast Brains and Slow Computers
A neuromorphic artificial cortex will probably have data connections that allow its synaptic structures to be saved to external storage, but a study of current theory and designs in the field dispels some common myths: an artificial cortex will be a very separate specialized type of computer hardware, and will not automatically inherit a digital computer’s supposed advantages such as perfect recall. It will probably not be able to automagically download new memories or skills as easily as downloading new software. The emerging general theories of the brain’s intelligence, such as the heirchachial bayesian network models, all posit that learned knowledge is stored in a deeply non-local, distributed and connected fashion, very different than say a digital computer’s random access memory (even if the said synapses are implemented in RAM). Reading (or accessing) memories and writing memories in a brain-like network intrinsically involves thinking about the associated concepts – as memories are distributed associations, and everywhere tangled up to existing memory patterns. An artificial cortex could be designed to connect to external computer systems more directly than through the senses, but this would have only marginal advantages. For example, we know from dreams that the brain can hallucinate visual input by bypassing the lowest layers of the visual cortex and directly stimulating regions responsible for recognizing moving objects, shapes, colors, etc, all without actually requiring input from the retina. But this is not much of a difference for a posthuman mind already living in a virtual reality – simulating sound waves and their conversion into neural audio signals and simulating the processing into neural patterns representing spoken dialog is not that much different than just directly sending the final dialog representing neural patterns into the appropriate regions. The small differences will probably show up as what philosophers call qualia – those subjective aspects of consciousness or feelings that operate well below the verbal threshold of explanation.
Thus a posthuman with a neuromorphic artificial cortex will still depend heavily on traditional computers to run all the kinds of software that we use today, and to simulate a virtual environment complete with a virtual body and all that entails. But the posthuman will essentially think at the computer’s clock rate. The human brain has a base ‘clock rate’ of about a kilohertz, completing on the order of a thousand neural computing steps per second simultaneously for all the trillions of circuits. A neuromorphic computer works the same, but the clock rate can be dramatically sped up to CMOS levels. A strange and interesting consequence is that a posthuman thinking at hyperspeed would subjectively experience its computer systems and computer environment slow down by an equivalent rate. Its likely the neuromorphic hardware will have considerably lower clock rates than traditional CPUs for power reasons, but they have the same theoretical limit, and running at the same clock rate, a posthuman would experience a subjective second in just a thousand clock cycles, which is hardly enough time for a traditional CPU to do anything. Running at a less ambitious acceleration factor of just 1000x, and with gigahertz computer systems, the posthuman would still experience a massive slowdown in its computer environment, as if it had jumped back more than a decade in time to a vastly slower era of of megahertz computing. However, we can imagine that by this time traditional computers will be much further down the road of parallelization, so a posthuman’s typical computer will consist of a very large number of cores and software will be much more heavily parallelized. Nevertheless, its generally true that a posthuman, no matter what level of acceleration, will still have to wait the same amount of time as anyone else, including a regular human, for its regular computations to complete.
Ironically, while posthumans will eventually be able to think thousands or even millions of times faster than biological humans, using this quickening ability will have the perceptual effect of slowing down the external universe in direct proportion – including their computer environment. Thus greatly accelerated posthumans will spend proportionally inordinate amounts of subjective time waiting on regular computing tasks.
The combination of these trends leads to the conclusion that a highly variable clock rate will be an important feature for future posthuman minds. Accelerating to full thought speed – quickening – will probably be associated with something like entering an isolated meditative state. We can reason that at least in the initial phases of posthuman development, their simulated realities will mainly run at real-time, in order to provide compatibility with human visitors and to provide full fidelity while conserving power. When quickening, a posthuman would experience its simulated reality slowing down in proportion, grinding to a near halt at higher levels of acceleration. This low-computational mode would still be very useful for much of human mental work: reading, writing, and old-fashioned thinking.
In the present, we are used to computers getting exponentially faster while the speed of human thought remains constant. All else being equal, we are now in a regime where the time required for fixed computational tasks is decreasingly exponentially (even if new software tends to eat much of this improvement.) The posthuman regime is radically different. In the early phases of ramp up the speed of thought will increase rapidly until it approaches the clock rate of the processor technology. During this phase the trend will actually reverse – posthuman thoughtspeed will increase faster than computer speed and from a posthumans perspective, computers will appear to get exponentially slower. This phase will peter out when posthuman thoughtspeed approaches the clock rate – somewhere around a million times human thoughtspeed for the fastest, most extremely optimized neuromorphic designs using today’s process technology (gigahertz vs a kilohertz). At that point there is little room for further raw speed of thought improvements (remember, the brain can recognize objects and perform relatively complex judgements in just a few dozen ‘clock cycles’ – not much room to improve on that in terms of speed potential given its clock rate).
After the initial ramp up regime, Moore’s law will continue of course, but at that point you enter a new plateau phase. In this second regime, once the algorithms of intelligence are well mapped to hardware designs, further increases in transistor density will enable more traditional computer cores per dollar and more ‘cortical columns or equivalents’ per dollar in direct proportion. Posthuman brains may get bigger, and or they may get cheaper, but the clock speed wouldn’t change much (as any process improvement in clock rate would speed up both traditional computers and posthuman brains). So in the plateau phase, you have this weird effect where computer clock rate is more or less fixed at a far lower level than we are used to – about a million times less or so from the perspective of the fastest neuromorphic posthuman brain designs. This would correspond to computer clock rates measured in the kilohertz. The typical computer available to a posthuman by then would certainly have far more cores than today, thousands or perhaps even millions, but they would be extremely slow from the posthuman’s perspective. Latency and bandwidth would be similarly constrained, which would effectively expand the size of the world in terms of communication barriers – and this single idea has wide ranging implications for understanding how posthuman civilizations will diverge and evolve. It suggests a strong diversity increasing counter–globalization effect which would further fragment and disperse localized sub-populations for better or worse.
What would posthumans do in the plateau phase, forever limited to extremely slow, roughly kilohertz-speed computers? This would limit the range of effective tasks they could do in quicktime. Much of hardware and software design, engineering, etc would be limited by slow computer speeds. Surprisingly, the obvious low-computational tasks that could still run at full speed would be the simpler, lower technology creative occupations such as writing. It’s not that posthumans wouldn’t be good at all computer intensive tasks as well – they certainly would be superhuman in all endeavors. The point is rather that they will be vastly, incomprehensibly more effective only in those occupations that are not dependent on the speed of general purpose computation. Thus we can expect that they will utterly dominate professions such as writing.
It seems likely that very soon into the posthuman era, the bestseller lists will be inundated by an exponentially expanding set of books, the best of which would be noticeably better than anything humans could write (and most probably written under pseduo-names and with fake biographies). When posthumans achieve 1000x human thoughtspeed, you might go on a short vacation and come back to find several years of new literature. When posthumans achieve 1 million X human thoughtspeed, you might go to sleep and wake up to find that the number of books in the world (as well as the number of languages), just doubled over night. Of course, by that point, you’re already pretty close to the end.
We can expect that the initial posthuman hardware requirements will be expensive and thus they will be few in number and of limited speed, but once they achieve economic parity with human workers, we can expect the tipping point to crash like a tidal wave, with a rapid succession of hardware generations increasing maximum thoughtspeed while reducing size, cost, and power consumption and huge economies of scale leading to an exponential posthuman population expansion, and their virtual realities eventually accelerating well beyond human comprehension.
