Bill Softky's Science Page

3-D Mind Theory

My most recent, complete, and readable work is available on the theoretical-physics site ArXiv, with the title Elastic Nanocomputation in an Ideal Brain. This work supercedes what is written below, which I only leave posted for reasons of personal history, and because I am unmotivated to keep this website fully synchronized with my research.

Vision

“We don't know what a neuron does, we don't know how neurons are connected, and we don't know how the connections change with learning. But most important is that we don't even know what the whole circuit is trying to do. Detect regularity? Detect change? Be stable? Be sensitive? Preserve information? Throw away information? Our best chance for understanding will come from lucky guesses about the general statistical structure of perceptual processing, and from practical, partial solutions which can be tweaked and "hacked" into ever-better systems which solve real-world problems.”

Unlike most neuroscientists, I've worked as an engineer and software architect, and thus have unusual and strong opinions about how to best approximate and build a model of the brain which really works and can do something useful. Below are outlines of my past research and links to some articles I've written: some are for laymen, some for specialists, and some are free-form discussions; some are published, some are not.

Layman's introduction

An online technology journal based in England--The Register--published two essays arguing that the most important problems in neuroscience are more likely to be solved by software engineers than by biologists. Whether or not you believe that claim, these articles may still be the best distilliation of my own viewpoints and prejudices:
Part I
Part II

Recent work

A minimal real-time brain architecture (2011, PDF, 600k). An outline from my 2011 talk at the Telluride Neuromorphic Engineering workshope titled "How the Brain Works", tying multiple "Compressor" modules together into a distributed system to compress, expand, predict, and control (using active sensing, optimal dithering, trans-sensory statistics, manifold-discovery, etc.).

Generic Sensory Prediction (2011, PPT, 4.3MB). My Telluride talk (in PPT form for the animations and notes) of my simple scheme for using dimensionality-reduction to map and predict the future motion of a simple but representative nonlinear sensory input, akin to the task faced by one of my "Compressor" modules. Sort of a "reference implementation" and tutorial.

A software architecture and design principles for brain-like sensory processing (2002, Powerpoint, 800k). I gave this talk years ago at the Redwood Neuroscience Institute, before working there. It outlines an approach for "solving" the brain by creating a specific kind of software infrastructure, learning/prediction modules, and Application Programming Interfaces ("API's") which might enable teams of engineers to create and improve upon the component algorithms separately, rather than leaving the entire project in the hands of a lone genius. (I think this approach is being used now by Jeff Hawkins in his company Numenta).

Feedforward/feedback hierarchy of self-learning "Compressor" modules (2003, Powerpoint, 3.8 MB). This is a specific, working example of the kind of architecture outlined above, in which each module learns by discovering shapes ("manifolds") in its input data, and represents each new data point by giving its location on that shape. This system has only been proved to "work" on simple toy problems, and not (yet) real-world data or time-varying data.

An engineered algorithm for using Wiskott's 'Slow Feature Analysis' to predict specific future input patterns (Feb 2005, Powerpoint, 800k). This is a first step in making an architecture which can learn and represent time-varying data: given a module which can learn or discover meaningful temporal patterns ("invariances") on its own, this new algorithm can take the temporal invariance and reconstruct or predict what the next input signal is going to be. The source code for the algorithm may prove useful, at least for understanding details of the algorithm which aren't clear from the PowerPoint presentation.

Research articles

Here are some of the articles I'm most proud of having written.

Outstanding puzzles of neural theory

Each of the following micro-essays discusses an underestimated issue in neural theory. They are meant to persuade and explain, not to survey previous work or create a scientifically unassailable edifice. Read them at your own risk.

Probabilities vs. Parameters
An analog signal can represent the probability of something being there (e.g. likely vs. unlikely), or a parameter of it (e.g. angle of tilt). What is the difference between these two approaches? What are the advantages and disadvantages or each?

Types of invariance
An "invariance" is some representation of a signal which doesn't change even when the signal does change... for example, you know a floor stays fixed even as you move your head and walk around. Here is a catalog of several types of invariance we expect to find in perception, and some of the tricks the brain may use for discovering them.

Energy use by brains
Our opinions about what neurons do are based almost exclusively on measurements of neurons which fire at least 10 spikes per second, which may be ten- or a hundred-fold faster than typical neurons during normal perception. Why are realistic spike rates so slow? What are the implications? Why are reports biased towards unrealistically fast-firing neurons?

'Belief particles' have the binding problem'
The "binding problem" is a well-known challenge for representations which break objects apart into complementary features (like shape and position). I argue that it is also a problem for probablistic representations which break objects apart into competing hypotheses (beliefs). Since real brains probably face both problems, its circuitry may solve both problems at once.

Mini-bio

Bill spent a year in West Africa imagining neural circuits and re-inventing the Hebb Rule before getting exposed to real neurobiology and computational theory at Caltech. His most impactful research has been in three fields: 1) the paradox that irregular cortical spiking probably represents an information-dense, precise-time pulse code, 2) the electro-chemical properties of neocortical neurons which make a precise-time code both possible and desirable, and 3) an interpretation of cortex (and thalamo-cortical feedback) as a system which learns to predict its own sensory input in real time. His recent focus has been on designing software "API's" which isolate the scaling and hierarchy issues of cortical-style signal processing from the modules which perform feature extraction/compression/reconstruction/prediction on smaller groups of signals.

Bill has taught physics in West Africa and "Physics for Poets" in junior college, and has Physics degrees from Haverford College (BS) and Caltech (PhD), where he worked in the Computation and Neural Systems Program under Christof Koch. He was a postdoctoral fellow at the Mathematics Research Group at the National Institutes of Health, and has worked at half a dozen Silicon Valley companies as scientist, engineer, and software architect, in many fields: statistical algorithms, biophysics, software tools, web applications, and finance.

Publication list