The visual system is the largest sensory system in the brain. It is sophisticated and complex. The number of visual neurons expands from about a million retinal ganglion cells to about 8 billion cells in the cerebral cortex. fMRI studies reveal two streams of information in the visual cortex, a ventral stream related mostly to "what" an object is, and a dorsal stream related to "where" it is. But even in the retina the visual signal is split up into multiple channels that convey real-time information which would be difficult to compute sequentially from an image in the allotted time.

Human Visual Field And Visual Acuity

The human visual field is shown in the figure below, it's about 200 degrees horizontally and about 130 degrees vertically. The fields from each eye overlap in the central 120 degrees or so, and the high resolution foveal area is actually quite small, covering about 5 degrees of horizontal arc.

Humans direct attention to specific locations in the visual field ("targets") with eye, head, and body movements. These movements are coordinated in the brainstem at the level of the superior colliculus, which we'll visit in the next section. The motor map is topographically aligned with the visual map, it self-organizes during development on the basis of both internally generated neural activity and visual exposure. In most cases the goal of a saccadic eye movement is to "foveate" the area of interest, bringing it into the high-acuity area in the central 5 degrees.

Humans perceive motion from continuous images at around 12 Hz (hence the early standards for movies and video at 24 fps). However human visual perception is sensitive to contrast-related motion at frequencies into the 100's of Hz, and even beyond. It is important to note that the critical flicker frequency of 50-60 Hz only pertains to whole-field flicker. Alternating flicker increases the fusion limit to 400 Hz, and checkerboard flicker increases it even more. All of these upper bounds are important for the human perception of time, which is very much related to the ranges of motion we find in the environment around us. For example, if input exceeds the flicker fusion frequency it will not be perceived, that is to say, it will not be individuated from the background. You can try this for yourself by tracking a fly as it flies around. The fly goes in and out of perception when it's in motion, we have to "find it again" as it flies. (And in spite of this idiosyncracy, humans can predict with great accuracy where the fly will be at any given moment, and in many cases can reach out and trap it without much effort). In context, it is worth studying insect eyes, which have multiple lenses called "ommatidia" arranged in a mosaic pattern. They are spatially diffuse but very fast with motion, again up into the 100's of Hz. The velocity specialization of visual processing is an excellent example of extracting and maintaining very precise information from noisy and unreliable neurons. A rapid perception of motion is biologically selected for and essential to the survival of the organism.

Some of the peculiarities of human vision include interruption by eye movements and eye blinks. The eyes are constantly moving, the retinal image is constantly shifting. Retinal neurons adapt very quickly to changes in background illumination, and part of the constant eye motion is to to prevent perceptual adaptation in interesting situations. In turn, the internal wiring of the visual portion of the human brain transforms these interruptions into a continuous visual perception.

Overview Of The Retina

The human retina emits a little north of a million fibers from its ganglion cells. The ones we're discussing now, travel from the retina through the thalamus and into the occipital lobe of the cerebral cortex. (Others connect with the superior colliculus, and other structures like the suprachiasmatic nucleus which is involved in light-related circadian rhythms, and the olivary pretectal nucleus which is involved with light-driven pupil constriction). The figure shows the general plan of the early visual system, specifically the pathway from the retina to the primary visual cortex.

The nerve fibers from each retina cross in the optic chiasm. The nasal hemifield from one is combined with the temporal hemifield from the other, and the result is that the left side of the brain processes the right side of the visual field, and vice versa. The retina "knows" its orientation and its subdivisions early in development, based on chemical gradients of proteins called "ephrins". Later in development there is also oscillatory neural activity that helps to self-organize the connections.

The retina is a layered structure, and it is inverted with respect to light - in other words the photoreceptors are on the inside, nearest the brain. Light has to travel through the ganglion cells to get to the photoreceptors. This figure shows the general orientation of the neurons within the retina.

The number of layers in the retina keeps increasing as our experimental methods get better, Ramon y Cajal said 5 but newer results bring the number closer to 10. Some layers contain cells, some layers contain fibers and synapses, and some contain both. The general connection pattern of the retina is photoreceptors => bipolar cells => ganglion cells. Laterally between the photoreceptors and the bipolar cells are the horizontal cells, and laterally between the bipolar cells and the ganglion cells are the amacrine cells. This pattern is schematized in the figure.

There are four main types of photoreceptors, a dozen kinds of bipolar cells, and about 20 different varieties of ganglion cells. Here is another view, showing the neatly organized layers.

The electrical activity from the retina associated with an optical stimulus is shown in the figure below. Historically it was detected as early as the mid-1860's by placing electrodes on the cornea. This "electroretinogram" is also visible in other places (it can be detected with a special contact lens, and it is also visible using ordinary electrodes on the forehead and temple). The ERG is subject to volume conductor effects from the vitreous humor and other parts of the eye and nearby tissue.

Photoreceptors

The right eye is shown schematically in the figure below. The typical human eye is about 2.4 cm from front to back, it is approximately spherical but slightly wider than it is deep. The retina is in back of the eye, light has to travel through the cornea, aqueous humor, lens, and vitreous humor to get to the photoreceptors. The photoreceptors are in back along the choroid wall and the nerve fibers from the ganglion cells exit in the front and travel laterally to enter the optic nerve at the blind spot.

Humans have about 126 million photoreceptors, of which 120 million are rods and 6 million are cones. The distribution of photoreceptors in a human retina is shown in the figure.

The retinal photoreceptors hyperpolarize in response to light, they have a baseline membrane potential that goes negative as a result of ion influx. This process is shown in the figure below.

There are four main types of photoreceptors, their characteristics are shown in the figure. The density of small color-sensitive photoreceptors is highest in the foveal region of the eye, while the density of wider field contrastive cells is highest in the periphery. The cones are called S, M, and L for blue, green, and red respectively, based on their preferred wavelength of light. They are arranged in a mosaic pattern within the retina.

The various photoreceptors form a mosaic pattern. The mosaic is organized in a radial manner, with a very high concentration of cones at the fovea, that tapers off in the periphery. The figure shows the mosaic pattern of color-sensitive cones in a normal retina (left) and a retina with protanopsia (in this case, no red cones). The foveal area is outlined. Careful observation reveals, in both cases, a lack of blue cones in the center of the fovea.

This electron micrograph shows how rods relate to the cones.

Photoreceptors of all kinds synapse on bipolar cells, using glutamate as a neurotransmitter. In addition, they form gap junctions and triadic synapses with horizontal cells, of which there are several different kinds. Generally speaking only one or a few photoreceptors connect to a bipolar cell in the high resolution area of the fovea, whereas the number may be slightly higher in the periphery.

In addition to the synapses in the feed-forward path, there are gap junctions between rods, between cones, and between rods and cones. These gap junctions are associated with various connexin proteins, that allow ions and small molecules to move between cells. Electrically, the movement of sodium, calcium, potassium, and chloride between cells creates a syncytium, whose properties can be modeled as one large membrane. Some of the gap junctions are rectifying, they allow movement in only one direction. Gap junctions trade off a small amount of visual acuity for better contrast enhancement. In the cone syncytium, the extent of voltage penetration away from a cone is only about 0.5 cone widths, indicating that a single cone at most affects its nearest neighbors via gap junctions (DeVries et al 2002). In contrast, the syncytium formed by horizontal cells is more permissive. Horizontal cells respond to point stimuli in a radius approximately 25 times the corresponding extent of their dendritic trees. This responsiveness is enabled by rapid transmission through gap junctions. There are not yet any waves at this level, the syncytia are mostly involved with contrast adaptation and contrast maintenance.

Bipolar Cells and Horizontal Cells

Photoreceptors feed forward to bipolar cells. There are two kinds of bipolar cells, those that respond to increased illumination, and those that respond to decreased illumination. These two responses are provided by two different kinds of neurotransmitter receptors that process the glutamate released by photoreceptors. The sign-preserving (off-center) synapses use the ionotropic kainate-type GRIK-1 glutamate receptor, whereas the sign-reversing (on-center) synapses use the metabotropic mGlu-R6 (GRM6) receptor. These two response types are sometimes called "ON" and "OFF", but that's really a misnomer at this level. An ON bipolar cell will respond whenever the level of illumination in the center exceeds that in the periphery, and an OFF bipolar cell will respond when the center is darker than the periphery. Thus the primary axis of differentiation is spatial rather than temporal.

Between the photoreceptors and the bipolar cells are the horizontal cells, which connect laterally in the outer plexiform layer. There are broadly two types of horizontal cells from a response standpoint, those that respond to luminance and those that respond to contrast. In humans there are three different kinds of horizontal cells anatomically, called H1, H2, and H3. These axes are independent of the response axes. Some of the H1 and H3 cells have axons that travel laterally and synapse onto rods, whereas others are axonless. Axons from H2 cells synapse on cones only, and blue cones preferentially. Information from the blue pathway remains segregated throughout the early visual system. The horizontal cells are inhibitory, they inhibit photoreceptors via the neurotransmitter GABA.

The surround field can be loosely modeled as a "sum of Gaussians", where the middle peak is determined by the viewing field of the photoreceptor and the edges are determined by the extent of the horizontal cell. In real life the situation is a bit more complicated, because of both of these patterns are dynamic, they undergo adaptation and other forms of plasticity that help the eye adjust to changing light conditions.

Already at this early stage of retinal processing, we see 4 types of photoreceptors and 2 x 3 kinds of horizontal cells being organized into 12 bipolar cell channels, the entire mosaic being organized topographically. This is one of the areas where modeling can really help us, because the regular geometry is part and parcel of the response. The large receptive field of horizontal cells is proof enough of this assertion. The geometry, topography, and even topology are important to understand, and later we'll look at some tools that are available for modeling purposes.

Horizontal cells generate an inhibitory surround field for the bipolar cells. Neighboring horizontal cells are coupled by gap junctions that are much more permeable than those between photoreceptors. The surround does not depend on the bipolar cells, it is generated by a cooperative process between the photoreceptors and the horizontal cells. The coupling between neighboring horizontal cells adjusts the size of the receptive field in the electrical syncytium, and therefore the size of the surround in subsequent processing. The figure below shows the modulation of horizontal cell coupling. In (a) the receptive field is largest in conditions of dark adaptation, and in (b) it shrinks a bit with medium intensity light. In (c) it's inhibited completely by a chemical agent that turns off the gap junctions, and in (d) it's modulated by dopamine, from a particular kind of amacrine cell (discussed below).

In humans and trichromatic primates, the horizontal cells contribute to color opponency. The H1 class of horizontal cell shapes M and L cone output (red and green), and the H2 class shapes S cone output (blue) and contributes to S-M/L opponency (Chapot et al 1997). The surround is not necessarily color-specific, for example the red and green cones both contact the H1 horizontal cells without regard to color specificity. There is no "green horizontal cell", the same horizontal cell generates the surrounds for both red and green cones. In humans, the axons from the H1 cells terminate mainly on nearby rods, and can be directional. Directionally sensitive cells are important for things like the looming reflex, which is mediated by the superior colliculus. Axons from retinal ganglion cells directly innervate the superficial layers of the superior colliculus monosynaptically, and in many cases these arise from specific types of ganglion cells not projecting to the lateral geniculate nucleus.

Though there are only three main types of horizontal cells, there are at least 12 different kinds of bipolar cells in humans. One of them is a specialized rod bipolar cell that only contacts AII amacrine cells (20-80 per cell) and does not connect directly with ganglion cells. The others feed forward to retinal ganglion cells. The inhibitory surrounds of bipolar cells are generated by specialized synapses between the receptors, horizontal cells, and bipolar cells. These surrounds are generated when activity in one photoreceptor activates the connected horizontal cell, which then inhibits nearby receptors. During changing light conditions, the horizontal cell synapses are part of the circuitry related to adaptive gain control (Scholl et al 2012, VanLeeuwen et al 2009). Bipolar cells also participate in adaptive gain control (Beaudoin et al 2007).

The synaptic web between the photoreceptors and horizontal cells sends signals with a very small delay, in a similar range to synaptic delays, around or under 1 msec (Trenholm & Awatramani 2025, Gollisch & Meister 2010). Therefore the inhibitory surrounds of bipolar cells activate without much delay, after an optical stimulus. The early processing in the retina is done by graded transmission in the absence of action potentials. Photoreceptors and horizontal cells do not generate action potentials at all, and of the 12 types of bipolar cells only the three fastest (transient) types generate action potentials (Baden et al 2013). In the retina, it is mainly the amacrine and ganglion cells that generate action potentials (amacrine cells may or may not, and ganglion cells always do). In this context, the organization of gap junctions and the creation of electrical syncytia within the retinal neural network is important to understand.

Bipolar cells excite ganglion cells, they use glutamate as a neurotransmitter. Of the 12 identified types of bipolar cells in humans, 5 are on-type cone bipolars, one is an on-type rod bipolar, and 6 are off-type cone bipolars. The different types of bipolar cells segregate local information into at least 20 independent "channels" that exit the retina at the level of the ganglion cells. Each of these channels is topographically consistent, and the result is an overlay of various "views" of the retinal input. Each view is filtered in a different way, some carry color information, some carry motion information, and some carry general luminance. The channels remain segregated on their way to the visual cortex. This is a great resource for visualizing cells in the retina.

Ganglion Cells and Amacrine Cells

The bipolar cells send outputs to a bewildering array of amacrine cells and ganglion cells, and the amacrine cells in turn connect back with the ganglion cells, both synaptically and electrically. Over 20 different kinds of retinal ganglion cells have been identified in humans and primates, and many of these obtain their properties (both static and dynamic) from the amacrine cells. Each different combination of ganglion and amacrine cells performs a local filtering operation on the input from bipolar cells. The filter acts in both space and time, as many of the inhibitory effects from amacrine cells are synaptic and have time constants on the order of milliseconds.

The inner plexiform layer is where the retinal output acquires its dynamics. Many of the retinal ganglion cells (especially the M-type cells) respond transiently to visual stimuli. These transient characteristics are introduced at the level of the amacrine cell, and research to date has shown that these amacrine-induced dynamics are more significant in determining ganglion cell output, than the differences in the time courses of the various glutamate receptor types in the bipolar cells. The figure shows the stratification of the inner plexiform layer. Off cells are on top (closest to the photoreceptors), on cells are on the bottom (closest to the ganglion cells). In between the sustained types are layers with transient response. These differing responses come from various combinations of bipolar and amacrine cell input.

The figure below shows some of the varieties of amacrine cells. Particular varieties can use GABA, glycine, acetylcholine, dopamine and other neurotransmitters. One of the well studied amacrine cells in the inner nuclear layer of the retina is the A-II cell, which gets inputs from rod bipolars and distributes them to cone bipolars (Kolb, 1997). The figure shows the stratification of some of the 40+ different kinds of retinal amacrine cells.

Amacrine cells participate in retinal gain control, and they help report the direction of movement to ganglion cells. In addition, the rapid response properties of certain kinds of wide-field amacrine cells throttle the visual output during ballistic eye movements, helping to avoid blurred images.

The stratification of the inner plexiform layer is important for the behavior of the ganglion cells. Most of the synapses between bipolar cells, amacrine cells, and ganglion cells are carefully organized within this layer. The IPL is further subdivided into at least 5 distinct strata, each of which contains a particular type of connection, or multiple types of connections. Kolb reviews at least 40 different kinds of amacrine cells, each of which has specific connectivity. As an example of IPL organization, the high-level stratification at the level of on and off responses is shown in the figure.

The information from the photoreceptors is combined in different ways into the retinal ganglion cells. Some examples are shown in the figure. There are over 20 different kinds of retinal ganglion cells that pass filtered versions of the retinal image into the brain. Many of these pathways remain segregated in the central areas, and it is not entirely clear whether and how they may be eventually combined into a unified visual perception. However it is know that the human brain performs correlations between the information in the various retinal channels, to extract the statistical properties of the visual input. This is similar to the goal of machine vision, which endeavors to do the same, often using multiple channels and serial topographic convolutional architectures.

There are only a very few centrifugal fibers that enter the retina from the brainstem and the central nervous system. Most of the ganglion cells send projections into the optic chiasm, with a few of them traveling to the superior colliculus. The ganglion cells are organized into "filtered streams" that convey different kinds of information. Many are static color sensitive or color insensitive cells that respond to luminance and luminance contrast. Others are highly dynamic and may be both motion and direction sensitive.

The richness of dynamic behavior in the retina is important to understand. It is reflected not only in the dynamic behaviors of individual neurons, but also in the form of traveling waves that augment precision in the population and become very important during development. In adulthood the wave-like activity continues, but it transitions from spontaneous to stimulus-driven. An example is the retinal shift effect, which is mediated by amacrine cells. Most of the dynamics of ganglion cells are provided by the amacrine cell layer(s).

Finally, there is a small subset of intrinsically photoreceptive retinal ganglion cells, that contain the photopigment melanopsin. These cells feed into the midbrain areas related to circadian rhythms.

Retinal ganglion cells in primates are broadly classified into three categories: P cells, M cells, and K cells. These correspond approximately, to the X, Y, and W cells in cats. The P (parvocellular) cells exhibit mostly tonic responses to light, whereas the M (magnocellular) neurons respond transiently to onset and offset. The K (koniocellular) neurons have distinct targets, they project to different areas of the LGN and superior colliculus, and the LGN pathways have different endpoints in the primary visual cortex.

Ganglion Cell Center-Surround Organization

With respect to light stimuli, the ganglion cells in the retina act like filters. The filters have a "Mexican hat" shape, they have a tall central area that responds to input with excitation, and smaller peripheral lobes that respond to input with inhibition. These filter shapes translate directly to response shapes in the visual field. This relationship is called the "receptive field" of the neuron.

While the retinal photoreceptors respond to luminance (light intensity), the center-surround filters are well suited for processing contrast. It is the contrast that is most meaningful, it has to be extracted in both low light and bright light. (Object recognition should not depend on light intensity). When studying the center-surround organization of retinal receptive fields, it is important to understand that they are not static entities. Stimulation of a single spot on the retina creates a multiplicity of responses in retinal ganglion cells, some when the light turns on, some when it turns off, and some during the entire duration of the stimulus. Additionally, some retinal ganglion cells are direction sensitive, and some of the surrounds are too. Both the center and the surround take time to develop. Generally speaking the surrounds involve more (synaptic or ephaptic) transmission, so they take longer (Turner et al 2018, Wienbar & Schwartz 2018).

The filters in the retina pre-process the visual signal before it ever gets to the brain. The output of the retina is already gain-adapted, and subject to the centrally controlled effects of pupil constriction and lens accommodation. Many of the filters cause ganglion cells to respond well to edges and transients, as depicted in these drawings. In the retina the scale of this effect is mainly local, the wide-field edge detection related to the orientation of corners and the curvature of surfaces doesn't occur until the cerebral cortex.

Any or all of the P/M/K ganglion cells can be color sensitive and direction sensitive. Color sensitivity is conferred by a particular combination of retinal cones, and direction sensitivity can be generated by geometric specificity of both axon and dendrite branching patterns. In addition to the time-varying elements of receptive fields, there is also local and global adaptation mediated at the synaptic level. (There is something in between local and global, which I'd call "regional", organized at the level of the retinal honeycomb and connecting neighboring cells).

By the time the visual signal leaves the retina, it has been considerably modified, although its original character is still recognizable. Filtering by the retinal circuitry guarantees that the image leaving the retinal ganglion cells is not the same as the pattern of light that created it. So how then, does the brain reconstruct the original visual input from multiple channels that distort its character?

To answer this question, we have to keep going, and look at what happens to the visual information as it gets transmitted centrally from the eye. The first stop along the way to the cerebral cortex is the lateral geniculate nucleus of the thalamus, which receives inputs from retinal ganglion cells that feed the optic nerve.