The visual system has to develop in order. The proper sequence of development is crucial. In neural networks, systems usually develop from the outside in, the periphery develops first because that's where the data is, and the central networks wait until the properly formatted data is available. And the oculomotor system has to coordinate with the visual system, because it uses retinotopic maps and maps of the visual field.

Visual System Development

In humans the visual pathways develop before birth. The retino-geniculate fibers start forming the optic nerve around 7-15 gestational weeks, and cross the optic chiasm to reach the lateral geniculate nucleus at about 20-22 weeks (Hevner 2000). The synaptic layers of the LGN begin forming around 20 weeks, and cellular lamination follows shortly thereafter. Once the eyes open there is activity-dependent refinement of the synaptic connections, involving both the pruning of excitatory connections and the development of inhibitory connections, the latter being assisted by wave-like activity from the retina. The distinct firing patterns of on and off ganglion cells help organize the final synaptic pattern in the LGN.

Ganglion cells are the first to form in the fetal retina. The sequence of neural development is fascinating in the sense that its dependencies aren't immediately obvious. The order of retinal neuron development and connection stratification is shown in the figures. Note the retraction of the basal horizontal cell process.

We've already discussed the important of genetic markers, in determing the axes of the retina. There are two important markers that determine the horizontal and vertical axes respectively, and one that specifies the center of the fovea. In addition there is the radius of the fovea and the location of the optic nerve, which are all determined before the first exposure to light. When the retino-geniculate pathway forms, the optic chiasm is an important choice point for retinal axons, and the direction they travel is influenced by transcription factors within the axons themselves, that identify the type of source cell they're from. Therefore the gradients that determine retinal orientation are somehow preserved in the ganglion cells, and influence subsequent connectivity. Proteins called ephrins are involved (mainly in retinotopic mapping), as are members of the teneurin adhesion family (which are involved in eye-specific mapping).

One of the first things that happens in human visual development is that starburst amacrine cells start generating waves of oscillations in the retina. T2 amacrine cells are also involved in this process. The waves begin in the periphery of the retina and travel centrally towards the fovea. Such waves are in alignment with the "optic flow" that the organism will eventually experience when it's moving forward. The retinal waves are transmitted to the LGN and the superior colliculus by retinal ganglion cells, where they help align the connections to propertly map the visual field. The wave-like activity from the retina continues through adulthood, contributing to things like the retinal shift effect, which is also thought to be mediated by starburst and T2 amacrine cells. The starburst cells are cholinergic and excitatory, whereas the T2 cells use GABA and begin as excitatory but eventually become inhibitory when visual input starts arriving. The figure shows the time course of a retinal wave, visualized over 2 mm² using calcium imaging.

The wave-like activity in the amacrine cells generates waves of bursting in the retinal ganglion cells. As the wave propagates across the retina, neighboring ganglion cells start bursting in order. This activity contributes to the development of direction-sensitive ganglion cells, which can send axons to the superior colliculus or the LGN or both. The wave-like activity in the retina occurs in three phases during development. First there are infrequent bursts of patterned activity that appear embryonically and are likely mediated by gap junctions. In the second phase the cholinergic starburst amacrine cells become active, and this phase begins before the eyes open and continues through initial visual exposure. In the third phase there is ganglion cell bursting driven by glutamatergic bipolar cells, and it's due at least in part to an important mechanism called "glutamate spillover" which we'll revisit when we talk about the computational properties of dendritic spines.

As mentioned, some interesting things happen with neurotransmitters during development. For example the neurotransmitter GABA is excitatory early in development, it helps horizontal cells and amacrine cells form connections, but later it acquires an inhibitory function, responsible in part for tuning the ganglion cell surrounds.

In the lateral geniculate nucleus, the retinal axons begin with a rather wide proliferation, each axon will branch across 20 or more thalamocortical relay neurons, even after eye-specific zones are formed. After a while though, this connectivity is reduced by pruning, to one to three inputs per relay cell. The remaining inputs are strengthened, by up to 50 times, partially through an increase in neurotransmitter quantal size, but primary by an increase in the number of release sites. The change in connectivity as well as the change in synapse strength contribute to the development of the receptive field characteristics of the geniculo-cortical relay neurons. Some of these mechanisms are summarized in the figure.

The development of the retino-geniculate system has been modeled by Markowitz et al 2012. In its final form the retino-geniculate pathway is shown below, as visualized by MRI diffusion tractography.

Types of Plasticity

There is both data-driven plasticity and structural plasticity. An example of data-related plasticity is the adaptive retinal gain control discussed below. An example of structural plasticity is the formation of precise topographic maps during development. This latter process is both structural and data-driven, insofar as there is a competitive process between targets for nearest neighbors that depends on sufficient signal input over time.

During development there is cell proliferation and migration, as well as the extension of processes ultimately destined to become axons and dendrites. Besides the addition of neurons, there is communication between neurons. Before synapses even form (that is, before axons and dendrites extend), growing neurons may express gap junctions that form an electricaly syncytium and allow the exchange of small molecules (including signaling molecules). Once axonal and dendritic processes begin to extend, these gap junctions are sometimes retracted, but sometimes they're left in place.

The formation of new synaptic connections is a multi-stage sequential process that depends on both nature and nurture. Axons typically climb up concentration gradients of marker molecules that are released by their targets. Once they're "in the area", they sprout, and begin to form connections with many different kinds of neurons. Gradually, some of these connections are pruned, usually this process begins before actual synaptic throughput but continues through the initial period of electrical responsiveness. Once pruning is complete some of the plasticity may be turned off, although some may remain.

The mechanisms of synaptic pruning are mostly the same as those of synaptic plasticity, except that the results are different. The cooperative and competitive processes ultimately result in synaptic death for those synapses that don't cut it (so to speak), and if an axon doesn't have enough synapses it may retract or it may re-sprout and look for new connections. The sprouting geometry at the tip of the axon is called the "growth cone", and the ultimate connection scheme of an axon is called its "divergence". Axons may diverge geometrically, they may diverge to multiple cell types, and they may have specific effects on each postsynaptic cell type depending on the configuration of receptors and the relationship of the synapse to neighboring glial cells. Glial cells are important in many ways. They help regulate the homeostatic environment for neurons, and many or most of them display electrical activity as well (for example the B wave in the retinal ERG is impacted by the Muller cells, which are glial cells that extend all the way across the layers of the retina from the photoreceptors to the ganglion cells).

The biochemical pathways underlying synaptic pruning are quite complex. Part of that is because programmed synaptic death is related to programmed cell death (apoptosis). Some of the same underlying mechanisms are at work. One recent model postulates that individual synapses send out signals when they reach certain states. Xing et al 2025 characterize these as "eat me" and "don't eat me" signals, as shown.

Synaptic pruning can involve intracellular calcium, LTD and LTP, and neurotransmitters of all kinds. Weight scaling can occur on the basis of temporarily enhanced sensitivities to inputs. In humans and other mammals, synaptic development is associated with "critical periods" during which growth and pruning occur, followed by stabilization. Synaptic maturation in the early visual system is activity dependent. Putting vertical prisms on a kitten during the critical period results in a cat unable to climb stairs, because the horizontal orientation columns in the visual cortex never formed, only the vertical ones. In humans, once the connections have stabilized, they are still plastic, but with a time constant of days instead of hours. NASA experimented with the astronauts in the 60's, and found that they took about 10 days to adjust to inverting prisms, and upon removal, it took another ten days to adjust back.

Topopgraphic Maps

The axons leaving the retina (via the ganglion cells) need to form retinotopic maps with at least two very important targets, the lateral geniculate nucleus in the visual stream, and the superior colliculus in the oculomotor system. In addition there are at least a dozen other targets, some of which are retinotopic (like the pulvinar in the thalamus) and some of which aren't (like the pupillary light reflex).

The mechanism of topographic mapping initially involves genetically programmed marker molecules. The axes of the retina can be aligned with the axes of a target by two marker molecules. An interesting example is found in the separation of fibers in the optic chiasm.

When first forming connections, an axon will typically have a wide target area. The target area is then reduced by a competitive process with neighboring neurons. The parameters of this process depend on the time constants of the underlying proteins, however the signal levels may include information derived from active sensory input. Ultimately the programming of this process is genetically derived, even though its actual execution may depend on incoming data from the environment. The process is assisted by the retinal waves present during development, which help solidify the synaptic maps. (Retinal waves occur during development, and contribute to the retinal shift effect in adult humans).

Adaptive Gain Control

For the visual system to develop properly, the retina has to perform certain important functions so the signals that drive upstream synaptic formation are present when they're needed. One of the most important retinal functions is adaptive gain control. The retina works on contrast, the ganglion cells signal the differences between the centers and the surrounds. The range of contrast can change dramatically in bright light versus low light, and the rest of the visual system doesn't care (much) about the light level (except for the pupillary reflex and etc), all it really cares about is edges and contours and boundaries that define surfaces, so the retina helps the upstream systems extract this information by providing a consistent contrast range so features can be easily detected. The figure shows an example of gain control on a sunny day. The gain-controlled image is on the left, the original image is on the right.

Gain adaptation mechanisms have been included in convolutional network models like "Deep Retina" (Idrees et al 2024). It turns out that incorporating photoreceptor adaptation improves CNN performance in predicting retinal ganglion cell responses to naturalistic movies. Which we probably could have predicted, based on the information on these pages.

Additional contrast and luminance gain control occurs in the LGN (Raghavan et al 2023). This adaptive filtering is necessary for downstream systems to develop properly. The visual pathways have to develop in order, because proper development is data dependent.

Synchronization

While topographic maps are forming, systems dynamics are also forming. The system dynamics will typically change as the synaptic locations and strengths change, during synaptic growth and pruning. Part of the input for synaptic determination may be the dynamics of individual neurons, which in turn determine and are determined by the population dynamics. Certain kinds of dynamic behavior may be needed during development, that are no longer needed once the connections are determined. So during and after synaptic determination, there are typically changes to the molecular machinery inside the synapse, including the receptor organization that ultimately determines a neuron's response to arriving signals.

An example of dynamic synchronization early in development is the interplay between saccades and smooth pursuit movements. Early in human development, infants are drawn to bright lights, their eyes move to those locations. This occurs about the same time the topographic map between the retina and the superior colliculus is forming, and this map supports things like the looming reflex we discussed earlier. Not only do retinal axons need to be properly aligned with their targets in the SC, but there is a directional map overlaid on top of the retinotopic map that serves as a target for a subpopulation of directionally sensitive ganglion cells. The connectivity of this sub-map is assisted by the retinal oscillations ("waves") we discussed earlier.

Once the retina is properly connected, the visual system can begin driving saccades to selected targets. This ability requires a functional pathway through the cerebral cortex, and in turn the cerebral cortex requires a great deal of visual input to develop properly. Voluntary saccades to targets begin in the first month of human life, and are well developed by three months. Very young infants localize distant targets in multiple saccades of roughly equal amplitude, unlike adults who make one large saccade followed by corrections. However infants as young as 3-4 months can plan saccades based on attentional goals, and by 7 months can adjust saccade amplitude based on visual feedback from the previous trial.

Once the saccadic system is in place, reaction time decreases with age, maximum amplitude increases over time, stimulus dependence tends to disappear, and scanning efficiency increases even across scenes.

Newborns can initiate smooth pursuit movements, but they are limited in scope, as they primarily involve the superior colliculus before the cortical pathways have developed. The fovea is still immature when smooth pursuit first appears. Smooth pursuit gain increases with age until about 6 months, and at 5 months the pursuit movements begin leading the stimulus (this function is ascribed to development of the cerebellum).

Now that we've established a basic framework for vision and related it to both biology and machines, let's move on to look at the oculomotor system. The visual and oculomotor systems work together, they're part and parcel of the same effort. The closed-loop feedback from the oculomotor system to the visual system occurs on the basis of the position of the retinal image in the visual field. When the image is not aligned with the desired image, an error occurs. The error needs to be calculated, and the calculation is not necessarily simple. Let's take a look at how the oculomotor system works, and then try to understand how the error signal is created and applied.