In 1959, David Hubel and Torsten Wiesel, two Harvard neuroscientists, conducted a series of experiments on anesthetized cats to understand how visual perception works in animals.

They found that neurons in the visual cortex are specialized to detect features like specific orientations, shapes, and movements. These cells have “receptive fields” organized into excitatory and inhibitory regions which filter and highlight patterns, very much like a convolutional algorithm.

We think we see RGB pixels but we actually see in representation space. The brain encodes the world as abstractions, edges, shapes, objects, not raw data. Both CNNs and the visual cortex use convolutions to extract meaning from raw input.

This should be obvious to anyone who has dabbled in psychedelics. See these images generated by Google’s DeepDream.

Contrary to what you might think, DeepDream is not making these images trippy on purpose. All it does is exaggerate the existing patterns in the image to help us visualize how CNNs “see”.

This is also what psychedelics do to our perception of reality i.e. amplify the patterns that are already there. Normally, the brain integrates sensory data into a cohesive narrative by ironing out inconsistencies. Psychedelics weaken this integration, letting unprocessed data flood consciousness. DeepDream’s feedback amplifies features; psychedelic feedback amplifies experience.