Hubbry Logo
Visual musicVisual musicMain
Open search
Visual music
Community hub
Visual music
logo
8 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Visual music
Visual music
Visual music
View on Wikipedia
from Wikipedia
Lichtspiel: Opus I (1921) by Walter Ruttmann

Visual music, sometimes called color music, refers to the creation of a visual analogue to musical form by adapting musical structures for visual composition, which can also include silent films or silent Lumia work. It also refers to methods or devices which can translate sounds or music into a related visual presentation. An expanded definition may include the translation of music to painting; this was the original definition of the term, as coined by Roger Fry in 1912 to describe the work of Wassily Kandinsky.[1] There are a variety of definitions of visual music, particularly as the field continues to expand. In some recent writing, usually in the fine art world, visual music is often conflated with or defined as synaesthesia, though historically this has never been a definition of visual music. Visual music has also been defined as a form of intermedia.

Visual music also refers to systems which convert music or sound directly into visual forms, such as film, video, computer graphics, installations or performances by means of a mechanical instrument, an artist's interpretation, or a computer. The reverse is applicable also, literally converting images to sound by drawn objects and figures on a film's soundtrack, in a technique known as drawn or graphical sound. Famous visual music artists include Mary Ellen Bute, Jordan Belson, Oskar Fischinger, Norman McLaren, John Whitney Sr., and Thomas Wilfred, plus a number of contemporary artists.

Instruments

[edit]

The history of this tradition includes many experiments with color organs. Artist or inventors "built instruments, usually called 'color organs,' that would display modulated colored light in some kind of fluid fashion comparable to music".[2] For example, the Farblichtspiele ('colored-light-plays') of former Bauhaus student Ludwig Hirschfeld Mack. Several different definitions of color music exist; one is that color music is generally formless projections of colored light. Some scholars and writers have used the term color music interchangeably with visual music.

The construction of instruments to perform visual music live, as with sonic music, has been a continuous concern of this art. Color organs, while related, form an earlier tradition extending as early as the eighteenth century with the Jesuit Louis Bertrand Castel building an ocular harpsichord in the 1730s (visited by Georg Philipp Telemann, who composed for it). Other prominent color organ artist-inventors include: Alexander Wallace Rimington, Bainbridge Bishop, Thomas Wilfred, Charles Dockum, Mary Hallock-Greenewalt and Kurt Laurenz Theinert.[citation needed]

On film

[edit]

Visual music and abstract film or video often coincide. Some of the earliest known films of these two genres were hand-painted works produced by the Futurists Bruno Corra[3] and Arnaldo Ginna between 1911 and 1912 (as they report in the Futurist Manifesto of Cinema), which are now lost. Mary Hallock-Greenewalt produced several reels of hand-painted films (although not traditional motion pictures) that are held by the Historical Society of Philadelphia. Like the Futurist films, and many other visual music films, her 'films' were meant to be a visualization of musical form.

Notable visual music filmmakers include: Walter Ruttmann, Hans Richter, Viking Eggeling, Oskar Fischinger, Len Lye, Mary Ellen Bute, Jordan Belson, Norman McLaren, Harry Smith, Hy Hirsh, John, James Whitney, Steven Woloshen, Richard Reeves and many others up to present day.

Computer graphics

[edit]
Oscilloscope showing a single pitch, a sine wave

The cathode ray tube made possible the oscilloscope, an early electronic device that can produce images that are easily associated with sounds from microphones. The modern Laser lighting display displays wave patterns produced by similar circuitry. The imagery used to represent audio in digital audio workstations is largely based on familiar oscilloscope patterns.

The Animusic company (originally called 'Visual Music') has repeatedly demonstrated the use of computers to convert music — principally pop-rock based and composed as MIDI events — to animations. Graphic artist-designed virtual instruments which either play themselves or are played by virtual objects are all, along with the sounds, controlled by MIDI instructions.[4]

In the image-to-sound sphere, MetaSynth[5] includes a feature which converts images to sounds. The tool uses drawn or imported bitmap images, which can be manipulated with graphic tools, to generate new sounds or process existing audio. A reverse function allows the creation of images from sounds.[6]

Virtual reality

[edit]

With the increasing popularity of head mounted displays for virtual reality [7][8][9] there is an emerging new platform for visual music. While some developers have been focused on the impact of virtual reality on live music [10] or on the possibilities for music videos,[11] virtual reality is also an emerging field for music visualization[12][13][14][15] and visual music.[16]

Graphic notation

[edit]

Many composers have applied graphic notation to write compositions. Pioneering examples are the graphical scores of John Cage and Morton Feldman. Also known is the graphical score of György Ligeti's Artikulation designed by Rainer Wehinger, and Sylvano Bussotti.

Musical theorists such as Harry Partch, Erv Wilson, Ivor Darreg, Glenn Branca, and Yuri Landman applied geometry in detailed visual musical diagrams explaining microtonal structures and musical scales.

See also

[edit]

Science

[edit]

Industry

[edit]

Similar types of art

[edit]

Notes

[edit]

Further reading

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Visual music

View on Grokipedia
from Grokipedia
Visual music is an artistic practice that translates musical compositions or sounds into visual forms, emulating the original auditory syntax through abstract imagery, color, rhythm, and motion to evoke spatial and temporal experiences akin to hearing music. This intermedia form often draws on , where sensory experiences like sound and sight overlap, and encompasses a range from static paintings to dynamic films and installations. The history of visual music spans over three centuries, originating in theoretical ideas from ancient philosophers like and , who explored correspondences between sound and color, and advancing through 18th-century inventions such as Louis-Bertrand Castel's Clavecin pour les yeux (Ocular Harpsichord) in 1734, an early designed to project colored lights in response to musical notes. In the 19th and early 20th centuries, the movement gained momentum with the development of mechanical s and influenced by composers like and , whose atonal music inspired visual abstractions. Pioneering exhibitions, such as the 2005 "Visual Music" show at the Museum of Contemporary Art in and the Hirshhorn Museum, highlighted over 90 works by more than 40 artists, underscoring its evolution from painting to multimedia. Key figures in visual music include Wassily Kandinsky, whose synesthetic experiences led him to associate specific colors with musical tones—such as bright yellow with high notes—and to produce seminal abstract works like Fragment 2 for Composition VII (1913), as detailed in his 1912 treatise Concerning the Spiritual in Art. Other influential artists encompass Oskar Fischinger, known for films like Ornament Sound (c. 1932), which directly translated sounds into visual patterns; Viking Eggeling and Hans Richter, founders of the 1920s Absolute Film movement in Germany; Len Lye, with his kinetic animations; and Norman McLaren, a pioneer in animated sound visualization. Later contributors include John and James Whitney, who developed computer-generated visuals in the mid-20th century, and contemporary creators like Jordan Belson with Epilogue (2005) and Jennifer Steinkamp with interactive installations such as SWELL (1995). Notable works further illustrate visual music's breadth, including Mikhail Matiushin's Painterly-Musical Construction (1918), an early experiment in color-musical synthesis, and Georgia O'Keeffe's Blue and Green Music (1921), which evoked undulating rhythms through form and hue. Thomas Wilfred's Study in Depth: Opus 152 (1959) exemplified lumia, or "light music," using projected colored lights to mimic orchestral dynamics. In the digital era, artists like Scott Draves have produced algorithmic pieces such as Dreams in High Fidelity, leveraging software to generate evolving visual patterns synchronized with sound, reflecting the medium's adaptation to technology since the late 1990s. Visual music continues to influence , bridging disciplines like , installation, and , while facing challenges such as scholarly inaccuracies and varying interpretations of its nature. In recent years, particularly since 2020, has emerged as a tool for generative visual music, enabling dynamic, algorithm-driven abstractions synchronized with sound. Its enduring appeal lies in the pursuit of universal sensory harmony, as seen in ongoing exhibitions and research that expand its historical canon.

Definition and Origins

Definition

Visual music is an art form that creates time-based visual imagery analogous to the temporal and structural elements of , which is non-narrative and devoid of programmatic content. This practice translates musical forms into abstract visual compositions, emphasizing non-representational shapes, colors, and movements to evoke synaesthetic experiences where visual elements resonate with auditory perceptions. Unlike narrative-driven media, visual music prioritizes pure to mirror music's intrinsic qualities, such as progression and , without literal depiction or storytelling. The term "visual music" was coined by British art critic in 1912, specifically in reference to the improvisational abstract paintings of , which Fry described as evoking musical rhythms through color and form. This nomenclature highlighted the emerging idea of visual art operating like music in its temporal flow and emotional immediacy, independent of representational subjects. Central to visual music are characteristics like the precise synchronization of visual rhythms, colors, shapes, and movements with musical components including pitch, tempo, harmony, and dynamics, creating a direct analogue between auditory and visual stimuli. This distinguishes it from conventional music videos or representational animations, which often prioritize narrative or illustrative elements over abstract structural equivalence. Synaesthesia serves as a foundational concept in visual music, with historical claims by artists such as Kandinsky, who described experiencing auditory sensations—like specific tones or harmonies—from visual stimuli such as colors and forms. However, debate persists over whether Kandinsky truly possessed synaesthesia or employed it metaphorically to articulate his artistic process, as analyses of his accounts reveal inconsistencies with clinical definitions of the condition.

Historical Development

The concept of visual music traces its origins to the early , when French Jesuit scholar Louis Bertrand Castel proposed the "ocular " in the , an instrument designed to associate specific colors with musical notes to create a synesthetic experience of "color music." This theoretical innovation, inspired by analogies between sound and light, laid foundational ideas for linking auditory and visual elements, though no functional prototype was built during Castel's lifetime. In the , these ideas advanced toward practical devices, exemplified by British painter Alexander Wallace Rimington's invention of the Colour Organ in 1893, a keyboard-controlled apparatus that projected colored lights in synchronization with music to evoke emotional responses through visual harmony. Rimington's device, patented and demonstrated publicly in by 1895, represented an early attempt to perform "color symphonies," influencing subsequent light shows and organ-like projections that bridged music and abstract visuals. Early 20th-century breakthroughs emerged within the Italian Futurist movement, where artists Bruno Corra and Arnaldo Ginna created the first hand-painted abstract films in 1911–1912, including titles like The Rainbow and The Dance, directly translating musical rhythms and moods into non-representational colored patterns on film strips. These pioneering works, though largely lost, anticipated synchronized audiovisual abstraction. Shortly after, in 1921, German filmmaker Walter Ruttmann released Lichtspiel: Opus I, recognized as the first abstract film explicitly synced to music, using oil paints on glass to generate fluid, rhythmic light forms that visualized tonal progressions. The through 1940s marked a golden age of visual music, driven by innovative animations and projections. produced a series of abstract Studies in the , synchronizing geometric forms and colors to music through hand-drawn and mechanical techniques, establishing visual music as a cinematic art form. In the 1930s, American animator Mary Ellen Bute created color organ-inspired films like Rhythm in Light (1934), employing oscilloscope-like visuals to abstractly interpret classical compositions, blending electronic with musical structure. Paralleling these efforts, Norwegian-American artist Thomas Wilfred developed Lumia projections from the into the 1960s using his Clavilux device, which generated evolving, music-accompanied light compositions performed in theaters and galleries. Post-World War II expansion in the 1940s–1960s incorporated technological advancements, as seen in John Whitney Sr.'s analog computer animations, where he repurposed military equipment to generate parametric patterns synced to music, producing works like Film Exercises that automated complex visual harmonies. Similarly, at the , pioneered graphical sound techniques from the 1930s through the 1950s, drawing waveforms directly on film to create synthetic scores for abstract animations, such as Synchromy (1971), which reversed traditional image-sound hierarchies. By the mid-20th century, visual music gained institutional recognition through exhibitions in the –1970s, culminating in retrospectives like the 2005 "Visual Music" show, which highlighted this era's milestones in synesthetic art and .

Creation Techniques

Visual Instruments

Visual instruments represent a pivotal development in the early history of visual music, comprising mechanical and optical devices designed to produce synchronized light displays in response to musical performance. These pre-digital apparatuses, often resembling traditional musical instruments like harpsichords or organs, aimed to translate auditory input into visual output through direct physical linkages, enabling real-time synesthetic experiences. Emerging from 18th-century theoretical proposals, they evolved into practical hardware by the late 19th and early 20th centuries, primarily using keyboards or pedals to control lights projected onto screens or walls. One of the earliest conceptualizations was the ocular proposed by French Jesuit mathematician Louis-Bertrand Castel in 1725. Castel's design theoretically mapped musical notes to specific colors, drawing on Isaac Newton's spectrum theory, where each key of a would lift a curtain to reveal light passing through colored glass pieces, creating "visual melodies" that could be appreciated by the deaf or in silent performances. Although Castel never built a functional , his ideas inspired 19th-century realizations, such as devices using prisms or rotating wheels to generate color sequences aligned with musical scales, establishing a fixed correspondence between pitches (e.g., C to ) and hues. In the late , practical implementations advanced with inventions like Bainbridge Bishop's in the . This American artist's attached lighting instruments to a , using carbon-arc lamps to project colored beams onto a screen, modulated by keyboard inputs to mimic musical phrases in light. Bishop's patented device (1877) employed gels and filters for hue control, with organ keys directly triggering light intensity and color changes to parallel note durations and harmonies. Similarly, British painter Alexander Wallace Rimington's Colour Organ, patented in 1893, featured a five-octave keyboard integrated with arc lamps, prisms, and diaphragms; pressing keys activated rotating discs for hue and intensity variation, synced to organ pipes, while stops and pedals adjusted and strengths to evoke rhythmic visual patterns. A landmark in this lineage was Thomas Wilfred's Lumia suite and Clavilux instruments, developed from the 1920s through the 1960s. The Danish-American artist's Clavilux projected ever-changing colored forms—resembling auroras—onto screens, with keyboard controls modulating light via mechanical linkages to gels, motors, and shutters, allowing performers to compose visual "phrases" that echoed musical structures. Wilfred patented multiple versions, emphasizing fluid motion and form alongside color, and composed over 30 Lumia works for live presentation, where operators used the device to synchronize visuals with accompanying music in theater settings. These instruments operated on direct mechanical principles, linking sound production mechanisms—such as keys, pedals, or organ stops—to light generation components like gas lamps, carbon-arc sources, colored gels, prisms, or early projectors. For instance, a key press might mechanically open a shutter to allow through a specific filter while simultaneously sounding a note, ensuring temporal alignment between auditory and visual elements; intensity was often varied via rotating wheels or diaphragms to reflect dynamics, and hue selection relied on fixed mappings derived from natural analogies like the color to the musical . Early power sources included gas illumination, later transitioning to electric arcs for brighter projections, though setups required manual operation and were prone to hazards like overheating. Despite their innovations, analog visual instruments faced significant limitations, particularly rigid fixed associations between notes and colors, which constrained expressive flexibility—for example, assigning a single hue like to all C notes across s, regardless of context or performer intent. These constraints, rooted in early theories like Castel's or Newton's, limited adaptability in real-time performances, as mappings did not account for perceptual variations in pitch cyclicity or emotional nuance, often resulting in repetitive visuals that failed to fully capture musical complexity. Their legacy endures in the influence on light concerts in theaters, where devices like the Clavilux enabled immersive live spectacles, paving the way for synesthetic art forms while highlighting the need for more dynamic technologies.

Graphic Notation

Graphic notation emerged in the modernist era as a means to represent musical structures through abstract visual forms, distinct from conventional staff-based systems. Wassily Kandinsky's paintings from 1911 to 1913, such as Composition VII, served as proto-notations, translating synesthetic experiences of sound into colors, lines, and shapes that evoked auditory sensations like chords and rhythms. Influenced by his chromesthesia, where music triggered vivid visual imagery, Kandinsky described color as a "keyboard" for the soul, using dynamic forms to parallel musical progression and intensity. This approach laid foundational principles for graphic notation by prioritizing emotional and structural equivalence between visual and sonic elements. In the mid-20th century, advanced these ideas with the UPIC (Unité Polyagogique Informatique du CEMAMu) system, conceived in the mid-1970s and first realized in 1977, which enabled composers to draw sound waves and timbres directly on a graphic tablet for computer-assisted synthesis. Users sketched curves and lines to generate wavetables, mapping horizontal axes to time (from milliseconds to minutes) and vertical dimensions to and , thus visualizing and probabilistic musical processes. 's first UPIC composition, Mycènes Alpha (1978), demonstrated this by converting hand-drawn graphics into complex electronic textures, emphasizing the system's role in democratizing composition for non-specialists. Prominent examples include John Cage's notational experiments in the 1950s, culminating in the Concert for Piano and Orchestra (1957–1958), where 63 pages of abstract graphics—featuring lines, numbers, and symbols—allowed performers to interpret pitch, duration, and amplitude through chance operations without a fixed score. The piano part employed 84 types of notations, such as note sizes for amplitude and spatial arrangements for timing, fostering indeterminacy and performer agency. Similarly, György Ligeti's micropolyphonic works from the late 1950s, like the Kyrie from Requiem (1963–1965, developed from earlier ideas), used dense clusters of lines to notate overlapping voices, where visual proximity conveyed textural fusion and rhythmic complexity beyond individual pitches. Techniques in graphic notation employ lines for directional flow (e.g., rising for ascending pitch), shapes like circles or boxes for timbral clusters or registers, colors to differentiate instrumental families or dynamics, and spatial layouts to indicate and duration, as seen in Earle Brown's December 1952 (1952), where vertical positioning suggests pitch and horizontal spread denotes time. Density of marks often represents intensity or polyphonic layering, such as in Morton Feldman's Projection 4 (1951), where grouped squares evoke sustained textures without precise note values. Unlike traditional notation's linear specificity on staves, these methods embrace ambiguity, prioritizing perceptual and interpretive freedom to capture non-Western or aleatoric structures. Graphic notation fulfills a dual role as both performative art and analytical tool: in live settings, scores like Cornelius Cardew's (1963–1967) are projected as expansive drawings, inviting improvisers to respond to relational visuals such as curving lines for melodic arcs. As an analytical aid, it visualizes intricate forms like stochastic music in Xenakis's works, where probabilistic distributions appear as scattered points or gradients, helping composers model chaos and density. Modern extensions in the , such as Eric Wenger's MetaSynth software (initially released around 1996), built on these principles by converting images into sound via pixel-based synthesis, where controls envelopes and RGB values map to pitch and , allowing graphical sketches to generate audio directly. This tool preserved the focus on visual abstraction, enabling composers to treat photographs or drawings as scores for experimental sound design without traditional interfaces.

Digital Methods

Digital methods in visual music emerged in the mid-20th century as computational tools enabled the transition from analog hardware to programmable systems for synchronizing sound and visuals. John Whitney Sr., a pioneering figure in computer animation, bridged analog and digital approaches during the 1960s and 1970s by adapting slit-scan techniques—originally mechanical processes involving moving slits to create distorted motion—into early digital frameworks using military surplus computers like the M-5 Antiaircraft Gun Director. This allowed for precise control over parametric patterns that responded to musical rhythms, as seen in works like Catalog (1961), where algorithmic variations produced abstract, music-like visual forms. By the 1980s, oscilloscope-based visuals advanced this lineage, with artists employing vector graphics to draw waveforms directly from audio signals on cathode-ray tubes, creating Lissajous figures that mirrored harmonic structures in real time. Jerobeam Fenderson exemplified this in later iterations, using analog oscilloscopes interfaced with digital audio to generate intricate, sound-driven vector patterns, as in his Oscilloscope Music series, where left and right audio channels control horizontal and vertical deflections for synchronized geometric displays. Software tools proliferated in the 1990s, facilitating user-friendly creation of audio-visual compositions without specialized hardware. MetaSynth, developed by Eric Wenger, introduced spectral image-to-sound mapping, where users paint frequency-time images that the software resynthesizes into audio, and conversely, audio spectra are visualized as editable images for granular manipulation, enabling composers to sculpt sounds and visuals interchangeably. Real-time patching environments like (introduced in 1990 by ) and its open-source counterpart (developed by Miller Puckette in 1996) allowed artists to build modular networks of audio and video objects, routing signals for live synchronization—such as modulating video textures with oscillator outputs—to create responsive installations. For larger-scale interactive works, node-based platforms like (since 2004) and (from in 2009) support real-time rendering of visuals driven by audio inputs, ideal for immersive environments where sensors or controllers trigger procedural effects. Core algorithms underpin these tools, transforming audio data into visual elements with mathematical precision. The Fast Fourier Transform (FFT) decomposes sound into frequency components, mapping amplitudes to colors or shapes in spectrum analyzers—for instance, low frequencies as warm hues and high harmonics as angular forms—to visualize musical spectra as dynamic, layered abstractions. extends this through particle systems, where note data governs particle birth, velocity, and decay; in environments like , incoming triggers simulate flocking behaviors or explosive bursts aligned with beats, producing emergent visuals from simple rules. Post-2010 advancements integrated , enhancing generative capabilities. Neural networks, particularly Generative Adversarial Networks (GANs), analyze audio features like and to produce abstract visuals, as in ML's tools, which train on datasets to output synchronized animations from sound inputs, allowing artists to iterate on styles without manual coding. In live contexts, software like Resolume (evolving through the 2020s) incorporates these for , where AI-assisted effects layers respond to DJ sets in real time, blending clips with audio-reactive distortions for performances. By 2024–2025, AI-generated have further advanced this, with tools creating fully synchronized visuals from audio tracks, as seen in releases like Linkin Park's "Lost" and remixes of classic tracks, enabling rapid production of immersive audiovisual content. Compared to analog methods, digital approaches offer infinite variability through algorithmic recombination, enabling endless permutations from finite inputs; heightened interactivity via real-time user control; and scalability for multi-screen installations or web distribution without physical degradation.

Media Applications

In Film and Animation

Visual music found early expression in abstract films of the 1920s and 1930s, where filmmakers synchronized non-representational imagery to musical structures. Oskar Fischinger's Studies series, comprising 13 short black-and-white films produced in the late 1920s, exemplified this approach through cutout animations and cameraless techniques, such as drawing geometric forms on white paper with charcoal to create fluid, rhythmic movements aligned with musical rhythms. These works, including Studie Nr. 5 (1930), emphasized orchestral density and spatial depth, transforming musical phrases into dynamic visual patterns. Similarly, Len Lye pioneered direct-on-film scratching in the 1930s, etching and painting abstract designs directly onto celluloid strips to produce vibrant, syncopated animations synced to jazz and dance tunes, as seen in films like Rhythm (1936), which captured asymmetrical rhythms approximating swing phrasings. The graphical sound era advanced visual music by integrating image and audio production on the same film strip. At the (NFB), conducted optical soundtrack experiments from the mid-1930s through the 1950s, etching, drawing, and photographing patterns directly onto the film's soundtrack area to generate both synthetic sounds and corresponding visuals. This technique, used in works like Dots (1940) and Loops (1940), allowed precise synchronization of graphical waveforms to produce percussive and tonal music, effectively making the film a dual medium for visual and auditory abstraction. McLaren's innovations predated electronic synthesizers, treating the optical track as a canvas for "graphical music." Hollywood and experimental cinema intersected in the 1930s and through efforts to commercialize visual music. Mary Ellen Bute's Seeing Sound series, produced from the mid-1930s to the , visualized classical compositions using abstract animations generated with mechanical devices and early electronic oscillators to translate sound waves into luminous, oscillating forms. These shorts, such as Synchrony (1938), screened in theaters as preludes to feature films, bridging experimentation with mainstream audiences by rendering music as geometric light patterns. In the 1950s and 1960s, Jordan Belson's psychedelic films, including those derived from concert series (1957–1959), evoked cosmic immersion through layered abstractions of color and motion, often projected in planetariums to accompany electronic scores. Belson's works, like Allures (1961), intensified visual music's sensory impact, influencing the era's countercultural light experiences. Key techniques in these films included frame-by-frame painting, multiple exposures, and selective rotoscoping to ensure precise alignment of visual motifs with musical phrases. Fischinger's Kreise (1934), for instance, employed frame-by-frame painting on punched paper and multiple exposures in the three-strip GasparColor process to produce pulsating circles that responded to classical motifs, creating a luminous, rhythmic interplay of form and sound. , though less common in pure abstraction, was adapted in some hybrid works to trace musical waveforms or subtle motions for enhanced synchronization. These methods prioritized analog precision, allowing artists to craft non-narrative sequences where visuals directly mirrored musical structure, , and . By the , visual music in film evolved toward hybrid analog-digital formats, building on the multi-media spectacles of the 1967 Montreal Expo. The Expo's light shows and experimental presentations, such as those in the pavilion, combined analog projections with emerging electronic controls for immersive, synchronized environments that foreshadowed digital integration in abstract cinema. These hybrid approaches expanded visual music's scale, using multiple screens and automated lighting to amplify musical responsiveness in live settings.

Computer Graphics

The application of to visual music began gaining prominence in the 1970s and 1980s, building on early digital experimentation to produce synchronized abstract animations. John Whitney, recognized as a foundational figure in , shifted from analog techniques—such as those in his 1961 Catalog—to digital using systems like the IBM 360 computer by the late 1960s and 1970s. These efforts created parametric patterns that visualized musical rhythms through looping geometric forms, marking some of the first instances of digitally rendered visual music. By the 1990s, projects like , developed by Wayne Lytle starting in the mid-1990s, advanced this tradition with physics-based simulations of self-playing instruments. These computer-generated animations depicted robotic mechanisms—such as drum machines and string synthesizers—that appeared to autonomously produce sound, with motions precisely modeled using and to align with composed music tracks. Rendering techniques in for visual music often exploit optical principles to parallel auditory ones, enhancing synesthetic expression. Ray tracing, a core method for simulating light propagation, has been adapted to generate visual representations of harmonic interference, where virtual light rays mimic sound wave superpositions to form evolving patterns akin to musical overtones. Complementing this, shader programming in (GLSL) enables real-time manipulation of visuals driven by audio input, such as modulating colors and textures based on spectral analysis from fast Fourier transforms (FFT). This allows for fluid deformations of shapes— like warping meshes or blending hues—that respond instantaneously to bands in music, facilitating live performances. Key software ecosystems have democratized procedural generation of visual music. MilkDrop, released in 2001 by Ryan Geiss as a Winamp plugin, pioneered hardware-accelerated visualization through per-frame FFT processing, which analyzes audio waveforms to drive parametric equations deforming 2D and 3D primitives into hypnotic, beat-synced forms. Similarly, Houdini's Channel Operator (CHOP) networks process audio signals to parameterize procedural models, generating particle systems or geometry that evolve with musical amplitude and pitch, while Processing—an open-source Java-based environment—supports custom sketches integrating audio libraries for algorithmic visuals like oscillating grids tied to sound spectra. MIDI integration further enhances these tools, enabling live synchronization where note data from instruments directly controls graphic parameters, such as scaling fractals or triggering particle bursts, in real-time rendering pipelines. Notable contemporary works, particularly from the 2000s to 2020s, leverage these advancements for large-scale applications. teamLab, an interdisciplinary collective established in 2001, employs in interactive installations where projected visuals—rendered via Unity and custom engines—pulse and morph in harmony with ambient music, using sensor-driven algorithms to adapt patterns to sonic cues. Hardware evolution has underpinned this progression, from cathode ray tube (CRT) oscilloscopes in the mid-20th century, which displayed basic vector Lissajous figures as audio visualizers, to modern GPU-accelerated rendering. GPUs now handle parallel computations for intricate simulations, such as geometries (e.g., Mandelbrot iterations) that expand and contract in sync with beats, enabling high-frame-rate outputs unattainable on earlier systems.

Virtual Reality

Virtual reality (VR) has expanded visual music into fully immersive environments since the , enabling users to inhabit abstract audio-visual spaces where movement and sound synchronize in three dimensions. Early experiments, such as Char Davies' Osmose (1995), pioneered this integration by using body gestures to navigate ethereal virtual landscapes accompanied by interactive 3D soundscapes, creating a symbiotic relationship between physical motion and evolving musical visuals. In this installation, participants floated through grid-like and organic realms, with spatialized audio—composed with elements of natural and synthetic tones—triggering visual transformations, thus blending somatic immersion with auditory rhythms. Advancements in consumer VR hardware during the 2010s and 2020s, particularly with platforms like Oculus and , have democratized 360-degree visual music experiences, allowing real-time synchronization of abstract graphics to music in head-tracked environments. Applications such as Fantasynth VR on transform user-selected tracks into dynamic particle systems and waveform visualizations that respond to head movements and gestures, enveloping the viewer in pulsating, music-driven geometries. Similarly, (2018), a gamified VR rhythm title for Oculus and , exemplifies this evolution by syncing neon block-slicing mechanics and environmental effects to (EDM) beats, where visual cues like glowing sabers and rhythmic light pulses heighten the perceptual fusion of . Core techniques in VR visual music leverage spatial audio mapping to drive 3D visuals, such as binaural soundscapes that animate particle fields or volumetric effects in response to musical frequencies and panning. For instance, tools like Ableton's Envelop for Live enable ambisonic audio to position sounds in virtual space, which in turn modulates visual elements like scattering particles or morphing structures, enhancing the sense of depth and directionality. Haptic feedback further enriches these experiences by integrating tactile responses to music, with wearable devices delivering vibrations synced to basslines or melodies, simulating physical impacts that align touch with auditory and visual pulses in VR concerts. Prominent works in the 2020s, showcased in 's VR art galleries and Immersive Pavilions, utilize engines like Unity and Unreal for real-time rendering of interactive abstract worlds that react to user-input music. Projects in 2022's Immersive Pavilion, for example, featured multi-sensory VR musical journeys where participants co-create evolving visual symphonies through gestures, with Unity-powered environments generating patterns and light orchestrations tied to live audio inputs. Recent advancements as of 2025 include explorations of AI to synchronize real-time music generation with visual elements in VR, enhancing immersive audio-visual experiences. Despite these innovations, challenges like persist in VR visual music, addressed through smooth visual rhythms that align frame rates and animations to musical tempos for reduced disorientation. Joyful or soft music tracks have been shown to significantly mitigate cybersickness symptoms in VR sessions, promoting calmer of dynamic visuals. Post-2020, collaborative multi-user VR concerts have emerged as a key , enabling shared spaces for synchronized audio-visual performances, as seen in platforms like NOYS VR where remote audiences interact in real-time musical metaverses.

Scientific Foundations

The scientific foundations of visual music rest on perceptual and acoustic principles derived from , physics, and , which explain how auditory and visual stimuli can be integrated to create unified experiences. Synaesthesia research provides a key neurological basis, where sensory experiences involuntarily cross modalities, such as perceiving sounds as colors. Early investigations in the , led by , documented these phenomena through surveys of mental imagery, revealing familial patterns and consistent associations like numbers evoking specific hues. In artists like , debates persist on whether such experiences were innate—rooted in hyperconnectivity between sensory brain areas—or acquired through prolonged exposure to music and art, as evidenced by analyses of his writings describing auditory-visual equivalences. Psychoacoustics further elucidates correspondences between sound attributes and visual qualities, where listeners intuitively map auditory features onto visual ones. For instance, higher sound frequencies are commonly associated with brighter colors and lighter tones, a pattern observed across cultures and supported by experimental studies on cross-modal matching. This principle informed Alexander Scriabin's 1911 score for Prometheus: The Poem of Fire, which specified a "color organ" to project lights corresponding to musical keys—e.g., the key of triggering red hues—drawing on his personal synaesthetic perceptions and early psychoacoustic theories of sensory . From a physics perspective, analogies between sound waves and spectra underpin visual representations of , as both are oscillatory phenomena governed by wave equations. Sound waves, propagating as pressure variations in air, can be visualized using oscilloscopes to display harmonics as complex waveforms, revealing overtones and timbres. Lissajous figures exemplify this, formed by superimposing two perpendicular sinusoidal oscillations at harmonic ratios (e.g., 2:1 for an ), producing closed curves that illustrate phase relationships and interactions, thus providing a geometric analog for musical intervals. Cognitive science contributes through Gestalt principles, which describe how the brain organizes disparate sensory inputs into coherent wholes, facilitating audio-visual unity. Principles like similarity (grouping similar sounds and visuals) and proximity (temporal alignment of stimuli) enhance cross-modal perception, as shown in experiments where synchronized auditory rhythms improve visual grouping. (fMRI) studies from the 2000s further reveal neural mechanisms, with activations in the during music-color associations indicating integrated processing in multisensory areas, supporting the perceptual binding essential to visual music. Microtonal and mathematical notations extend these foundations by employing geometry to visualize non-Western scales, transcending . The 20th-century Bohlen-Pierce scale, dividing the 3:1 (a "tritave") into 13 equal steps of approximately 146 cents, uses circular diagrams to map intervals, highlighting just like 9:7 and 7:5 without equivalence, thus geometrically representing alternative structures. Visual music has significantly influenced arts, particularly through its integration into performance-based movements like in the 1960s, where artists drew on musical and visual experimentation to create multimedia experiences. Nam June Paik's development of video synthesizers, such as the Paik-Abe synthesizer, exemplified this by enabling real-time manipulation of video signals in response to sound, blending electronic music with dynamic visuals in live settings. These innovations extended to connections with and , where perceptual illusions of movement and color vibration, as pioneered by , inspired abstract visual patterns that evoke rhythmic responses akin to musical structures, fostering synesthetic interplay between sight and sound. The evolution of music videos from the 1980s to the 2000s owes much to visual music's emphasis on abstract and synchronized imagery, transforming into a platform for experimental audiovisuals. For instance, a-ha's "Take On Me" (1985) pioneered rotoscoped techniques to blend live-action with hand-drawn sketches, creating a seamless fusion of and abstract motion that synced visual transitions to the song's pop rhythms, influencing subsequent hybrid formats. This approach paralleled the rise of VJ culture in nightclubs, where real-time visualizers manipulated generative graphics and video loops to respond to DJ sets, enhancing immersive club experiences through software like VJamm and Resolume, which treat visuals as a performative extension of music. In industrial applications, visual music principles have permeated advertising and , notably through features like Spotify's , introduced in 2017, which allows artists to attach short, looping vertical videos to tracks, replacing static album art with dynamic, music-synced visuals to boost listener engagement. Additionally, tools like the software use audio-reactive algorithms to generate visuals from sound inputs for live performances and . Visual music intersects with related art forms such as , exemplified by James Turrell's installations, which manipulate to evoke auditory-like perceptions, as in his "Skyspace" series and the ongoing project (initiated 1972), where colored light fields create synesthetic experiences blending visual immersion with implied sonic resonance. Overlaps with appear in Alvin Lucier's experimental works, such as "Music on a Long Thin Wire" (1977), where physical vibrations produce evolving tones visualized through spatial acoustics, bridging auditory phenomena with perceptual mappings that echo graphical representations of sound waves. Post-2020 extensions of visual music have emerged in digital marketplaces, with NFT platforms enabling audio-visual pieces that combine generative music and synchronized animations as unique collectibles. For example, Adventure Club's 2020 NFT drop on Blockparty featured limited-edition audio-visual works, including a 1/1 "Genesis" piece, hailed as a landmark in merging electronic with blockchain-verified visuals, paving the way for artist royalties and fan ownership. Concurrently, AI-driven has fueled high-profile auctions, such as "Augmented Intelligence" sale from February 20 to March 5, 2025, where algorithms produced artworks that sold for a total of $728,784, highlighting the fusion of computational creativity with art traditions.

References

  1. https://www.centerforvisualmusic.org/Ox_Keefer_VM.htm
  2. https://hirshhorn.si.edu/explore/the-hirshhorn-presents-east-coast-premiere-of-visual-music/
  3. https://www.denverartmuseum.org/en/blog/wassily-kandinskys-symphony-colors
  4. https://arxiv.org/pdf/2508.11694
  5. https://www.farsidestudio.com/articles/what-is-visual-music
  6. https://www.tate.org.uk/research/in-focus/abstract-kinetic-collage-painting-sound/artistic-intellectual-influences
  7. https://accad.osu.edu/sites/accad.osu.edu/files/real-time-music-visualization.pdf
  8. https://www.researchgate.net/publication/10583961_Was_Kandinsky_a_Synesthete
  9. https://www.awn.com/mag/issue2.1/articles/moritz2.1.html
  10. https://ora.ox.ac.uk/objects/uuid:c24ea518-fb73-4f34-874e-f9398d678903/files/dm039k5094
  11. https://www.atlantis-press.com/article/125957529.pdf
  12. https://lukemckernan.com/2014/08/21/colour-music/
  13. https://www.sensesofcinema.com/2011/feature-articles/cinema-and-abstraction-from-bruno-corra-to-hugo-verlinde/
  14. https://www.medienkunstnetz.de/works/opus1/video/1/
  15. https://muse.jhu.edu/article/268996
  16. https://harvardfilmarchive.org/programs/seeing-sound-a-mary-ellen-bute-retrospective
  17. https://artgallery.yale.edu/press-release/lumia-thomas-wilfred-and-art-light
  18. https://www.awn.com/mag/issue2.5/2.5pages/2.5moritzwhitney.html
  19. https://academic.oup.com/book/2681/chapter/143118146
  20. https://www.centerforvisualmusic.org/News.htm
  21. https://www.theatlantic.com/technology/archive/2015/11/color-organs/414460/
  22. https://www.lindahall.org/about/news/scientist-of-the-day/louis-castel/
  23. https://rhythmiclight.com/timeline/
  24. https://americanart.si.edu/exhibitions/lumia
  25. https://pmc.ncbi.nlm.nih.gov/articles/PMC9099070/
  26. https://www.guggenheim.org/articles/checklist/synesthesia-a-visual-symphony-art-at-the-intersection-of-sight-and-sound
  27. https://www.iannis-xenakis.org/en/dictionary-upic/
  28. https://www.johncage.org/pp/John-Cage-Work-Detail.cfm?work_ID=48
  29. https://www.jstor.org/stable/10.7757/persnewmusi.49.1.0004
  30. https://openaccess.city.ac.uk/6476/1/Graphic%20Notation%20%282007%29.pdf
  31. https://escholarship.org/content/qt5h3764n4/qt5h3764n4_noSplash_b867ebe1f7f9612aac8896dacb1e4042.pdf
  32. https://uisoftware.com/metasynth/
  33. https://www.vice.com/en/article/original-creators-visionary-computer-animator-john-whitney-sr/
  34. http://preparedguitar.blogspot.com/2014/10/palais-de-mari-john-whitney.html
  35. https://oscilloscopemusic.com/
  36. https://cycling74.com/products/max
  37. https://musichackspace.org/a-guide-to-seven-powerful-programs-for-music-and-visuals/
  38. https://interactiveimmersive.io/blog/technology/resolume-vs-touchdesigner/
  39. https://musichackspace.org/product/interactive-visuals-with-touchdesigner/
  40. https://sangarshanan.com/2024/11/05/visualising-music/
  41. https://alltd.org/audio-reactive-particle-systems-in-touchdesigner-feat-yaima-its-written-in-the-wind/
  42. https://musichackspace.org/tag/gan/
  43. https://www.resolume.com/
  44. https://fitdigital.co.uk/best-ai-music-videos/
  45. https://www.fluance.com/blog/analog-vs-digital-music-impact-playback/
  46. https://www.centerforvisualmusic.org/Fischinger/OFBio.htm
  47. https://openlab.bmcc.cuny.edu/history-of-animation-mes160/week-4/
  48. https://www.researchcatalogue.net/view/457239/457240
  49. https://www.sensesofcinema.com/2007/great-directors/lye/
  50. https://www.nfb.ca/film/norman_mclaren_animated_musician/
  51. https://thecinemaofattractions.wordpress.com/2011/02/16/norman-mclarens-animated-sound/
  52. https://www.bfi.org.uk/sight-and-sound/features/how-write-film-piano-norman-mclarens-visual-music
  53. https://www.awn.com/mag/issue1.2/articles1.2/moritz1.2.html
  54. https://www.centerforvisualmusic.org/Belson/
  55. https://www.centerforvisualmusic.org/BelsonAJ.htm
  56. https://filmcolors.org/galleries/kreise-1933-34/
  57. https://www.fxguide.com/fxfeatured/the-art-of-roto-2011/
  58. https://smallworldmusic.com/artists/expanding-cinema-expo-67-reconstructions-and-other-finds-program-i-with-imax-man-belongs-to-earth/
  59. https://www.westland.net/expo67/map-docs/cinema.htm
  60. https://midi.org/animusic-midi-driven-computer-animation
  61. https://opus4.kobv.de/opus4-filmuniversitaet/files/343/StimbergMA.pdf
  62. https://codetv.dev/series/learn-with-jason/s5/build-your-own-audio-visualization-in-a-shader
  63. https://www.geisswerks.com/about_milkdrop.html
  64. https://digitalshowcase.lynchburg.edu/cgi/viewcontent.cgi?article=1170&context=utcp
  65. https://www.teamlab.art/concept/co-creative-music/
  66. https://hackaday.com/2023/03/04/a-crt-audio-visualiser-for-when-leds-just-wont-do/
  67. https://www.geeksforgeeks.org/computer-graphics/fractals-in-computer-graphics/
  68. https://www.immersence.com/osmose/
  69. http://www.medienkunstnetz.de/works/osmose/
  70. https://www.youtube.com/watch?v=g9_IvcpdGLE
  71. https://www.roadtovr.com/beat-saber-early-access-review-vr-rhythm-game-budding-jedi-knights/
  72. https://www.ableton.com/en/blog/free-tools-live-unlock-3d-spatial-audio-vr-ar/
  73. https://www.frontiersin.org/journals/virtual-reality/articles/10.3389/frvir.2025.1592652/full
  74. https://www.siggraph.org/wp-content/uploads/2022/08/SIGGRAPH-22-ACM-SIGGRAPH-2022-Immersive-Pavilion.html
  75. https://ieeexplore.ieee.org/document/11112588
  76. https://www.freethink.com/ar-vr/cybersickness
  77. https://woodsidecap.com/the-next-arena-an-indepth-analysis-of-the-immersive-live-events-market/
  78. https://www.nature.com/articles/021252a0
  79. https://pmc.ncbi.nlm.nih.gov/articles/PMC7446036/
  80. https://www.researchgate.net/publication/315973713_Was_Kandinsky_a_Synaesthete_Examining_His_Writings_and_Other_Evidence
  81. https://www.researchgate.net/publication/360467871_Coloured_hearing_colour_music_colour_organs_and_the_search_for_perceptually_meaningful_correspondences_between_colour_and_sound
  82. https://pmc.ncbi.nlm.nih.gov/articles/PMC11061078/
  83. https://doi.org/10.1162/00240940152549357
  84. https://www.exploratorium.edu/snacks/vocal-visualizer
  85. https://academo.org/demos/lissajous-curves/
  86. https://www.researchgate.net/publication/12283946_Sound_enhances_visual_perception_Cross-modal_effects_of_auditory_organization_on_vision
  87. https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2015.00103/full
  88. https://digitalcollections.lipscomb.edu/cgi/viewcontent.cgi?article=1162&context=jmtp
  89. https://www.ebsco.com/research-starters/communication-and-mass-media/nam-june-paik-exhibits-video-and-television-art
  90. https://americanart.si.edu/artist/nam-june-paik-3670
  91. https://www.theartstory.org/movement/op-art/
  92. https://www.animationmagazine.net/2022/04/take-on-me-revisiting-the-making-of-a-has-trailblazing-animated-music-video/
  93. https://www.amazon.com/VJ-DVD-Audio-Visual-Art-Culture/dp/1856694909
  94. https://artists.spotify.com/en/canvas
  95. https://synesthesia.live/
  96. https://www.researchgate.net/publication/337603750_The_eye_listens_light_music_and_visual_perceptions_by_James_Turrell
  97. https://socks-studio.com/2016/07/12/music-on-a-long-thin-wire-by-alvin-lucier-1977/
  98. https://edm.com/gear-tech/adventure-club-teases-new-nft-release/
  99. https://www.forbes.com/sites/virginieberger/2025/02/19/christies-ai-generated-art-auction-who-profits-and-who-pays-the-price/
  100. https://www.artsy.net/article/artsy-editorial-controversial-christies-ai-sale-beats-estimates
Add your contribution
Related Hubs
User Avatar
No comments yet.