A.I., Algorithms, Art, and Allegro – The Story of Interactive Media

Janae Jean
Feb 22, 2021
11 min read

The Future Is Now.

Robot — Music, Art, and Technology are intrinsically linked.

When you think of the relationship between the arts and technology what do you imagine? Do you think of a keyboardist playing a musical synthesizer, an artist drawing on a digital tablets with a stylus, a playwright typing out his latest scenes on a laptop in a café, or a singer recording their voice with their smartphone? All of these are ways that tech meets the arts. But, the relationship between them is even closer than that. The artistic expression and technological accomplishment go hand-in-hand; they are both ways that people allow their imaginations to interact with the world around them.

In previous articles, such as What Is MIDI?, MIDI 2.0, the Sample This series, the What Is Music Production? series, the Harry Partch Instruments, The Art of the VJ, and Electronics Has Changed Everything?, we explored some of the ways the music and technology develop together. In fact, without technological developments we wouldn't have the arts. Without the invention of papyrus, we wouldn't have modern sketchbooks, notepads, or even laptops. Without the invention of amphitheaters in ancient Greece, we wouldn't have modern amplification. Without the first bone flutes from about 40,000 years before present, we wouldn't have musical instruments. The list goes on and on. Without people applying their imaginations, we don't have art nor the tools we use to create it.

Throughout the 20th century until now in the 21st century, innovation with electronic technological and later digital technology has revolutionized how people create, consume, interact with, enjoy, and discuss the arts. Right now, you are reading about the arts on a digital computer or smartphone, you can also use these tools to create your own art and share it with the world. You can also interact with other people's creative works via these tools. This includes through consuming what's called "New Media."

New Media refers to any media from websites, newspapers, magazines, blogs, music, video, and podcasts that is accessed via interactive electronic/digital means (generally the internet). It also includes interactive media that is accessed electronically, such as social media, video games, computer animations, human-computer interfaces, interactive computer installations, and virtual worlds. This media is in contrast to Old Media, which includes media that is not interactive, such as print newspapers, print magazine, traditional film, broadcast radio, and broadcast TV. This potential of New Media is expanding exponentially right before our eyes. We may not know what yet unimagined horizons lay ahead, but if we can trace through its development we (as creators) may be able to predict—and create—our future interactive world. In this post, we are going to learn how advances in animation, computer graphics, and computer music, have lead to interactive new media, in this case, computer games.

The Story of New Media Begins with the CRT – 1890s

CRT monitor with keyboards, Photo: Senad Palic, Unsplash, https://www.retro.wtf/

In the 1890s, the Cathode-Ray Tube (CRT) was developed. It is a vacuum tube containing one or more electron guns whose beams are manipulated to display images on a phosphorescent screen. The Cathode-Ray Tube was precursor to TV, oscilloscope, radar control panel. Imagine a bulky 1990s computer monitor or a 1950s television set.

The First Computer Graphic Artist

American mathematician, artist and draftsman, Benjamin Laposky (Sept. 14, 1914 – 2000) is credited with making the first computer graphics by using oscilloscopes to create abstract art. An oscilloscope is a type of electronic test instrument that graphically displays varying signal voltages on a CRT. In 1952, Laposky released a series called Oscillons with a companion thesis, Electronic Abstractions at a gallery exhibition of fifty pictures of the same name at Sanford Museum in Cherokee, IA. The images above from Ruins of Books are scans from a 1961 magazine.

Radiosity Algorithm – 1930 through 1940s

A modern version of the Utah teapot model — 21st Century Computer Art using the Radiosity Algorithm

In the 1940s, Massachusetts Institute of Technology professors (MIT), Parry Moon (1898–1988) and Domina Eberle Spencer (b. September 26, 1920) used their knowledge in their field of applied mathematics to calculate highly accurate global lighting models, a mathematical formula known as the "radiosity algorithm." These calculations were based on earlier work first published by Henry Harold "H.H" Higbie (1882 – 1947) in his 1934 book, Lighting Calculations. These

This ability to approximate the look of real light on objects that the radiosity algorithm provides to a computer generated image is essential to create realistic looking virtual worlds. When we look at an object in the real world, our eyes interpret that object as having a 3-D form due to the way the light creates highlights and shadows on that object. The example above is a modern rendering of the "Utah Teapot," which is a common 3-D test model and standard reference object in the world of computer graphics. The image looks 3-D though it is 2-D because the diffuse illumination was entirely determined by the radiosity algorithm. At It was first created in 1974 by computer graphics researcher and doctoral candidate Martin Newell at the University of Utah.

Visual Music – 1920s to 1970s

Oskar Wilhelm Fischinger (June 22, 1900 – Jan. 31, 1967) was a German-born American animator, filmmaker, and painter. He originally trained to be a musician but later moved into the new art of film. He created abstract musical animations which we referred to as "visual music." He His work is considered to be the forerunner to later computer graphics and music videos. Fischinger created a large artistic output including over 500 short films and 800 paintings. He is considered the first video artist.

In 1929, Fischinger created special effects for Fritz Lang's (Dec. 5, 1890 – Aug. 2, 1976) Woman in the Moon. one of the first science fiction films. Fischinger's visual music influenced Disney's 1940 hand-drawn animated full-length feature Fantasia. In fact, he designed the sequence set to Johann Sebastian Bach's (March 1685 – July 28, 1750) Toccata and Fugue in D minor, BWV 565 for the film but quit because his designs were simplified and altered to be more representational. Fischinger's only major Hollywood production to be released was An Optical Poem (1938) which features hundreds of paper cutouts that are manipulated and shot as it appear as if they are dancing to Franz Liszt's ( Oct. 22, 1811 – July 31, 1886) Second Hungarian Rhapsody.

MUSIC is not limited to the world of sound. There exists a music of the visual world.

– Oskar Fischinger

Fischinger innovated ways to create animations, better quality films, and bring art to life. He was involved with the development of the three-strip GasparColor film process, the European version to Hollywood's Technicolor process. He created a series of His Motion Painting No. 1 (1947) was created by applying oil paint on acrylic. In this animated short, the images are visual interpretations of J. S. Bach's Brandenburg Concerto No. 3, BWV 1048. This film won the Grand Prix for Experimental Film at the Brussels International Experimental Film Competition.

Fischinger also invented a "visual music" instrument called the lumigraph that created images on a screen and took two players to operate. One of the players manipulated the screen to create imagery and the other changed the colors of the lights on cue. The instrument did not create sound but was meant to accompany auditory music. It may be considered an ancestor to music visualizers and the art of VJing .

Others inspired by early video (film) artists like Fischinger continued to develop the artwork. A pair of brothers, John Whitney, Sr. (1917-1995) and James Whitney (1921 – 1982), were so impressed by Fischinger's films that they set out to create their own. However, they did not care for his choice to use pre-existing classical music pieces and chose to create their own new sounds to accompany their animations.

In their Pasadena, CA apartment, John constructed an animation stand and other equipment. James drew geometric shapes on small index cards and cut out positive and negative stencils that could be painted or air-brushed onto the cards. John was a trained composed and Inspired by the 12-tone music theory of Austrian-born American composer Arnold Schönberg (Sept. 13, 1874 – July 13, 1951), they used the shapes by inverting them, using them as clusters, using their in backwards order, and in other ways that a composer may manipulate the notes in a piece of 12-tone music. John invented a mechanism to create sound and James continued to make visual Variations, through hundreds of hours of hand animation. This exploration and invention culminated in avant-garde works such as 1944's Five Film Exercises. Five Film Exercises won the prize for best sound at the 1949 Brussels Experimental Film Competition in Belgium. The above video includes Experiments 2 and 3.

The brothers continued to work in the field of animation. John is known for creating the animated title sequence for Alfred Hitchcock's (Aug. 13, 1899 – April 29, 1980) 1958 film Vertigo with noted graphic designer Saul Bass. He also developed early computer-based animation technology on analogy computer and later was one of the first to develop art with digital computers. He was IBM's first artist-in-residence in 1966 and taught the first computer graphics course at UCLA in 1982.

John's work above, Matrix III from 1972, is a work of generative graphics. Generative graphics is a form of art wholly or in part was made with the use of a non-human autonomous system. This means that a computer is programmed to make for itself some of the choices that are normally made by the human artist. The music was composed for the film by American minimalist composer Terry Riley (b. June 24, 1935).

James continued to create animation as well, often combining art, technology, Jungian psychology, and Eastern philosophy into his art. His notable works include 1957's Yantra and 1966's Lapis. Both films were created by using dot pattern on cards. Lapis (above) took two years to create with the aid of John's development with analog computer animation techniques. James chose traditional Indian sitar music to accompany the film.

Innovations at Bell Labs – 1960s

While John Whitney was developing a new way to create visual art and film with computers, others were focusing on creating music using computers. Known as "The Father of Computer Music," Max Mathews (Nov. 13, 1926 – April 21, 2011) was the pioneer of the field. He is best known for creating the computer generated version of the song Daisy Bell for Stanley Kubrick's (July 26, 1928 – March 7, 1999) film 2001: A Space Odyssey. The scene features the computer, HAL 9000, singing via Mathew's remarkable speech software, as his cognitive function is disabled. His piece from 1960, Numerology, in the video above, demonstrates the ability of 1960s computers to create musical sounds.

Max Mathews playing one of the electronic violins he built, in his analog electronics lab at Bell Telephone Labs (c.1970) — Mathews playing an electronic violin he built, in his lab at Bell Telephone Labs (c.1970)

The COMPUTER may be potentially as valuable a tool to the ARTS as it has already proven itself to be in the sciences.

–A. Michael Noll

During the 1960s, the researchers Bell Labs, where Mathew worked, developed computer video technology as well. This early example of computer graphics below was developed there by American engineer and professor A. Michael Noll (b. 1939). Noll, Mathews, and others at Bell Labs believed from the beginning in the ability of computers to enhance artistic expression.

Early Computer Gaming – 1940s to 1960s

Enigma Encryption Machine, Photo Mauro Sbicego, Unsplash

While some creators were inventing technology for art, film, and music applications, it is only natural that others would develop this technology for amusement. Just as the piano could be a tool for "high art" composers and popular songwriters alike and be a source of home entertainment, computer technology was both a tool for artists and a source of entertainment.

The Cathode-Ray Tube Amusement Device – 1947

The Cathode-Ray Tube (CRT) mentioned above lead to many applications including the art of Ben Laposky, military radar screens and television sets. In 1947, inventor and physicist Thomas T. Goldsmith Jr. (Jan.9, 1910 – March 5, 2009) and Estle Ray Mann used the technology to construct the first computer game, the Cathode Ray Tube Amusement Device, drew from World War II radar technology.

TuroChamp – 1948

British mathematician and computer scientist, Alan Mathison Turing (June 23, 1912 – June 7, 1954) is popularly known for his contributions to cracking the Nazi code during World War II with the Enigma machine (see above photo). This story is wonderfully told in the 2014 film, The Imitation Game. Turning is known as the "Father of Computer Science" and "The Father of Artificial Intelligence." He is also known for the "Turing Test," originally known as "The Imitation Game," a set of questions which are meant to determine if a computer or artificial intelligence is capable of "thinking" in the way a human does.

In 1948, Turing along with British economist and mathematician David Champernowne (July 9, 1912 – Aug. 19, 2000) developed TuroChamp, a computer chess game, that is the first computer game. In the game, a human player would play at a low-level of play against a computer player.

Sometimes, it is the people no one can imagine anything of who do the things no one can IMAGINE.

– Alan Turing

OXO – 1952

Electronic Delay Storage Automatic Calculator in 1948

In 1952, British computer scientist Alexander Shafto "Sandy" Douglas (May 21, 1921 – April 29, 2010) at the University of Cambridge, UK developed a tic-tac-toe game (also known as "noughts and crosses") for computer. The game consisted of a human player playing against a computer opponent. In the original game, the human player entered their choices through a rotary phone dial. The game was only playable on the university computer, the Electronic Delay Storage Automatic Calculator (EDSAC), and was meant, not for amusement, rather academic research into artificial intelligence.

Early Commercial Computer Gaming – 1970s to 1980s

Computer Space – 1971

Photo of Spacewar! by Joi Ito, wikipedia

As computer technology advanced and went from bulky analog to digital designs, the commercial gaming industry emerged. In 1971, the first commercially available computer game was Computer Space, which had grown from an earlier game developed by three students at MIT called SpaceWar! The game featured simple graphics and as accompanied by electronically generated sounds of bleeps and noise.

Pong – 1972

In 1972, not long after Computer Space, the first commercially successful video game, Pong, was released. Pong is a tennis-inspired arcade cabinet game in which players move rectangles to hit a computer generated ball back and forth like a tennis match. The players may be two human players or a human player with a computer opponent. It was designed by American engineer and computer scientist Allan Alcorn (b. January 1, 1948) for Atari. The graphics and sound were still simple. The game was so successful that a home version was released through Sears stores in 1975.

Atari 2600 – 1977

One of the earliest home video gaming consoles, the Atari 2600, was first released on Sept. 11, 1977. While arcade cabinet games had built-in displays, this home console used the screen already present in most U.S. households, the television. It was the first to popularize the use of microprocessors. The Atari 2600 used memory cartridges, ROM (read-only memory) modules, that stored the game instructions to be executed by its microprocessor. Still, the graphics remained simple, there was color. The sounds, while still bleeps, were now musical ones. It remained a popular home gaming choice into the early 1980s. Popular games for the Atari 2600 included Pac Mac, Space Invaders, and a game based on the popular 1980 movie, E.T., the Extra-Terrestrial.

Mattel Intellivision – 1979

The toy company Mattel began developing its own home gaming system as soon the Atari 2600 was released. The resulting gaming console, the Mattel Intellivision was released in 1979. The device featured richer color graphics and a sound generator that could produce three-part harmony. It also included a keyboard component so that it could be used as an early home computer. Popular games for the Intellivision include Star Wars, a game based on the film, Auto Racing, and Major League Baseball.

Konami's Frogger – 1981

Frogger disk by Sierra On-Line for IBM PC

After the revelation of home gaming, the next step was incorporating adaptive music, background music whose musical elements, including volume, rhythm, harmony, or melody, change in response to specific events happening in game. The first video game to feature this advancement was Frogger, originally developed by the Japanese gaming company Konami in 1981. With adaptive music, the environment of the virtual game world is directly reacting to the actions of the player, adding an element of reality. It has remained one of the most popular computer games of all times and is playable on many gaming systems.

In this post, we explored some of the ways that computer technology developed during the 20th and 21st centuries has been used to create visual art, film, music, and games, to create new horizons for which artists may aspire. We focused on how advances in animation, computer graphics, and computer music, lead to interactive media—computer games. In future posts, we will look more in-depth into how New Media, especially, social media has changed the world for the arts.

PERENNIAL MUSIC AND ARTS