Home Computer Science
Encounters of Al and popular culture: the Global North
In the early 21st century, cultural industries, such as gaming, music, film, and webtoon, are taking advantage of AL Al has recently been “trying its hand at various human creative endeavors, from cooking to art, and from poetry to board games” (The Guardian, 2016). Other than digital gaming that started its massive adoption of Al earlier than other cultural genres, cultural creators in film, broadcasting, music, and webtoon have mainly begun to use Al to create their cultural content since the mid-2010s.
Among these, the film industry is showing unique growth in the encounters of Al and popular culture, as filmmakers started to create films and film trailers through Al full scale in 2016. After representing Al in several films discussed previously, filmmakers finally used Al in production. When the film studio 20th Century Fox created a movie trailer of Morgan, a 2016 British American science fiction horror film directed by Luke Scott, it was recorded as the first ever Al to produce a film trailer. The film studio used the Al supercomputer system called IBM Watson to make the trailer. As Heathman (2018) outlined in Wired, in order to create the movie trailer, IBM researchers fed Watson more than 100 horror film trailers cut into separate moments and scenes. It performed several visual, sound, and composition analyses on each scene to get an idea of how to create the dynamics of a trailer. Then Watson processed 90 minutes of Morgan to find the right moments to include in the trailer. Once it finished processing Morgan, it isolated ten scenes—a total of six minutes of video. A human editor was still needed to patch the scenes together to tell a coherent story; however, the Al shortened the process down to 24 hours, compared to taking around 10-30 days to complete a trailer.
The film studio 20th Century Fox also used Al to predict people’s patterns, as specific patterns help predict the future. In other words, the studio used Al to predict what films people will want to see. According to its own paper published in 2018 (Hsieh et al., 2018), several researchers from the company analyzed the content of movie trailers using machine learning. Hsieh et al. claim that
temporal sequencing of objects in a movie trailer (e.g., a long shot of an object vs intermittent short shots) can convey information about the type of movie, plot of the movie, role of the main characters, and the filmmaker’s cinematographic choices. When combined with historical customer data, sequencing analysis can be used to improve predictions of customer behavior. E.g., a customer buys tickets to a new movie and maybe the customer has seen movies in the past that contained similar sequences. To explore object sequencing in movie trailers, we propose a video convolutional network to capture actions and scenes that are predictive of customers’ preferences. The model learns the specific nature of sequences for different types of objects (e.g., cars vs faces), and the role of sequences in predicting customer future behavior.
More specifically, “machine vision systems examine trailer footage frame by frame, labeling objects and events, and then compare this to data generated for other trailers. The idea is that movies with similar sets of labels will attract similar sets of people” (Vincent, 2018). For example, Logan (2017) is a superhero movie, but it has darker themes and a plot that attracts a slightly different audience. The film studio employed Al to potentially capture those differences. To create their experimental movie attendance prediction and recommendation system, called Merlin, the film studio partnered with Google to use the company’s servers and open source Al framework TensorFlow (Vincent, 2018). “Machine learning is, at heart, the art of finding patterns in data. That is why businesses love it. Patterns help predict the future, and predicting the future is a great way to make money” (Vincent, 2018). Al’s role in the movies Morgan (2016) and Logan (2017) and numerous other creative endeavors proves “how far Al has come. Using techniques such as deep learning has enabled tremendous progress, but Al remains relegated to an assistant role—for now” (IBM, n.d.). Such automation has become common in the production of culture.
Another exemplary case in the film sector created in 2016 is the script and movie (Sunspring) that was the product of director Oscar Sharp and New York University Al researcher Ross Goodwin. A so-called recurrent neural network, named Benjamin, was fed the scripts of dozens of science fiction movies including such classics as Highlander Endgame, Ghostbusters, Interstellar, and The Fifth Element. From there it was asked to create a screenplay, including actor directions, using a set of prompts required by the Sci-Fi-London film festival’s 48-hour challenge. The resulting screenplay and pop song were then given to the cast to interpret and make into a film. “The actors were randomly assigned to the parts and set to it. The result is a
Al and cultural production 61 weirdly entertaining, strangely moving dark sci-fi story of love and despair” (The Guardian, 2016). Although it is not perfect, these attempts to create movie trailers, scripts, and movies themselves are certainly proofs of the use of Al in the cultural sector.
Music composition programs were also among the first cases of this development (Turner, 2019). Music has been one of the powerful arts that is “as core to the human experience as communicating”; however, in recent years, Al “has increasingly been making headway into some of the more creative pursuits” in music (Walch and World, 2019). In the music industry, there are a range of use cases for Al, including “creating backing tracks for video, helping an artist come up with melodies or lyrics, and automatically creating mood music” (Dredge, 2019).
More specifically, in 1997, a computer program called Mubert in California had written Mozart’s 42nd Symphony, and Mubert “continuously produces a unique music stream created by an algorithm based on the laws of musical theory, mathematics and creative experience” (GVA Capital, 2017). In the popular music sector, as of September 2018, several corporations were developing software that might help people write better songs, even hit songs (Donoughue, 2018). As Marr (2018) clearly points out, musicgenerating algorithms are now inspiring new songs: “Given enough input— millions of conversations, newspaper headlines and speeches—insights are gleaned that can help create a theme for lyrics.” In other words, there are machines such as “Watson BEAT that can come up with different musical elements to inspire composers. Al helps musicians understand what their audiences want and to help determine more accurately what songs might ultimately be hits” (Marr, 2018).
There is indeed an emerging body of activity around popular music production with AL
They do this by feeding heaps and heaps of data into a computer program to teach it about music, to the point where, eventually, it can whack out a tune on its own. A team at Google created Al Duet, software that will jam with you. Scientists at Sony’s CSL Research Lab went one step better, producing a whole song—in the vein of The Beatles—using Al, though with a little help from their friends (a human composer).
In Australia, one particular corporation named Popgun uses Al to develop software that it hopes will make music composition easy. “It uses a form of deep learning called unsupervised learning, where the Al learns the various features of songs—how a scale or a harmony works, for example—just by studying enough of them.” As is well discussed, people consider music to be a very human activity. Unlike in Go—the board game where expert human players were defeated by Al and—“a high-water mark in the world of machine learning—in music, there is no clear winner or loser. There are some notes that sound bad in a progression, but lots that sound fine” (Donoughue, 2018).
The rise of music created by Al software has furthermore continued. Sony’s Al lab has created software called FlowComposer that is capable of producing a song whose style matches whatever songs people have fed into it. In this case,
researchers fed it Beatles songs—and indeed, the result sounds a little Beatles-ish. Note that FlowComposer produced only a lead sheet (that is, a piece of music showing the melody and chord symbols); a human then arranged it for instruments, wrote lyrics, recorded it, and mixed it. There are some fresh and intriguing chord sequences in this song, but also some noodling, aimless ones that don’t really land.
As Ken Lythgoe, head of business development at creative Al technology company MXX, which was founded in 2015 by Al specialist and composer Joe Lyske and Philip Walsh (Davis, 2019) says, “In a world of personalization and on-demand services, music is one of very few remaining static artefacts.” MXX has created what it says is the world’s first Al tech that allows individual users to instantly edit music to fit their own video footage, complete with rises and fades. Lythgoe says,
There are two types of Al—the Al that is here to replace us and Al that is here to empower us—we are definitely in the empowerment camp. We are not about computers replacing musicians or editors; we are firm believers in the creative process.
What they want to develop is a new world where “music can be adapted to perfectly fit certain experiences—such as gym workouts and runs, gaming, user-generated content and virtual or augmented-reality experiences” (Davis, 2019). The convergence of Al and music like in other cultural spheres has been a growth area, and it is certain that the convergence of these two formerly separated areas is likely to last.
Meanwhile, the broadcasting sector has not been left behind. Although network broadcasters are not at the front line in Al-supported cultural production, several global broadcasters began to work to advance Al-supported programs. Al has become more mainstream across the entire entertainment ecosystem. From the front stage (e.g., recommendation engines) through the creative process, scripting, shooting, post-production, and all the way to the backstage (e.g., meta-tagging and distribution), Al is enabling industry newcomers to leverage new business opportunities (Natajaran and Baue, 2019). For example, the BBC project, “Talking with Machines,” is an audio drama that allows listeners to join in and have a two-way conversation via their smart speaker. Listeners get to be a part of the story as it prompts them to answer questions and insert their own lines into the story.
Of course, as mainly discussed in Chapter 5, in the broadcasting sector, Netflix fully uses big data analytics to predict what its customers will enjoy watching. Netflix is increasingly a content creator, not just a distributor, and uses data to drive what content it will invest in creating. Due to the confidence it has in the data findings, Netflix is willing to buck convention and commission multiple seasons of new television shows rather than just pilot episodes (Marr, 2018).
As such, the convergence of Al and popular culture has been widely actualized in the cultural industries, more than we expect. Several cultural forms, including film, music, broadcasting, and digital gaming in several Western countries, are beneficiaries of the interaction between Al and popular culture. From film companies like 20th Century Fox to digital platforms, media giants in these Western countries have vehemently invested in AI-related technologies, either developing these technologies or acquiring firms who already owned these technologies and produced cultural content. Al-supported cultural production is not yet a major force at the end of the 2010s and the early 2020s; however, cultural firms and digital platforms have agreed that the adaptation of Al for the creation of popular culture will be the future of these firms, and therefore, they aggressively attempt to produce cultural content based on further reliance on AL