Machine imagines music: HE composes new Wagner music, premiere in Dresden

On the evening of November 20, 2022, the Singakademie Dresden will hold a concert in the Lukaskirche, the program of which will come from an unidentified composer along with Wagner, Schubert, Draeseke and Zemlinsky. Anonymus is not an unknown 19th-century master, but a contemporary machine whose artificial neural network has been trained on music from the Romantic period and which is fine-tuned to the works of Richard Wagner.

Michael Käppler studied church music and choir conducting in Dresden. He directs Singakademie since 2021. In principle, artistic direction means being a conductor, i.e. directing rehearsals and concerts, as well as being responsible for the overall artistic concept. In theaters or concert halls, it is usually the case that an artistic director is responsible together with a chief conductor for the overall artistic concept and the conductor for the execution. Käppler can plan and then “do” himself, which is sometimes very practical. In his spare time he programs, for a long time contributing to the development of the music notation tool “LilyPond” (mainly Scheme, more precisely GNU Guile and Python). It also automated downloading and transcribing audio files using a crude mix of Docker containers, shell scripts and Python.

Michael Käppler, head of the Singakademie Dresden and conductor, has been running a music project for about a year, for which he was able to win the Technical University of Darmstadt and the Heidelberg AI startup Aleph Alpha as partners. His Meistersinger reloaded project provides a technical and intellectual foundation in addition to sensual and cultural stimulation. The afternoon before the world premiere, a scientific panel discussion from the fields of music, technology, machine learning and mathematics will examine the question of whether machines can compose: three men and a male expert discuss the choir director.

The origin of the project was the question of how such a problematic composer as Wagner can be interpreted today. Conductor Käppler wanted to “break Wagner’s music in a different way – with an AI commentary on Wagner,” he explained. heise developer. To do this, he turned to various universities and experts and found an enthusiastic comrade-in-arms in Kristian Kersting. Kersting is Professor of Machine Learning and Artificial Intelligence at TU Darmstadt, where he heads the Hessian AI Research Center.

He connected the musician with his TU colleague Prof. Iryna Gurevych, students, Hessian. AI and for research and engineering with Heidelberg’s startup research department Aleph Alphawho then worked on the project as a team. For Aleph Alpha, among others, Jonas Andrulis and AI researcher Koen Oostermijer were involved in building the AI ​​- much of the work done on the Heidelberg AI system. Kersting’s PhD student, Wolfgang Stammer, created a new transformer at the same time.

Aleph Alpha is best known for its multilingual, multimodal core AI model, Luminous, which understands text and images in context. For the Heidelberg AI company’s research team, the goal of the project was to change the modeling of data and relationships in the artificial neural network in such a way that AI, which is otherwise oriented towards conversation or finding information, instead. creates harmony and creates music that also follows a certain style.

Since the structure of music works differently from that of language, adaptations were necessary: ​​among other things, the research team had to expand the way in which words or notes are positioned in relation to each other in such a way that durations and time intervals can be translated mathematically in the language of AI. The work of an entire year and the contributions of many comrades have gone into the composition of approximately three minutes.

Jonas Andrulis, Samuel Weinbach, Koen Oostermijer von Aleph Alpha

Jonas Andrulis, Samuel Weinbach, Koen Oostermijer von Aleph Alpha

Representing the Aleph Alpha team, some contributors (from left): Jonas Andrulis (CEO and founder), Samuel Weinbach (co-founder and senior researcher), Koen Oostermijer (researcher)

According to Michael Käppler, they fed a Transformer model with music from the Romantic era, especially piano music. Training was performed on different datasets. Based on YouTube audio and a Google Magenta transcription model, Käppler himself trained a dataset of romantic piano music and called it the Great Romantic Piano Dataset (GRPDS). It contains around 12,000 MIDI (Musical Instrument Digital Interface) tracks with a total length of 1900 hours. During the conversation, the leader added that this data set can be greatly expanded. In addition, he and his partners have also used GiantMIDI-Piano, a dataset available on GitHub from ByteDance – the Chinese internet technology company behind TikTok, among others. The data set provided the signal the team used to continue the work.

In these larger data sets, the model not only learned harmonic rules, but also the intricacies of composition and construction of a masterpiece. Once AI had understood the rules of the music as a whole, the project team brought it closer to the specific style and character of Wagner’s works – to this end, the research team performed fine-tuning of Wagner’s works together with the conductor Käppler. . If the model had trained exclusively with Wagner, he would have learned it by heart and would not have been able to create anything new beyond the familiar pieces (the technical term for this is redundant).

“It’s important to find the balance between a model that parrots and one that only finds specific Wagner music,” Käppler said, explaining the procedure. As an audio example, he showed the Developer’s editorial team one of the many AI compositions that were created in the project and were not processed for the orchestral piece: the first 15 seconds are originally from Hans von Bülow’s Meistersinger Paraphrase, the rest . it’s car work (link leads to audio sample).

Transparency is important to him, that is, revealing which parts of the work are human-made and which come from the artificial neural network. In the panel planned before the concert, alongside Käppler and Kersting, empirical musicologist Miriam Akkermann (junior professor at TU Dresden) will talk, among other things, about post-processing. According to Käppler, the project allowed the AI ​​system to compose piano music while the musicians did the orchestration themselves—ie. transposing and changing registers for different instruments: “A good number of people have come in here and it’s important for me to reveal it and with it transparent,” says Käppler (you can listen to an audio teaser of “Meistersinger” here reloaded” – the audio prompt ends after the beep and AI continues the Wagner composition).

A related project would be Beethoven X – not the cryptocurrency portal of the same name, but an elaborate project to complete Beethoven’s tenth symphony using AI by teams of experts from Harvard, Cambridge and Rutgers Universities, sponsored by Deutsche Telekom, among others. The Dresden collaboration for the replenished Meistersinger managed with far fewer human and financial resources. Two to four people mostly worked on the project, and the long-term goal is not only to present a result, but also to reveal the workflow.

The automaton of the composition raises questions: Some may think of ETA Hoffmann’s terrifying Sandman – to which Sigmund Freud attached his definition of the uncanny, as it fascinates us as humans, but also begins to frighten us when mechanical objects have a soul or living beings also appear. similar to the slot machine. Others will dismiss it and question the authenticity of the car’s art.

It should leave no one completely unscathed when a new form of automation is now increasingly penetrating areas previously considered the domain of humans and dealing with feelings, such as expressive composition and art creation. What do composing machines mean for our creativity?

“The Machine Imagines Music. German Romantic Period, Oil on Canvas. Richard Wagner’s Genius Viewing the Stage.” Text request by Silke Hahn, created with Stable Diffusion

(Image: via Nightcafé)

A panel discussion on 20 November at 15:00 will present the collaborative project between Singakademie Dresden, TU Darmstadt and Aleph Alpha and discuss the boundaries between machine learning and human creativity. The following quartet will discuss whether humans can be replaced as artists – or whether new skills will create a new partnership between man and machine: alongside Michael Käppler and Kristian Kersting, the mathematician Christfried Brödel, who is not only the head of the New Bach Society in Leipzig, but also has a background in contemporary music, as well as Darmstadt AI master’s student and classical percussionist Matthias Lang.

As moderator, empirical musicologist Miriam Akkermann (junior professor in Dresden) brings together the topics of conversation. Aleph Alpha will be replaced by Koen Oostermijer. The panel will take place in the parish hall at Einsteinstraße 2 in Dresden, opposite the Lukaskirche. Entry is free and participation is independent of going to the concert – the round of talks will be recorded and published afterwards.

“AI systems can not only help find new drugs, they can do much more. They can be a muse for us and help to compose songs, write poems or paint pictures .It can make us more creative – and I see no reason why machines can’t be creative themselves at some point.

In any case, the topic of artificial intelligence and art brings us researchers into contact with the public. And that’s just wonderful. Thanks to Singakademija for the cooperation!”

– Kristian Kersting to appreciate the Developer on the motivation of the project

From 17:00 the evening ends with a concert in Dresden’s Lukaskirche, in which, alongside the works of Franz Schubert, Richard Wagner, Felix Draeseke and Alexander Zemlinsky, the Meistersinger reloaded by “Anonymus” will also premiere. The program can be viewed on the Singakademie website. As far as we know today, the concert will not be broadcast or recorded. If you are interested and can arrange to come to Dresden, you can buy tickets for the concert in advance. The exact program of the musical evening “Art Faith and Faith Art” as well as details on the panel discussion can be found on the website of the Singing Academy Dresden. Tickets for the concert are available from 7 to 20 euros (plus advance booking fee, otherwise at the box office).


On the main page

Leave a Comment

Your email address will not be published. Required fields are marked *