World Music's DIVERSITY and Data Visualisation's EXPRESSIVE POWER collide. A galaxy of INTERACTIVE, SCORE-DRIVEN instrument model and theory tool animations is born.

Sunday, September 4, 2016

Cantillate

World Music Visualisations: Musical Storytelling

The means will shortly be at hand to drive graphical animations -whatever their nature- directly from a musical score. This has potentially profound impact for storytelling across a swathe of applications, be they musical, artistic, 2D, 3D, on-screen or open-air. It will allow researchers worldwide to integrate and animate theory models, bringing them to a world-wide audience. The only limits are those of own imagination.

Big, brave, open-source, non-profit, community-provisioned, cross-cultural and off the reservation crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or simply register as a potential crowdfunder..

Musical Storytelling


Through song, saga, ritual dance, or celebration of the great cycles of nature, music has always been intimately associated with storytelling. Music is one of the vehicles by which cultural continuity is assured.

Cultural diversity implies distribution of power. Cultural continuity implies a sense of identity, and so wellbeing. Cultural understanding implies experience and openness towards others. Cultural mobility implies acceptance.

No stories, no culture? No musical diversity, no democracy? No understanding, no tolerance? Neither museums nor archives, but people, their sense of identity and their values keep culture alive.

We devalue cultural identity at our peril. Culture seems, even in more retrenched forms, an abstraction central to wellbeing. Nevertheless, to understand someone else's cultural values is not to sacrifice your own.

Understanding -and especially in context- is the foundation of respect and tolerance, and ultimately of mutual advancement.
Culture is catalyst, portal and key. It awakens curiosity, motivates, fosters identification and enriches. Moreover, there is no freedom greater than acceptance where none was asked or expected.

Cultural Continuity

Potential Crowdfunder?

Pre-Crowdfunding Registration (* indicates required)
Email Format

We live in challenging times. No matter where we look, we encounter fault lines. Discontinuities. Conflict.

Cultural memory and values, last remnants of a refugee's life before speculation-fuelled wars, come under attack: scapegoats for the real aggressors - armaments industry, hedge funds, banks and politicians locked in last-gasp, pre-singularity economic overreach. Raptors, psychopaths, hoarders, narcissists and tax evaders united in one goal: to beat each other to the other ninety-nine percent's last, if symbolic, tooth filling.

Cultural diversity implies distribution of power. Cultural continuity implies a sense of identity, and so wellbeing. Cultural understanding implies experience and openness towards others. Cultural mobility implies acceptance.

So is it worthwhile keeping culture alive?

Narrative Threads

Culture is a narrative. Where our forebears were masters in sustaining it's continuity, we seem to be losing it.

Too many threads, too many interrupts, too many demands. If honest with ourselves, we are losing even the willingness and ability to relax in other's company - central to sustaining the narrative in the first place.

Music, dance and community are fertile ground. If we can adjust our ways, narrative and continuity stand a chance of taking root once more.

Storytelling Tool

Part of the challenge at present are the limitations of the tools at our disposal. It's all very well talking community if these only compound our troubled, hectic lifestyles. Even a 200-page book on music theory?

With this framework, however, we are equipped to drive our musical narrative -whatever their storytelling payload- directly from the musical score.

Elegy. Dance. Theatre. Animation. Parody. Edutainment. Theory. Instruments. Esoterics. History. Anything that can find accompaniment in music can be brought to life on-screen. The only limit is imagination.

Narrative Threads or Arcs

There are, however, many, many ways in which musical score playback can be used to underpin and highten narrative tension.

Story Threads or Arcs

Multiple threads are a widespread device. The aggregator platform focussed on in this site is no exception.

In bringing together the many different aspects of a musical narrative, we foster immersion, immediacy and discovery.

With SVG at our disposal, we have the means to take these ideas much further.

Music Animation Examples

Interference Music Visualizer

In another context, I've already mentioned Richard Merrick's Interference Music Visualizer, which -in content- bears comparison with ideas being developed in the music aggregator platform at the centre of focus here.


Early music instrument and theory visualisation (or visualization) using Adobe Flash
Interference Theory's Visualisation Dashboard

This is musical storytelling in the sense that the development of a musical theme or (here) phrase is brought to pictorial life in time.
Appealing use is made of saturated colour, pitch, position on a timeline, instrument model, and a variety of theory tools in support of his underlying interference theory.
It's not clear if an interactive version was ever made available, but the demo in itself remains a little mesmerising.

Notation Mimickry

Another approach effectively duplicates the notation, adding form and colour to hold the viewer's attention. Again, some of these (there are many) can be quite beautiful. In the absence of any additional storyline or data, interest can be quickly lost.

Musical Engines

Musical engines exploit our child-like curiosity for the intricate, if the signal to our brains is confused. Is the engine producing the sound, the sound driving the engine, or both externally held in synch?


Narrative Support


Sergei Prokofiev's "Peter and the Wolf" is famous for bringing greater depth to story characters by associating each with a specific instrument and musical theme.

This supports the narrator (storyteller) by providing additional, musical clues as to the temperament and movement of the creature concerned. Earlier film used static pictures, later ones animation synchronised with the music. Nevertheless, all the work setting this up was strictly manual. 

Learning Reinforcement

Much as in earlier playground counting rhymes, song and music can be used to reinforce memorisation, as in this short, surgical "Brain Parts Song" by Aaron Wolf.


Physics

As a child, I was fascinated by the standing waves produced in water or using filings on a metal plate, musing if every piece of music might not have it's own, unique signature. Nowadays, designers experiment with the detailed audio response of entire rooms.


You may feel that the only story these patterns tell is their own. I feel they hint at much more. Here, for example, Lissajous standing waves are taken to an entirely new level: drawing with sound.



Summary

For all their flexibility, SVG animations driven from musical sources, whatever these may be, remain few. Decoding a time-based source such as MusicXML or audio, and manipulating complex SVG animations can be time-consuming to code, and bear an initially steep learning curve.

Many early adopters of libraries such as D3.js have been snapped up by industry. Nevertheless, the quality and breadth of data visualisation work continues to grow.

The main challenge to notation-driven animations remains, however, the complexity of music exchange formats such as MusicXML. Midi represents a simpler access mechanism, but one that sacrifices much of the musical expressiveness of MusicXML.

Beyond this, coding highly interactive yet robust music animations also has it's challenges, but where these find widespread use by a large community, this is time well invested. With the range of entry points (notation, audio etc) to controlling SVG on the rise, opportunity calls.

Cantillate

About Cantillate -

Autodidact. Laird o' the Windy Wa's. Serial Failure with Attitude. Bit of a Dreamer..

Comments, questions and (especially) critique welcome.