World Music's DIVERSITY and Data Visualisation's EXPRESSIVE POWER collide. A galaxy of INTERACTIVE, SCORE-DRIVEN instrument model and theory tool animations is born.

Wednesday, August 10, 2016

Cantillate

What IS Music Visualisation?

Despite it's potential for driving global understanding, exchange and social value creation, world music (instrument and theory) visualisation as a single, uniting discipline based on music data is more or less unexplored.

Asked what music visualisation is, most would -as does Wikipedia- envisage colourful, swirling and largely cosmetic patterns.

Clearly we can immediately distinguish between these and more literal music instrument and music theory visualisations. Where the first are predominantly decorative, the latter are models of physical or conceptual structures. They are recognisable, and have meaning.

Big, brave, open-source, non-profit, community-provisioned, cross-cultural and wax-a-chuckie crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or simply register as a potential crowdfunder..

Tonnetz Torus (Wikipedia)

Learning Modes

Potential Crowdfunder?

Pre-Crowdfunding Registration (* indicates required)
Email Format

Each learner has his or her own learning preferences: the means by which musical understanding and ability is developed. Some prefer to learn by ear, progressively picking up all the tension and detail as the tempo is increased, some from the written note, others using tablature, some directly by mimicry from an instrument display, or indeed face to face from a more experienced player. Advanced players may opt for improvisation in a simulated group or orchestra context.

Whatever the mode, studies show that the same brain centres are stimulated by both mental practice and actual physical movement. In other words, because the brain stores the information the same way, conscious visualisation is as valuable as actual performance. Indeed, newcomers to music learning actively seek visual patterns.

Regardless of learning mode, this is something to be actively explored, yet fully source-driven, dynamic, interactive instrument and theory tool visualisations remain a rarity.

Judging by Google Search, however, music visualisation can and does go beyond the purely cosmetic: from static interpretive artwork, through design of alternative music notations, to real-time music industry commercial visualisations, standalone and very specific theory animations, visualisations of the impact of music on the functioning of the brain, the on-screen simulation of stage or studio hardware, to the perceived visual relationships between mathematics, music and cosmos.

How much of this is relevant to or effective in musical learning is a matter of debate. Keeping in mind our overall goal of generating social (as opposed to purely commercial) value, let's survey the field of engagement..

The Entirely Literal

 

Cosmetics & Aesthetics

At the simplest level (with little or no information content), we have audio-driven graphics animations with focus on visual expression.

Not, however, necessarily trivial to create. Some rely on complex algorithms to achieve real-time 3D animation.

Cosmetic Music Visualisations


Ramped up a dimension.. LOTS of animation, but still no information.. Here we are in the realm of added, cosmetic visuals for event atmosphere.

Light Shows

So far, though, apart from sampling of sound level and/or frequencies, no attempt at deriving or mapping deeper musical data such as note or tone names, octave or even identification of the underlying music system.

While in the realm of social-commercial eye candy, here a playback / visualisation interface the 'Reactable', a project from the Music Technology Group at the Universitat Pompeu Fabra in Barcelona:

Industry Insights


Radically tangential are the standard business presentations reflecting industry trends. Derived from industry data using off-the-shelf visualisation packages, the stuff of suits. Essentially a little bland.

Music Industry Insights

Music Theory Visualisation

Huge cache of these on the internet, but perhaps also the first real surprise: hardly any animation. Information-rich, pretty, but predominantly static images.

Music Theory Visualisations
 

Music Sequencers

In the browser? Yup. A few. Visualisation? Only of the bits that go bang if dropped..

Software Sequencers

 

Synthesizers

In the browser? Yup. A few. Visualisations on a conceptual / engineering level only..

Software Synthesizers

 

State of Mind

Music and wish or ideal? Of course.. Retro, for example..

Hippy Retro


Psychotrance..

Psychotrance

 

Music Esoterica Visualisations

Even if we venture into the at times obsessive mathematical landscapes of music esoterica enthusiast, all is static. In compensation, colourful!

Music Esoterica Visualisations

Music and (Pure) Mathematics

Presumably a little more money behind these. Some dynamic elements creeping in..

Music and Mathematics

 

Off the Shelf Data Visualisations

Not all music visualisations need necessarily be custom built. Many useful representations can be built with more or less standard, or 'off-the-shelf' animations.

Data Visualisation icons representing various data views
Data Visualisation Catalogue

In many cases these can be more or less chosen from a prepackaged library, with the data populated from (generally JSON format) file.
There are, however, many liberally-licensed example files around which can be extended for own use. Generally you need only look for the name of the underlying visualisation library (such as d3.js), along with the term 'gallery'.
As illustrated here, there are also a wide range of icons representing these visualisations.
"Off the shelf" visualisations have wide application in instrument modelling, theory tool creation and (above all) supporting visualisations. 

To the latter belong those representing relationships between various community groupings, calendar and scheduling functions, hierarchies and workflows.
Lots of potential.

Music Apps

Never assume you know more than your kids. They have it good - if simple.

Music Apps

Seriously Good Software

Ok, we all know search engine limitations. How's about YouTube and the grapevine? Interactive, solidly useful - but forget visualisation. Here some picks.

Soundslice

 

SoundSlice


What it does, though, is allow time-tagged synchronisation with video. Enter commercial synergies.

 
As tabs are introduced for other instruments, Soundslice is likely to grow. Rather dramatically.
 
This must seriously frustrate at least one developer, whose Guitar Tab Player was around far earlier..

Mathemusic4D



This may not have been implemented in the browser using SVG, but in combining a base lattice or tonnetz structure with animation, this is the kind of rabbit we want to be chasing.

Perhaps a graphing library built on D3.js for the hexagonal tonnetz, and WebGL (three.js?) for the gyroscopic 4D thingy on the right.

Mapping Tonal Harmony Pro

Another novel visual approach, this time splitting the circle of fifths into tonal regions then used to lay bare potential improvisation paths.



This example comprises a nice explanation, model and tool, reinforcing the conviction that there may be many possible models or views to a given theory topic.

In the end, however, these model's final destination are the regions of the brain responsible for visual processing. Here an intriguing glimpse into just one of the possibilities unfolding.

Finish


So there you have it. State of the Art / Where We Start. If data visualisation's role in data science ("communcate, visualise, report" below) is well understood, where does that leave music visualisation?


Relationship of Data Visualisation to Data Science processing.
Data Visualisation in Data Science Process


By analogy, we are talking of the full-blown visualisation of every aspect of music - it's structures, patterns, evolution, instrumentation, culture, genres and values, but -insofar as it has a temporal dimension and is linked to music- a wider ecosystem of events, storytelling, animated art and esoterics.

At it's most abstract, however, this is reduced to just two elements:
Music Plus Data Visualisation Equals Music Visualisation #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory
Music Plus Data Visualisation Equals Music Visualisation
The project in focus here simply aggregates the results into a single platform.

Source-Driven Music Instrument and Theory Tools Visualisation or Visualization. #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory
"Literal" Music Visualisation


At a high level I see it as the foundation for music learning workflows based on progressively more insightful animations, under the guidance of a genre-knowledgeable, remote (P2P-connected) mentor or tutor.

At another level, I see it as an opportunity for much needed / just-in-time social value creation prior to the singularity.

Overlaid with a remote, P2P teaching capability, this has the potential for driving musical diversity, and for the dirtiest of reasons: the process of learning from others 1:1 is replete with interpretational noise, and hence subtle change. This is the very lifeblood of musical evolution.

To close, I draw your attention to the characteristics of a good data visualisation as formulated by David McCandless, a leading visualisation practitioner. The diagram seems to me as valid for music- as data- visualisation.

https://www.informationisbeautiful.net/visualizations/what-makes-a-good-data-visualization/
From the Information is Beautiful Website


Music visualisation is a journey which has barely begun. It is perhaps the storytelling and metaphor areas which open up the most intriguing possibilities.

Music Visualisation: Exploit the Platform, Free The Mind #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory
Music Visualisation: Exploit the Platform, Free The Mind

The world music visualisation aggregator platform will free us of the hassle of manipulating raw data, allowing us to concentrate on our storytelling goals and chosen metaphors.
Exploit the platform. Free the mind.

Cantillate

About Cantillate -

Autodidact. Laird o' the Windy Wa's. Serial Failure with Attitude. Bit of a Dreamer..

Comments, questions and (especially) critique welcome.