World Music's DIVERSITY and Data Visualisation's EXPRESSIVE POWER collide. A galaxy of INTERACTIVE, SCORE-DRIVEN instrument model and theory tool animations is born.

About Me

My photo
Autodidact. Laird o' the Windy Wa's. Serial Failure with Attitude. Bit of a Dreamer.
Andrew Douglas Hope 2016. Powered by Blogger.

Search This Site

This is not a blog. Posts (below) are continually updated, routinely integrated into the menu structures above (check 'em out), remain relevant, do not expire, and form a progressively more rigorous documentation set. Encompasses the entire spectrum of world & experimental notation, instrumentation and theory, and best explored on a large screen. COMMENTS WELCOME.

Wednesday, June 14, 2017

Cantillate

World Music Visualisation Aggregator Platform: Key Features


Peer To Peer Teaching And Learning Via Video Chat

Regarding interworking between a teacher and one or more learners, we are certainly NOT suggesting some form of simultaneous play will be possible, but more a two-way musical conversation, with the opportunity to listen, absorb, repeat - and of course question.

This mimics the style of conventional (offline) musical workshops, where students are accustomed (for example) to exploring melody, fingering and chord progression alternatives, getting a tighter grip on the specifics of style and ornamentation, or absorbing a wide range of anchoring cultural information. These important and personal aspects of musical immersion are entirely ignored in many conventional online video courses.



Clearly, ease of configuration and synchronisation are central, but beyond that, our focus must be teacher, student, music (exercise), and instrument. Simplicity and transparency is all.

A Galaxy Of Supporting Music Visualisation Possibilities

It's useful to think of platform integrations in terms of relevance to our central goal of a toolset supporting P2P teaching and learning.

If the core animations are notation, instrument models and theory tools, secondary to these (we can imagine them as clustered around them) are supporting applications, handling, for example, teacher, student, band and other affiliations, and related planning or scheduling tools.

A third (and outer) zone of relevance would be that of related integrations - in which we can include standalone applications derived from core platform components. Here we might encounter integrations focussed on the impact of music on the psyche, esoterica such as music-astrological mappings, or simply personal 'Internet Of Things' preferences associated with specific learning environments or pieces of music.



With open source (commons-based peer-produced) models, the only limitation is the collective imagination. What one mind can visualise and articulate, others will be able to implement.

Simple, Flexible, Personal

It's not rocket science. The development tools and libraries central to a comprehensive world music teaching platform such as this have for the most part long been proven. Scalar vector graphics (SVG) has, for example, been around for approaching two decades, and has long been supported in all major browsers.



The truly transformative link in our aggregator platform's technology chain is the visualisation library D3.js, which provides a consistent and DOM-affine foundation for SVG manipulation.

With a surprising economy and elegance of expression, D3.js provides the means to dramatically augment the range of tools available to teachers and learners. D3.js tends to hide tedious detail (the 'how to do'), allowing developers to concentrate on tasks (the 'what to do'). It has a reputation for a steep learning curve, but this is dramatically eased with access to the code for established, open source instrument and theory tool models.

Simple Core Architecture

The DOM-based structures as shown in the banner image at the head of this blog are at heart built around a few simple, interlocking mechanisms. These allow us, from raw data, to build static, on-screen models which can then be overlaid with fully dynamic, score-driven and synchronised animated graphics.



This model is trivial in the sense that it resembles that of a number of existing notation systems offering basic instrument (guitar, piano) display.

What distinguishes this system are the mechanisms for personalisation: instrument model and theory tool selection and exchange. Again there are clear structures and processes to these. In as far as useful to open source integration work, these will be revealed following successful crowdfunding and initial release.

Big, brave, open-source, non-profit, community-provisioned, cross-cultural and batshit crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or simply register as a potential crowdfunder..

Instrument Model Diversity - In Surfeit



With time, the platform can be expected to feature score-driven, interchangeable models of almost every world music instrument -of which there are certainly many thousands- and theory tools -of which there are hundreds- and allow straightforward behavioural comparison between them. How is this possible?

As Few As Five Steps To An Instrument Configuration

Using the 'lute' instrument family as an example, here we see how we might configure a Turkish bağlama or saz:

Modelling steps can be expected to mimic historical instrument construction and evolution. For simpler instrument such as those of the lute family in equal temperament (the default), configuration can be completed in as few as five steps. More complex instrument families may require a few more, but will always be manageable.

Complex wind instruments such as the modern flute, clarinet and saxophone can be expected to present among the greatest challenges, but manufacturers may be willing to assist. Our goal here is pragmatic visual modelling - and certainly not the fine tolerances associated with manufacturing.

Social Value Wellspring

Each configuration can be saved for immediate, general use by the community. In this way, any generic instrument base can be used to generate hundreds of configuration variants subsequently available to the community via the classification trees (visual hierarchies) accessible only through the top-level menus row.



These entirely dynamic, interactive models represent a wellspring of social value in three distinct ways:
  1. used directly on the aggregator site, they provide a secure base toolset for remote P2P teaching by anyone
  2. embedded in an own, external site, they will allow users to build custom offerings, nevertheless around proven models
  3. because the models are to be open source and liberally licensed, they can be reused and adapted at will
The former will further understanding and community among musical cultures ('comparative musicology'), the latter two will allow anyone to develop and showcase own teaching content around specific instrument models. This might include entire instrumental courses, where even the simplest of exercises drives networked and synchronised understanding across animated notation, instruments and theory.

Serving The 2D And 3D Web

The exchange file format 'MusicXML' represents a piece of music as data in a serialised form suitable for streaming. In this sense each exchange file can be thought of as a rather wilfully structured database.

In contrast to a conventional database, though, to get 'useful' data (a melody line, the notes suggesting a chord), you are obliged to reconstruct the score and apply filters to it. Depending on the intended application, these may target (say) a score part associated with a specific instrument, a note selection at a particular position during playback, the scope of a loop, the tonal delta between two notes, or indeed the range of dynamics glyphs applied.

In this sense, the filters act as a form of database query, the score something of a query playground. Literally anything can be selected.

Note: there is another approach using XSLT stylesheets to query streaming MusicXML with XPath expressions. As this would be more difficult to steer and we are at no point dealing with actual streaming MusicXML, this approach is discarded.

The decisive factor is likely to be user convenience and freedom of choice: it is easier to handle filtering visually, on a reconstructed score. D3.js's selection mechanisms do just this - flexibly, precisely and concisely. Filtered data is then passed to dependent animations.

This data could also, however, be passed -via API- to other, external, applications.


The platform has the potential to exploit three transformational technologies: music visualisation, artificial intelligence, and augmented or virtual reality. The above diagram attempts to illustrate use of MusicXML in two of these scenarios.

On the top left-hand side we have predominantly (but not exclusively) 2D music visualisation, which are score-driven animations of music instrument models and theory tools built using SVG (scalar vector graphics) on the browser's DOM (domain object model). On the top right, we see how the same base musical data might be used to drive dramatic or gamified 3D narratives in a WebGL-based virtual world.

There are many possible configurations (directly via the API, using an embedded and interactive score display, or indeed app to app), but the central point is simply that any selections made on the notation could be shared with the 3D WebGL environment and hence fulfil considerably wider needs.

Learning Speed And Focus

The brain is by all accounts a remarkably elastic organ. Given the right field of specialisation and learning environment, a strong and free will and steady support, I have the impression we may all be capable of something approaching the extraordinary.

Our goal is as musically open and immersive an environment as possible, but with the live, virtuoso / teacher firmly at the centre. To that end, here an overview of some of the key musical qualities (key performance indicators or KPIs, if you will) aspiring musicians can be expected to hunger after, and which any learning environment should consciously support.

By structure, incidentally, the mental maps underlying instrumental navigation. Clearly, these can be literal or figurative.
With technical advance goes cultural erosion. We are left -on a purely technical level- dissatisfied with the media used in the direct teachings of earlier, culturally iconic players. In discarding the media, we discard what are often the last remnants of a culture.
a
Moreover, no single technology can replicate in all it's subtlety the directness of face-to-face teaching of a specific example of fine motor movement. This is why -despite technical barriers- it is so very essential that where we work online, it is with live, virtuoso musicians.

At the end of the day, we have feeds needing synchronised and merged to give our best shot at a semblance of immersion. Looking in a little more detail, we see how those feeds might flower into 2- and 3D animations across a whole range of technology stacks.

A Natural Step In The Move Towards Immersive Musical Learning

The traditional, face-to-face approach to teaching and learning mentioned at the bottom LHS of the following diagram embodied more or less everything needed to master an instrument over time. Indeed, in as much as there was also companionship and wide personal engagement, it also served social ends.

Until now online music learning has more or less just mimicked, step for step, that old mode of learning - if at distance. Technology will only really come into it's own when it breaks with conventional learning, and develops new, unimagined forms. How this happens -and whether, with diluted forms of contact, this proves a good thing- is anyone's guess.



For the meantime, immersion in online music learning is technology-led, and any gain in immediacy simply a reflection of how it is applied. There is, however, always this sense of increasingly 'distancing' oneself from the original, living source. This makes me uneasy. Nevertheless, like it or not, technology is with us to stay, and so it is in our interest to make the best possible use of it that we can.

Establishes An End-To-End, Readily Enhanceable Value Chain

The musical data available at various points in the platform workflows are a resource to which both algorithm and machine intelligence can be applied, and which could be made available on-demand.



Liberate The Emotional, Delegate The Banal

Whether through soaring genre fluency, subtle contrasts in tempo, pitch, timbre or rhythm, or simply triggering each other's creativity through clear role punctuation, musical virtuosity is an act of profound empathy and intuition, of listening and reacting to other's play is ways that develop the emotional tension of the whole.

'Artificial intuition' is something of a hot topic in advanced computing, but is likely to take decades to find general application. Until then we have models such as the following for relationship between the emotional and the (for want of a better word) banal.

There is a threshold above which algorithm (and here I include artificial intelligence's pattern matching orgies) cede ground to simple human intuition and emotional intelligence. This is an area where creativity has free rein, and, naturally, where musicians love to hang out..

Driving Social Value: The 'Open Everything' Community Provisioning Stack

In the context of the web browser and DOM, there is a well-established (if perhaps hitherto not explicitly documented) open source resource pyramid providing a solid base for commons-based peer production. It lies in our hands to ensure that, for world music, the upper, as yet only marginally explored tiers are also brought into the realm of commons based peer production.

Our aggregator platform has the potential to provide the skeleton or framework upon which to integrate such services. How technologies such as artificial intelligence or virtual reality are integrated will at least be open to exploration and experiment.

Fully Configurable - From User Menus Down To Note Colours

The entire system has been conceived for personalisation, both in terms of own (side) menu content, but also generally applicable conventions such as note colourings and fine settings such as tunings, custom fingering layouts and the use of aids such as capo.

Perhaps contrary to expectations, this is no rocket science, but simply reflects the power of modern data visualisation libraries.

For all our technological advance, we are barely able to replicate millennia-old face-to-face (F2F) learning practices at distance - via video chat. Soundslice has set the bar high for synchonised video. Our aim is to go a little further, add as much value, immersion and immediacy to that dialog as current means allow.

Cross-Cultural, Immersive, Global

Working with our hands seems to answer a fundamental human need - satisfied both in the high craftsmanship of instrument building, and in musicianship itself. With the inevitable arrival of a post-work society, I feel fairly sure both will enjoy a thoroughgoing renaissance.

Anyone familiar with the smell of freshly freshly shellacked wood, cured leather, machined synthetic ivory or the twang of a set of brass-wound steel strings will know just how satisfying a visit to an instrument builder's workshop or an instrumental workshop can be. It is our intention to catch and ride this wave, bringing what we can to the home, but more importantly, motivating people to get together.

Supports Any Music System (Notation), Instrument Or Theory Tool

The prize is sensational: the ability to configure the platform to your own needs, and then have the animations dance together -in tight synchronisation- as the score plays.



The freedom to load one's own or a teachers immediate choice of exercise or score, have it drive an own, immediate selection of instrument models and theory tools, and the ability to visually follow and understand mappings between each.

Just as with the original, instrument models will be fine-configurable to fit own, immediate needs. Choose between different African Kora tunings, just bağlama or equal temperament oud, Venezuelan diatonic or classical chromatic harp, Bb, C or other clarinet (and particular historical form), A B- or C- system chromatic accordion, or simply between instruments of different scale lengths.

Visual Classification Of Theory Tools

Going by Google Blogger's (anonymised) statistics, few visitors take the time to explore the Instrument and Theory Tool menus above, yet together with the notation, these form the core, if exchangeable, visual elements of the framework.

2 and 3D music-theoretical models normally represent some form of abstraction (for example reducing octave equivalents to some 'octave-agnostic' value) with the aim of exposing lean, underlying structure. Carefully aligned for manipulation by advanced, data-driven visualisation libraries such as D3.js, though, data can be transformed and mapped in ways that we can barely imagine.

Mirroring Hornbostel-Sachs' visual classification for musical instruments, out aim is a simple, pragmatic classification system for theory tools - all of which, in the context of our aggregator platform, are all of course graphical in nature.

Shared P2P Interworking Controls

Shared, P2P Interworking controls are foreseen on three levels>
  1. Configuration
  2. Environmental
  3. Playback


Configuration controls focus on the setting or sharing of instrument and theory tool definitions governing form and function.

Environmental controls govern the setting or sharing of preferences such as colour schemes, sound libraries and layouts.

Playback controls govern the speed, scope, and other behaviours of shared score playback.

Each control type could be applied across several peers. It would be up to the peers themselves to decide if they wish to retain these settings for future sessions.

Value Added Core Strengths

The platform's core strengths are ripe for augmenation by two transformative technologies: artificial intelligens, and, by preference the Web3D forms of virtual and augmented reality.

The focus of the initial crowdfunding, however, is the browser DOM, as that is where this journey starts. Any expertise won there can be reapplied in other the other areas.

A Foundation For Musical Storytelling

To date, timeline applications using D3.js or it's derivatives have been focussed on conventional data sets, but (analogously to our instrument and theory tool animations and using using either SVG or WebGL) we could equally well envisage stories being animated during score playback.



This suggests the merging of data supporting storyline development with that of data generated during score playback.

How 'creatives' develop this opportunity - and especially if it leads to new tools blending score and storyline - is something I'll be following with great interest.
Read More

Sunday, June 4, 2017

Cantillate

World Music Visualisations: Platform Aspirations


This post is a RIOT of good intentions. Overriding goal? Get everyone on the same page..



Big, brave, open-source, non-profit, community-provisioned, cross-cultural and feather duster crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or simply register as a potential crowdfunder..

Potential Crowdfunder?

Pre-Crowdfunding Registration (* indicates required)
Email Format

I invite you to wander through, mentally picking off the stuff you like, dislike or frankly couldn't give a toss about. Do let me know (in all the time this blog has been live, there has been only one comment, and yes, it was positive).

'Why this list?'. Well: because all the points touched on have been mentioned in past writings, but sometimes in rather specific contexts. I feel this pressing urge to gather, abstract and generalise.

The list items have been loosely categorised as 'Emotional', 'Organisational' and 'Technical', broadly reflecting three interworking levels: that of users, tools and technologies. Further ordering and prioritisation seems unlikely to serve any purpose.

Visual models are fine, but once in a while it can be helpful to give the imagination free rein, right? In contrast to earlier 'overview' posts this time without a swathe of accompanying images.

One last thing.. The moment this monster gets crowdfunded, everything is going to go into overdrive. For the moment, though, you have my full and undivided attention. If you are reasonably familiar with the blog's content and can suggest areas you feel would add value, feel free to say so. Like, now. :-)

Platform Aspirations

Emotional

  • Counter the drift towards a global musical monoculture: celebrate instrumental diversity with world music instruments, tools, cultures and perspectives
  • Grass-roots, person-to-person teaching and enablement, connecting emotional intelligence, musicality and virtuosity
  • Driving global, cross-cultural social value: hence non-profit, largely open-source, with commons-based peer production (CBPP).
  • Center all activity around the user's choice of teacher and music
  • The re-democratisation of music: encourage the playful, communicative and socially immersive aspects of music over the competitive
  • Exploit the brain's formidable visual and spatial reasoning
  • Fuel a renaissance in participatory community dance and music
  • Help render work irrelevant: providing occupation for idle hands and liberated minds
  • Data, story, goal, metaphor. Enable musical storytelling across a range of disciplines
  • Mutual empowerment through the gift of own time and skills
  • Support comparative musicology across a wide range of world musical cultures
  • Provide the tools to secure grassroots teacher earning potential - at lowest possible cost
  • provide an antidote to the music industry's more toxic business practises and role models
  • offer a familiar route into a musical future otherwise dominated by exponential technological advance and bewildering change

Organisational

  • Provide in as far as possible culture-agnostic visual models, leaving their interpretation to the teacher
  • Management of musical complexity: a catalyst for communication between musicologists, engineers and musicians
  • A toolset in support of P2P video chat, using ANY music source, instrument configuration, theory tool, mentor or location
  • Liberate musical learning and interworking through global visibility, ease of access and interactivity
  • An integration framework for artificial and machine intelligence with on-demand insights
  • Expose relationships between notation, instrument fingerings and abstract theory through score driven visual animation and interworking
  • Help users take instrumental and theoretical 'ownership' by giving them equal prominence with notation
  • Aggregate everything into one highly configurable and personalisable online platform
  • Power of the crowd: from experiment to experience and consensus, with 'best of visual class' instrument and tool implementations
  • Reduce superfluous and toxic travel
  • Take a decisive stance against raptor economics and wealth hoarding. Get earnings working in and for the grass-roots music community.
  • Where visual classification trees are missing (as for theory tools), provide them
  • No advertising, but if there is a demand, links (via shopping-cart icon) to curated, user-rated directories of hand-built products
  • Provide sufficient but dependable, genre-authentic reference models

Technical

  • Focus: first and foremost graphical, thereafter -in as far as possible- multimedia-immersive
  • Simple, flexible and robust reuse mechanism favouring easily extendable visual model(s) over absolute dimensioning accuracy
  • Real-time single page application (SPA). Everything dynamically loaded: no installations, no 'plugins'
  • Event-driven javascript controlling data-driven, in-browser SVG- and CSS using cutting-edge visualisation libraries.
  • Device-responsive timeline-based but application agnostic -hence reusable- framework
  • Consistency through hardware timers, algorithmic placement and styling guidelines
Because it's focus is entirely graphical, don't expect this platform to replace the many other excellent sites dedicated to mainstream instruments. Moreover, while providing profound visual insights, it leaves musical their interpretation (which may differ depending on culture or genre) to those who adopt it as a teaching platform, toolset and reference.

As with other music sites, subscription-based usage is envisaged. Our aim is to keep the costs as low as possible, and, as a non-profit, where surfeit exists, re.in.vest. There is nothing stopping a non-profits generating earnings: they just have to put it back to use - generally in the same year.

While the P2P aspects have only been dealt with conceptually, the basic top-to-bottom mechanisms for instrument and theory tool configuration, population (personalisation), docking and synchronisation have been prototyped, demonstrate feasibility, and now beg to be fleshed out and scaled. Help us take delivery of social value to the next level.

What it will deliver is
o motivation (worldwide choice of teacher, instrument and tools)
o immediacy (cittern, CGDAD tuning, capo 2nd fret, driven by the user's own music source: no problem), and
o immersion (notation, instrumentation and theory tools woven into one coherent, synchronised dynamic whole - layered with stylistic cues from synchronised P2P audio and video).

I have no idea if everything can be done, but dammit, let's give it our best shot.
Read More

Tuesday, May 9, 2017

Cantillate

World Music Aggregator Platform vs Soundslice


The world music visualisation platform in focus here takes an approach quite different to that of Soundslice, perhaps the currently fastest-growing (solo-)instrument learning environment.

Many instrumental learners -especially those in heritage, folk, and traditional circles- get by learning from slowed audio. This mimics to a certain degree the face-to-face rote learning of earlier generations (and glorious accidents of memory likely responsible for much of the tune diversity).

Even so, alone the time overhead in finding, chopping up and slowing tunes can be daunting. This was, indeed, the motivation for 'Soundslice'. It can be thought of as a round-trip learning tool, in the sense that you start with an original recording, slow it down to a point where you can imitate it, then progressively speed up until playing along in perfect synchrony at the original speed.

Soundslice's focus is on simplifying life for the notation-oriented, 'learning-through-mimicry' instrumentalist, bringing a variety of learning sources into a single workflow - clear audio, simple, single-voice notation, a tiny selection of models of the most popular instruments, and synchronised video of 'virtuoso' (professional teacher's) play.

Soundslice takes a number of similar systems (such as InstantNotation or Knowtation) just a tick further in the drift towards on-demand immersive music environments.

Soundslice's primary achievements are: giving some visual context to what is essentially an audio feed; in time-synchronising the various other feeds with this audio.

A particular success are:
  1. the synchronisation of video with external (as opposed to recorded, 'in-video') notation, and the notation's robust, well-synchronised looping controls. These provide both accurate positional and fingering clues, and very useful stylistic audio cues
  2. The reuse of fingering information gleaned from the various instrument-specific input file formats
It is difficult to overemphasise the value of the latter, but it immediately exposes one of the major weaknesses of current music exchange file formats: the inability to map to exchangeable, external fingering files. Even MusicXML, the W3C musical crown jewel, uses hard-coded, inline fingerings. In a world music context, this is cultural assassination.

Nevertheless, taken together, these to some degree plaster over one of the major shortcomings of notation - it's inability to adequately describe the subtler elements of style. Be it accenting, subtle rhythmic attack or delay, or the fine detail in ornamentation, these are all qualities notation is notoriously poor at conveying. Nevertheless, audio is far from unique to Soundslice. More on that below.
For all it's effectiveness, with the exception of looping controls, I haven't featured Soundslice's notation on this diagram, as there are many similar offerings elsewhere. (Had the notation been data-driven and the notes individually addressable/interrogable, I would have, as this opens whole new interaction possibilities). Moreover, Soundslice's approach more or less undermines the possibility of exploiting an area of huge and untapped potential: music visualisation.

What it does, though, it does extremely well, and at first glance, appears to answer pretty much all needs.

Today's needs are, however, not tomorrow's. We stand at the threshold to multiple technological paradigm changes:
  • music visualisation
  • augmented and virtual reality
  • machine learning and artificial intelligence
These draw us inevitably from 'mobile first' towards 'data-driven', 'on-demand' (or, better still, 'need-of-the-moment'), immersive and ultimately, it seems, 'AI-first'. A comparison at this point seems both timely and interesting - on a number of levels.

Without distracting from Soundslice's successes, I'd like to focus in on a couple of opportunities it's design impedes - or even obstructs.

Big, brave, open-source, non-profit, community-provisioned, cross-cultural and batshit crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or simply register as a potential crowdfunder..

Learning By Ear

Potential Crowdfunder?

Pre-Crowdfunding Registration (* indicates required)
Email Format

Prior to that, however, I feel obliged to concede that for many older / 'traditional' players, even notation is overkill: slowed audio (mp3, more recently AAC) is widely accepted as entirely sufficient:
  • It mirrors the older (and much revered) practice of face-to-face learning around the hearth during long winter months.
  • It is well supported by 'slow-down-while-maintaining-pitch' tools
  • Audio is the fastest and shortest route to the brain. With our brains naturally (and with impressive accuracy) able to distinguish pitch, everything put in the way of ear learning simply slows down the mapping of audio via brain to instrument. Neural pathways take time and practice to form, yet folk virtuosos around the world are stunning testimony as to the power of learning by ear. Many don't read a note of music.
  • Hands-free play-along. Once a piece of music is set to automatic looping playback on an audio device, no further intervention is required. The mind is free to concentrate, the hand to play.
  • Location independent. With any good mp3/AAC player - down by the river, in a park, perhaps during a lunch break (effectively a mini-holiday, given the switch of brain hemispheres).
  • Learners develop a simple, intuitive sense of harmonic relations - applicable both in equal temperament and just intoned music cultures.
That may seem a daft admission coming from something of a visualisation 'evangelist', but where audio supports musicality, music visualisation brings deep understanding. Poles apart.

Immediate Needs

There is a sense in which all current online offerings fail to deliver, and that is in satisfying immediate needs. While easy enough to find a modal scale, there may be no help with fingering on an instrument of your choice, or in viewing how it fits in with the broader modal landscape of the musical culture. Often, just to get at this information, you are obliged to scour online book & CD offerings, commit to an unnecessarily broad online course, or indeed follow something created for an entirely different instrument.

The counter-argument is that the brain develops according to the challenges we feed it. Accordingly, even a marginally immersive teaching environment should help develop multiple skills, in parallel, and across a broad front.

Yet notation's massive cognitive load has to be overcome before a note of actual music is played. This has clear impact on the speed at which we learn. Indeed, notation could be argued as having nothing to do with the actual music.

Younger players have grown with the internet, and so are perhaps more open to even slightly more immersive approaches. Soundslices's video and 'hardcoded' (non-configurable) instruments to some degree bridge the gap in providing limited visual orientation. These, however, remain a drop in the instrumental and music visualisation ocean.

Soundslice In Detail

Business Model

Soundslice's focus is squarely on a limited range of music exchange formats -mainly those preferred by guitar players- with consequences throughout it's entire business alignment. Orchestral or even small ensemble music scores for example, are out.

Soundslice exploits a canny mix of franchising and subscription. In providing the tools needed to synchronise professionally-produced video with notation generated from a variety of exchange file formats, it not only drives own and other's (franchised) subscriptions, but transfers the responsibility for provisioning and promotion to the individual franchises. With a 30% commission on franchise sales, beyond the initial implementation for a given instrument, for it's owner a golden egg laying goose.

At the simplest level, Soundslice allows users to load their tune in a variety of formats and have it played back with synthesised audio. This is precise, if lacking in the fine musical tensions that distinguish a human player from a mechanical plonker. This does, however, provide for solid rhythmic accuracy - which for some beginners is perhaps no bad thing.

This form of usage allows display either in conventional music notation or so-called 'tablature', a form of pictorial shorthand specific mainly (but not quite entirely) to strung instruments. For this form of use, basic subscription rates apply.

So what help in expanding the platform's functional range is offered? Soundslice's Acceptable Use Policy states:
"The main goal of Soundslice is to help you learn to play specific pieces of music. It's primarily for educational use, and we hope our tools make you become a better musician.

Beyond that, Soundslice is rather open-ended, in that it allows arbitrary annotation of sound and video. We expect users will discover and invent many wonderful, unexpected ways to use it. Those are encouraged, as long as they're consistent with this policy".

This seems to hint some form of open / win-win further development strategy, but as it turns out: only as long as the subscriptions are paid. → This is a closed source, franchise model. Moreover, the most obvious area in which musicians can make a useful contribution is that of tablature glyphs for other melody instruments, and in providing fingerings. The former is reusable, the latter a unicum.

Technical Overview

Soundslice has been built using javascript in conjunction with recent HTML5 web multimedia elements. It reflects efficient and economic -if from a graphics point of view slightly dated- good practice, achieving not just responsive page elements but also responsive content. Where a learner might earlier have worked with an audio recording, a page of ABC, tablature or a melody line and a few notes, Soundslice generates tablature and/or notation from one's own choice of (single-voiced) exchange file, some simple fingering diagrams and (for an extra fee) time-synchronised video and tablature/notation of some professional mentor's play.

It integrates and fully synchronises notation, video and simple instrument animation. It's appears to rely on tablature information for diagrammatic finger positioning.

All files necessary to full offline use can be retained in the browser's cache, making it useful for those times you are out of mobile or WIFI signal range.

As touched on above, for traditional or folk learners who have worked predominantly with slowed audio, this is already possibly too rich an offering.

For advanced folk musicians seeking a deeper insight into music-cultural structures and for notation-based learners interested in deeper aspects of harmony, it may prove too little.

Nevertheless, Soundslice occupies an odd niche, strangely isolated amongst technologies currently attracting a lot of attention.

In particular, the indications are that is neither 'live', nor 'real-time', nor strictly 'data-driven' - nor (curiously) does it rely on WebGL, which underpins much of the current work on WebAR and WebVR, (the web-based forms of augmented and virtual reality, collectively known as Web3D). Nevertheless, it is reasonably immersive.

How easy it would be to integrate elements of Soundslice into these virtual worlds is moot, but it's an entertaining thought..

Soundslice Screen Shot
Soundslice Screenshot
It has a simple, intuitive interface and reacts quickly to modification. It gets round some problems inherent in it's implementation using simple but effective (creator's own word..) 'hacks'.

Though allowing Soundslice to do what it does very well, as we will see below, these will clearly hamper the product in it's further development.

Soundslice's Strategic Limitations

Several Soundslice design decisions bar it's further development into a comprehensive aggregator platform for all instruments and other animations. Let's try to summarise some of these.

Notation
  • Notation appears to be limited to the treble and bass clefs
  • Currently, piano is the most complex instrument (MusicXML 'Part') setting offered. Synchronisation between left (bass) and right (treble) hand voices is possible only because -exceptionally- all the relevant information is present within this instrument's 'Part'.
  • With note selection apparently by offset rather than class or id, it could be technically challenging to extend the system to play multiple parts simultaneously.
  • The above limit harmonic exploration.
Audio And Other Libraries
  • Instrumental audio 'synthesis' is achieved not by generative techniques, but by calculating offsets into mp3 files containing fixed-length soundbites of given instruments at various pitches. A hack, but as it happens, a very effective one..
  • Because of the above, however, other than instrument choice there is no mechanism for selection of alternative audio libraries
Video
  • There is a distinct cognitive load associated with video: mentally mapping positions and fingerings from an opposing, dynamic instrument view. Our immediate need is for orientation and detailed positional information.
Animations
  • Because controls are time-offset-based rather than linked to specific on-screen notation elements, the scope for animation and/or P2P interworking is severely limited (there seem to be hints on the Soundslice site, however, that other interworking models may be in the pipeline).
  • Instrument displays are 'one-offs' - effectively hard-coded island solutions with no scope for nicities such as alternative tunings. In serving only a tiny range of already hugely popular instruments and attracting predominantly professional musicians into it's franchising model, Soundslice could be said to be feeding the trend to something of an instrumental monoculture. No doubt this will ease as a wider range of genres, instruments and styles are added.
  • Though likely to improve, the number of instruments for which tablature exists is very limited.
  • Instrument displays appear to be prescribed rather than user-selectable.
  • Tablature and instrument model fingerings and chord diagrams could be said to represent redundancy.
Technical

Soundslice creates it's notation in bitmap form using the simplest of graphical elements 'hard-coded' onto a DOM 'canvas' element. Once there, they are more or less fixed.
  1. In bitmap form, the visual mapping and transformation possibilities are severely limited. Where a bitmap is needed (for speed on a mobile device, or exported as an image to a virtual reality context), it could equally well be derived on-the-fly using SVG-to-image conversion on a server - at any size, and if desired transformed.
  2. Bitmapped notation tends to undermine the clearest motivation for parsing MusicXML using Javascript (to prepare notation glyphs for algorithmic placement, using javascript-based data visualisation libraries).
  3. Given the above, there is no means of selecting own, preferred note colouring schemes
  4. Soundslice's focus is not on data, but on processing speed. It was built to the 'Mobile First' mantra. Tomorrows's immersive applications, however, will run on dramatically faster, high-resolution augmented and virtual reality devices, will be real-time, 'on-demand', data-driven, and in no sense pre-baked.
  5. The instrumental focus is on the popular, not the diverse. We are in an age of musical individualism, of experimentation, world music and self-expression. Yes, of group play - but preferably accommodating the exotic. Moreover, the platforms of tomorrow will connect real people learning from each other in social groups. If real-time play together over the internet is still a pipe-dream, on-line face-to-face teaching and learning are already being done. They just need the supporting toolset..

In sum, a tour-de-force in pragmatic scaling down for the mobile app market, but which undermines much of it's potential in multi-voiced and advanced educational environments. The following diagrams more or less sum it up.

Omitting any mention of virtual / augmented reality or machine intelligence, here we see how Soundslice, our Aggregator platform and the earlier and less sophisticated InstantNotation and Knowtation products are placed relative to each other in the move towards immersive online learning.

Soundslice's Achilles Heel. #VisualFutureOfMusic #WorldMusicInstrumentAndTheory
Instrument Model and Theory Tools Diversity
Soundslice's Achilles Heel
By diversity, we mean a wide range of world music instruments and theory tools animations. But why a 'dubious' model? Because animation diversity and platform speed are not necessarily in conflict with each other. It all depends how that diversity is achieved. More on that in a mo..

Meanwhile, as you may see:
  • Over time (and especially as devices and browser technology advances), Soundslice may quickly get left behind. It will, however, continue to serve well on legacy devices.
  • For Soundslice to achieve greater instrumental or music theory diversity, the current franchise model could lead to high organisational overhead.

Rationale For A Clear Alternative

While readily acknowledging that much of folk music is monaural, the following will allow us to work with all manner of score: from single voices through band works, choral pieces, chamber music, orchestral scores - and across many cultures:
  • the widespread availability of good traditional music scores, as collected by enthusiasts worldwide.
  • the appearance of powerful online notation editors such as Noteflight, which contribute to the wealth of open-source scores, and can be used to create exercises illuminating theoretical and harmonic principles.
  • new forms of microtonal notation.
  • mechanisms for finely differentiated instrument and theory tool configuration.
Our as yet unnamed aggregator platform focusses on notational, voicing, instrumental and theory tools diversity. I hope this post has demonstrated that it is in no sense modelled on the Soundslice approach, but on social value, data and the possibility of reuse across multiple technologies.

Coming back briefly to that 'dubious' model (see similar graphic earlier in this post), can we now put the aggregator platform's potential capabilities into perspective?
Aggregator platform vs Soundslice. #VisualFutureOfMusic #WorldMusicInstrumentAndTheory
Aggregator Platform vs Soundslice
Instrument & Theory Tool Diversity Versus Speed
This graph is attempting to tell us that through careful configuration of world music instruments, we can reduce the processing overheads involved in recovering and displaying any instrument to much the same level as those Soundslice requires to show one.

What is not perhaps immediately clear is that it will allow the user to populate menus with their own selection of instruments, theory tools and supporting applications, and that the automatic interworking mechanisms open up a galaxy of potential score-driven music visualisations. Another way of looking at things is in the context of the immediacy and immersion of the user experience. Over time, these have developed from (for immediacy) simple face-to-face learning through to the point where people are already learning (albeit with little tool support), P2P (peer-to-peer) across the internet, and for immersion from simple audio through to immersive Web3D. In that user's acceptance of new technologies is delayed, immediacy and immersion can be viewed as alternately 'leap-frogging' each other.
If we zoom in a little on the top RHS, we can perhaps see how Soundslice and our aspiring aggregator platform might fit in. In introducing synchronised video, Soundslice is leading in an era of on-demand, media-synchronised learning.

Nevertheless: 'on-demand' is not 'immediate-need-driven', and 'synchronised' is not 'real-time'. The central distinction is simple: whether or not media is synchronised with a live teacher. A peer-to-peer system linking live teacher and students with gamer-style controls would greatly simplify one-way synchronisation - which, if not enough to allow play together, is certainly enough to teach.

Entirely automating synchronisation of video stream with a running score during playback is, however, much easier said than done. This is an area where artificial intelligence can perhaps, one day, prove really useful. Until then (and even with the help of game protocol and back-end server tricks) any synchronisation of a P2P teacher's play with notation on the student's end is likely to remain a simple, rule-of-thumb affair.

Pre-synchronised audio and video of the kind used in Soundslice is howeve perhaps not such a problem. Much of the groundwork has likely already been done for tools such as VLC (video) and Audacity (audio). Perhaps someone will take the time to investigate this soon: an open-source video (time) tokeniser program - similar to that apparently offered by Soundslice could be very useful all round.

Finally, our aggregator platform, in supporting music visualisation in the widest sense and in sharing it's various feeds, bridges the data divide between the conventional browser DOM and the dramatic, gamified world of WebGL and Web3D.

To sum up, while more than enough for learning new tunes on a few mainstream instruments, viewed from the perspective of musical diversity, visual modelling and understanding, immediacy and immersion, there is abundant scope for advance. The next step is crowdfunding, for which, having read this far, I fervently hope I have your support. :-)

Finally finally, a blow-by-blow summary of the main differences between Soundslice and our anticipated aggregator platform:

Comparison between Soundslice and the Music Visualisation Aggregator platform #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory

Read More

Sunday, May 7, 2017

Cantillate

Music Visualisation, Machine Learning And Artificial Intelligence

Artificial intelligence has arrived, no question. Already we are beginning to see the first AI-assisted music apps appearing in, for example, Google Store.

The new machine age is "digital, exponential and combinatorial" and characterised by "mind, not matter; brain not brawn; and ideas, not things" (Erik Brynjolfsson "Race with the machines").

I'm not so sure about the 'brain not brawn' bit. Humans are complex, emotional, self-aware beings. Artificial intelligence (AI), machine learning or neural networks are basically just logic trees allowing brute-force siege-style evaluation of all mathematically possible outcomes, and algorithms trained to recognise which paths constitute successful outcomes. What a machine achieves through meticulous attention to detail, a human can achieve through leaps of intuition. How long will it take to arrive at artificial intuition?



There is a technological threshold below which chores can be delegated, and above which human empathy and emotional intelligence are freed.

At the same time, we have to accept the likelihood that in a not too distant future, we will interact with highly empathetic automata equipped to replicate the finest of human motor skills, musical styles, creativity, tension, and hence emotion - and that these will be indistinguishable from the real thing. What will be, will be. Let's stay grounded :-)



Though I have misgivings (see my closing remarks way below) about delegating too much of those things related to human wellbeing to them, kept focussed, artificial and machine intelligence have the potential to greatly enrich the music learning value chain.

We have perhaps to make a distinction here between machine learning, which is broadly understood as comprising classification, clustering, rule mining and deep learning, and artificial intelligence, comprising logic, reasoning, rule engines and the semantic web. Underlying both, however, are large collections of data. Together, they are known as 'machine intelligence'.

Both have been around for a while now. There are already many open-source software libraries around for machine learning. To name just a few: Torch. Berkeley Vision released Caffe. The University of Montreal's Theano. Slowly but surely, our web browsers are being infused with artificial intelligence. Google open-sourced their TensorFlow AI engine last year, which is claimed to combine the best of Torch, Caffe and Theano. The curious can even dabble online.

"Machine intelligence can do a lot of creative things; it can mash up existing content, reframe it to fit a new context, fill in gaps in an appropriate fashion, or generate potential solutions given a range of parameters". -Carlos E. Perez

In a musical context, they can be used to solve problems spanning from the more or less cosmetic all the way down to the logical. For example, from musical motif recognition during work analysis, over aspects of music technique, down to supporting automated tool configuration processes.

As the diagram above suggests, there are a wide range of potential applications in music, but for the moment (and especially until data handling and interrogation is consistent across the entire stack, these are necessarily focussed or island solutions. A general purpose, layered music intelligence engine -and especially one suited to web applications- is still a long way off.

The to my mind currently most pressing need possibly lending itself to an AI solution is the automatic synchronisation of video, audio and notation feeds from disparate, near-but-not-quite-simultaneous sources whose speed may be subject to drift. This wearying manual task is undertaken daily by thousands of musicians around the globe.. Fully automated -and especially if open sourced- it be of huge assistance in getting P2P teaching off the ground. An approach tolerant of variation in playing speed is based on a so-called beatmap file, but still involves several manual steps.

With it's clear focus on data, it is hoped the music aggregator platform will act as a honeypot for AI and machine intelligence solutions in music. It can, in this sense, be understood as an integration platform. Have a lead? Feel free to get in contact. ;-)

Avidly Seeking Sponsors #VisualFutureOfMusic #WorldMusicInstrumentsAndTheory

Why Incorporate Artificial Intelligence?

Potential Crowdfunder?

Pre-Crowdfunding Registration (* indicates required)
Email Format

AI has the potential to solve challenges at many levels in an end-to-end music learning environment. Some of these relate to constructing the platform's own artifacts, others to supporting the learner, or, indeed, a P2P teacher.
Big, brave, open-source, non-profit, community-provisioned, cross-cultural and batshit crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or simply register as a potential crowdfunder..
So let's see how some of these might have practical and visible impact on our aggregator platform...



Away from the browser (and with it, our aggregator platform), there are further possibilities..
Finally, a cautionary word. Since the early days of the internet, we have been obliged to police the boundary between machine and man. Increasingly, both attack and policing are done by largely autonomous software built on deep learning principles. The former scours defenses for potential weaknesses, the latter monitors interaction behaviours, looking for unusual patterns. Already, our online security is passing out of our hands.

Under threat, we revert to predictable, tribal behaviours. If man-made, Alt-Right Twitter bots can already endanger, and billionaire-sponsored social media profile analysis straight out of military psychological warfare decide national elections, it takes no great leap of the imagination to see the potential impact of autonomous, self-replicating and learning AI bots on the internet. How and where they turn up is anybody's guess.

However, if digital goods can be replicated in perfect quality at nearly zero cost, be delivered instantaneously and in doing so wipe out half the available jobs, then surely the goal must be -for an ever-increasing number unable to keep pace- to make work irrelevant. How? By supporting human strengths, and creating tools that drive social value into the community. Amongst these, machine intelligence will inevitably find a role - but we need to be involved.
Read More

Saturday, May 6, 2017

Cantillate

World Music Visualisations And The 3D Web

This post is unapologetically technical. Normally I make an effort to keep the jargon to a minimum, but this time it's so pervasive I'm not even going to provide a glossary. As with all other posts, it will be updated from time to time to reflect fresh, spicy developments and insights.

I'd like to briefly revisit the overview of Augmented and Virtual Reality) stacks introduced in a much earlier post, this time focussing in on the 3D web offerings (those on the right-hand side of the diagram to the left).

These comprise 'conventional' -and in some cases repurposed- 2D web stack technologies such as CSS and SVG, but also open (but not necessarily W3C-governed) 3D standards based libraries such as XML3D, X3DOM, and the various WebGL-based (Web3D, ie WebAR and WebVR) approaches. For reasons of space and time, the focus is on a tiny, but hopefully reasonably durable selection.

There are three contexts under which a music source might be used as a 'driver' of on-screen events. Clearly, music notation may be helpful in the first two contexts, but is likely superfluous to the last:
  1. 'Pure' (visually structured) music learning
  2. Gamified music learning
  3. Music-driven gaming

Potential Crowdfunder?

Pre-Crowdfunding Registration (* indicates required)
Email Format

Music-driven games are those where a music data stream has an impact on the game narrative equivalent to and on a par with user actions: in effect an additional control stream. There are many ways of exploiting this, from associating object colour, shape, family or other behaviour with any of pitch, octave, note position in octave, through to memory challenges involving note names in various cultures, or familiarity with individual parts or voices.

It's been a challenge writing this post. My 2010 MacBook Pro is on it's last legs, it's graphics card toast. Hence, incidentally, so few screenshots.. Oh for a 💚 sponsor 💚 to bridge the crowdfunding ramp-up gap. :-)

So why, if our main focus is the desktop, it's generous screen real estate and better bandwidth, the browser and 2D visualisations, should we interest ourselves for 3D?

For one, visually simpler forms of 3D are reasonably easily integrated into our aggregator platform's envisaged Theory Tools arsenal (see the Theory Tools menu below the banner image, or the related Pinterest pages). Moreover, even visually less structured WebGL-based 3D needs to get it's musical data from somewhere - preferably in real time, and clearly sliced and diced.

BTW, though it doesn't touch on other than navigational interactivity, here a gentle intro to 'real-time' rendering for the 3D web.

DOM or WebGL: Choices And Challenges

The browser's DOM provides us with an exceptionally powerful selection mechanism targeting each DOM element's id and/or class. These allow us not only to maintain behavioural consistency across a wide range of animations, but to finely target individual elements for change. This capability is more or less lost when we move across to WebGL.

It would be helpful if all graphical processing -including that associated with conventional 2D- could be done on the GPU. It turns out that without resorting to WebGL, this isn't so easy. There are libraries claiming to do this, but they don't yet appear to have gained much traction.

At it's simplest, you need only a browser to experience 3D on the web. Stripped of cosmetics and kept (from a rendering point of view) simple it can also be reasonably easy on the GPU.

To experience Web3D you -additionally- need a headset. With the arrival of high resolution and frame rate WebVR and WebAR capable devices, 'mobile-first' is losing ground to the notion of 'data first'. In the background, and despite web standards, the ongoing nuisance of proprietary technologies tying users to a particular hardware or software ecosystem.

The data extracted from MusicXML is a prime candidate, both within our aggregator platform and via public API, as an additional driver of dramatic, gaming-style virtual storylines. We're talking on-demand, freely transformable, musical data: making (for example) music notation bendable, hideable, peek-a-boo visible, twistable, flippable, riot-of-colourable, capable of being turned into jungle flowers, racing cars, dancing dolphins, rolling waves, or classic Disney-style songbirds on telegraph lines. Rigorous method and learning needn't be at odds with imagination and whimsy. Everything, really everything, must be data-driven.

Transformations as part of an ongoing and developing storyline are one thing, responding to user actions in real time another. The next challenge? Instant, onscreen, programmatic reconfiguration according to changing user needs. For instruments, for example, the ability to freely change core music visualisation characteristics such as number of notes per octave, temperament / intonation, or (lute family) to add a capo.

Another challenge: where 2D SVG (used for instrument models, see menus at head of page) is prepared on the CPU, embedded-canvas, WebGL-based content (potentially, 3D theory tools: again, see menus), on the GPU, can both be synchronised via hardware clock, using (for example d3.js)? I certainly think so.

The last challenge? Whether, under some dual DOM/WebGL scenario (of which there are a few) the system as a whole can maintain synchronisation under load, such as for example between animations and their associated audio. For want of a better term, think 'load balancing'. For me, it's a bit of an unknown.

So, in summary, with these clearly in mind:
  • Selection for visual consistency
  • Graphical processing (including conventional 2D) if possible on the GPU,
  • Real-time, data-driven visualisations
  • .. and hence on-screen reconfiguration
  • hardware clock synchronisation across 2- and 3D technology stack branches
  • 2D and 3D load balancing
..we are ready to start thinking about their application. Is your goal teaching and learning transparency - or gaming and infotainment? Does 3D offer any gain over 2D in the browser? Best thought through carefully before prototyping.

DOM or WebGL: Technology Review

Our world music aggregator platform supports anything that can be dynamically loaded and run in a single page application, and javascript libraries can wrap just about anything. So: pure DOM? HTML canvas? WebGL on the DOM via canvas? WebAR or WebVR on the DOM? With d3.js or it's declarative wrapper, Vega? R? XTK? X3DOM or XML3D? What actually works - and how can we narrow the field?

Incidentally, I'm just curious as to future animation integration possibilities - and though as adept a faker as the next, certainly no expert. If you spot mistakes, do let me know.



As a reminder, on the left, the (on the whole) declarative, browser DOM technologies, on the right the (on the whole) procedural / imperative 3D polygonal wireframe world of AR/MR/VR. The web DOM space is focussed (on the whole) on structured data and understanding, the VR space on ad-hoc, many-to-many-related objects, the dramatic experience, suspension of disbelief and immersion in the storytelling flow. As concepts go, poles apart..

As can be seen, the Unity (and, by proxy, other) content delivery platforms for 3D games, augmented and virtual reality can also target (export content to) the WebVR, and, increasingly, WebAR stacks.

Big, brave, open-source, non-profit, community-provisioned, cross-cultural and kickass crazy. → Like, share, back-link, pin, tweet and mail. Hashtags? For the crowdfunding: #VisualFutureOfMusic. For the future live platform: #WorldMusicInstrumentsAndTheory. Or simply register as a potential crowdfunder..

3D Web Closer Up


Let's immediately distinguish between the term Web3D (referring to all interactive 3D content embedded into web pages html, commonly powered by WebGL, that we can see through a web browser, and often known by one of WebVR or WebAR) and Augmented and Virtual Reality (A & VR), which tend to run on dedicated devices.

VR objects are modelled, textured, and rendered in lengthy pre-production workflows using (in many cases free) 3D modelling tools such as Maya, Unity, Unreal and Blender. WebVR and WebAR are an attempt at bridging the gap - directly in the browser.

Web3D's convenience lies in lower development overhead and that WebVR and WebAR can be experienced both in the browser, yet also on many full-blown VR devices. The buzz? Faster delivery, more users. The biggest obstacle is the lack of native inputs, significantly impeding user interaction. Indeed, many examples handle the VR's "six degrees of freedom" (forward/back, up/down, left/right, yaw, pitch, roll) beautifully, but can end up leaving object data content inaccessible. As affairs go, "it's complicated".

Meanwhile, with WebGL now widely supported by default in modern browsers, further technologies such as X3D, X3DOM, Cobweb, three.js, glTF, ReactVR and A-Frame VR are steadily extending 3D capabilities within the web stack.

Can these match or enhance the 2D visually transformative capabilities of D3.js + SVG? Let's take a little stroll from left to right through the possibilities.

WebGL

WebGL is founded on OpenGL, and provides a base for a small range of 3D modelling libraries, such as Playcanvas, Turbulenz, BabylonJS and Three.js. As WebGL is canvas-based, the graphical sophistication is not on a par with SVG, but for solids and planes, the results can still be impressive.

The Role of D3.js

In the conventional 2D web, SVG graphics elements can be scripted with JavaScript for CSS and DOM interactivity (onclick, onmouseover etc). Javascript is, however, procedural. Vega, a 'visualisation grammar' (and wrapper for D3.js), provides us with a declarative, data-driven interface. "One unique aspect of Vega is its support for declarative interaction design: instead of the “spaghetti code” of event handler callbacks, Vega treats user input (mouse movement, touch events, etc.) as first-class streaming data to drive reactive updates to a visualization".

With it's declarative and reactive approach, Vega hints at the possibility of HTML-embeddable and highly interactive scores.. More on that in a later post.

For the 3D web, D3 eases manipulation of the likes CSS, XML3D, X3DOM and CSS3D, but can also be used in conjunction with web component, routing and XHR libraries such as React, Vue and Mithril.

D3.js or Vega can, incidentally, be used directly in conjunction with WebGL or (fallback) canvas.

Three.js

Three.js is a cross-browser JavaScript library/API used to create and display animated 3D computer graphics in a web browser. Three.js uses WebGL.

In an enablement sense, Three.js is to WebGL as D3.js is to the DOM. As with D3.js, it spawned a remarkable ecosystem of derivation and enhancement libraries, some with substantial followings of their own.

CSS

HTML describes structure and CSS described visual presentation. (In the diagram I've grouped CSS under 'Structure' - in the sense that it forms part of the arsenal of tools surrounding the highly structured browser DOM).

CSS has something of a reputation as a 'poor cousin' in the animated graphics world, but is a powerful tool in maintaining style consistency across a wide range of graphical objects. Moreover, should the choice be made to manipulate CSS styles directly in code rather than using style sheets, this can be managed with a high degree of control within D3.js, via the style attribute in conjunction with class and id attributes.

In effect these allow CSS styles to be inherited much as in the accustomed style sheets. You can get a feel for the simplicity and elegance of CSS use in conjunction with D3.js here.

In recent times, CSS has been the focus of more advanced graphical tools development. Here, for example a tool for advanced CSS path clipping.

SVG

Scalar Vector Graphics (SVG) is a declarative language associated with seamlessly scalable (pin-sharp at all sizes) graphics. Basic shapes such as those used in our instrument and theory models are supported across pretty much all browsers.

With all the overhead of the underlying DOM manipulations, however, the CPU is quickly overwhelmed by SVG processing, yet getting the processing to execute on the GPU can be quite a challenge. In some respects, the overheads mirror those of Web3D pipeline workflows.

SVG is more suited to limited but data-rich animations, such as interactive maps, board games - and of course music visualisations. The range and complexity of graphics that can be produced is huge - and they scale seamlessly to any size. SVG is the basis of the aggregator platform in focus here.

Incidentally, conventional 2D graphic (web) designer's animation tools (used, for example, in the non-programmatic creation of SVG art) are unsuited to the creation of dynamic, data-driven music visualisations as in focus here, and of only very limited use in a VR or, indeed, WebVR context. Nevertheless, sophisticated 2D programmatically controlled visualisations are well established in the web browser.

The main problem with a SVG (here D3.js-driven) is there is currently no way of forcing the DOM and graphical donkey-work to be done on the GPU. With all the overhead of the underlying DOM manipulations, the CPU is quickly overwhelmed. To execute on the GPU, D3.js can be used in conjunction with WebGL or (fallback) canvas, but thereby loses much of it's visual expressiveness.

SVG usage is likely to increase as (especially mobile device) CPU and GPU clout improves in the wake of the current augmented and virtual reality experimentation. The real breakthroughs will come if and when DOM and WebGL technologies are successfully brought together on a workable scale, and if the W3C can revitalise the ongoing SVG standardisation roadmap.

Mozilla maintains a page providing guidelines for SVG coding, with the aim of cutting down on client-side processing overheads.

Dec3D (Declarative 3D): With XML3D Or X3DOM

Dec3D (Declarative 3D) is a collective name for XML3D Or X3DOM technologies, and is described here.

xml3d

Judging by the images to be found using Google search, xml3d seems to have found favour as a modelling language in manufacturing industries, but does not seem to have caught on amongst web developers. The online examples, though promising, are few.

In the current implementation, xml3d is delivered as a javascript file, xml3d.js, and runs on any browser with WebGL support. The documentation (a Wiki) appears concise, sufficient and well structured.

xml3d supports JSON and (needless to say) XML input formats, the latter of which raises some interesting thoughts in connection with MusicXML.. :-)

With support for events, I see no reason why xml3d should not be used in conjunction with D3.js to create comparatively complex and interactive animations.

x3dom

For the web, such a thing does indeed exist - in the form of x3dom. D3.js's creator Mike Bostock was quick to demonstrate the union of D3.js and x3dom. Others followed, culminating in examples that begin to show promise in the context of lightweight, graphical music theory modelling. More examples here and here.

As the x3dom web site puts it:
"The goal here is to have a live X3D scene in your HTML DOM, which allows you to manipulate the 3D content by only adding, removing, or changing DOM elements. No specific plugin or plugin interface (like the X3D-specific SAI) is needed. It also supports most of the HTML events (like "onclick") on 3D objects. The whole integration model is still evolving and open for discussion.

We hope to trigger a process similar to how the SVG in HTML5 integration evolved:
  • Provide a vision and runtime today to experiment with and develop an integration model for declarative 3D in HTML5
  • Get the discussion in the HTML5 and X3D communities going and evolve the system and integration model
  • Finally it would be part of the HTML5 standard and supported by every major browser natively"
While not providing quite the control we have using SVG, it does give us access to CSS styling and events, is declarative, part of the HTML document - and hence integrated with the DOM.

CSS3D

Despite early interest in and some interesting experiments in conjunction with D3.js, CSS3D has yet to really take off. Though very good at handling simple pythagorean and platonic solids, perhaps the challenges of putting together complex shapes has discouraged it's use in comparison to the wireframe shapes of WebGL. With the arrival of CSS3D graphical editor such as Tridiv, perhaps that will change.

For the meantime, to see what's possible on CSS3D's cutting edge, wrap yer jellybeans round this. A couple of years old, but still seething with promise.

There have been numerous attempts at creating libraries of basic shapes - either via web components, or using visualisation libraries, as in the case of d3-x3dom-shape.

Declarative? Procedural/Imperative?

Before taking a look at finer-grain developments in the 3D web, a look at the distinction between declarative and procedural (a.k.a. imperative) languages. Just skip this block if you are comfortable with the distinction.

Declarative approaches describe what you want to do, but not how you want to do it. In a graphics context you would simply state your graphic object's properties, such as position, shape and size, the rest being done for you. In 'hiding' unnecessary detail, this can lead to very much more compact code, but is often associated with a steeper learning curve.

Both Vega (a wrapper library for D3.js) and Scalar Vector Graphics (SVG) are declarative languages. Given their succinctness and power, it is little surprise that they are associated respectively with complex data-driven applications, and high-precision graphics.

In a 3D DOM context, declarative approaches include those of XML3D, X3DOM and CSS3D. Of these, X3DOM appears to have found the most resonance in conjunction with D3.js / Vega. CSS3D is adequate for manipulating simple pythagorean and platonic solids, but there are no facilities for handling more complex graphics objects such as Bezier curves.

A procedural approach executes a series of well-structured -and often fairly detailed- steps. In a graphics context, a procedural language allows you give commands to draw from scratch - as if directing a paint brush's moves. Though the workings can be more transparent, reusability and integrity can more difficult to manage.

Though it's core technologies (HTML and CSS) are declarative, the HTML 'canvas' element is a fast, procedural / imperative interface to bitmap-based, low-level graphics. JavaScript is used in conjunction with Canvas to paint things to the screen. It offers little in the way of graphical elements, and these do not scale well. Suited to bitmapped games, Canvas found use in Soundslice, a fast and successful synchronised notation/instrument/video learning environment for a few of the most popular musical instruments and genres.

A declarative approach is perhaps half as verbose as an imperative one, but can easily take twice the time to master. The results, however, are much more elegant and easily supported code.

Finer-Grain 3D Web Focus

Finally, we try to get a handle on the more recent and specific initiatives in the 3D web space, and especially those of possible use in a music context.

In many cases, these build on or combine the technologies already introduced. As it's easy to get lost in the forest of experimental libraries, we'll stick to the main ones. That means some of what you see in the diagram may find no mention below.

In contrast to augmented and virtual reality, which is often build on platform- rather than web-oriented languages such as C#, the 3D web space tends to be stitched together with javascript.

This is an important distinction, both in exposing us to a vast, rich and growing development ecosystems such as web components (shadow DOM and reuse), but also wider developer availability. Moreover, whatever happens in the more rarified AR / VR space, web connectivity is likely to remain central.

ReactVR

Working from the bottom up, we quickly arrive at the first newcomer, ReactVR (or React VR).

Much as D3.js allows us to build complex dynamic and interactive graphics on the browser's DOM, ReactVR allows us to build and position 3D graphical components in a 3D space. From the initial React VR release statement: "Expanding on the declarative programming style of React and React Native, React VR lets anyone with an understanding of JavaScript rapidly build and deploy VR experiences using standard web tools. Those experiences can then be distributed across the web—React VR leverages APIs like WebGL and WebVR to connect immersive headsets with a scene in a web page. And to maximize your potential audience, sites built in React VR are also accessible on mobile phones and PCs, using accelerometers or the cursor for navigation.

With ReactVR, you can use React components to compose scenes in 3D, combining 360 panoramas with 2D UI, text, and images. You can increase immersion with audio and video capabilities, plus take full advantage of the space around you with 3D models. If you know React, you now know how to build 360 and VR content!" ReactVR basically allows front-end developers currently creating websites to transition to creating WebVR sites.

A-Frame React

In principle this is two topics in one. A-Frame is a web framework built on top of the DOM, which means web libraries such as React, Vue.js, Mithril.js and d3.js sit cleanly on top of it.

A-Frame is an entity-component-system (ECS) framework exposed through HTML. (ECS is a pattern used in game development that favors composability over inheritance, which is more naturally suited to 3D scenes where objects are built of complex appearance, behavior, and functionality).

In A-Frame, HTML attributes map to components which are composable modules that are plugged into s to attach appearance, behavior, and functionality.

A-Frame provides an actual DOM for React to reconcile, diff, and bind to.

aframe-react is a very thin layer on top of A-Frame to bridge with React, and with it web components and reuse.

aframe-react lets A-Frame handle the heavy lifting 3D and VR rendering and behavior. A-Frame is optimized from the ground up for WebVR with a 3D-oriented entity-component architecture. And aframe-react lets React focus on what it's good at: views, state management, and data binding.

Pixi.js

From the Pixi.js website: "The aim of this project is to provide a fast lightweight 2D library that works across all devices. The Pixi renderer allows everyone to enjoy the power of hardware acceleration without prior knowledge of WebGL. Also, it's fast. Really fast".

Accessed through a data-driven visualisation library such as D3.js, it can be coerced into visually detailed and expressive behaviours. Certainly, for the simple shapes employed in our instrument models, Pixi.js looks like a viable alternative to SVG - with the advantage of being processed on the GPU. This is definitely something I'd like to see tested with an eye to use for our open source animations templates.

Pixi.js has been used within Google Chrome's own Music Lab (showcasing experiments with the Web Audio APi) to provide fast and smoothly animated graphical elements.

For all that, though, the range of basic shapes offered is limited.

Hybrids

To round off, attempts have been made (for example html-gl.js) to get DOM-located graphics such as SVG to be be calculated on the GPU rather than the CPU, or to bring powerful but essentially platform-dependent solutions to the web (RGL, xtk.js).

At the end of the day, a data-driven notation feed can be both to DOM and Web3D, but the way they are applied will differ radically. Notation, instruments and theory on the DOM, action, adventure and whimsy on the other. In both, however, a clear chance to better anchor music learning.

My gut feeling is that D3 can add a level of sophistication to pretty much any of the tools covered in this post. That is valid as much for SVG as WebGL based approaches. It's worth taking a look at the various 2- and 3D animation demos available on the internet. A number of examples bring remarkably similar results

You can gain a little more insight on the transition from 2D web to 3D virtual reality from the user's perspective from this thesis publication.

There are countless finely nuanced technical approaches other than those touched on above, and as we know, wide-spread adoption is sometimes just a matter of luck. A spectacular example, the right moment, a sympathetic and key information broker, a surge of interest in some related field. Nevertheless, if there is one thing we can take away from the last few years experiences with frameworks, it is that less is generally more.

The Future: OpenXR And The 3D Portability API

Producing content for use across multiple environments is a pain. Happily, for Web3D, VR and AR, there is some hope of the various platform APIs being homogenised and unified under a common 'OpenXR' layer, and for these to form the basis of future javascript libraries allowing cross-platform development.

OpenXR will offer two APIs, one application-facing, the other device-facing, acting (depending on viewpoint) as an integration or defragmentation layer for VR and Web3D.

OpenXR will be augmented by a so-called 3D Portability API intended to harmonise various hardware interfaces, which, together with initiatives around open source compilers and translators is hoped will lead to an updated 'WebGL Next'.

Why all the blether? WebGL 2.0 has already brought desktop and mobile platforms 3D capabilities much closer together. WebGL Next is expected to continue this trend.
Read More