HTML 5 Video with a Chroma Key (and a bit of Parallax Scrolling)

2010-09-04T22:00:00.001+01:00

There’s a lot of talk of HTML 5 bringing the death of Flash, Silverlight and their kind. I’m not so sure, but it might make Flash justify it’s currently excessive use.

One of the things that Flash does do well, and HTML 5 may compete with, is video playing. However rarely does a Flash based website just play video. The OlogyWorld website, supporting the Ology series of children's books, (most famously Dragonology,) is a example of a Flash website using video with a bit more to it than just playing video in a box. There are two different effects involved on the OlogyWorld website: Chroma keying, generating the transparency in the video; and parallax scrolling, i.e. several layered images scrolling at different speeds to give the impression of depth. Both of these are possible in HTML 5.

Parallax Scrolling

Parallax scrolling has been around for decades - check out pretty much any 16 bit console games on something like the Super Nintendo and you’ll see it in action. Several layers scroll at different speeds, the further the layer are supposed to be from the viewer or camera, the slower it scrolls.

Parallax has been accomplishable since HTML 4 with reasonable ease. It just requires a div acting as a viewport with it’s overflow set to hidden and a number of images inside it with their position set to absolute. By setting the left margin of the images they can be positioned so the viewport clips them to only show the require part. It’s then just a matter of updating the margins at different rates to scroll the images within the viewport div and create the parallax effect.

The HTML for three parallax layers in a view port:

   1: <div class="Viewport">

   2:     <img src="Layer0.png" width="600" />

   3:     <img src="Layer1.png" width="1000" />

   4:     <img src="Layer2.png" width="1600" />

   5: </div>

The CSS to define the viewport size and set the parallax layers to absolute position:

   1: div.Viewport

   2: {

   3:     width: 500px;

   4:     height: 400px;

   5:     position: absolute;

   6:     overflow: hidden;

   7: }

8:

   9: div.Viewport img

  10: {

  11:     position: absolute;

  12:     display: block;

  13: }

And the JavaScript to animate it all:

   1: var nViewportWidth = $("div.Viewport").width();

2:

   3: var arrLayers = [];

4:

   5: var oLayerSet = $("div.Container div.Viewport > *");

6:

   7: for(var nIndex = 0; nIndex < oLayerSet.length; ++nIndex)

   8: {

   9:     arrLayers[nIndex] = {};

  10:     arrLayers[nIndex].node = oLayerSet[nIndex];

  11:     arrLayers[nIndex].width = $("#" + oLayerSet[nIndex].id).width() - nViewportWidth;

  12: }

13:

  14: // When the mouse moves, position the parallax proportional to it's horizontal position in the HTML document

  15: $(document).mousemove

  16: (

  17:     function(e)

  18:     {

  19:         var rPageX = e.pageX / $(document).width();

20:

  21:         for(nIndex = 0; nIndex < arrLayers.length; ++nIndex)

  22:         {

  23:             arrLayers[nIndex].node.style.marginLeft = "-" + (arrLayers[nIndex].width * rPageX) + "px";

  24:         }

  25:     }

  26: );

If you haven’t see code like $(“div.Viewport”) in JavaScript before then you should definitely check out of JQuery. It’s basically a library that wraps a lot of common JavaScript functionality - and in doing so removes all the nasty cross-browser compatibility issues. The $ is just a function call - in JavaScript $ is a string character just like any letter so a function can be called $. $(“div.Viewport”) returns the element, (well sort of - it’s actually the set of elements,) of type div and with a class name of Viewport - just like a CSS selector.

The first 12 lines are just set up - grabbing the parallax layer images and storing them in an array. Line 15 onwards attaches an event to the mouse’s movement, (see how easy JQuery makes this?) When the mouse moves the parallax layers are moved with it - at different speeds proportional to their width creating the parallax scrolling effect.

Video with a Chroma Key

HTML 5 introduces the video tag - which allows video to be embedded in an HTML page without requiring external plug-ins such as Flash. Unfortunately it’s a little limited - the video doesn't currently support an alpha channel. There is also the slight pain-in-the-bum of different browsers requiring different video formats. Licensing issues I believe.

The different video format requirements can easily be solved - although it’s a hack because it does require the video to be encoded twice. Firefox happily plays OGG video and Safari plays MP4 video, so the solution is to add both sources to the video element. Each browser will play the format with which it is compatible.

   1: <video>

   2:     <!-- Video for Firefox -->

   3:     <source src="VideoFile.ogg" type="video/ogg" />

   4:     <!-- Video for Safari -->

   5:     <source src="VideoFile.mp4" type="video/mp4" />

   6: </video>

This video cannot have an alpha channel - it will always play as a solid rectangle. This is where a chroma key is useful. A chroma key is green, (or blue,) screening - filming something in front of a background colour that can later be removed and replaced with something else.

When playing a chroma keyed video we can grab a frame using JavaScript and copy it onto a HTML 5 canvas. Once on a canvas it can be manipulated and the chroma key colour replaced.

The actual manipulation is fairly simple: Loop through each pixel on the canvas, if it’s colour matches the chroma key colour, (e.g. green, #00FF00 or 0, 255, 0,) set it’s alpha value to zero making it fully transparent. Once this is done for each pixel in the frame on the canvas the whole canvas is copied to a second canvas element - which is where the image with it’s alpha channel data in place is actually displayed to the user. A second canvas is required because the frame data in the first canvas contains the chroma key colour at the start of the frame, then is systematically updated. This process needs to be hidden from the user so only the final, updated, pixel data for the frame is presented.

The HTML for the two canvases:

   1: <canvas id="WorkCanvas"></canvas>

   2: <canvas id="PresentationCanvas"></canvas>

The JavaScript which sets up the canvases to the size of the video and invokes the function to process each frame when the video is played:

   1: var oChromaVideoSet = $("#" + strChromaVideoID);

   2: var elmChromaVideo = oChromaVideoSet[0];

3:

   4: var elmPresentationCanvas = $("#PresentationCanvas")[0];

   5: var oPresentationCanvasContext = elmPresentationCanvas.getContext("2d");

6:

   7: var elmWorkCanvas = $("#WorkCanvas")[0];

   8: var oWorkCanvasContext = elmWorkCanvas.getContext("2d");

9:

  10: oChromaVideoSet.bind

  11: (

  12:     "play",

  13:     function()

  14:     {

  15:         nChromaVideoWidth = self.elmChromaVideo.videoWidth;

  16:         nChromaVideoHeight = self.elmChromaVideo.videoHeight;

17:

  18:         elmWorkCanvas.width = elmPresentationCanvas.width = nChromaVideoWidth;

  19:         elmWorkCanvas.height = elmPresentationCanvas.height = nChromaVideoHeight;

20:

  21:         //elmChromaVideo_Tick function process the video

  22:         elmChromaVideo_Tick();

  23:     }

  24: );

The JavaScript tick function which processes the video. Note that it isn’t tied to the frame rate. I don’t think this is possible as of yet so it just repeats as fast at it can.

   1: // Avoid using local variables in the tick function because after a while the garbage collector struggles and kills performance

   2: var oFrame = null;

   3: var nR = null;

   4: var nG = null;

   5: var nB = null;

   6: var nIndex = null;

7:

   8: function elmChromaVideo_Tick()

   9: {

  10:     if(!(self.elmChromaVideo.paused || self.elmChromaVideo.ended))

  11:     {

  12:         // Grab the current frame of video into the work canvas

  13:         oWorkCanvasContext.drawImage(elmChromaVideo, 0, 0, nChromaVideoWidth, nChromaVideoHeight);

14:

  15:         // Get the pixel data from the work canvas

  16:         oFrame = oWorkCanvasContext.getImageData(0, 0, nChromaVideoWidth, nChromaVideoHeight);

17:

  18:         // Loop though pixel data. Each pixel consists of 4 element sin the array, RGBA

  19:         for(nIndex = 0; nIndex < oFrame.data.length; nIndex += 4)

  20:         {

  21:             nR = oFrame.data[nIndex];

  22:             nG = oFrame.data[nIndex + 1];

  23:             nB = oFrame.data[nIndex + 2];

24:

  25:             // Check pixel colour for the chroma key and set alpha to 0 (transparent) if equal to chroma key range

  26:             if (nR == 0  && nG == 255 && nB == 0)

  27:             {

  28:                 oFrame.data[nIndex + 3] = 0;

  29:             }

  30:         }

31:

  32:         // Draw the pixel data to the visible canvas

  33:         oPresentationCanvasContext.putImageData(oFrame, 0, 0);

  34:     }

35:

  36:     // Call the tick again, but without causing infinite recursion

  37:     window.setTimeout

  38:     (

  39:         function()

  40:         {

  41:             _elmChromaVideo_Tick();

  42:         },

  43:         0

  44:     );

  45: }

And finally a line of JavaScript to play the video:

   1: elmChromaVideo.play();

And that’s basically it. The two bits of functionality, the parallax scrolling and the chroma keyed video, need stitching together - although that’s pretty simple because the final presentational canvas element of the video exists in the DOM and obeys CSS so can easily be dropped in to the parallax as a new layer.

Below is a complete example. Before you click on the link however, I’m going to qualify the result with a few points:

The scrolling is just set up to move based on the mouse position. The original Flash of the OlogyWorld website uses some form of tweening to make the scrolling smoother - it accelerates and decelerates to the correct position rather than just tracking the mouse exactly. Functionality like that isn't too complex to write - it’s just not included in this example.
The chroma key video is just a random video. The actual chroma key colour, (in this case green,) has quite a bit of variation - so the edge of the transparency isn’t perfect. In a real world situation it would be worth cleaning up the video to make sure the chroma key was as close to a perfect colour as possible before using it on a website.
And perhaps most importantly: you need a compatible browser. I’ve tested this in Firefox 3.6.8 and Safari 4.0.5. It’s a little flaky in Safari - sometimes the page needs refreshing before the video plays. I’m sure this is solvable, I suspect that the play action may need delaying until the video has begun to download or something similar. It will not work in any version of Internet Explorer because of it’s lack of HTML 5 support. Role on IE9 and let’s hope it has full canvas support.

Click here for the example.

A final thought - if JavaScript can process each pixel in a video then I’m sure there is much more cleverness to come yet. Specific chroma key colours could be used as anchors - maybe to attach images or even other videos to; or colours adjusted on the fly. In essence videos can be customised in real time.

Grab the source here.

Notes by a geek

HTML 5 Video with a Chroma Key (and a bit of Parallax Scrolling)