Etiketter - Flash

Is it possible to synchronize sounds with good precision in AS2 to create a beat box? You want those bass drum sounds pounding on 1, 3 while digging a nice groovy tight hi-hat. And sync problems: no thanks! Lots of people have experienced problems, but read on to get some code and 7 strategies to get there. Or…browse to the end and download some code.

Some time ago, we faced the problem of building a little audio/video sequencer in Flash. With this sequencer the user would produce his/her own music video that was published on YouTube. As for video, there wasn’t much problems, but synchronizing audio turned out to be a real beast. Following the debate on Andre Michelle’s blog and http://www.make-some-noise.info/2008/04/01/response-to-a-bunch-of-questions-from-adobe/ Adobe hasn’t really been the Flash sound community’s dear friend recently. In short, they introduced severe bugs in Flash player 9.r115, but they changed the whole API for version Flash Player 10. With version 10 you can do really cool stuff writing directly to sound mixer buffers. In particular: see Hobnox’s Audiotool which is probably the coolest Flash application I’ve ever seen (ReBirth anyone? :-)).

Anyhow, unfortunately our project couldn’t go for Flash Player 10 as the market penetration wasn’t large enough. Even AS 3 was not an option due to platform restrictions. So, back to thinking in terms of the old API:s and playing around. The idea was for the user to click in a grid and play different samples, just like a beat box or drum machine. Furthermore, each sample had a video so, in effect, the user made a little video beat box loop. Ok, cool…but could we accomplish this?

After trying just naive Sound.play calls together with matching timer values from getTimer() and noticing quite large rythmic glitches I was disappointed. Clearly, we needed better timing information. But reading the article Advanced Flash MX Sound Timing made it clear. There is one exact timing for sounds in Flash, namely the Sound.position property. In order to use this for common timing information, we needed to play a really long sound, and then we could check where we were in this reference sound. As we’d see, we’d get position increases in chunks of 46.4 ms. It turns out that this corresponds to the buffer size used by Flash (2048 samples 44.1 KHz) and sometimes (NOTE: not always) when the soundcard calls back to get more data to play, Flash updates Sound objects’ position values. So I started a really long, totally quiet, sound to play in parallel for common timing, but it quickly struck me that a smart approach would be just to loop something like a 1 second sound. I’d found my first conclusion:

1. Use a silent reference sound and loop this sound. For each loop we can increase a counter and thus we can always reach the overall position. A reasonable length is probably 1 second. Make sure it’s mono by the way.

Ok great! Things were clearly better. Only - quite many sounds still came slightly wrong and misaligned musically. After thorough investigation, it turned out that when you say Sound.play to Flash, this doesn’t really happen immediately. Not even on the next sound card callback. But rather on the next sound card callback which Flash happens to like. (My guess: where some low prioritized task queue copy function for some other thread has completed).

2. So the solution is to use some silence in the beginning of the audio, and schedule the sound to play “in the future”. We thus introduce a little latency, but we really have the ability to sync sounds given a little latency of say 200 ms. If we schedule audio every 200 ms, we need at least a 200 ms silent part before each sound, and then we can just check the position of the reference sound and use Sound.start(OFFSET) to start a sound with a little offset that has been calculated from the position. This is actually much better explained at Advanced Flash MX Sound Timing. Great! But it turned out that there were still glitches.

3. Enter 1 short 1-sample silent mono sound that we don’t loop but re-trig all the time. Yes, just as it had been suggested by Advanced Flash MX Sound Timing and others. It turns out that callbacks at onSoundComplete are really good for starting new sounds, even though they don’t always come when the sound has actually finished (as pointed out by very disappointed people at http://www.make-some-noise.info). Anyhow…still great for our purposes! To be clear: the idea is basically to just start this one-sample sound and use its onSoundComplete handler. In that handler, we just trigger the little sound again. And so on. This way we get a callback on “good” times, instead of using setInterval or onEnterFrame. Super! So we now have a longer sync sound that we loop for position, and a shorter sync sound for good callback times. But…happiness didn’t last for long. Even though the situation had clearly improved there were still glitches every now and then. If you weren’t picky or perhaps playing some rather soft/smooth sounds, perhaps you didn’t care. But we had this smackin’ snare and hand clap we wanted to make a groove of and glitches aren’t grooves’ best friends.

4. It turned out some old fashioned pickyness, good system design and loads of asserts were bringing in some predictability. Starting a Sound in Flash isn’t just a few-instructions synchronous little thingie. It actually takes a couple of milliseconds. And by starting a lot of sounds, the master position could have changed, and then those recently started sounds were out of sync. Even on direct callbacks from the onSoundComplete handler previously mentioned the master clock sometime changed directly. Ok, so by just avoiding scheduling sounds at those positions seemed to do the trick quite well. But we now needed to increase the quite part in the sounds, since REALLY GOOD callback times occurred more seldom. Ok, still cool, but….still glitches. Shit! But some more coffee revealed a more serious approach to solving this.

5. Enter error detection. Yep, we could always check the positions of Sound objects that we schedule to see if they match what we expect them to have compared to the master position. This meant that we had to rewrite the scheduling mechanism. Instead of just using play, we now added sounds to a queue and then removed them when they had been deemed “OK” by some error detection handler. But…now it turned out the sync was really good. But…there were drop-outs. Yep, some sound could never be synced back, but an even more common problem turned out to be that it was really hard to read a sound’s position. Say you used the same kick drum sample on each beat, but when you were to read the position of that sound object, it might return the old sound object’s position and vice versa. This really exposed a nasty detail of the Sound API: namely that the Sound object plays two roles: to wrap some audio file and to provide information about currently playing material. A more clean solution more in line with how things work would probably be for the play function to return a PlayingSound (or similar) object which had the position. Anyhow, as that wasn’t available, we decided to build…

6. …a SoundPool class. The idea is to load a bunch, say 30, similar Sound objects in a pool. Whenever a new playing sound is needed, we could ask the pool for a “fresh sound” and the pool would just circle round the sounds and return them in circular order. How many sounds you need depends on how many sounds you play simultaneously, and also how long your silent intro is. We decided that ~30 was probably a quite good number. Notice that the browser cache would normally take care of not issuing more requests than necessary, but it’s probably wise to make the server return an Expiration Data in the HTTP header or similar for the mp3 files in question.

Now we had quite some solutions to deal with this. There was just the problem that when you filled the sequencer grid with beats (I know, not the grooviest of beats, kind of repetitive) sounds vanished and there was nothing heard. Aarrghh…going crazy again! When investigating this it turned out to be because of the sound channel limitation in Flash (currently 32 channels). So what happens is actually that if you have 32 channels playing and play another sound, one of the first sounds just stop. Unlucky for us all sounds start with silence, so we were listening to 32 channels of groovy silence. Not cool.

7. Enter cancel mechanisms on scheduling when too many sounds are playing. Yes - we adapted our scheduling mechanism so that when a sound was OK in sync we moved it to a playing collection and continuously removed sounds from that queue when they finished. Thus we could keep track of how many sounds were playing and cancel any scheduling attempts. We just simply had to wait for some sounds to complete. Sure, this meant that we had fewer times to sync audio, but it still worked quite well.

Ok, but there were still occasional drop-outs

Yep, we decided to live with it. It really happened quite seldom, and we set a limit on 100 ms drift to regard a sound as playable even though not in sync.

Next thing: sync with video. We needed to provide some video along with the sounds as well. We just used an onEnterFrame callback and calculated the approximate position. When we had the latest callback we saved the position using a getTimer() call and then we could just diff a getTimer() to the last call to calculate a continuous position which we could then use for video. Our video actually started 85 ms before the audio se we showed the corresponding video on the first frame ~85 ms before the sounds was going to be heard.

» Download Code

So - how can you use this code? Just check out the Main.run function which demonstrates an example. Furthermore, check each classes header to see some documentation in comments.

If you really want, I could probably include a demo too =) Just contact me or leave a comment below.

Good luck!

Staffan