3D Guitar Player Pro – a deeper look

I realized, later of course, that I hadn’t even gone over the guitar technology and explained *how* I created it. I was so wrapped up in writing code for the conference that I completely spaced any discussion on it. Well, let’s make that up, shall we? Hopefully, this will make it up to all the people that actually showed up 9am Sunday morning for the show! Please forgive me for not going over the details right there!

A BIG HUGE THANK YOU TO LUKE HUBBARD AND JOACHIM BAUCH – I bugged and bugged and bugged them to help me make this thing work and they came through for me like you wouldn’t believe. They’re both VERY nice people and extremely talented developers. They’re also core developers on the Red5 team since day 1 of it’s creation.

First off, I used a Roland Guitar setup consisting of the GK-3 pickup, the GI-20 midi processor. If a Roland guy is reading this, I REALLY could use a sponsorship! This is turning into a real roadshow :)

GK-3GI-20

Next, I used the midi application with Red5 that everyone now gets when the install Red5 server. Just connect to “midiDemo” with a NetConnection object, and bingo, you get back a list of compatible Midi in and out devices (separate lists).

[as]
var nc:NetConnection = new NetConnection();
nc.connect(“rtmp://localhost/midiDemo”);[/as]

When the connection shows successful, you can then request the midi lists:
[as]nc.call(“getMidiInDeviceNames”, new Responder(updateMidiInDeviceList));
nc.call(“getMidiOutDeviceNames”, new Responder(updateMidiOutDeviceList))[/as]

[as]private function updateMidiInDeviceList(data:Array):void
{
deviceListIn = data;
}

private function updateMidiOutDeviceList(data:Array):void
{
deviceListOut = data;
}[/as]

Next, you connect to your devices by telling Red5 which 2 you’d like to use:
[as]public function setMidiDevice(midiDeviceIn:String, midiDeviceOut:String):void
{
nc.call(“connectToMidi”, null, midiDeviceIn, midiDeviceOut);
}[/as]

At this point, you’ll have 2 lists: 1 for Midi In devices, the other for Midi Out devices on your system and you’ll be able to receive midi events from Red5 ;)

Finally, you’ll receive midi in calls by creating a “midi” method and be able to send midi messages via sendMidiShortMessage3() call to Red5:
[as]public function midi(time:Number, data:Array):void
{
dispatchEvent(new MidiDeviceEvent(MIDI_MESSAGE, data));
}

public function sendMidiMessage(note:Number, velocity:Number):void
{
nc.call(“sendMidiShortMessage3″, new Responder(handleMidiSendResponse), 144, note, velocity, 0);
}[/as]

This setup is EXTREMELY fast and works well. I was playing close to 64th notes on the guitar and Red5 just smiled mildly ;) AS3/Flex2 provide a MUCH better rendering experience than the AS2 drum demo I created.

Papervision3D provided the next valuable chunk to this project. The idea was to create a learning tool for learning chords, scales and patterns people use on solos. The absolute mind blow of this thing is that when you rotate the guitar to the “guitarists” view (over the neck from behind the guitar), it’s about 1000x’s easier to follow along and learn.

I had to create a decent guitar model in 3D Studio Max that was as low on the poly’s as possible. I needed the performance to be retained for the playback of the notes on the fret board. I was able to get a good looking model at about 646 polys and keep the neck, body and head separate.

3D StudioMax screenshot

All I had to do was create groups in 3DS and name them appropriately. Then save out as Collada file format. In Flex2, I was able to simply use the getChildByName(“neck”) and I’m able to hide/show the various parts of the guitar. It’s almost like creating flash movieclips, but I’m actually doing the work in 3DS!

Body onlyNeck only

Ok, now we’re actually getting to the cool part: The skin. This is what makes it all happen! Here’s the actual texture I started with. I had to go outside and lay the guitar down on a white towel on a cloudy day – that helped in getting a great diffused shot of the guitar! HUGE thanks to Peter Kapelyan on the PV3D list for giving me a million tips on how to photograph stuff for use as textures! He’s a genius!! To bad he uses Lightwave ;)

Guitar Texture

After I successfully mapped this bitmap to the guitar in 3DS, I was able to import the image into Flash CS3, and create a movieclip where I could then create my “Note” and “Fret” movieclips and supporting classes to handle the incoming midi notifications. I imported the skin, resized the stage to the dimensions of the skin (important since we’re going to use the Flash movieclip to skin the guitar, and it has to be the exact same size). Then proceeded to create a Fret movieclip and Note movieclip (which goes inside Fret). Fret contains 6 copies of Note – make sense? Then I just name them appropriately: fret_0 – 21, and then inside of Fret, note_0-5.

Flash CS3 Skin

Now, this is where you glue it together – the big question in your mind is: How did I communicate with the SWF that’s loaded for use with a PV3D skin AND register events with it -am I right? is that your question?! good question.

Thanks to Grant for coming along at the EXACT moment I was dealing with this (no, i’m not kidding, literally, I was JUST starting work on it and 5 minutes later, Grant pops up with an IM). He’d just released FlashLib for this very issue. So, all I had to do was select “Create FlashLib” from the commands menu in CS3, and BAM, it created the SWF and FlashLib.as class for accessing assets from the library. The Note class I created was completely in tact and accessible from Flex2! All I had to do was call FlashLib.getInstance(LinkageName:String) and that’s all I needed!

[as]var texture:DisplayObject = FlashLib.getInstance(“guitar”);[/as]

DONE. Now, I could use that with my Collada object with Papervision3D for the skin, AND I had all the access to the fret board that I needed to marry the midi note values to the proper frets.

I’ll be going over this and showing code at the classes (RMI) in June, so if you’re interested, you might want to book a seat now ;) I’ll also post a demo so that you can rotate the model around and see the interface.

About these ads
  1. This is the best and most interesting experimental Flash project i’ve seen for a long time. Fscking First Class. Mr. Grden, we salute you.

  2. Really nice explantion!!! I feel like fiddling, think I can plug in my kazoo somehow? j/k
    And…………………………
    Too BAD I use lightwave? haha that’s any oxymoron :)
    -Pete

  3. Thanks for the info, John. Right before you posted this, I had just got Red5 working with my Midi Controller/Keyboard(Axiom 49). I used basically the same methods, except I think the midiDemo Application I’ve got must be an earlier version, because it doesn’t have the “getMidiInDeviceNames” and “getMidiOutDeviceNames”, just “getMidiDeviceNames”. Either way, it’s very fun to play with! I was at your presentation at FITC and now I’m inspired to get a Godin xtSA and a GI-20… Awesome work!

  4. Hey Matt! excellent work! Yeah, the VERY latest version, which fixes the issue with sending midi messages OUT, is in the trunk, but I had thought it was included in V0.6 Final, which we just released.

    Here’s the link to the actual class in the repository:

    http://mirror1.cvsdude.com/trac/osflash/red5/browser/java/server/trunk/webapps/midiDemo/WEB-INF/src/org/red5/demos/midi/Application.java

    Then you’re calls will make sense :) I’ll probably post my MidiConnection class as well, just want to give it a once over and put it in the correct package location

  5. \m/ on Clark!

  6. *NICE*

    • Brad D
    • May 1st, 2007

    This is really cool and easy to use, but it appears that this doesn’t work under OS X. If I trace back the ins and outs I get this “deviceListIn = Real Time Sequencer
    deviceListOut = Real Time Sequencer,Java Sound Synthesizer”. On my PC I get those plus all the other hardware attached to the machine.

    Is there anyway to get this running under OS X?

    Nice presentation @ FITC by the way. I wish I would have went to your instead of the one next door (though you ould be heard through the walls ;). I guess I wasn’t awake that morning. Thanks for posting the vids though.

  7. It should work just fine – Luke Hubbard originally wrote this on OSX and tested both midi in and out and both worked from what he indicated.

    What devices do you have installed? Is there any initialization that has to happen with any of those devices? Red5 should be started AFTER all devices are up and initialized. Otherwise, you won’t see them in the lists.

    Thanks man! How loud WAS it next door? They came in and told us to turn down – which TOTALLY rocked \m/

    • Brad D
    • May 2nd, 2007

    I’ve got it triggering the Java Sound Synthesizer, but that just makes a generic piano sound. I still can’t seem to access any of my other devices. I restarted Red 5 several times. the devices show up in Audio Midi setup and can be triggered both ways in OS X. I have a standard M-Audio USB keyboard plugged in, and I have tried other USB midi controllers that I have, but no luck. I wonder if it may has something to do with the Intel build of the Java JRE (I’m on a MacBook Pro)…don’t know. I’ll have to spend some more time with it later.

    Do you know of any example source files or more in depth info on midi capabilities the Red 5? That would help me a lot. Thanks!

    As far as your noise complaints at FITC, they were very much deserved – YOU WERE SHAKIN
    THE WALLS! We could barely hear the guy yell over your screaming solo. I was clearly in the wrong room… rockOn \m/

  8. well, at least we have consistency – that’s the midi device that Luke tested with if I remember correctly. He doesn’t have any other midi instruments, so it wasn’t possible for him to test and I’m on XP, so neither of us had any clue if OSX would show other devices.

    They’re going to lynch me for starting something people apparently want ;)

    At this point, we need to find a java dev who’s somewhat familiar with the Javax api and has OSX AND has some midi devices to test with.

    Got anyone in mind? Anyone care to volunteer?

    As for source, I pretty well posted most of it and I was going to post the actual MidiConnector class I use. I have a Flex2 component I created (very very basic) for use with this stuff that presents the user with a TitleWindow and 2 list boxes for selecting your gear. There’s been nothing done in the way of error handling etc, so I kinda need to clean it up before posting

  9. Hey John, thanks tons for posting this. I’ve connected my MIDI controller / DJ Mixer (M-Audio’s X-Session PRO) and am starting to play around with mapping it to a nifty flash interface. Too much fun! I was wondering, did you end up posting that MidiConnector class you mentioned somewhere? Wanted to go through it for reference.

    Anyway, great work!

    Cheers,
    Greg

  10. John, you’re making me want to unpack my drum set from the basement and hook my Roland back up – Stop It! :)

  11. Crazy!! Cheers!

  12. hi, this is really great work! its inspired me with my Midi Flash endeavors. i’ve put together a small Air app that allows you to configure your Midi devices for flash… something that can be handy on top of your work.

    http://www.julapy.com/blog/2007/11/05/midi-config-air-application/

    • John
    • December 13th, 2007

    That guitar design is awesome! I want it for a real guitar!

    • John
    • December 28th, 2007

    Hi, great work John! I’m probably being stupid here but I’ve made something similar and am setting up a server for red5 to run on, do all the ‘clients’ need red5 too or will just running the swf with their midi input connected be sufficient? At the moment it’s looking pretty bleak i.e. I’ve tested it from another computer and it’s not even connecting to the server (I tried using both localhost and external IP values in the .as file)

    If anyone has any idea about what I’m rambling on about and can get back to me that would be fantastic, Thanks.

    • tim
    • February 28th, 2008

    Hi, Thanks for sharing this John!
    I had been using FlashMIDI proir to finding this, but i found the response too slow for what i needed.
    I have got this up and running so far with MIDI in, but i keep getting the error “Access of undefined property handleMidiSendResponse.

    • tim
    • February 28th, 2008

    .. when i try to get MIDI out working.

    Any help would be much appreciated.
    thankyou

    (sorry, that cut off half my comment)

  13. @tim: can you send me what’s in that method? have you tried commenting out each line in that method to narrow down which property this error message is referencing?

    • tim
    • February 28th, 2008

    Hi John, thanks for getting back. I’m not much of a developer, i’m another musician who likes messing about with flash , so you’ll have to forgive me :)

    i just copied your code (i’m in cs3 so it’s a little different, but everything else i have changed works fine. please tell me if i’m being an idiot..)

    function sendMidiMessage(note:Number, velocity:Number)
    {
    nc.call(“sendMidiShortMessage3″, new Responder(handleMidiSendResponse), 144, note, velocity, 0);
    }

    and i get the above error message.

    incidentally, when i use..

    function midi(time:Number, data:Array)
    {
    dispatchEvent(new MidiDeviceEvent(MIDI_MESSAGE, data));
    }

    ..i get two errors, “1180: Call to a possibly undefined method MidiDeviceEvent.

    • tim
    • February 28th, 2008

    ..i get two errors, “1180: Call to a possibly undefined method MidiDeviceEvent.

    • tim
    • February 28th, 2008

    and “1120: Access of undefined property MIDI_MESSAGE”

    but when i use the code copied from Lukasz Karluk’s page..

    function midi(time:Number, data:Array)
    {
    trace(“MIDI Test :: time : ” + time + ” data : ” + data);
    }

    (something along those lines, his page appears to be down since last night)
    ..i get a trace of the midi in note and velocity, which has served me fine for controlling Movie Clips

    i hope this is of some use to you!
    thanks for your help

    ps i don’t know why my comments keep getting cut in half :)

    • tim
    • February 28th, 2008

    sorry, above error message refers to..

    “1120: Access of undefined property number.

    • tim
    • February 28th, 2008

    sorry to flood your board with posts, but i’ve fixed it.
    i had neglected to make a function called “handleMidiSendResponse”

    i’m still getting a couple of errors, but nothing that seems to be affecting the performance.
    hope this might be of use to someone else!

    • paulo
    • March 10th, 2008

    Hi John, don’t suppose you could give us a quick note on how to send CC data instead of Notes please?

    thanks very much

  14. @Paulo: you mean using the Midi methods? Well, even if you didn’t mean that, I’ve not dealt with cending CC Data – did you mean CDATA?

    • paulo
    • March 10th, 2008

    hi, i meant MIDI Continuous Controller data, such as a pitch bend or modulation wheel on a keyboard would send, sorry i wasn’t clearer.
    do you know if this would be possible using Red5?

  15. Hello John,

    This is amazing… In fact, Ive always dreamt with the posibility of connecting an as3 project with an external midi/audio editing software like Ableton Live, Reason, Cubace etc and hence control Vst instruments from certain parameters in a flash animation. I managed to do this using as3 and a windows application called FlashMidi, but the lag was terrible, plus there seems to be nothing around to do it with as3. However if you could accomplish this, that means that you know of a way to accomplish what Im talking about.
    Could you PLEASE shed me some light in the matter? Note: I dont even know what Red5 is or does.

  16. @Li: Red5 is a Java based server that runs on Mac, windows, linux systems and is capable of listening and communicating with Midi devices. It passes Midi note information to and from the Flash Player vis RTMP connection (like streaming video). It’s fast and works great ;) If you install Red5 it comes with the midi application already. You can use the code references in this post to get a list of Midi devices from Red5 and start playing!

  17. Hey John, thanks for the reply. Ive been cracking my head with this. I managet to get the midi IN/OUT lists, which read:
    IN >> Real Time Sequencer
    OUT >> Mapeador Microsoft MIDI,Sint. SW de tabla de ondas GS,Real Time Sequencer,Java Sound Synthesizer
    Now what? I think that my problem is that I dont have a midi input device, so Im trying to simulate it in some way…

  18. you need to select the in and out that you want to use, then pass that to the server. there’s a method that accepts both of them, and I cant’ recall what it is at the moment. but here’s how to tell Red5 which one you want to listen to:

    1.
    public function setMidiDevice(midiDeviceIn:String, midiDeviceOut:String):void
    2.
    {
    3.
    nc.call(“connectToMidi”, null, midiDeviceIn, midiDeviceOut);
    4.
    }

  19. LOL never mind, I should have read that closely – that IS the method for setting both in and out ;)

  20. John,

    Thanks! I managed to get some sound out of the code!!
    Now, what I want to do is use flash to trigger sounds in Ableton Live, so my next logical step was to activate Mac OSX’s IAC Driver to set up a midi loopback and connect flash’s midi output to live’s midi input. This was done succesfully, flash sees the “IAC Driver – Bus 1″ output and live sees the “IAC Driver – Bus 1″ input… The problem is, Red5 reports an expeption when I trigger the notes from flash: “javax.sound.midi.MidiUnavailableException: IAC Driver – Bus 1 – no Receiver available”.

    Got any idea on how to solve this or what is happening?

  21. let me add something else…

    Im thinking that I need a more advanced java app than the one provided with red5 “midiDemo”. Plus, I suppose that when this problem is solved, I will also need to control more complex midi events than just triggering notes… I hope I can find something around there, otherwise I will have to go ahead and learn java :S

  22. It’s works quite well!
    I have been using it for 7 years now and It did make my life much easier!

  23. It does give a bit of troubles with latency but all in all it’s a good tool if you need to write a track and cannot play piano….like me

  1. April 30th, 2007
  2. October 23rd, 2007

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 794 other followers