September 22nd, 2011
This project was created for my Advanced VFX II class at The Art Institutes International Minnesota, one of the last core classes as I finish my senior year. In the class, we have the entire quarter to work on one shot. That may seem like a long time for one shot (which it is), but the bar is raised in that we have to do a “matchimation” type shot—3D geometry tracking and compositing. A common route is an arm or head replacement of a character, perhaps making them grow a reptilian arm.
Our instructor also gave us a few options to, perhaps, stir up the pot a little—ideas to pose an even greater challenge. One of them was performance-capture. If you’re unfamiliar with the concept, it involves tracking features of an actor that are shot with multiple cameras. The multiple cameras allow for the tracking software (in this case PFTrack) to triangulate the data between the multiple viewing angles in order to calculate where the points are in 3D space. I’ve been interested in performance-capture (or facial performance-capture) since I saw the DVD extras for “Pirates of the Caribbean: Dead Man’s Chest” where there was a brief breakdown of the Davy Jones character where the digital prosthetics were added to Bill Nighy’s face (“do ya fear death-nya?!”)
I asked my good friend Jose Ortiz to develop a character for my project, because although I claim to do everything, he’s far more talented in the area of concept art than I am. He dubbed the sketch “metalhead” which ended up sticking, as dictated by my project directory for the endeavor. The head was modeled in Maya (with the use of front and side orthogonals that Jose created after I approved the initial sketch) and further sculpted in ZBrush. I rigged the model with splines within Maya using influence curves, which allowed me to plan for the shoot. Knowing where the points were on my model, I then applied markers in the same locations on my actors face.
I decided that my friend Sean, who’s already-effervescent personality becomes even more lively on camera, would be the perfect fit for the character. My friend Charlie helped out on the shoot as well, and also documented various aspects with my phone’s camera. We shot with three cameras, and then I spent nearly an entire weekend tracking the three sets of footage, partly automated, partly manual “frame-to-frame.”
I had some trouble applying the animation to my model early on in the post-tracking stage of things. Many facial performance-capture endeavors involve head-mounted cameras to zero out head movement, and then re-apply it later. This was not the case for me, and so my tracking data had the head-movement baked into it as well. What saved me in “capturing” this head movement was the 2 points I drew on Sean’s face at the last minute as a “just in case” factor—one on his chin and one on his forehead. I created a cylinder, parent-constrained it to the forehead point and then aim-constrained it to the chin point which then gave me the XYZ rotation and translation data. I was then able to apply this to the head itself so that the tracking data alone didn’t rip the skin out of place as the animation progressed through time.
This was a great learning experience for me as a digital artist. I had a pretty good plan of attack from the get-go, and from the rather-minor problems I ran into throughout the course of the project, I feel well-equipped should I ever attempt another performance capture in the future (which I certainly hope to do)!