Wikia's own Director of Programming for Entertainment, Eric Moro, was part of a press interview in honor of The Avengers Blu-ray release. On the panel were Jeff White, Jason Smith and Marc Chu -- all members of the visual effects team from Industrial Light & Magic that worked on The Avengers.
Read on for excerpts from the interview!
Jeff White—Visual Effects Supervisor, Industrial Light & Magic
Q: When working with Mark Ruffalo, did you have that curve with him where he took some convincing, but then after he got into the process he started to realize how much [of the] performance he was actually going to be [contributing]?
Jeff White: I think so. You know the nice thing is we started with him coming up and just working on it talking with Joss, but not for the movie at all. Really all we were trying to get out of that day was a sort of training library of facial shapes so that we could accurately, you know recreate his kind of major expressions.
But for the rest of the day he got in the mocap suit and, you know we were doing kind of a live retarget onto an old Hulk model. Just so he could kind of play around and feel like what is this performance capture stuff really gonna turn into in the end. And then, you know I think it’s difficult on set, you know those first couple of days when everybody’s in cool costumes and you’re in the – you’re in the gray spandex.
And we actually – you know he had a tiara on his head that we used to track his head motion and then dots all over his face. So it’s hard – I would think it would be hard for any actor to kind of get beyond that. But once he got into I think it really started to work. We did – on set we also did a – we had a couple hours with him where we were just off to the side and had him run through the shots without any tracking.
It was just, you know just kind of wild performance and got some really great, you know reference material out of that. So at that point I think he was comfortable with it and we had the edit kind of sort of locked together by Octoberish. And then he came back up to Island again with Joss and they did the full like mocap suit, the dual head camera, which is its own sort of uncomfortable part of it. But by that time I think he was pretty comfortable with the process.
Q: Speaking of differences with the Hulk, you know the Hulk has been established on screen quite a few times before – what were you most eager or excited to change about the Hulk when you found out that was what you were gonna be dealing with? And how important was it for you to make the Hulk your own?
Jeff White: I think, you know we were really excited about the initial previews that we got because Josh clearly understood like the kind of performance people wanted to see the Hulk do. And I think for us, you know we’ve had a lot of developments in terms of the technology side of it, both in terms of performance capture and in terms of rendering and muscle simulation and everything. So for us it’s kind of this ultimate like okay, let’s throw everything at it that we’ve got.
And then my background is working as a character TD, so really kind of like rigging and simulation and anatomy and so, you know you can’t think of a better problem to try and solve than working on something like the Hulk.
So, you know we were very – you know I was the most worried about it, but when we saw the initial designs where we’re going with the more de-saturated scan and we’re incorporating Mark Ruffalo into it, that gave us some really tangible goals in terms of how we needed to make this Hulk different than the previous ones.
Q: If you were to do the Hulk right now to improve on him, what would you do at this moment – at this time?
Jeff White: I think, you know one of the things – a lot of the hair was difficult, especially, you know his hair design and how different it came out in each shot. I think there’s more that we can push as far as the simulation of his skin and the muscles underneath. And it’s always a matter of like the level of detail that we can incorporate into him.
And some of the shots, you know especially where he’s at a mid-ground, those are some of the toughest for us to solve in terms of, you know the sweat starts to look a little plasticy and things like that. So, you know if we can start from the basis that we have now – you know there were shots like the close-up smile where we got a long way, but we wanted to add little spit bubbles in between his gums and, you know things that we see in photo reference that there just wasn’t time for. So I think it would be great to have another opportunity to keep pushing that forward.
Q: What was the most challenging shot to complete in the film?
Jeff White: Definitely the tie-in – we called it the tie-in shot, but it’s essentially the part with all The Avengers characters in it together. It was right in the original previs and we knew it was gonna be huge. And when you see it, the only plates that we started with were Black Widow kind of riding on the chariot, Hawk Eye on the rooftop and then Thor hitting guys with the hammer at the end. Everything else we had to create including Captain America, only because we had shot plates of a real Captain America, but the camera move was too slow. And we wanted to – the momentum of the shot kind of died in there.
So a big problem of that one was keeping momentum. Like even Hawk Eye on the rooftop, as we come up there – they had built a set of the corner of the building but again, the camera kind of died there and the shot really slowed down. So we actually ended up replacing the set altogether with a CG rooftop and then had to roto him off there. And that one took from the beginning and we finished it the last day and it was right down to, you know the wire. It was such a huge shot to pull together.
Asker: It’s a great shot.
Jeff White: Yeah. Thank you. It was really – there’s – I don’t know if you saw the breakdown reel, but there’s a neat breakdown of it where you can see, you know how little is real and all the different – it was a great shot because we could really feature everything about New York that we’d built. You know all the destroyed cars and the fires and all that stuff. So it was a lot of fun.
Jason Smith—Associate Visual Effects Supervisor, Industrial Light & Magic
Q: It seems like one of the big challenges – obviously creating the Fantastic’s is something I love that’s gone very well for a very long time. But in recent movies like Dark of the Moon with Chicago or this movie in New York, you’re dealing with things that we are intimately familiar with from our every day life, and there is a higher degree of failure if you miss.
Jason Smith: Yes. I think that that is exactly true. And the fact that so much of this movie took place right in the viaduct in front of the Grand Central Station, such a recognizable spot. And another challenge along those lines was the Chrysler Building. You know everybody knows exactly what the Chrysler Building looks like. And how it’s supposed to react to light and everything. So creating the digital Chrysler Building really was kind of a challenge in that way that we had to get the warble of those panels just right, we had to have just the right amount of weathering.
And they’ve done kind of patching on it over the years: there’s like these little lines of tar that you don’t know about until you really start to look for them. But if they’re missing, you can kinda tell it’s just not really the Chrysler Building. So there were a lot of things like that, as we were working on the New York stuff, that we kept looking back at the pictures of New York to make sure that we were nailing it. And then asking our New Yorkers too: “is this okay? Does that look okay?”
Another thing that we did is we drove around a car with ultima arms sticking up filming footage of New York as we drove around. And we had that kind of exposed as good reference for what the exposure level should be and everything. And that was just priceless. Later, when we’d be doing shots, we could go back to that footage and say “okay, this building should actually be a lot dirtier. And this one over here should actually be a little bit cleaner.” And just use reality as a little bit of a guide there to try to make sure that we were doing it right. If you do everything from the hip, you can get something, but to do it really right and to really nail reality, you’ve got to be looking at the reference all the time.
Q: A couple of sites have reported, I think actually today, that they announced Avengers 2 will be coming out in 2015. How far along in the process is that at all, I mean even just conceptually? It’s actually something about we don’t even have details to share at this point. What is like your reaction to that, though?
Jason Smith: The reaction is yeah, I’m very excited. I mean because after going through this process with Joss and seeing that there’s somebody out there who understands how to take these characters that we all love and make it work without getting too serious or too light hearted about everything; just kind of really weaving it together in a way that you care about the characters, you can take them seriously but still have fun. I think the idea of having that happen again – especially now that we’ve been introduced to them all working together. You know, what kinda stories might happen? I think it’s very exciting.
Q: Well, one of the things that I think Marvel’s been pushing for is a sense of unity. So they’ve now got in four, five films before they could do the Avengers 2. Is it important for them to keep you guys involved in each of those films, as well, so that there’s a sense that the world that they’re building is the same world? That it still fits together? Because they’ve got such, I mean to do Guardians and Galaxy, Ant Man and pull all that together, seems like a huge trick.
Jason Smith: Yeah. I think Marvel really has their finger on the pulse of keeping all that stuff in line. There are some people at Marvel who are always looking at each project and making sure that the mythology works between each film. And they take it really seriously. They really care about making all of those things work. So I think that happens at the Disney and Marvel level.
Q: Speaking of that, how did you co-ordinate with all the different effects companies? How did everybody – I mean, it looked seamless on the screen, so how did that, what had to happen behind the scenes for that?
Jason Smith: I think that came down to Janek Sirrs, who is the visual effects supervisor overall on the film. He really helped keep us all in line because he had visibility of every shot at any given time. So it was not uncommon to get a phone call that the skies in the heli-carrier sequence that they’re cutting with the skies from another company right before that need to match up. So then we’d get on the phone with him and those other companies and make them match.
Another example is all the digital doubles like Captain America, Thor, Black Widow, all the digital versions of the Avengers that we built, we developed the assets and then gave them to Weda, who then used them. And in one case at least the heli-carrier, they damaged the wing and then gave that back to us. So I think the different companies on the film were very interested in making this a smooth process, as smooth as possible. So I would say that it really came down to Yannick making sure that we were matching. And he would call out those areas where we weren’t and we would get in line.
Q: So what happens afterwards as far as stowing the assets for either future productions or for archives?
Jason Smith: Yeah. Marvel has all those assets. You know they keep their own archives of everything. So I think they’re all ready for what happens next.
Q: Is that getting more and more common to have that many houses work on these giant films? Is it simply a matter of time? Or is it specifically like – I know there’s one company that did the user interface stuff for this and Prometheus. That seems to be their specialty.
Jason Smith: Yeah. And I think that sometimes that’s what it comes down to is that there is a company that is really good and fast at iterating on a certain kind of work. And sometimes it comes down to capacity; that there’s just only a certain amount of time and a given house only has a certain number of artists during that time. So then you have to split up the work. So yeah, it happens for a lot of different reasons. But I definitely think it’s the way that things will continue.
Q: It seems like at this point almost everything’s been – you guys have been called upon to do character work, to do environment work and things like that. Is there anything that you really feel like you haven’t been able to do yet but that you’re itching to try? Is there anything that like you guys are just working with general software and going “oh my God! I hope somebody has something that we can do this on?”
Jason Smith: Yeah. I think for me it’s just – I really feel there’s a lot we can do in character work. So I’m hoping that we get more character work; things like the Incredible Hulk where we can show off the ability to pull off those things.
Q: What would be like, I mean what would you do sort of different than what you did in the Hulk that you’d want to push by?
Jason Smith: The Hulk only had one line in the film so I think that there is a challenge with dialog that would be great to explore with digital characters more. I mean we’ve done that with Davy Jones in the past. But I think that’s, as far as looking at something that we haven’t done before, that’s kinda difficult because I think almost anything you think of: like giant robots, yes, explosions, yes, fire, yes, water, yeah. You know, almost everything that you can think of has been done. It’s just a matter of what can we do that we can take it to a new level. So it’s something we’ve already done but maybe there’s another step that we can take there.
Q: Well, it seems like real world weather effects and things like that are still hard just because of the scale and miniature work and things like that. So is that something that you guys are continually working to push further, in terms of how, just make that reality to sell it?
Jason Smith: Yeah. I remember when I was a kid and I went and watched Jumanji; then I saw The Making of. And they had done hair on the lion and I remember thinking to myself all right! They’ve done hair. But that was so early. That was very, very early on and so many advances have been made in hair since then. And we’re still working on hair. We’re still working on skin. We had brand new eyes for the Incredible Hulk that were more anatomically based eyes than we’ve ever done before.
So it’s always getting better, even when we’ve done something. There’s always something new we can do. Like on the Incredible Hulk for the first time, he had beard stubble. And when we looked at his face, we realized it just looked CG. It looked like the stubble just goes to the face and stops and then there’s green skin.
And then we looked at photos of people close-up and you can see, even though you think of skin as opaque, you watch that thing go into the skin and you see it underneath the skin. You see it continue. You can actually see the follicle, if you look close enough. And so with the Hulk, we had to incorporate that. We had to say all right the light bouncing around, the scattering inside his skin, darken it right at the root of the hair. So it’s a bunch of little incremental things like that where we notice some small piece of reality and say okay, let’s focus on that for a second. And then move on to the next one.
Q: Del Toro’s been talking about possibly doing a Hulk series. And it seems like one of the challenges if they wanted to use the Hulk and you guys developed is simply figuring out how to do that in a timely manner. Or what kind of schedule it would take to pull something like that off. How much of that decision would be driven by your work, in terms of what they would have to do time wise?
Jason Smith: I’m not sure; I don’t know anything about it. Unfortunately.
Q: Are there any shots in New York that you guys eventually had to scrap because of time or things like that? Any sequences that you you had to remove?
Jason Smith: No, I think there was back and forth throughout the whole process about what action was taking place and where it would be. But yeah. I think the fact that Marvel and Yannick actually, Yannick Souris, had planned out where this action would take place and what photography we needed, with us early on and kind of really put that investment in ahead of time, I think really paid off.
Marc Chu—Animation Director, Industrial Light & Magic
Q: [Regarding the Iron Man effects with a physical suit that doesn't need to be CG] Is that a - do you approach that any differently than you do, say, character animation for any character that is fully CG rendered, or?
Marc Chu: You know, it's a different challenge and after the first movie, when Robert saw, oh, wow, look at that, he realized he could not wear as much as he needed to. It gave him freedom. He was like, wow, I don't need to wear half of the suit and I could get the same effect. And that allows him to be more comfortable for sure because that thing is not built to be worn any long duration. It'll pinch you, I can't even put the helmet on. I tried one time. We got the helmet up here and I was like, oh, good, I'm going to put it on and I couldn't even get it over here. They have to cast - the first person they had to cast, the stunt guy had to just be able to put the helmet on and that's Clay, Clay is the stunt guy who always plays Iron Man. But the approaches to doing, for this movie, when he takes the suit off, he didn't even wear anything for that. He just wore what he was going to be revealed in.
Asker: In the end?
Marc Chu: No, no, in the beginning.
Asker: [What about] the end result, what he looked like after.
Marc Chu: The t-shirt and the pants. So that's a different challenge. We have to imagine his movements exactly and then place the armor on him. And then figure out ways for us to take the armor off. And everything, I think when they try to plan these things, looks great on a drawing. When they actually build the platform and you go, the platform doesn't even look like the drawing, it becomes our responsibility to figure it out. We need to figure out where the arms go, what gets taken off, how the mechanisms work and fortunately, we have lots of people who are well versed in robots and things to create that from scratch.
Q: You mentioned tailoring the Iron Man CG with Robert Downey's movements. How did you assist him, or did you need to assist him on how to walk a little bit like he's wearing the suit because he had nothing on at the time?
Marc Chu: Well we never interfere with what Robert will do, so he's just going to walk the way he does. And we just want him to be natural. In the end, all those shots are tight shots, right, they're all going to be like this on his body, on his shoulder or whatever. So it wasn't that important. When he first lands and he starts walking, that's completely CG. And that's something that we did on our mobile cap stage. And that's us interpreting his walk and putting it on the CG character. And then once it gets to him, it's just a matter of brute force, imagination, let's figure out how this thing works.
Q: I think there's a big misconception that performance captures, I think that you just push a button and it captures a performance and there you go, that's the whole thing. But so much of it is still dependent upon genuine key frame animation techniques. And I know there's some directors that – Del Toro talked a little bit at Comic-Con about how he refused to use any performance capture because he prefers that it be animated and that you guys be able to bring character to it yourselves. When you're bringing young animators in, do they come from strong animation backgrounds? Do they know that when they come in or is there a sense for some of them that they think the actors will drive the entire thing?
Marc Chu: For this movie, for Iron Man, for Avengers, for the Hulk, it doesn't matter, it's a combination of - sometimes it's MOCAP, the suits that we use on set when we capture the stunt performer's actions. Sometimes it's MOCAP on our stage, sometimes it's pure key frame animation. I think each one of those is a tool. And each one of them has their place to be in the movie. And the sequence where he's running after black widow and he's running down that little grating, we're using a combination of key frame shots and motion capture shots. And we're cutting between, back and forth to those things. And you can't tell. Each one, I think has their place. I think motion capture is a valuable tool but as you said, you'll always need an - I firmly believe it's always the case. You will always need an animator to pulse it out.
Either because you need to preserve the size and the weight of the Hulk, for instance, so you might need to adjust it, different proportions. It's always going to be an animator that gives it that level of final polish. With the actions, with the face, especially, we had Mark as a reference, we used solves of his face as performance, baseline for performance but it's always going to be the animator to go, you know what, I could push that expression a little bit more in the Hulk brow. And sometimes they would aks for to be a little more wild or this is the specific tool that we give him. He's fully CG. We can start with Mark's performance. If that's not enough, we can push it. Or we can completely change it based on what we've captured with him before. We've captured all his expressions so we can come up with a performance that's from scratch that is going to look like Mark because I it's based on his expressions.
Q: It feels like the definition of performance is changing. Like I would argue that say, for Attack the Clones, Yoda's performance is Rob Coleman and Frank Oz, like it's really a combination of their skills and what they did. And obviously Mark [inaudible] is part of the Hulk, but it's not the only person's who's playing that role. So it feels like it's not - like the word itself, it always doesn’t feel like it fully captures how these characters are created. There really is a combination of many people's skill sets and many people's opinions and sort of to get to that final result.
Marc Chu: It is. And Joss came up here to tell us, he gave us a low down. He said, this is, this needs to be the best Hulk ever. Everybody here, when we created this, we worked on the Hulk, we knew everybody would be judging that single character in all our work. It didn't matter how good the character looked or how good our digital doubles look or anything. It was the Hulk that was going to sell everything o we knew everybody was going to be critical about that and when he came up here, he told us, he actually read us a letter from Mark and it said, basically don't be constrained by what I do. If I do something, that's great, perfect, you can use it, but don't be afraid to push things if you need to. And that kind of gave us the freedom to go, okay, we're not - we don't need to follow his performance 100% but in the case that Joss wants something bigger, we can push those things and not be afraid that we're ruining some sort of - his acting.
We don't want to insult his contribution. And so that was liberating and then it becomes a process of, does it look cool enough? We had Hulk Daily, so we had daily, specifically just for Hulk shots and that consisted of every discipline, so we would have models, painters, rigors, animators, all in one room, the VFX supervisors and there was no restriction. I mean it was, does this shot look cool? You as a view painter, in charge of the skin, do you like the animation? Does that work for you? So we're all approaching it from, is this the best product that it can be? is this the best Hulk that it can be? So that is a lot of voices ultimately going into a character.
Q: What's more challenging for you, like when you are working on sort of like organic characters, or something like Iron Man where it's a lot of machines that you kind of like have to figure out how they look underneath the shell, how his arms come up.
Marc Chu: I don't know, I'm kind of a geeky guy, so I like mechanical stuff too. And for Iron Man it's always like, let's look at reference, let's look online like when he's putting this little thing on the pipe that diverts energy at the beginning, we looked at a ton of stuff that people were doing with underwater, I don't know. I don't even know what they're called but let's look at all this pipe cutting reference underwater and see what they do. Okay, let's make a conversion of that. It's - I think the more you always kind of hinge things off of reality, the more things look believable. For the suit that, for the Mark 7 suit, when it kind of unfolds and sticks onto him and transforms, we wanted to make sure that didn't feel too magical, that the volume of the pieces didn't feel like they were coming from nowhere. Like a magical tortoise of a suit, I don't know. Geeky, yeah. Sorry.
But - and that's what, that's how I felt the suitcase suit was in Iron Man too. It felt like it was unfolding from nowhere. So we kind of paid attention and wanted to make sure this felt like it was physically possible, that this thing, X amount of size, could actually hold a suit and it didn't actually. The guys upstairs are crazy technical.