Double Trouble: The Making of Oblivion

Trevor Hogg chats with visual effects supervisor Bjørn Mayer, digital effects supervisor Paul Lambert, animation supervisor Steve Preeg and previsualization supervisor Nick Markel about bringing the world of Oblivion to the big screen.  Beware there are spoilers…

Joseph Kosinski

For his sophomore effort filmmaker Joseph Kosinski shifted from an established franchise to an original project which he wrote as a 12 page story five years before directing Tron: Legacy (2010).  “I have seen a lot of concept art which was derived from that but not the novel itself,” states Bjørn Mayer (The Hunger Games) who collaborated with Eric Barba (The Curious Case of Benjamin Button) to orchestrate the visual effects for Oblivion (2013).   There is a booklet from Comic-Con three years ago and there are a lot of pictures.”  The science fiction thriller which revolves around security repairman Jack Harper (Tom Cruise) who discovers that he is not as alone as previously thought on an Earth decimated by an alien war; 804 visual effects shots were required and evenly split between Digital Domain and Pixomondo with Barba dealing with his colleagues in the former and Mayer with his co-workers in the latter.  “We did work together with Joe and Steve Gaub (Terminator Salvation), the visual effects producer.  Eric and Joe worked together on Tron: Legacy so he knows about the workflow of Digital Domain.  The main separation was that we were doing the Skytower and the Bubbleship, and Digital Domain did the Jack vs Jack Fight, the drones and the TET.”

“I was working on Red Tails [2012] and Joseph Kosinski came by with his producer to have a look at some of the cloud work that we did for that movie and he wanted the surroundings of the Skytower done with CG clouds,” remembers Bjørn Mayer of incident which occurred back in August 2011.  “Joseph had some concept art and explained what he wanted and I needed to find out how difficult it was to do.  The cloud projections we did on the Skytower were hard to make in CG; the idea came quickly that we were going out to shoot them somehow somewhere.  I came up with this location Haleakala, to shoot the clouds in Hawaii.”  Before principle photography commenced, Mayer and his crew spent four days at 10,000 feet from dust to dawn capturing what would be used as the sky plates.  We had three Epic cameras with us to get 5K images that could be stitched together and with an overlap we made 10 different times of day and forming cloud setups, they were divided into 2K by 13K image sequences each four minutes long so we ended up with 40 minutes of 13K material to be placed all around the building.”  20 HD projectors, 34 video feeds, 19 different layered image zones and 42 foot by 500 foot-long screens were utilized. “The most time at the Skytower when they’re sitting and talking, the background is in camera because the visual effects had been done before the shoot and gets projected. When you see the Skytower from the outside then it’s again a lot of HDRs, lighting tricks and compositing.”

The flying machine which transports the character portrayed by Tom Cruise (Collateral) was physically constructed.  “The Bubbleship is standing around in the background a lot so therefore they made a practical one,” notes Bjørn Mayer who had to recreate the machine digitally.   “There was a LiDAR scan and we got some data from Wildfactory which built the thing and then we matched it, reviewed the geometry and realized everything so it was a flawless rendering.”  Usually, invisible effects are considered the best; however, with the signature device it was meant to be noticed.  “When you have a flying Bubbleship that’s obvious that is a visual effect.  We did some shots where we had to have a CG bubble ship land and on-set we had a part Bubbleship and Tom Cruise is getting out.  We needed to make a proper handover so you don’t see the difference between the flying one and the one standing there.”  The futuristic transporter is a hybrid of a Bell 47 helicopter and a jet fighter.  “When you think of the Bubbleship it is made of bits and pieces that could be nowadays technology with a few steps forward which makes it more believable.”  Mayer observes, The Bubbleship and the Skytower were straightforward because it was like car or architecture visualizations so we’re used to that how to mimic a car finish or contour glass and how it reflects we did a lot of that with Eric as well.  The Skytower is the same thing.  All of the materials in the Skytower were straightforward.  It was how to illuminate them with the nice surroundings that we got from Hawaii.”

“The testing started a few years ago with Joe and when we flew to that location to shoot a plate for that we found a glacier was filling up the canyon so the canyon was not there anymore,” remembers Bjørn Mayer about unexpected turn events which affected the Canyon Chase Sequence.  “Then time had passed and it had closed completely.  We tried to find another canyon which we could fly a helicopter through and we found one that Joe liked but it wasn’t fast enough.  It was too dangerous for the helicopter to fly fast through it.  It was feeling too slow and Joe wanted to speed up with it because it gets goofy with that look.  We went back to the ice idea and I was talking to our R&D team how to generate it with other consistencies, dirty patches, and a blue glacier ice scene.  We did some geometry to work it out and imported it into Houdini to get the plates; it looked like the glacier.  We made folds and had the wrinkles looking like ice and putting some spikes on it as well.  It got re-exported into 3ds Max to render it.  It was five minutes so it was really challenging.

“We created the waterfalls,” says Bjørn Mayer who was faced with recreating something which was easily recognizable thereby complicating the design process.  “But when you figure it out how the water should stay together in the upper part and slowly drift apart in the lower part [as well as the size and scale] we were able to create the look we wanted and change a few settings.  By putting a rock in the way of the water and it bounces off it makes a little believable touch.”  Interestingly there are not any holograms in Oblivion.  “The heads up display isn’t holographic.  It’s on the glass.  Sometimes there is a gap between the glass and the heads up display. The camera is in-between there and it looks like its floating.”  Like with the sky projections the table computer graphics navigated by Andrea Riseborough (Happy-Go-Lucky) were done mostly on-set.  “The table she is hitting on is four 54 inch plasma screens.  We did some interactive stuff when she is moving bits and pieces around or when she is talking to Sally [Melissa Leo] we put Sally’s face in there.  That was subtle media pass. The glitch look was developed to make it look like a data stream and it gets constantly interrupted.”  Mayer discusses a significant plot twist.  “There is another Skytower.  Skytower 49 is the one you see in the trailer where he [Tom Cruise] says, ‘I’m ready to go.’  There’s Skytower 52 where there is another Tom clone.  52 was easy because we just needed to put it into existing plates from the Iceland shoot.   49 was the more complicated because there is way more story happening there, and we needed to match the projection look and the Hawaiian clouds.  Those cloud covers for the Skytower that peak through it is really a tower on a long spire.”

“Eric was the visual effects supervisor but here at Digital Domain we have a position just beneath the visual effects supervisor side,” states Digital Effects Supervisor Paul Lambert who is based at the Venice, California headquarters of the company.  “I helped to keep everything on schedule and had an input in all of the departments.  Mainly I expanded upon Eric’s role.”  Assisting the effort at Digital Domain was Steve Preeg who along with Lambert had previously helped out on the Tron sequel.  “I was the Animation Supervisor so I was working with Joe [Kosinski] in figuring out how a drone moves,” explains Preeg.  “It was difficult to give some character to a flying sphere without making it Pixary.  And the same with the Bubbleship, we worked not only with Joe but with the other vendor Pixomondo in making sure that we were both on the same page on how the Bubbleship and drones move.  There is a quite a bit of digital double work in the movie which I don’t think was totally planned for originally.”

“Eric’s role was to discuss all of this early on with the production team how they were even going to shoot the shots and plan out the principle photography part of it,” states Steve Preeg.  “By the time it gets into post hopefully there is a good idea of what was shot, how and why and what we’re going to put in there.  In the meantime while he was on location we were building the assets and getting everything ready to start shot production once the plates showed up.  Definitely being involved and having those early conversations with Joe, seeing the artwork early on, getting scripts and figuring out what the movie was about definitely helped.” It was important to filmmaker Joseph Kosinski to have everything grounded in reality.  “He will always question, ‘Is this actually simulated correctly?’ From everything from how a character moves to cameras and lenses he’ll be the first person to pickup on a camera doing anything wrong.  Anything from physical movement to how the world was designed and how everything worked we relied on Joe to make sure that the foundation of the movie was like that.  A flying sphere is not the most aerodynamic system so there was a lot of back and forth with, ‘How do we want this drone to behave?  How is it that this thing can fly around and look correct when it is difficult in terms of reality to think about how that thing would work?  Going back and forth with animation studies looking at options and if you have to modify here and there about how this engine would work versus that engine or how much they wobble or not wobble.  It got tricky in space because if you watch footage in space everything moves perfectly in unison without bobbing or moving; that starts to have that CG feel.”

“There was a lot of effort put in making sure that, ‘How would you shoot this thing? Can you shoot this thing?  Maybe we need to make something look different from what we got from previs because once you start getting it rendered it doesn’t look correct,” reveals Steve Preeg.  “Joe is particular about lighting things up in a specific way and his framing so it is not only the motion of the camera that has to be thought about but exactly where everything is in frame.  The cockpit of the Bubbleship was on a gimbal and there is only so much you can do having the actors sit in a gimbal and rotate the gimbal around because it’s not really flying through space.  There are a lot of post 2D moves done on those plates to make it feel like you’re in a bubble plane where there is a bit of bobbing and weaving.  The ship is getting closer than getting away from you even though it’s a static camera on a gimbal which is rotating on its own axis.”

Designing the space station known as the TET became a mind-bending experience.  “That was a sequence which was fun,” states Paul Lambert.  “We did have some trouble trying to convey it on screen.  Joe wanted this upside down pyramid which was 30 miles wide and deep.   In the animation stage we could see we were going to have an issue trying to tell story points.  When we put the Bubbleship up against this huge ship it would be less than a size of a pixel. There are instances where we had to cheat ever so slightly trying to define that scale.  Also inside the TET is made up of an upside down pyramid inside another pyramid inside another pyramid.  You never have any horizontal or vertical lines.  Everything is diagonal.  There were times we would be in dailies with a shot we would constantly be questioning if we were in the correct space and if the orientation was correct. It was hard to get grounded as to where you were in the ship.  We were constantly going backwards and forwards trying to see if there was a mistake in the line up.”  Lambert continues, “Some shots in particular where you see a tunnel connected to one of the sloping walls and it just doesn’t look correct.  We went back a couple of times to make sure of that the line up was correct and it was.” NASA footage was watched as part of the visual research.    “There is no atmosphere in space but when we didn’t have any atmosphere or anything in the room,” remarks Steve Preeg.  “There was nothing that suggested distance so we introduced this space dust to push back certain walls to give a sense of distance and to have something that the light could play off of.”

Circular flying drones play a key role in Oblivion.  “They had a built a practical one on-set so sometimes when it is sitting down there was a real one so as far as visually we were matching something that really existed,” says Steve Preeg.  “They’re supposed to be good guys in the beginning but they always have that threatening feel.  Joe was particular about the fact that Jack is always a little bit scared of these things.  He’s like, ‘Jack Harper.  I’m Tech 49.’  But then again the thing does look like it was made by Apple and that tends to be non-threatening.  We tried doing things like, ‘How fast does it move?  Does it bob up and down?  Is it moving linearly?’  We tried to think of it in terms of the Terminator like it was determined which meant it didn’t do a lot of bouncing around and wobbling but then the more you take out that movement the more you’re inviting that it’s CG.   It’s not moving according to physics.  We ended up playing a lot with gun movement. There was nothing much else articulated on it except the eye and guns.  If you watch the movie they tend to take their guns in and out an often lot.  You do really think that they are controlled by a super computer otherwise they were these static things flying around.”

The drones had to be given personality especially 166 which plays the part of the main protagonist.  “We had full control of the focusing of the lens and the eye,” explains Steve Preeg.  “You could do little movements of making it feel that it’s change its focus.  The threatening part of the drone had to be a combination of the movements of the gun and drone, positioning and where it is in frame.  They tend to hover about seven feet above the ground.  They had to do a lot.  There times they’re coming in real low and kicking up dust off the ground.  For the most part, when the drones are facing off against Jack they tend to be above and looking down on him.”  There was practically not artwork on the drone point of view.  “We wanted to keep some red because the drones had a red eye,” says Paul Lambert.  “We went through various iterations of doing an inverse of the imagery, trying to make it heat seeking or Infrared.  What it ended up being was a combination of all three. One of our colleagues Christian Schermerhorn did all of them; he went through the iterations and came up with the final look which Joe liked. He added a look when the drone recognized a character it would push the image out like a pin cushion.”  The scanning effect was done practically, reveals Preeg.  “They had a white laser on-set that was on a little gimbal head they could tilt it down or up.”

Towering structures know as Resource Gatherers or Hyrdo Rigs populate the surface of the Earth.  “They’re a thousand foot tall giant water sucking things that are in a couple establishing shots,” says Steve Preeg.  “The only time you see them close-up is when they’re destroyed.  But it had to have the feeling of sucking up water from out of the ocean so we played around a lot with that.  We took videos of waterfalls and tried inverting those to see what they looked like.” A column of water which was 200 feet wide had to be depicted being pulled upwards a couple of hundred feet into the air.   “We tried to physically fib it to get it to look correct,” states Paul Lambert.  “When we simulated it everything was too slow and Joe wanted to see more movement.  We forced it to go faster and we also added real elements in there.”  As for one of them getting destroyed, Lambert remarks, “It was supposed to look like a small nuclear explosion.  You only really see a resource gather blow up from the Skytower.”  Clouds were pushed away by the blast.  “There are some simulated clouds in there.  You see the resource gather destroyed close up when he flies out to it.  We had some close ups of where it is all burning and there is wreckage and bits falling off of it.  You don’t see the explosion at the point where there are big chunks flying around.”

When Jack gets captured the drones follow him into the Scavs base camp resulting in a dramatic fly-through the environment.  “That was the big scene where three drones come in and make havoc inside Ravins Rock,” states Steve Preeg.  That’s where we have our biggest shot which is a 500 frame shot which is all CG where we follow a drone flying through the entire structure killing masses of people.  It was something which got prevised heavily and they fell in love with it so we had to bring that one to life.  There are lots of explosions, painting out stunt doubles and putting in practical explosions and other explosions and blowing people up.”  Not everything was initially done in high resolution.  “We did our development on one section of it to get the look down.  Then we would do lower resolution renders of the entire thing to make sure that it all blended together and once everything looked like way we wanted it to look we launched a full render of it.”  Paul Lambert remarks, “There were only a few practical elements in there.  I would say that 90 per cent of it is CG.  We would augment some of the fires and explosions.  Like Steve said we worked on a lot of low resolution versions for a good period and once Joe was happy with how everything looked we would then kick off some of the high resolution renders and some of those renders only finished on our last days.”

Jack Harper ends up fighting against himself.  “A fair amount of that was plate photography,” states Paul Lambert.  “Tom would do the Jack role and then he would do the clone role.  A lot of the shots were with a locked off camera so all we had to do was a bit of paint and roto.  There were eight shots where the footage from the other take didn’t line up so that’s where we did a face replacement.  We did not do his entire head.  We redid his eyes and mouth and the surrounding area.  They scanned Tom after the shoot so we had good takes of his physical appearance and actual look.  It all worked together well.” No previs was made of the Jack vs Jack Fight Sequence.  “That was a stunt coordinator’s rehearsing it with him,” says Steve Preeg.  “We did have one shot where we had to do a full head because he was wearing a tracking hoodie.  We did have one shot where we replaced the hair.”

A destroyed New York Public Library appears in the movie.  “They built the entire set where all the action takes place in New Orleans and it was gigantic,” says Steve Preeg.  “We didn’t have a chance to LiDAR that set so we had to do a photo reconstruction of it for the stuff we needed to do CG or patch things.  For the most part other than the big chasm he fully crosses over which was fully CG and patching regions of it I don’t think there are any all CG shots in the library.”  Digital characters needed to be inserted.  “They had originally shot a bunch of the Scavs running around in places that were expected to be use the stunt guys but it didn’t end up working out well so we did a lot of CG Scavs in that sequence.”  The crash scene of the space shuttle Odyssey needed to be enhanced.  “That one was effects heavy because there was very little smoke and fire on-set and Joe wanted it be almost blacked out by smoke and fire. There was one steadicam shot that was about a 1000 frames that we needed to roto Tom off, and put smoke and fire simulations.  That sequence was a dramatic transformation if you look at the original plate photography to what is in the movie.  In some cases there was no smoke or fire in the plate and by the end it is full of smoke and fire.”  Preeg makes a comparison.  “On the Ravens Rock Attack it was all digital so it was more of animation, rendering and compositing, whereas the live-action shot that took a lot of time in camera tracking, roto and paint.”

The Third Floor supplied the previs, postvis and techvis.  “We developed the look and shape of the canyon,” explains Bjørn Mayer, “and gave our rough geometry base to Third Floor so they could work on the flight and plot points.”   Along with Pixomondo, Digital Domain was assisted by the creative partnership.  “We worked closely with Third Floor on this,” states Digital Domain Animation Supervisor Steve Preeg.  “Shots that were previs fully CG we made sure that we were working within the same scale.  We gave them our rigs so we could take some of the mainframe data off of their scenes and quickly get a version into archive blend we could render out and started working with that.”  The Third Floor Previsualization Supervisor Nick Markel remarks, “We often sat with Joe in editorial looking at the sequences as they developed,” remarks “Joe would also come by my desk just lining up shots and figuring out angles.  Joe is very hands on and collaborative — for me pretty much the perfect director.  He used previs as his first pass on the film and also as a way to wrap his head around the music.”  Not a lot of visual research was required when dealing with the futuristic elements such as the drones and the Bubbleship.  “A lot of the assets were built in 3D by the Art Department so it made the transition from Art Department to previs seamless.  The TET presented an interesting challenge of scale that required tests to see what it would look like from earth and in space.  The look developed for the TET was done with concept from the Art Department team.”

25 scenes were given the previs treatment.  “I don’t think there was ever an issue of major concern, mainly because Joe is very methodical about creativity developing a sequence,” states Nick Markel.  “There were a few tweaks here and there for the aerial battle, but nothing that couldn’t be figured out in postvis.”  The majority of the film was given a postvis run through.  “In some cases, scenes like the Act 3 battle continued to develop even into postvis since many of the shots are 100% CG.”  Markel recalls, “Working from storyboards, we cut together an animatic with sound.  From there, we worked closely with Joe on what he wanted in terms of the action and angles for the scene. One challenge in particular was determining how Jack would interact with the drones and destroy them.”  Techvis needed to be created.  “Our techvis included programming the gimbal rig that simulates the hero vehicle in the film.  While most of the work was done with a real time approach, our team helped prep for what would be shot.  We also assisted with a system spearheaded by PRG to help view digital backgrounds in camera.”

“We did share shots but we didn’t share assets,” says Paul Lambert.  “We did the drones and Pixomondo did the Bubbleship.”  Steve Preeg remarks, “We did a lot of shots with the Bubbleship in it.  When they finished modelling and texturing it we took it and brought it in and ran it in our pipeline.  We had to change a lot of stuff to make it fit our workflow but at least it was the same model they were using and we had their texture.  We were all using the same renderer.  We had hundreds of shots of the Bubbleship.”   Visual Effects Supervisor Bjørn Mayer when reflecting on the work of Pixomondo notes, “There are a lot of shots that I’m proud of.  We did one test shot that made it into the movie at the end when we were developing the look of the canyon and the waterfalls. It is when Jack and Julia [Olga Kurylenko] enter into the canyon and they are making a 180 and fly through a waterfall to hide from the drones.  That’s one I’m proud of because it’s a nice shot and it worked out well.  Joe liked it so much that he put it into the movie.  The whole Canyon Sequence looks nice.  We had a romantic night time sequence where we show off the Skytower at night that looks nice because the Skytower at night looks beautiful.  Overall we did over half an hour on the movie and five minutes of that is the Canyon Chase sequence.  All the other stuff was all over the map.  It could be set extensions, other spaceships and paint outs.”  The biggest challenge for Digital Domain was the interior of the TET.  “Trying to convey the scale and where you are in that thing,” remarks Preeg.  “It was even a mind game to even know where you are looking in that thing.  It is so vast and empty. It did play some games with your mind from a geometric standpoint.”  The experience of collaborating with Joseph Kosinski on Oblivion had not changed from the time spent on Tron: Legacy.  “I’ve never seen Joe raise his voice and he has a good sense of what he wants.”

Production stills © 2013 Universal Studios. All Rights Reserved.

VFX images and videos © 2013 Universal Studios. All rights reserved. Images and videos courtesy of Universal Studios, Pixomondo, Digital Domain, and The Third Floor.

Many thanks to Bjørn Mayer, Paul Lambert, Steve Preeg and Nick Markel for taking the time to be interviewed.

Make sure to visit the official websites for Oblivion, Pixomondo, Digital Domain and The Third Floor.

Image Conscious: A conversation with visual effects supervisor Eric Barba

Trevor Hogg is a freelance video editor and writer who currently resides in Canada.

  • Rohan Michael Morbey

    AMAZING PIECE!