This tutorial was written by the amazing Sean Kennedy and appeared in issue xx of 3D Artist
Set extensions are the kind of visual effect that every single movie released today has. Not just the big blockbusters, but comedies, little indie dramas and even just about every TV show has some type of set extension or background enhancement happening in it. It can be anything from alien landscapes and futuristic buildings to simply replacing a traffic sign or a storefront. Usually the same techniques are also useful if you want to remove something from a scene, like a wayward crew member or a car driving through the background of your medieval village.
While set extension tasks can be accomplished using both 2D and 3D techniques, in this tutorial we’re going to focus on 2.5D to add an important exterior location to our shot.
Perhaps the actual location was a protected park, and we were not allowed to build any kinds of structure out there during shooting. If the interior scenes are all shot on a soundstage or another location, then we only need a couple of establishing exterior shots, which is where matte painting and set extensions can save the day. And if you’ve not heard of 2.5D, it is exactly what it sounds like – a blend of 2D and 3D. We’ll use 2D cards with matte paintings on them and position them around our 3D space.
We are going to utilise Blender’s 3D space for much of this project, but keep in mind that the work we are doing is set up for compositing, not a 3D render. Matte painting, which isn’t the scope of this tutorial, goes hand in hand with set extensions. Matte paintings are usually given to a compositor in layers so the compositor can make adjustments to (or completely remove) each individual part of the painting and layer them into the matchmoved scene as needed. These layers can range from very simple (as in our case with this tutorial) to very complex.
Do not rule out 3D completely, though. Naturally, you are welcome to create any 3D elements you want. Having the matchmove will make it relatively easy to drop things into place, but be sure to take great care when texturing and lighting. Look at light and shadow placements in the footage, and the overall tonality of the shot, and do your best to get your 3D in the same visual world.
Step 01 – 3D track the scene
Bring the footage into the Movie Clip Editor (MCE), then Set Scene Frames and Prefetch so it’s loaded into memory. This is a simple shot, so there should be no problems finding at least eight points to track. Try and select points that are both close and far from the camera. I tracked 16 different points, finding contrast areas on the hillside and the pathway. In this project I tried to stay away from the very edges of the image.
Step 02 – Solve the camera motion
After choosing Solve keyframes and setting Refine to Focal Length, K1, K2, my 16 points generated an error of 0.3. In the Scene Setup tab, press Set as Background, so Blender automatically places the frame sequence in your camera view, then click Setup Tracking Scene. There are some buttons in the MCE editor for aligning the orientation, but I usually find it just as easy to select the camera, set the pivot to the 3D cursor (at centre), and rotate in the various views until it is aligned properly to the world.
Step 03 – Zero-weighted tracks
Having only 16 points makes it tough to get an idea of the geometry of the scene, so let’s use zero-weighted tracks to create many more. In the MCE, under the Track Settings tab, open Extra Settings and change the weight to 0. Go to the first frame and use Detect Features. Press F6 to bring up the options and change the threshold and margin until you have tons of tracks all over the shot. Track forward, then do the same from the last frame, tracking backwards. In the MCE graph editor, select the most erroneous curves and delete them.
Step 04 – Create rough geometry
Solve the camera again, and your 3D scene should now look like a point cloud of the landscape. In the Geometry tab, press 3D Markers To Mesh. In the 3D view, you’ll see a vertex created for every track point. Turn off the track points by unchecking Motion Tracking in the Properties panel. Create geometry from the new vertices by going into Edit Mode, selecting four vertices, and pressing ‘F’ to create a face. Keep building new faces off that one, choosing vertices that seem to make sense for the hillside.
Step 05 – Bring in matte paintings
In File>User Preferences>Add-Ons, turn on Import Images as Planes (which you’ll now find under Shift+A>Mesh and begin importing your matte painting elements. In this case, we’re using PNG images with alpha channels. When importing, be sure to use an Emission material and Use Alpha – Premultiplied. Place and scale them as necessary. Be sure to place the planes near vertices that lie in 3D space where that matte painted element should actually be.
Step 06 – Spice up the sky
The sky was clear on the day of shooting, but we can add some drama by replacing it or adding clouds to it. Import a sky image as a plane and make it very large. Push it back away from the camera, beyond the farthest track point. We don’t need a real sky dome because all the elements have light painted in to them from the matte painting department. This sky will be just another layer we composite in.
Step 07 – Arrange the layers
We have to split these paintings into the correct layers for compositing. For this particular scene, we could leave the hut and the bridge on the same render layer, but if we want to colour correct only one of them later, it will be easier if we just place everything on its own layer. Once you have split objects onto their own layers by pressing ‘M’, you then have to set up the corresponding render layers in the Render Layers panel. Place the rough landscape geometry on a layer that won’t render.
Step 08 – Rotoscope foreground objects
Obviously we’ve got to bring the actor and some of the pathway back over the matte paintings. Back in the MCE Tracking Properties, turn off the checkboxes for Pattern and Disabled, to keep the view less cluttered. Press Tab to switch into Mask mode and Cmd/Ctrl-left click to begin drawing your first mask around the actor’s hair, pressing ‘I’ to set keyframes. As you create the masks, be sure to label your mask layers clearly. I’m creating one mask for the actor and one for the pathway in front of the bridge. And don’t forget you can parent masks to tracks!
Step 09 – Moving to the compositor
Once in the compositor, Enable Nodes should already be activated because setting up the tracking scene creates a basic node layout for render layers and shadow passes. Delete whatever you won’t need, duplicate the Render Layers node for every layer you set up in Step 7 and layer them so the sky is first (using a Mix node set to Screen), followed by the bridge, hut, and hut shadow (all using Alpha Over nodes). Create Mask nodes for your rotoscoping, combine them with a Math node set to Add (enable Clamp), and plug it into the factor of the Alpha Over node for the bridge.
Step 10 – Adjust the alpha channel
You’ll have to invert the matte on the bridge, and add a Blur to each mask before they are combined. You could also use a Dilate/Erode node if your roto isn’t accurate enough once blurred. For the sky, run a Separate RGBA node off the Movie Clip node and plug a Color Ramp into the blue channel. Experiment with adjusting the sliders until you get a nice matte for the sky, then subtract the actor’s hair roto with a Math node. In order to get the perfect clouds, you’ll need to plug that Math node into the sky Mix factor.
Step 11 – Add 3D element
Now let’s dip into real 3D to bring in some added realistic details. I chose to make a bird flying in the background, which I rendered from a separate project file to use as an alpha channel on a 2D card in our main scene. Other fun details could be a clothes line hanging outside with fabric blowing in the breeze, or maybe some added trees or bushes. You could even shoot an actor against a greenscreen and place that footage on a 2D card out there in front of the hut, if you liked.
Step 12 – Finalise the shot
Now that all the elements are there and layered correctly, feel free to dial things in to your liking. Maybe add some light wrap to the hut, and make sure the colour correction is working on all the individual elements. Check your black levels and brightness levels by making the composite overly bright or overly dark temporarily (throw a Color Correction node at the end of the node tree and crank up the Gamma). If you’re using footage from a camera that has noticeable film grain, you’ll have to add matching grain to your matte painted elements.