TUTORIAL: Layer Masking in Photoshop (inc. Rendering)

[MARKER]

**Originally Posted in LK's resource thread.. but Reed suggested I post here too**

Masking Images (Rendering)

---=== PART 1 ===---

I was going to do a very detailed tutorial in Masking Images (Rendering) but currently a bit busy ------ got a few requested printable covers to make which I've been putting off like Puppy Luv and Kawasaki Jet Ski for Wii.. Zzzzzzzz.. not to mention finish off my Fifa Street 3 cover -- and got to edit a lot of wedding photos for my friend.. so stuck for time, thus... here's a quickish version of it.

As the title states.. this is a MASKING tutorial, and made for Photoshop users, although users of GIMP can use it too. Sorry, those that use Paint.Net can look away, because it doesn't have layer masks and main reason I would NEVER use paint because every time I import an image I more or less add a layer mask to it.

Before we start... if using photoshop, here's a few keyboard shortcuts that I ALWAYS use, and might be helpful for anyone that use it, for this tut.

'd' key - defaults your palette to black and white.
'x' key - swap the background colour with foreground colour.
'b' key - brush tool
'1'-'0' - keys 1 to 0 determines your opacity. 1 being 10% and 0 being 100%
'[' ']' - square brackets decrease and increase brush size.


THE QUICK AND DIRTY METHOD!
This method is not recommended by itself but it's useful for a fast render of an image that has primary a single colour background.




(See PICTURE 1.)
A. Here, I've loaded up an image of sackgirl from Little Big Planet which has a white background which we'll remove. Since you can't add a layer mask to a background image, you should duplicate the image - just drag the layer onto the "Create a new layer" icon (A)

B. Since you're don't want to see the background layer now, click on the eye icon (B) to hide the layer.

C. Add a layer vector mask to your duplicated layer, by pressing "Add Vector mask' (C) A white box with a link on it will appear next to your layer image. Now, with it highlighed..

D. Select the "Magic Wand Tool" (D)

E. Note the Tolerance at the top of the screen (E). It's default as 32. You can adjust this if you are selecting too much or too less of the colour range. In this case, we'll leave it at 32.

F. With the Magic Wand Tool.. click on the any White space around the Sackgirl and a dotted line will highlight all the white area (F)



(See Picture 2.)
G. Now, to remove that white area, we need to fill that area in black. The easiest way is to press ALT+Delete which will fill the selected area with your foreground colour. So, if you have the default colour set (press d), you'll fill the area on the mask black, and the white area will disappear. You'll see this in (G). You might be asking why I don't just delete the area instead of using a layer mask. You could, but with layer masks, drawing in with white will bring back parts of the removed image, so if I accidentally removed sackgirl's hand I can simply, select a white brush and draw it back. If like me, you render images inside your composition, then later on you change your mind, you can draw back what you removed. If you deleted it.. it's gone!
Anyway... the image looks pretty good. Be aware however, that because you filled INSIDE the selection, you will get a 1 pixel white line around the edge of the image. so crop the image, or run the ERASER round the edge. One negative about this method!

Although it's not perfect (if you zoom in like 400%.. you'll notice there's a very slight edge to it) but placing it on a light background helps.. or even adding an INNER GLOW using the blending option in Photoshop using the same colour as the background can help.

How do I use this now? Just drag it into photoshop which will keep the layer mask intact. Or you can save it out as a PNG file.

---=== END OF PART 1 ===---

[MARKER]

Masking Images (Rendering)

---=== PART 2 ===---

The Better and Precise Way

If you used the "Quick and Dirty Method" I posted, then you will probably have to use this method to clean up.



(See picture 3)
Okay, here I've loaded up a picture of Sagat/Ryu from SF2. He's infront of a fairly colourful scenery, so using the Quick & Dirty Method to render him just won't work. Well, it would, but selecting different areas would takes ages, and it won't look good idea.
So, the first thing to do, like in part 1, duplicate the layer, hide the original background layer, then add a layer vector mask.

Next we need to remove as much of the background as we can quickly. To do this you can do a number of things.
1. You can get a BIG Brush (b key). On the top of the screen, on the Brush: bar, there's a drop down menu arrow (1). Here you can adjuect the diameter and hardness of the brush. Set the diameter size to whatever suits you.. and use a hardness of 100%



(See picture 4)
I've used a brush size of 125 and as you can see, using black on the layer mask, I've quickly brushed out most of the left side of Sagat, including under Sagat's right armpit using a smaller brush. If you accidentally remove something, then switch to white, and re-brush it back in.

Alternatively, you can use the pologon lasso tool (press "L" key)(2 on pic). I prefer the straight line one (second option) where you can make a rough selection round the area you want to remove, and like in "Quick & Dirty" press ALT+Delete to fill the area with foreground colour which should be black. I've done this on the right side of Sagat.



(See picture 5)
Now... to remove the rest of it...which requires a bit of time.
Zoom in on the image (depends on how big your image is -- my example is 300%). Select the brush, and use a small diameter and set the hardness to somewhere between 80-90%. Then slowly brush out the background as close to the edge of Sagat as possible. Remember, you can use the square bracket keys to increase/decrease the size of your brush... so along his shouder, I've used a brush size of around 20px but as I got close to his ear, I reduce the size of the brush to 9px, and real small around his ear. Now do this all the way round until you have rendered Sagat out completely. Well.. not completely in my example... as you don't see all of Sagat at the bottom



(See picture 6)
Sometimes it's difficult to see what you've missed or not, especially if parts are grey/white. To make it easier, add a layer under the layer you're working on (3 on pic). Then select a bright colour that your image doesn't have (I like to use Green, Red or Blue), then fill the entire layer with it (press ALT+Delete).
As you can see, just above Sagat's ear, I missed a few tiny spots which are grey.. and along his neck there's a couple of white notches which need smoothing out.

Another way to see wayward pixels is to go to the blending options, and add a stroke to the image using a bright colour. So, in my image, those pixels above sagat's ear will light up!



(See picture 7)
And Here we have the finished render. Okay.. the bottom is not done because Sagat/Ryu's feet are really dark and you can't tell what's what. So, if we were to use this, we would need to hide this area, or blend it in.



(See picture 8)
And for a bit of fun, there's the Sagat/Ryu Render added onto the Darkness picture. To do this, just drag the render into the Darkness image.. resize if required (make sure you hold the SHIFT key when you do to get the same scale proportions). I would reduce the opacity of the layer down so you can see what's underneath, and then remove the bottom parts using black brush on the layer mask to reveal the monsters.

---=== END OF PART 2 ===---

[Star89er]

Dude, that last pic almost makes me wanna do a Street Fighter - Darkeness crossover! Lol. Great tutorial, would come in handy.

[RunawayRed]

I'm kind of new to Photoshop, so this helps a lot. Thanks!

[HolyCowfartsBatman]

Great tut, however, I need to know how to render special effects, like hair, fire, electricity ect.

[MARKER]

Great tut, however, I need to know how to render special effects, like hair, fire, electricity ect.

Obviously, "rendering" special effects is more tricky and will depend on the "effect".

• HAIR -- Two methods I usually use if not zooming in and brushing out by hand is to -
1. Once you have rendered best you can around the hair, use the smudge tool. select a small brush size, and drag black into the gaps on the mask.
2. Another method is to use the colour selection. Make a selection of only the hair area that's troubling you and copy to clipboard. Create a NEW file, click okay for the size (as it'll use your clipboard image size), and then paste your copied hair to the new file. Now, use SELECT--> COLOUR RANGE, and select the hair.. then move the colour selection slider bar to get as much of the hair as possible. You can then copy this back to your original image, and then remove (mask out) the original hair. This is best for long fine hair like on my Soul Calibur IV Cover.
http://www.vgboxart.com/boxes/360/14...r_iv-print.jpg
If you see Mitsu (top left/front -- black hair guy), you'll see his fine flowing hair 'rendered', or colour selected, copied and pasted.

• Fire/electric -- To render these, you use a soft brush instead of a hard one when going round the edges with a low opacity too, so you can get a transparent/fading out edge.

[Tucker1140]

Great Tutorial. Could you make a Smash Bros. Brawl logo tutorial, like how you made the Super Smash Gays logo. I would appreciate if you could.