Selection to Mask Workflow

From Photoshop
Revision as of 11:59, 2 July 2016 by WikiSysopPS (talk | contribs) (Overview)
Jump to: navigation, search

Selecting is a key aspect of the compositing workflow and one that is not without a good deal of complexity. There are many selection tools, many reasons to select and many things you should know about it. In the compositing skill-set selection is most vital when it is used in order to separate a figure or object from its background and it is in this context that I shall adress it. If you get good at this then all other selecting tasks shall seem trivial.


Overview

The selection to mask workflow is, in principle, simple enough. Selecting can not be separated from masking. Here is the general workflow:


Stage Task
Review Review the image to determine what region needs to be masked, and if this region is naturally defined by a single color.
Arrow down.png
Select Select the region.
Arrow down.png
Make the mask Press the mask button. This will automatically make a mask based on the selection.
Arrow down.png
Edit the mask (removing junk) The mask that you have just created will almost certainly require editing: refining and correcting. There are separate process for removing junk, editing soft and hard masks and painting.
Edit the mask (refining soft edges)
Edit the mask (refining hard edges)
Edit the mask (painting)

Review

At this stage, you should review the image to determine exactly where the mask is to be located. Is the object, or its background, defined by a single flat color? If yes, then you are in luck. This will make the whole process easier.


Select

At this stage, you simply make the selection. This you do using one of the selection tools. For a hard edge object the Quick Selection tool is sufficient. For regular objects like boxes, buildings etc the Polygonal Lasso Tool is a good choice. For things like hair the Select / Color Range is the only option. Using the Color Range is simple, but also critical.

Here the sampled color is the greenish background. The selected region is visualized as white in the preview. The 'Fuzziness' value is too low for this image, and the selected region is very 'noisy'.
Here the 'Fuzziness' value is very high. See how the edge of the face is now selected. This over-selection would create problems later on.
Here the 'Fuzziness' value is as good as it can be at this stage. The face is not overly selected. However, the region behind the boy's shoulder is dark, indicating the it is under-selected .
Here the 'Add to Sample' button has been pressed, and clicked on the area behind the shoulder. This adds to the selection, improving the mask.

Make mask

Press the Add Layer Mask at the bottom of the Layers Palette. This will make a Layer Mask from the selection. Depending on what you have selected and what you wish to mask, it might be that the mask you are now looking at will require inverting (Command I).

The 'Add Layer Mask' at the bottom on the Layers Pallet.

Its possible that you now have the perfect mask. However, most likely, the mask will require editing. To get a better idea of how good your mask is, 'Alt click' on the mask in the Layers Pallet. In this case, the face needs some editing.

Alt click on the mask in the layers Pallet to reveal the mask.

To 'stress test' the mask, press 'Command L' to summon the Levels adjustment. Move the middle slider (the 'gamma' to either side. This reveals any flaws in the mask. This temporary adjustment is only for testing purposes. Do not press the 'OK' button. This test is known as a 'Gamma Slam'. Moving the gamma slider to the left will test the black of the mask, and to the right the white of the mask. In this case, the background looks a bit noisy... however, the background is probably good enough to require no more work.

A gamma slam showing up the noise in the mask.

Hard-edge masking

A hard edge mask might be a thing-in-itself: suitable for non-organic objects such as houses or cars. A hard edge mask might also be a part of the larger workflow needed to mask out a soft-edge thing such as a human figure and anything involving hair or fur.

Clean-up

Having made the preliminary mask you now need to get rid of the obvious noise and imperfections. Junk selections (for tidying up the black background area) and holdout selections (for tidying up the white foreground area) can be made using the Polygonal Lasso Tool and then filled with the foreground color via an Option Delete.

The trouble with such selections is that they can sometimes be too hard-edge. They can easily be softened via a Quick Mask Mode blur.

If the black portion of the mask is dark grey or its white portion is light grey then the following technique might be useful.

Go to Curves (Command M) or Levels (Command L) and look for the eyedropper tool (circled in red below).

Selecting 08.png

Click once on the white eyedropper and then click on a portion of the image that requires setting to white. Bingo! It should magically jump to white. If it misbehaves at all then it might be because it is sampling on the single pixel value that lies underneath the eyedropper. This eyedropper (and in fact all of PS's eyedroppers) takes its settings from the Eyedropper tool that is found in the Tools. The default is a single point value, but you would be advised to change it to a 3 X 3 average or something more realistic.

You might not get the perfect white point set immediately. In which case, close the adjustments dialogue box and then open it up again. This technique does not work iteratively in one go. It has to be done incrementally.

Repeat the process for the blacks.

Clamping and clean-up

Clamping is another way to remove noise from a mask. Consider the simple black and white image below:

Clamping 01.png

Let it represent the edge of a mask. Its Histogram will look like this (below):

Clamping 02.png

A histogram is a graphical representation of data. PS's histogram charts the distribution of the light values of an image. The histogram above shows a spike at white, a spike at black and nothing in between. Now lets blur the image (below):

Clamping 03.png

The histogram (below) shows that the blur has resulted in a full set of grey values.

Clamping 04.png

As well as charting data, the histogram can also be used to move the data around. If the black and grey values are moved towards the right, everything from light grey to dark gray will be converted to black.

Clamping 05.png

This will give us (below):

Clamping 06.png

So... a clamp can move a simple black shape out or in. In a mask a clamp can be used top shrink or disappear any noise and imperfections in the mask. Be warned! It might also change the outline of your masks! This can be a good thing (it can clean up noisy edges) but also a bad thing (it can shrink them or grow them).

Clamping and dilate / erode

It might be that your mask is too big / small by a few pixels. A clamp can also be used to dilate (make the white area smaller) or erode (make the white area bigger). See the Clamping page for more details. Caution: this can damage or soften details.

Test the mask

As you proceed, you will need feedback on how clean or noisy your mask is. A mask can be 'tested' by placing the masked layer on top of a black, white or colored layer (depending on the masked color value). Any defects should will up more clearly.

Another way of testing it is to do a Gamma Slam. Here it is:

The black area of the mask below looks fine.

Selecting 02.png

However, an extreme Curves adjustment like the one below will convert the grays into white (I have emphasized the curve in red for clarity).

Selecting 03.png

A curve running in the oposite way will convert all the grays into black, therby showing all the defects in the white area.

Selecting 04.png

Ultimate Test: the ultimate test is, of course, to use it in your composite. A mask will often need tweaking as the composition develops and your sense of what is required in it matures.


The hard edge mask should now be finished and good to go. A soft selection subject (hairy head, fur etc) might require both a hard mask and a soft mask and some sort of way to combine them.

Soft edge masking

Problem
A hard mask is clean but its edges, especially on hair, are too hard.
A soft mask has great edges but is noisy in the black and white ares.
Solution
Combine the two!

Erode 01.png

Procedure
  1. Make two masks: a soft one (using a Color Range) and a hard one (using whatever you want).
  2. Duplicate the hard one and erode one by a few pixels and dilate the other.
  3. Copy the erode and paste it into soft. Then go to Edit / Fade and set the blend mode to Screen. This will render the black of the pasted erode invisible as it is the neutral color for the screen mode (edit Fade dialogue box below).

Erode 02.png

  1. Copy the dilate and paste it into soft. Then go to Edit / Fade and set the blend mode to Multiply. This will render the white of the pasted erode mask invisible as it is the neutral color for the multiply mode.

The image below is a graphical representation of this process. The green area represents the hard, interior eroded mask (applied via a screen blend), the red area represents the hard, exterior dilated mask (applied via a multiply blend). Between the two the original edge of the soft mask can be seen.

Erode 04.png

Things to remember

Masking as drawing

Masking is a drawing activity. Organic shapes in particular can be subject to a lot of creative refinement in the process of masking. In the before (left) and after (right) below the shape of the girl has been clarified and the unfortunate wiggly pattern on the outline of her right sleeve has been removed.

Selecting 05.png


Where are you?

Whilst masking, be aware of where you are. After having made the mask from a selection you will be taken automatically to the mask component of the layer (B). Any painting or filling will be done to the mask. If you accidentally navigate to the layer contents component then you may well accidentally painting into the layer contents component of the layer (A). This is a BAD thing. Option Click on the mask to get into and out of it.

Selecting 01.png

Negative to positive cutting in

This hand is to be masked out manually. The sharp edge folds in the cloth and the small negative shapes are very hard to paint inside of.
Here is a poorly painted mask (junk mask shown as green). Its edges are very 'rounded'. The temptation is to refine the mask using a tiny brush (laborious and inaccurate).
The solution is to employ the 'cutting in'® method. Stage one: paint-out with a black, ignoring the fact that you are overpainting.
Stage two: cut-in with white:.
Stage three: bake until cooked.

See also

Channels

Clamp, Dilate and Erode

Taking a Photo

Quick Mask Mode