Portrait Retouching Using Masks
Updated: Aug 8
If you want to make friends, learn how to retouch a portrait. Nobody likes themselves as-is, despite what they may say. It’s often been said that your goal should be to make a person look 10 years younger, but not more. If you go too far with retouching, you’ll make a portrait that looks completely fake. You won’t get thanked for that.
On the other hand, it’s commonly expected that portrait photographers are also dermatologists, plastic surgeons, dentists, and opthalmologists.
You will be doing yourself a favor if you shoot your photographs using camera picture controls such as “portrait” or “neutral”. Avoid “vivid” like the plague. My favorite is “neutral”. Use a slightly-long lens, like the classic 85 mm, to get a pleasing perspective. Back in the day, the 105 mm was king; it’s still a great choice for portraiture.
Face parts have many completely opposite requirements; some need sharpening and others need softening. Some parts need more saturation, some need less saturation.
To meet these contradictory retouching needs, the best tool is the mask. Many image editors support masking. I’m still a die-hard Nikon Capture NX2 fan, so I’m going to concentrate on how that program uses masks.
Lightroom (ala the ‘Adjustment Brush’) has masks. Lightroom’s “Auto Mask” can select non-circular shapes by looking for similar coloration, but I find their masking a bit too limiting. This is probably my least-favorite program for masks.
Photoshop, of course, has masks. It has always struck me as being just a bit too complicated and time-consuming for my taste, but that may be because I haven’t invested sufficient time in it. If you’re comfortable with it, then by all means use it.
Zoner Photo Studio Pro supports masks via the “Selection Brush”, lassos, circles, rectangles, etc. along with “Mask: Do Not Show”, “Mask: Normal/Inverted” etc. Similar to Capture NX2, you can soften the edges of the selection and erase your selection mistakes. Once you make your mask, you can apply softening, sharpening, or other effects that only affect what’s inside the mask.
Zoner Photo Studio Pro mask in the “Editor” tab
Zoner Photo Studio Pro masking example using the Brush Selection
Portrait Retouching Using Nikon Capture NX2
I want to show you how I use Capture NX2 to accomplish retouching, but I’ll let you pick your own favorite editor to get the same job done. I realize that Capture NX2 is now un-supported by Nikon. If you want to keep using it with your RAW images, then you can check out this article to convert your newer camera files into a RAW format that Capture NX2 can understand.
In this program, you perform mask selection/ adjustment pairs. After the adjustment, you click on “New Step”. Next, you select another mask and the adjustment associated with that mask.
You need to know how to add a selection mask and, just as important, how to erase a selection mask. Some picture details, such as the corner of an eye, would be extremely difficult to accurately select in a single step. The mask selection brush is typically too wide to easily get into little nooks and crannies. Constantly adjusting the mask brush diameter is horribly inefficient and a losing proposition.
It’s much easier to paint outside of the lines, and then switch to the mask “eraser” to clean up your mask.
Capture NX2 mask controls: Add and Subtract
Don’t be afraid to use a mask that goes beyond the area you want
Switch to the mask “eraser” and clean up the corners
Finished mask after erasing around the nooks and crannies
Few people have really white teeth. Nobody has perfectly-white teeth. When retouching, you need to “de-saturate” them and also brighten them. Don’t go too far with this. You want to reduce yellowing by lowering their color saturation, which will leave teeth looking gray. Next, you need to brighten the teeth (without making them pure white).
You want to select only the teeth to whiten and brighten them
Paint the mask (green) over the teeth in Capture NX2
After you mask an area, you will need to make it invisible while you apply an adjustment. In Capture NX2, you hide the mask by changing the selection from “Show Overlay” to “Hide Selection” as indicated by the arrow in the picture above on the right-hand side.
Fine-tune the saturation
After you use the “Selection Brush +” to mask only the teeth, select the Saturation/Warmth adjustment. Avoid 100% opacity, or your edits will look a bit “fake”. Also adjust the mask feathering, to avoid hard edges. Change the mask selection from “Show Overlay” to “Hide Selection” to see the progress of the saturation effect.
If you make mistakes while painting the (green) selection mask, simply click on the “Selection Brush –“ to erase the parts of the mask that you don’t want. Masks don’t have to be continuous, so that you can do things like selecting both eyes.
Teeth after de-saturation. Gray is better than yellow, but not by much.
Change the mask selection to “Hide Selection” while adjusting the saturation (or to see any effect you’re working on). The teeth may still look a little disappointing, since they changed from yellow to gray. Not to worry.
Increase brightness, but with a mask selecting only the teeth.
Choose the Brightness adjustment, while using the mask over the teeth. Change the mask to “Hide Selection” again, while increasing the brightness. Avoid the temptation to over-brighten the teeth; real teeth are slightly yellow and slightly gray.
When you’re happy with the way the teeth look, click the “New Step” to finish (assuming you’re using Capture NX2).
Most eyes need three different adjustments. The iris typically looks better when its color is more saturated. Similar to teeth, the whites of the eyes may need some de-saturation and they always need brightening. You also want the eyes, brows, and lashes to be very sharp (via the ironically-named un-sharp mask).
Eyes with typical issues that need improvement
Mask used for brightening and de-saturation of any red color
Eyes and brows need extra sharpening
A portrait just won’t look good if the eyes, lashes, and brows aren’t sharp. Make a mask for them and apply the Unsharp Mask.
Repair Eye Bags
Bags under the eyes are typically a two-step process. First, they usually need more Gaussian Blur than the rest of the face, and possibly even some Healing Brush. They usually need extra brightness adjustment, too.
Make a separate mask for enhancing under the eyes
If makeup isn’t used under the eyes, then they usually need to be brightened and have a little healing brush applied. The brightening needs a mask, but the healing brush doesn’t.
Here’s some advice: don’t go crazy with the “Healing Brush”. You can waste a lot of hours trying to heal every blemish on an entire face. Try this instead: Gaussian Blur. You’ll find that you can usually hide skin blemishes in a single step by simply blurring the skin.
Moderation in all things. You really, really don’t want “Barbie Skin”. When you apply Gaussian Blur, remember to adjust the opacity away from 100%. Skin shouldn’t look blemish-free. And feather the mask edges, too. For males, you’ll generally use much less blur.
Make a face mask that avoids the eyes, brows, nostrils, and mouth
Perhaps the biggest improvement in most portraits is getting the skin blurred. This does not include the eyes and mouth, however.
Gaussian Blur for the skin
The Gaussian Blur can be pure magic. Again, don’t forget to allow a little of the original skin to show through. Keep the opacity around 80 percent, and use a generous “feather” for the face mask. Use a large enough blur radius to hide blemishes, but avoid making the skin look fake.
Portraits typically take more editing work than any other type of picture. Most pictures work just fine with ‘global’ adjustments, without any masking at all, but pictures of people rarely look good with this treatment.
A good job of portrait editing leaves the viewer with a sense that something’s different, but they can’t really put their finger on it.
Cosmetics and good lighting can certainly help portraits and reduce the retouching labor, but there’s really no substitute for skilled retouching.