Vertus Fluid Mask

This powerful tool speeds up mask creation– but has a few downsides

By Ctein Back to


Many folks don’t understand how valuable selections and masking are for working on their digital photos. It’s true that compositing is an important use of masking, but that isn’t the only or even the primary use of selections and masks. I’m entirely a fine- art photographer these days, and I frequently use masks to improve my photographs.

Masks are way of exercising local control over a photograph. They let you control what parts of an image get affected by your manipulations. A mask is nothing more than a grayscale image that has the same dimensions as the photograph you’re working on. Black areas in a mask completely block whatever change you’re making to the photograph; white areas permit the change to work at 100% strength. Gray values in the mask produce intermediate- strength changes—the lighter the tone in the mask, the stronger the effect of your manipulation on the photograph. Masks can even modulate filters and adjustment layers in Photoshop.

Sometimes a mask is simple and contiguous, like the backdrop behind a product. Other times it can be complicated and non-contiguous, like pieces of sky seen between branches and foliage in a landscape. In some cases, I want to select for a particular characteristic, like tone or color. Or I may select all the highlights or shadows in a photograph so that I can adjust their appearance without altering the rest of the photograph. Sometimes a mask is amorphous and ill-defined, like the tarnish in an old photograph. (Check online at Promo/DR_Excerpt3.htm for instruction on how remove the bluish sheet of tarnish from black-and-white photographs using a mask.)

Using masks isn’t hard. The big trick is creating a good one in the first place. Building a perfect mask by hand, pixel by pixel, is time-consuming and tedious. Third-party tools can do very sophisticated masking with fewer clicks of the mouse and strokes of the stylus, saving hours of work on a single photograph.

One such tool is Fluid Mask 3 from Vertus ( view.htm), a $239 program for both PCs and Macs. It runs under Windows XP SP2 or Vista and Mac OS X 10.3.9, 10.4.9, or later. You can run it as a stand-alone program or as a Photoshop plug-in under Photoshop CS2 or CS3. It’s a universal Mac application, running on both Power PC and Intel processor machines.

Why another masking tool?

Why do we need yet another masking tool? Photoshop comes with a handful of its own, including lassoes, magic wands, and color selections. In the past I’ve reviewed Asiva Select and Mask Pro 3 (PT July/August 2005 and September/October 2006, respectively), which each has its own bags of tricks for helping you create masks. What does Fluid Mask offer that the aforementioned tools don’t?

For starters, Fluid Mask offers a different approach to making masks. It divides a photograph into regions called objects—contiguous areas of the photograph that have similar tone, color, and texture characteristics (Figure 1). A photograph parsed by Fluid Mask looks a lot like a paint-by-number picture. Fluid Mask masks a photograph object by object. You can customize the masking criteria for individual objects or groups of them. Part of your photograph might demand a soft, fuzzy mask defined by a very low-contrast edge, while another part needs to be controlled by contrasty, hard edges. When the same colors or tones appear in both foregrounds and backgrounds, it’s hard to use global settings and tools to get the selection right. Fluid Mask lets you localize those choices so that, for example, selecting someone’s soft white shirt won’t also select white fluffy clouds.

Fluid Mask is a complicated and somewhat novel tool and you are not going to master it quickly. I recommend watching all the online tutorials for Fluid Mask versions 2 and 3. They do a really good job of helping you wrap your head around this new way of masking.

Getting up to speed

As a stand-alone program, Fluid Mask is very limited in the file types it can read. If you want to work with Photo- shop or on 16-bit files, launch it as a plug-in. I tested Fluid Mask using Photoshop CS3 on a MacBook Pro with a 2.2 GHz Core Duo processor, 4 GB of RAM, and a very fast hard drive under MacOS X 10.4.11. That’s a fairly highend machine—which is what a program like Fluid Mask demands (more about that later).

You can download a 14-day free trial version. If it’s to your liking, you then can purchase Fluid Mask and activate it. Activation schemes are often a pain; this one was worse than usual. It’s normal to need administrator privileges to install new software. But Fluid Mask’s activation process also demands administrator access. This is bad design; a registration scheme should not be mucking about on that level, as the person who is authorized to install software on a computer is frequently not the end-user. Usually, you don’t want someone who’s simply using a computer to have administrator privileges.

Worse, there was a bug. After I entered my registration information, a pop-up window appeared asking for my system password . . . and it automatically closed itself about 10 seconds later. That happened several times before I realized why activation was failing. It took me several more tries before I could get my administrator information entered; I am not that fast a typist.

I’ve complained about this to the pro- gram developers. In the meantime, this only reinforces my opinion that activation schemes are one of the less brilliant concepts programmers have come up with. Find a less aggravating way to deal with piracy, folks!

Once I got past this hurdle it was smooth sailing, but I will admit it did not start me out favorably disposed toward Vertus.

The four-step program

Building a high-quality mask with Fluid Mask is basically a four-step process:

  1. Use the Edge Finding function to divide the photograph into objects.
  2. Use the Keep and Delete brushes to assign those objects to the “keep,” a.k.a., “foreground” mask or the “delete,” a.k.a., “background” mask.
  3. Refine the “blending” mask (the blue region) that separates the foreground and background masks.
  4. Use local patch controls to improve the results wherever the global settings don’t produce an ideal mask.

There are a lot of adjustments you can make to optimize the results at each stage. Until you’re very skilled with this program, you’ll find yourself trying out different settings and back-tracking a lot (Fluid Mask has an unlimited number of undos). You can save your work at any point as a “project” that you can come back to later. I’m still pretty much of a duffer and nowhere close to being able to quickly find the best settings for making my masks. Even so, I can do a credible job on difficult photos.

Walking through a real example is the best way to provide a feel for how this program works. The photograph is the same one I used in my review of Mask Pro 3. It’s an extremely difficult photograph to mask. The details of the hair are very complex, and there’s a great deal of “noise” from the myriad cracks, dust specks, and faded spots (Figure 2).

First, divvy up the photograph into objects. The goal is to find most of the important edges in the photograph automatically, but not create so many objects that they’ll be hard to work with. Don’t worry about doing a perfect job; you can fix it up later. I set Edge Contrast to low to ensure that Fluid Mask could find all the wisps of hair. The Edge-Width Threshold determines how small a region the program sees as a distinct object. Figures 3a and b show the edge-finding results for 1-pixel and 8-pixel settings.

I changed the texture filter to “Coarse” to eliminate a lot of unnecessary regions. I wasn’t too happy with how the hair got delineated, so I decided I needed more regions and more closely defined ones. Figure 3c shows the edges produced by a 3-pixel threshold, low contrast, and coarse- texture settings, and a 70% edge number. This is busier than needed to build a good mask, but it’s useful for illustrating how the program works. In practice, I’d normally go with many fewer objects and use finer local patch settings along the hairline.

Next, assign those regions to fore- ground and background masks. Paint over those objects with the local Keep and Delete brushes to add them to the appropriate mask. Increase the strength of the brush and it will select similar adjacent objects. If the areas you’re trying to keep and delete are strongly differentiated, a high strength setting works well; for photographs like this, leaving it at zero is better.

Figure 4 shows my results midway through this process, and after I finished assigning the objects. Note that Fluid Mask lets me turn the object edge view on and off and vary the mask opacity to see the underlying image more clearly.

The thick blue line between the red and green masks is the all-important blending mask. It’s where Fluid Mask works its magic. Here, that boundary doesn’t conform very well to the girl’s hair. “Exact Brushes” add parts of the photograph to the background and foreground masks precisely where you draw. Figure 5 shows my results that conform more closely to the hairline. Now it’s time to try rendering the cutout. That isn’t very good at all! The edges of her arms are ragged, and the hairline isn’t separated from the back- ground. The problem is that the blend- ing mask that Fluid Mask generated automatically is just a thin ragged line. What’s needed is one that covers the full width of the wisps of hair and flows more cleanly down the girl’s arms.

The Blend Exact brush is the tool for that. I quickly painted in a blue blend mask to cover all the wispy hair (Figure 6). The cutout preview looks a lot better. The left arm is pretty good and so is the hair on top of the head. Considering the photo I started with it’s amazing, but it’s not perfect. The hair on the sides has problems; a lot of the back- ground is showing. The girl’s right arm isn’t very good, either; the edges are jagged. Now it’s time for real local control via patches.

Figure 7a shows a closeup of part of the cutout where the background is bleeding through. I can improve that using the Patch brush to create a small region within which I can custom-tune the blending settings. Figure 7b shows those controls at work. The yellow- bounded area is the patch. To get a better cutout I moved the Intelligent Blending slider toward the Smart setting, which produced a less blurry selection. I used the Color Workspace palette to assign individual colors and ranges of colors to the keep, delete, and blend masks, giving Fluid Mask even more information about just what parts of the image I wanted preserved. The color workspace is very powerful; you can display colors either as a map or as a histogram, and you can set different qualities for the X and Y axes, such as saturation or channel value.

Figure 7c shows the enlarged view of the cutout after I made the patch adjustments. Note that the strands of hair are now more clearly delineated within the patch, but the selection out- side of the patch hasn’t changed. It’s this kind of local correction that makes Fluid Mask so powerful; you can use overall Edge Detection and Edge Blending settings that give the best results for the image as a whole, modifying those settings in local areas where they don’t work well.

Click on Save and Apply and Fluid Mask applies the cutout to the image in Photoshop, as shown in Figure 8. To convert the cutout to a mask, control- click on the thumbnail in the Layer palette and choose Select Pixels. This creates a selection that you can save as a mask via the Select menu.

Performance issues

One feature I did not make use of in the Patch Properties was Complex Hair Blending. This filter picks hair out from backgrounds much better than any of the manual controls. The problem is that most of the time it failed to function. When I attempted to preview the results of applying this filter, more often than not the cutout operation hung. Sometimes it was impossible to escape from this and I had to force a shutdown of Fluid Mask, losing any work that I hadn’t saved.

This filter was more likely to run successfully on small, 8-bit images with very small patch selections, but even then it hung about half the time. Complex Hair Blending will be an amazing tool when the programmers get the bugs worked out of it, but in the meantime I caution you against even trying to use it.

I’m impressed with Fluid Mask’s capabilities, but they come at a serious price. Fluid Mask is a phenomenal resource hog. The sample photograph I used in this article is not an unusually large image; it’s about the same size I get from my 6-megapixel digital camera. Professional-quality DSLR’s pro- duce files two-to-three times as large; a

digital back or film scan can run five- to-10 times as large.

Still, working on this modest-sized image taxed my powerful machine. Opening the image in the Fluid Mask plug-in gobbled up 10 times the file size in RAM. By the time I had finished constructing the mask, Fluid Mask’s RAM share was approaching 1.5 GB! Some edge-finding operations took 20 sec- onds or more to complete. All that with a 35 MB file. Time scales proportionately with file size, as does RAM consumption. With larger images, it wasn’t hard to bring my whole system down to a crawl.

Most of that RAM likely goes to the unlimited number of undos that Fluid Mask offers. That’s a nice feature, but Vertus should offer a preference to let the user limit the number of backup steps and conserve RAM. Vertus’s programmers also need to work on speeding up the code. Fluid Mask will save hours of time, but that doesn’t mean I’ll like waiting for minutes for operations to complete.

I recommend this program for its phenomenal results; I’m going to make use of it regularly. But I would urge prospective buyers to take advantage of the 14- day free trial to find out if Fluid Mask is right for them. The systems of many photographers simply won’t be adequate for the demands of this program. Photographers used to working on very large files will have to decide if very long pauses are an acceptable trade-off for superlative masking capability.

About the Author

Ctein is a technical writer and expert printmaker. He is also the author of Digital Restoration and Post Exposure—Advanced Techniques for the Photographic Printer.