Trying to optimize performance, the question of whether or not to use the alpha of a normal map for hiding a mask in order to minimize texture look-ups. The other option would be to leave the normal itself and create a separate, lower resolution mask file. In that case, the normal could be compressed as DXT1 to halve it's memory size.
A little visual test on the accuracy of texture compression in UE4 using both 16-bit and 8-bit PSD's. TC_Normal compression uses BC5, a successor to DXT5, for imported normal maps. BC5 uses only the red and green channels of the texture, rebuilding the blue channel in the texture node while disregarding the alpha channel. In order to be able to use the alpha channel (for a mask, roughness map, or any grey-scale texture), a compression like DXT5 (TC_Default in UE4) can be used—at the cost of clarity. For both DXT5 and DXT1, the idea is to use these maps for larger normal changes on an environment mesh while layering detailed tiling normals on top to create finer detail anyways. The cleaner way would be to use BC5 compression and create a separate mask texture (or compile multiple masks or grayscale textures into an RGBA mask texture).
Simplified:
- BC5 - highest quality, no alpha
- DXT5 - lower quality, with alpha, same memory size as BC5
- DXT1 - lower quality, no alpha, half the memory size of BC5/DXT5
A little post on converting the normal's color space from (0-1) to (-1-1) HERE
8 Bit BC5
8 Bit DXT5
8 Bit DXT1
16 Bit BC5
16 Bit DXT5
16 Bit DXT1