Rendering 12

Refactoring My Shadows

In order to take transparency into account, we have to access the alpha value in the shadow caster shader pass. This means that we'll need to sample the albedo texture. However, this is not needed when using the opaque rendering mode. So we're going to need multiple shader variants for our shadows.

Right now we have two versions of our shadow programs. One version for cube shadow maps, which is required for point lights, and one for the other light types. Now we need to mix in even more variants. To make this easier, we're going to rewrite our My Shadow include file. We'll use interpolators for all variants, and create a single vertex and fragment program.

First, move the definition of Interpolators out of the conditional block. Then make the light vector conditional instead.

struct VertexData {
	float4 position : POSITION;
	float3 normal : NORMAL;
};

struct Interpolators {
	float4 position : SV_POSITION;
	#if defined(SHADOWS_CUBE)
		float3 lightVec : TEXCOORD0;
	#endif
};

Next, write a new vertex program, which contains copies of the two different versions. The non-cube code has to be slightly adjusted to work with the new Interpolators output.

Interpolators MyShadowVertexProgram (VertexData v) {
	Interpolators i;
	#if defined(SHADOWS_CUBE)
		i.position = UnityObjectToClipPos(v.position);
		i.lightVec =
			mul(unity_ObjectToWorld, v.position).xyz - _LightPositionRange.xyz;
	#else
		i.position = UnityClipSpaceShadowCasterPos(v.position.xyz, v.normal);
		i.position = UnityApplyLinearShadowBias(i.position);
	#endif
	return i;
}

Do the same for the fragment program. Then get rid of the old conditional programs.

float4 MyShadowFragmentProgram (Interpolators i) : SV_TARGET {
	#if defined(SHADOWS_CUBE)
		float depth = length(i.lightVec) + unity_LightShadowBias.x;
		depth *= _LightPositionRange.w;
		return UnityEncodeCubeShadowDepth(depth);
	#else
		return 0;
	#endif
}

//#if defined(SHADOWS_CUBE)
//	…
//#endif

Clipping Shadow Fragments

We'll take care of cutout shadows first. We cut holes in the shadows by discarding fragments, like we do for the Coutout rendering mode in the other rendering passes. For this we need the material's tint, albedo texture, and alpha cutoff settings. Add variables for them to the top of My Shadows.

#include "UnityCG.cginc"

float4 _Tint;
sampler2D _MainTex;
float4 _MainTex_ST;
float _AlphaCutoff;

So we have to sample the albedo texture when we're using Cutout rendering mode. Actually, we must only do this when we're not using the albedo's alpha value to determine smoothness. When these conditions are met, we have to pass the UV coordinates to the fragment program. We'll define SHADOWS_NEED_UV as 1 when these conditions are met. This way, we can conveniently use #if SHADOWS_NEED_UV.

#include "UnityCG.cginc"

#if defined(_RENDERING_CUTOUT) && !defined(_SMOOTHNESS_ALBEDO)
	#define SHADOWS_NEED_UV 1
#endif

Add the UV coordinates to the vertex input data. We don't need to make that conditional. Then conditionally add the UV to the interpolators.

struct VertexData {
	float4 position : POSITION;
	float3 normal : NORMAL;
	float2 uv : TEXCOORD0;
};

struct Interpolators {
	float4 position : SV_POSITION;
	#if SHADOWS_NEED_UV
		float2 uv : TEXCOORD0;
	#endif
	#if defined(SHADOWS_CUBE)
		float3 lightVec : TEXCOORD1;
	#endif
};

Pass the UV coordinates on to the the interpolators in the vertex program, when needed.

Interpolators MyShadowVertexProgram (VertexData v) {
	…

	#if SHADOWS_NEED_UV
		i.uv = TRANSFORM_TEX(v.uv, _MainTex);
	#endif
	return i;
}

Copy the GetAlpha method from My Lighting to My Shadows. Here, whether the texture is sampled has to depend on SHADOWS_NEED_UV. So check for that instead of whether _SMOOTHNESS_ALBEDO is defined. I marked the difference.

float GetAlpha (Interpolators i) {
	float alpha = _Tint.a;
	#if SHADOWS_NEED_UV
		alpha *= tex2D(_MainTex, i.uv.xy).a;
	#endif
	return alpha;
}

Now we can retrieve the alpha value in the fragment program, and use it to clip when in Cutout rendering mode.

float4 MyShadowFragmentProgram (Interpolators i) : SV_TARGET {
	float alpha = GetAlpha(i);
	#if defined(_RENDERING_CUTOUT)
		clip(alpha - _AlphaCutoff);
	#endif
	
	…
}

To make this actually work, add shader features for _RENDERING_CUTOUT and _SMOOTHNESS_ALBEDO to the shadow caster pass of My First Lighting Shader.

		Pass {
			Tags {
				"LightMode" = "ShadowCaster"
			}

			CGPROGRAM

			#pragma target 3.0

			#pragma shader_feature _RENDERING_CUTOUT
			#pragma shader_feature _SMOOTHNESS_ALBEDO
			
			…
		}

Refactoring My Lighting

Before we move on, let's tweak My Lighting a bit as well. Notice how we've used UnityObjectToClipPos to transform the vertex position in My Shadows. We can use this function in My Lighting as well, instead of performing a matrix multiplication ourselves. The UnityObjectToClipPos function also performs this multiplication, but uses the constant value 1 as the fourth position coordinate, instead of relying on the mesh data.

Interpolators MyVertexProgram (VertexData v) {
	Interpolators i;
//	i.pos = mul(UNITY_MATRIX_MVP, v.vertex);
	i.pos = UnityObjectToClipPos(v.vertex);
	…
}

The data supplied via the mesh is always 1, but the shader compiler doesn't know this. As a result, using a constant is more efficient. Beginning with version 5.6, Unity will give a performance warning when using an unoptimized multiplication with UNITY_MATRIX_MVP.

Dithering

Shadow maps contain the distance to surfaces that block light. Either the light is blocked at some distance, or it is not. Hence, there is no way to specify that light is partially blocked by semitransparent surfaces.

What we can do, is clip part of the shadow surface. That's what we do for cutout shadows. But instead of clipping based on a threshold, we could clip fragments uniformly. For example, if a surface lets half the light through, we could clip every other fragment, using a checkerboard pattern. Overall, the resulting shadow will appear half as strong as a full shadow.

We don't always have to use the same pattern. Depening on the alpha value, we can use a pattern with more or less holes. And if we mix these patterns, we can create smooth transitions of shadow density. Basically, we're using only two states to approximate a gradient. This technique is known as dithering.

Unity contains a dither pattern atlas that we can use. It contains 16 different patterns of 4 by 4 pixels. It starts with a completely empty pattern. Each successive pattern fills one additional pixel, until there are seven filled. Then the pattern is inverted and reverses, until all pixels are filled.

VPOS

To apply a dither patter to our shadow, we have to sample it. We cannot use the UV coordinates of the mesh, because those aren't uniform in shadow space. Instead, we'll need to use the screen-space coordinates of the fragment. As shadow maps are rendered from the point of view of the light, this aligns the patterns with the shadow map.

The screen-space position of a fragment can be accessed in the fragment program, by adding a parameter with the VPOS semantic to it. These coordinates are not explicitly output by the vertex program, but the GPU can make them available to us.

Unfortunately, the VPOS and SV_POSITION semantics don't play nice. On some platforms, they end up mapped to the same position semantic. So we cannot use both at the same time in our Interpolators struct. Fortunately, we only need to use SV_POSITION in the vertex program, while VPOS is only needed in the fragment program. So we can use a separate struct for each program.

First, rename Interpolators to InterpolatorsVertex and adjust MyShadowVertexProgram accordingly. Do not adjust MyShadowFragmentProgram.

struct InterpolatorsVertex {
	float4 position : SV_POSITION;
	#if SHADOWS_NEED_UV
		float2 uv : TEXCOORD0;
	#endif
	#if defined(SHADOWS_CUBE)
		float3 lightVec : TEXCOORD1;
	#endif
};

…

InterpolatorsVertex MyShadowVertexProgram (VertexData v) {
	InterpolatorsVertex i;
	#if defined(SHADOWS_CUBE)
		i.position = UnityObjectToClipPos(v.position);
		i.lightVec =
			mul(unity_ObjectToWorld, v.position).xyz - _LightPositionRange.xyz;
	#else
		i.position = UnityClipSpaceShadowCasterPos(v.position.xyz, v.normal);
		i.position = UnityApplyLinearShadowBias(i.position);
	#endif

	#if SHADOWS_NEED_UV
		i.uv = TRANSFORM_TEX(v.uv, _MainTex);
	#endif
	return i;
}

Then create a new Interpolators struct for use in the fragment program. It is a copy of the other struct, except that it should contain UNITY_VPOS_TYPE vpos : VPOS instead of float4 positions : SV_POSITION when semitransparent shadows are needed. The UNITY_VPOS_TYPE macro is defined in HLSLSupport. It's usually a float4, except for Direct3D 9, which needs it to be a float2.

struct InterpolatorsVertex {
	…
}

struct Interpolators {
	#if SHADOWS_SEMITRANSPARENT
		UNITY_VPOS_TYPE vpos : VPOS;
	#else
		float4 positions : SV_POSITION;
	#endif
	
	#if SHADOWS_NEED_UV
		float2 uv : TEXCOORD0;
	#endif
	#if defined(SHADOWS_CUBE)
		float3 lightVec : TEXCOORD1;
	#endif
};

Dithering

To access Unity's dither pattern texture, add a _DitherMaskLOD variable to My Shadows. The different patterns are stored in layers of a 3D texture, so its type has to be sampler3D instead of sampler2D.

sampler3D _DitherMaskLOD;

Sample this texture in MyShadowFragmentProgram, if we need semitransparent shadows. This is done via the tex3D function, which requires 3D coordinates. The third coordinate should be in the 0–1 range and is used to select a 3D slice. As there are 16 patterns, the Z coordinate of the first pattern is 0, the coordinate for the second pattern is 0.0625, the third is 0.128, and so on. Let's begin by always choosing the second pattern.

float4 MyShadowFragmentProgram (Interpolators i) : SV_TARGET {
	float alpha = GetAlpha(i);
	#if defined(_RENDERING_CUTOUT)
		clip(alpha - _AlphaCutoff);
	#endif

	#if SHADOWS_SEMITRANSPARENT
		tex3D(_DitherMaskLOD, float3(i.vpos.xy, 0.0625));
	#endif
	
	…
}

The alpha channel of the dither texture is zero when a fragment should be discarded. So subtract a small value from it and use that to clip.

	#if SHADOWS_SEMITRANSPARENT
		float dither =
			tex3D(_DitherMaskLOD, float3(i.vpos.xy, 0.0625)).a;
		clip(dither - 0.01);
	#endif

To actually see a pattern, we have to scale it. To get a good look at it, magnify it by a factor of 100, which is done by multiplying the position by 0.01. A spotlight shadow allows us to get a good look at it.

		tex3D(_DitherMaskLOD, float3(i.vpos.xy * 0.01, 0.0625)).a;

You can inspect all 16 dither patterns by increasing the Z coordinate in steps of 0.0625. The shadows get fully clipped at 0, and are fully rendered at 0.9375.

Approximating Semitransparency

Instead of using a uniform pattern, we have to base the selection of the dither pattern on the surface's alpha value. As full opacity is reached at 0.9375, multiply the alpha value by this factor, then use it as the Z coordinate.

			tex3D(_DitherMaskLOD, float3(i.vpos.xy * 0.01, alpha * 0.9375)).a;

The dithering now varies based on the surface opacity. To make it look more like a true shadow, we'll have to scale down the pattern size. Unity uses a factor of 0.25, so we'll use that as well.

			tex3D(_DitherMaskLOD, float3(i.vpos.xy * 0.25, alpha * 0.9375)).a;

This looks a lot better, but it's not perfect. How obvious the dithering is depends on the resolution of the shadow map. The higher its resolution, the smaller and less obvious the patterns.

Dithering works better with soft directional shadows. The screen-space filtering smudges the dithered fragments to such a degree that they're no longer obvious. The result is something that approaches actual semitransparent shadows.

Unfortunately, dithering is not visually stable. When things move, you can get very obvious shadow swimming. Not just along the edge, but across the entire shadow!

Toggling Semitransparency

To enable semitransparent shadows again, we have to add an option for it to our custom shader UI. So add a DoSemitransparentShadows method to MyLightingShaderGUI.

	void DoSemitransparentShadows () {
	}

We only need to show this option when using the Fade or Transparent rendering mode. We know which mode we're using inside DoRenderingMode. So invoke DoSemitransparentShadows at the end of this method, if needed.

	void DoRenderingMode () {
		…

		if (mode == RenderingMode.Fade || mode == RenderingMode.Transparent) {
			DoSemitransparentShadows();
		}
	}

As this is a binary choice, we can represent it with a toggle button. Because the label Semitransparent Shadows is wider than Unity's default inspector window width, I've abbreviated it. For clarity, I gave it a tooltip that isn't abbreviated.

	void DoSemitransparentShadows () {
		bool semitransparentShadows =
			EditorGUILayout.Toggle(
				MakeLabel("Semitransp. Shadows", "Semitransparent Shadows"),
				IsKeywordEnabled("_SEMITRANSPARENT_SHADOWS")
			);
	}

Like with the other keywords, check whether the user makes a change and set the keyword accordingly.

	void DoSemitransparentShadows () {
		EditorGUI.BeginChangeCheck();
		bool semitransparentShadows =
			EditorGUILayout.Toggle(
				MakeLabel("Semitransp. Shadows", "Semitransparent Shadows"),
				IsKeywordEnabled("_SEMITRANSPARENT_SHADOWS")
			);
		if (EditorGUI.EndChangeCheck()) {
			SetKeyword("_SEMITRANSPARENT_SHADOWS", semitransparentShadows);
		}
	}

Showing Alpha Cutoff for Shadows

When using cutout shadows, we might like to change the Alpha Cutoff threshold. Currently, it only shows up in our UI when using the Cutout rendering mode. However, it must now also be accessible in Fade and Transparent mode, when not using semitransparent shadows. We can support this by setting shouldShowAlphaCutoff to true in DoSemitransparentShadows, when appropriate.

	void DoSemitransparentShadows () {
		…
		if (!semitransparentShadows) {
			shouldShowAlphaCutoff = true;
		}
	}

The next tutorial is Deferred Shading.