I ran into a scene where I wanted to render an ambient occlusion (AO) pass, but certain objects were using transparency maps as cutouts. So when I rendered the AO pass, those objects with transparency maps turned out solid...not the desired result.
Disclaimer: This is some-what of a half solution for AO with transparencies. AO works based on geometry proximity and does not take into account mapping. So this solution is an approximation. With that said, lets take a look at AO with transparencies.
I'm going to explain how you can render an AO pass while keeping the transparency of certain objects, just using the mental ray and A&D shaders. The example I'm using here is a tree textured with A&D materials.
Disclaimer: This is some-what of a half solution for AO with transparencies. AO works based on geometry proximity and does not take into account mapping. So this solution is an approximation. With that said, lets take a look at AO with transparencies.
I'm going to explain how you can render an AO pass while keeping the transparency of certain objects, just using the mental ray and A&D shaders. The example I'm using here is a tree textured with A&D materials.
The leaves on the tree are simply planes, but when rendered you see the leaves, because the planes have a alpha-map of the leaves in the cutout slot of the A&D material.
When you render an AO pass (click here to go to my AO in mental ray posting), the transparency on the leaves go away and the occlusion shows the planes. Of course this is not what we're wanting.
So to render an AO pass with transparency, there are regular objects and there are the objects that we want with transparency. So we will need 2 AO shaders. The first is my regular AO shader, and the second is one with transparency. There is a trick to creating the AO shader with transparency. I create a mental ray material, and in the first slot I insert a mix map. In color #1 I insert a "material to shader". That shader is an A&D material, where I turned up the refraction to 1.0, the IOR to 1.0. For color #2, I use my regular AO shader, and for the mix amount, I use the leaves transparency map to mix the AO with the 100% transparent A&D shader.
When I render my AO pass the easiest way to do this is plugging in the AO material into the 
material override in the Processes tab of the Rendering Settings. But it's not enough to insert the AO material with the transparency into the material override, because we also have regular objects that shouldn't render with the leaves transparency. To overcome this problem I create "Sub-Object" material shader, with ID 1 being the regular AO shader and ID 2 being the AO shader with transparency. I plug the Sub-Object material into the material override.
material override in the Processes tab of the Rendering Settings. But it's not enough to insert the AO material with the transparency into the material override, because we also have regular objects that shouldn't render with the leaves transparency. To overcome this problem I create "Sub-Object" material shader, with ID 1 being the regular AO shader and ID 2 being the AO shader with transparency. I plug the Sub-Object material into the material override.
Then I select all of the objects in my scene (in this case it's the tree trunk), and apply a material modifier to assign the material ID to 1. Then I select my leaves in the scene and also apply a material modifier, and set the ID to 2.
Presto, the scene is ready for an AO pass with transparencies!
If you have several different transparency maps in one scene, you will have several AO shaders for each transparency, and put them all into the Sub-Object material, then you will need to apply the material modifier with the correct ID to each object for it's respective transparency maps.
