Hi,
I've been exploring a project with visionOS, and have been quite confused on capabilities and workflows for using custom materials in RealityKit & RealityComposerPro for visionOS.
Ideally I would be able to create / load / modify a model and its materials in Blender, export to openUSD and have it load in fully in RCP, but this hasn't been the case. Instead different aspects of the material don't seem to be exported correctly and that has lead me to investigate more into understanding MaterialX, openUSD and Metal, and how they work in visionOS, RealityKit and RealityComposer.
MaterialX was announced as a primary format for working with 3D materials, but the .mtlx file format doesn't appear to be compatible in RCP directly - specifically trying materials provided in the AMD OpenGPU MaterialX Library. (note: AFAIK, Blender does not currently support MaterialX) When downloading a material, this provides a folder with the textures and corresponding .mtlx, but currently in RCP (Xcode 15.6 beta) this file is ignored. Similarly, trying to load it using ShaderGraphMaterial fails with 'Error in prim' and no other details that I can see.
It also appears that there is a way of bundling MaterialX files within an openUSD file (especially implied by the error about Prims), but I haven't been able to understand how this can be done, or if this is the correct approach. Unpacking the Apple-provided materials in RCP from usdz to usda, these appear to define the shaders in openUSD and reference the RCP MaterialX Preview Shader (presumably created using the Shader Graph). There is also reference however from the official MaterialX.org and OpenUSD around using a USD / MaterialX Plugin to enable compatibility.
I've also tried, and followed along with the introductory tutorial on the in-built ShaderGraph, and find it difficult to understand and quite different from Blender's Shader Nodes, but it currently appears that this is the primary way promoted to create and work with materials.
Finally, I had expected that CustomMaterials using Metal Shaders would be available, as Metal was mentioned for Fully Immersive Spaces, and 'Explore Advanced Rendering with RealityKit 2' from WWDC21 covers using custom shaders, but this is not listed as included in visionOS and according to an answer here, it's not currently planned. (although the documentation here still mentions Metal briefly)
Overall, what are the suggestions for workflows with materials for RealityKit on visionOS?
- Is there a fully compatible path from Blender -> openUSD -> RealtyComposerPro? Do I need to export materials and models from Blender individually and rebuild them in RCP using the ShaderGraph?
- Can I utilise existing MaterialX materials in RealityComposerPro, and if so, how?
- Are there any other good resources for getting comfortable and understanding the nodes within the ShaderGraph? what WWDC talks would be good to revise on this?
Really appreciate any guidance!