[Reality Composer Pro] Is it possible to play video over a specific mesh like a material?

Hi!

I'm trying to play video on monitor 3D model like a material. I wanna know if it's possible work. I searched about it, but I couldn't get enough information. Thank you in advance.

Answered by Vision Pro Engineer in 812875022

Hi @PLAY4

Here's more detail on how you can get this to work by using a VideoMaterial.

Start by placing your video file somewhere inside of your Xcode project. In my case, I created a "Videos" folder in my project and placed a .mp4 file with the name "MyMovie" inside of it.

Next, define a RealityView that loads your video file and applies it to the surface of a model with a VideoMaterial, as shown below:

RealityView { content in
    // Create a URL that points to the movie file.
    if let url = Bundle.main.url(forResource: "MyMovie", withExtension: "mp4") {
        
        // Create an AVPlayer instance to control playback of that movie.
        let player = AVPlayer(url: url)

        // Instantiate and configure the video material.
        let material = VideoMaterial(avPlayer: player)

        // Create a new model entity using the video material.
        let modelEntity = ModelEntity(mesh: .generateBox(size: 1.0), materials: [material])
        modelEntity.position = [0, 1, -1]
        content.add(modelEntity)

        // Start playing the video.
        player.play()
    }
}

In this example, VideoMaterial is applied to a cube model positioned in front of the viewer, but you can apply it to the screen on your 3D monitor model instead.

We are currently developing a set of visionOS apps that detect and measure object surface geometries from MeshAnchor.

https://github.com/CurvSurf/FindSurface-visionOS

  • FindSurfaceST (Spatial Tap): Object surface detection by spatial tap
  • FindSurfaceRR (Responce-to-Request): autonomous object surface detection
  • FindSurfaceRT (Real-Time): Real-time object surface detection
  • FindSurfaceAD (Ads): rendering photos/videos on detected object surfaces. The corresponding iOS app is here; https://github.com/CurvSurf/FindSurface-SceneKit-ARDemo-iOS

The source code of the FindSurfaceAD app is planned to be released in December 2024. Photos/videos are planned to be selected in the Photos app.

Please keep GitHub CurvSurf running.

Hi @PLAY4

You can play a video on the surface of an entity with a VideoMaterial.

Let me know if you have any questions!

Accepted Answer

Hi @PLAY4

Here's more detail on how you can get this to work by using a VideoMaterial.

Start by placing your video file somewhere inside of your Xcode project. In my case, I created a "Videos" folder in my project and placed a .mp4 file with the name "MyMovie" inside of it.

Next, define a RealityView that loads your video file and applies it to the surface of a model with a VideoMaterial, as shown below:

RealityView { content in
    // Create a URL that points to the movie file.
    if let url = Bundle.main.url(forResource: "MyMovie", withExtension: "mp4") {
        
        // Create an AVPlayer instance to control playback of that movie.
        let player = AVPlayer(url: url)

        // Instantiate and configure the video material.
        let material = VideoMaterial(avPlayer: player)

        // Create a new model entity using the video material.
        let modelEntity = ModelEntity(mesh: .generateBox(size: 1.0), materials: [material])
        modelEntity.position = [0, 1, -1]
        content.add(modelEntity)

        // Start playing the video.
        player.play()
    }
}

In this example, VideoMaterial is applied to a cube model positioned in front of the viewer, but you can apply it to the screen on your 3D monitor model instead.

The source code of the FindSurfaceAD app is now accessible on GitHub CurvSurf. Photos/videos are imported from Photos app.

[Reality Composer Pro] Is it possible to play video over a specific mesh like a material?
 
 
Q