VisionFramework does not work with VisionOS2.0

I try vision frameworks with VisionPro but does not work only with VisionOS2.0.
When I perform requests, do not work and below error is caught. I try same code with VisionOS1.2, iOS18.0beta it works.

I try also new beta API but does not work and same error. ex.GenerateForegroundInstanceMaskRequest

do you have any idea? is it any permission for use vision framework with visionOS2.0.

This is my try list

with VisionOS2.0beta4

  • GenerateForegroundInstanceMaskRequest (not work error1)
  • VNGenerateForegroundInstanceMaskRequest(not work error1)
  • VNRecognizeTextRequest (not work error2)

with VisionOS1.2

  • VNRecognizeTextRequest (work)

with iOS 18beta

  • GenerateForegroundInstanceMaskRequest (work)

My Development Env

Env1 VisionPro: VIsionOS2.0beta4
Xcode: 16.0beta4,16.0beta2.
macOS: 14.5(23F79)

Env2 VisionPro: VIsionOS1.2.
Xcode: 15.4
macOS: 14.5(23F79).

Error1

Error Domain=com.apple.Vision Code=9 "Could not build inference plan - ANECF error: failed to load ANE model file:///System/Library/Frameworks/Vision.framework/subject_lifting_gen1_rev5_gv8dsz6vxu_multihead_int8.espresso.net Error= (DESIGN)" UserInfo={NSLocalizedDescription=Could not build inference plan - ANECF error: failed to load ANE model file:///System/Library/Frameworks/Vision.framework/subject_lifting_gen1_rev5_gv8dsz6vxu_multihead_int8.espresso.net Error= (DESIGN)}

Error2

Error Domain=com.apple.Vision Code=11 "VNRecognizeTextRequest produced an internal error" UserInfo={NSLocalizedDescription=VNRecognizeTextRequest produced an internal error, NSUnderlyingError=0x3001f6850 {Error Domain=CRImageReaderErrorDomain Code=-5 "Unknown error" UserInfo={NSLocalizedDescription=Unknown error}}}
Answered by DTS Engineer in 798458022

Hello,

Please file a bug report for this issue if you have not already.

As a temporary workaround while you wait for a resolution on your bug report, you can try limiting the compute devices of your request to only the CPU and GPU using the setComputeDevice(_:for:).

Remember to undo this temporary workaround once there is resolution on your bug report!

Best regards,

Greg

Hello,

Please file a bug report for this issue if you have not already.

As a temporary workaround while you wait for a resolution on your bug report, you can try limiting the compute devices of your request to only the CPU and GPU using the setComputeDevice(_:for:).

Remember to undo this temporary workaround once there is resolution on your bug report!

Best regards,

Greg

Thank you for your quick reply!
I posted bug report with this forum and I will try work around in my App

Thanks!

Finally, VNRequests worked successfully when I set the 'MLGPUComputeDevice' using the setComputeDevice(_:for:) method.

But, I can not use beta Vision Framework API because new API does no have setComputeDevice function or alternative function.

Anyway, thank you for advice!

@Frameworks Engineer

sorry for your misunderstanding. I know this method in VisionRequest protocol. but I can not access it from implement request class ex.GenerateForegroundInstanceMaskRequest

Value of type 'GenerateForegroundInstanceMaskRequest' has no member 'setComputeDevice'

Do you have any solution?

this is my code

        let request = GenerateForegroundInstanceMaskRequest(GenerateForegroundInstanceMaskRequest.Revision.revision1)
        let allDevices = MLComputeDevice.allComputeDevices

        if computeDevice == nil{
            for device in allDevices {
                print("device.description\(device.description)")
                if(device.description.contains("MLGPUComputeDevice")){
                    computeDevice = device
                    break
                }
            }
        }
        request.setComputeDevice(computeDevice,for: ComputeStage.main)

Sorry @Frameworks Engineer , I access it just now. This is my mistake. I will try it on device. thank you!

I try on device, but I have same build error(Value of type 'GenerateForegroundInstanceMaskRequest' has no member 'setComputeDevice') .

I guess VisionRequest needs some extension like perfrom function in ImageProcessingRequest.

Vision Framework for GenerateForegroundMask does NOT work on VisionOS 2.0 (Beta 5) See example code:

 func analyse() async {
        guard let selectedImage = selectedImage else { return }

        let ciImage = CIImage(image: selectedImage)
        let handler = VNImageRequestHandler(ciImage: ciImage!, options: [:])
        
        let request = VNGenerateForegroundInstanceMaskRequest { request, error in
            DispatchQueue.main.async {
                self.isProcessing = false

                if let error = error {
                    print("Error: \(error.localizedDescription)")
                    return
                }

                guard let results = request.results as? [VNPixelBufferObservation],
                      let pixelBuffer = results.first?.pixelBuffer else {
                    print("No results found")
                    return
                }

                let ciImage = CIImage(cvPixelBuffer: pixelBuffer)
                let context = CIContext()
                if let cgImage = context.createCGImage(ciImage, from: ciImage.extent) {
                    self.processedImage = UIImage(cgImage: cgImage)
                }
            }
        }

        // Configure the Vision request with preferred compute device
        do {
            // Query supported devices for each compute stage (handle potential errors)
            let supportedDevices = try request.supportedComputeStageDevices
            print("Supported Devices: \(supportedDevices)")

            // Check the available devices for the main compute stage
            if let mainStageDevices = supportedDevices[.main] {
                print("Main Stage Devices: \(mainStageDevices)")

                // Try to set the Neural Engine first, then GPU, and finally CPU (if applicable)
                var selectedDevice: MLComputeDevice? = nil
                
                if let neuralEngineDevice = mainStageDevices.first(where: { "\($0)".contains("NeuralEngine") }) {
                    selectedDevice = neuralEngineDevice
                    print("Selected Neural Engine: \(neuralEngineDevice)")
                } else
                if let gpuDevice = mainStageDevices.first(where: { "\($0)".contains("GPU") }) {
                    selectedDevice = gpuDevice
                    print("Selected GPU: \(gpuDevice)")
                } else
                if let cpuDevice = mainStageDevices.first(where: { "\($0)".contains("CPU") }) {
                    selectedDevice = cpuDevice
                    print("Selected CPU: \(cpuDevice)")
                } else {
                    print("No preferred device found, using default.")
                }
                
                // Set the selected compute device, if any
                if let selectedDevice = selectedDevice {
                    try request.setComputeDevice(selectedDevice, for: .main)
                }
            }
        } catch {
            print("Failed to configure Vision request compute device: \(error)")
        }

        #if targetEnvironment(simulator)
            request.usesCPUOnly = true
        #endif
        
        if #available(iOS 14.0, *) {
            request.usesCPUOnly = true
        }

        do {
            try handler.perform([request])
        } catch {
            DispatchQueue.main.async {
                self.isProcessing = false
                print("Failed to perform request: \(error.localizedDescription)")
            }
        }
    }```

Finally, I can access new Vision Framework API and VN prefix API without workaround on VisionOS2.0 release ver !!

Thank you so much!

VisionFramework does not work with VisionOS2.0
 
 
Q