Post not yet marked as solved
Same problem, I'm re-building an old ARKit project, and get the same error The app links to non-public libraries in Payload/XYZ.app/XYZ: /System/Library/PrivateFrameworks/AVFCapture.framework/AVFCapture, /System/Library/PrivateFrameworks/AVFCore.framework/AVFCore (ID: d6c054fd-1c55-40b0-8338-9f333e0d625b) building on Xcode 15 beta, for iOS 14. I don't know what image_picker_ios is, however I'm not using any packages/pods etc, and a search even for picker doesn't even turn up in my source.
Post not yet marked as solved
In iOS 17 you get great results with VNGenerateForegroundInstanceMaskRequest() - make your image a CVPixelBuffer, then feed it into something like this:
var maskRequest = VNGenerateForegroundInstanceMaskRequest()
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])
do {
try handler.perform([maskRequest])
if let observation = maskRequest.results?.first {
let allInstances = observation.allInstances
do {
let maskedImage = try observation.generateMaskedImage(ofInstances: allInstances, from: handler, croppedToInstancesExtent: false)
let maskImage = imageFromCVPixelBuffer(maskedImage)
sceneView.scene.background.contents = maskImage
} catch {
print("Error: \(error.localizedDescription)")
}
}
} catch {
print("Failed to perform Vision request: \(error)")
}
Post not yet marked as solved
This will turn the pale color output from a MTLTexture back to normal saturation. Example from Xcode default Metal project at line 178 in Renderer.swift, I've added an SCNView to my Storyboard for testing the output in the scnView.
if let texture = renderPassDescriptor.colorAttachments[0].texture { scnView.scene?.background.contents = texture.makeTextureView(pixelFormat: .bgra8Unorm_srgb) }
Have a look to see if your camera supports 4K with print(ARWorldTrackingConfiguration.supportedVideoFormats)
On an iPhone 13 Pro Max it’s only 4K camera is listed as <ARVideoFormat: 0x2834f2e90 imageResolution=(3840, 2160) pixelFormat=(420f) framesPerSecond=(30) captureDeviceType=AVCaptureDeviceTypeBuiltInWideAngleCamera captureDevicePosition=(1)>]
So in your config change to that format.
configuration.videoFormat = ARWorldTrackingConfiguration.supportedVideoFormats[12]
That’s on an iPhone 13 Pro Max 128gig - which did has the limited ProRes recording duration, so perhaps it works on the bigger storage models, or just future products.
Post not yet marked as solved
Incidentally noticed that confidence levels beyond Low are not currently working on latest iOS 14.5 beta 18E5178a on iPadPro 11", this is effecting Apple Point Cloud demo project too. This makes monitoring re-projection not too easy as the point cloud is very noisy.
Post not yet marked as solved
Thanks - I'm starting to get acceptable results with the worldPoint function. This is how I've implemented it:
for vv in 0..depthHeight {
for uu in 0..depthWidth {
let z = -depthValues[uu + vv * depthWidth]
let viewMatInverted = (sceneView.session.currentFrame?.camera.viewMatrix(for: UIApplication.shared.statusBarOrientation))!.inverse
let worldPoint = worldPoint(cameraPoint: SIMD2(Float(uu), Float(vv)), eyeDepth: z, cameraIntrinsicsInversed: intrinsics.inverse, viewMatrixInversed: viewMatInverted * rotateToARCamera )
points.append(SCNVector3(worldPoint))
}
}
Some points seem off - however I'll need to start checking each point with their corresponding confidence map to check. Thanks heap, this has been a great start.
Post not yet marked as solved
Hey thanks so much for the quick reply.
At line 28, this is what I'm trying, however the points are appearing off camera in a thin line distant from the camera - so I guess I need to submit a TSI to get any further help?
let worldPoint = worldPoint(cameraPoint: SIMD2(x,y), eyeDepth: z, cameraIntrinsicsInversed: simd_float3x3(intrinsics.inverse), viewMatrixInversed: frame.camera.projectionMatrix.inverse)
points.append(SCNVector3(worldPoint))
Thanks heaps for this so far.
Post not yet marked as solved
I'd say we're not the only ones... fingers crossed.
Post not yet marked as solved
Yeah, this is really annoying, it's a vital part of the AR chain - definitely fix.
Post not yet marked as solved
Changing the order around sure doesn't work for me.
