User Profile

Hello! We are writing an app which analyzes a real world 3D data by using the TrueDepth camera on the front of an iPhone, and an AVCaptureSession configured to produce AVDepthData along with image data. This worked great on iPhone 12, but the same code on iPhone 13 produces an unwanted "smoothing" effect which makes the scene impossible to process and breaks our app. We are unable to find any information on this effect, from Apple or otherwise, much less how to avoid it, so we are asking you experts. At the bottom of this post (Figure 3) is our code which configures the capture session, using an AVCaptureDataOutputSynchronizer, to produce frames of 640x480 image and depth data. I boiled it down as much as possible, sorry it's so long. The main two parts are the configure function, which sets up our capture session, and the dataOutputSynchronizer function, near the bottom, which fires when a sycned set of data is available. In the latter function I've included my code which extracts the information from the AVDepthData object, including looping through all 640x480 depth data points (in meters). I've excluded further processing for brevity (believe it or not :)). On an iPhone 12 device, the PNG data and the depth data merge nicely. The front view and side view of the merged pointcloud are below (Figure 1) . The angles visible in the side view are due to the application of the focal length which "de-perspectives" the data and places them in their proper position in xyz space. The same code on an iPhone 13 produces depth maps that result in point cloud further below (Figure 2 -- straight on view, angled view, and side view). There is no longer any clear distinction between objects and the background becasue the depth data appears to be "smoothed" between the mannequin and the background -- i.e., there are seven or eight points between the subject and background that are not realistic and make it impossible to do any meaningful processing such as segmenting the scene. Has anyone else encountered this issue, or have any insight into how we might change our code to avoid it? Any help or ideas are MUCH appreciated, since this is a definite showstopper (we can't tell people to only run our App on older phones :)). Thank you! Figure 1 -- Merged depth data and image into point cloud, from iPhone 12 Figure 2 -- Merged depth data and image into point cloud, from iPhone 13; unwanted smoothing effect visible Figure 3 -- Our configuration code and capture handler; edited to remove downstream processing of captured data (which was basically formatting it into an XML file and uploading to the cloud) (See Attachment) CameraService (edited).swift

Hello, we would like to use this API to track an image in our camera view. We have run the demo where we upload a static image and it finds it great. However, our target is actually a small pattern of shapes; triangles, circles and squares arranged in a rectangular pattern, which encircles some visual information that we want to take a picture of and upload (in other words, its the "payload" and is highly variable). Is there a way that we can describe such a target to ARImageTrackingConfiguration? Two things that might work are (1) can we provide an image with translucency, where basically those pixel are "don't care"... anything will match them. That way we can make the center of the target trasnparent/ignored, and it will not throw off the pattern matching. Our experiments indicate that this doesn't work, but something like this would increase versatility so much, that I can't imagine there isn't *some* way to achieve this. (2) provide irregularly shaped (not rectangular) images... I guess this is similar to 1. We would like to match certain polygons within the area, and ignore the rest. (3) Can we provide multiple images, preferably with spatial relationships, that are matched simultaneously? This reads like a wish list, but Im just thinking about how constrained this powerful tool is if it is limited to rectangular images. Is there any clever way to accomplish what we are trying to do? (Other than complex custom image analysis.)

Hello... we are just getting started with a project and would like a little help getting pointed in the right direction. We would like to create an App that, when the camera is pointed at a surface, it can scan for a certain pattern and when it sees it takes an picture (sort of like one of those check scanning bank apps, but a unique pattern involving fiducials). So we would need to analyze the live preview data for specific features, and then save the snapshot when we see the feature we are looking for. Is this (a) possible, and (b) is there a sample project that does this, in part or whole? Thanks very much!