I've converted pix2pix model whose input/output is Image(256x256)But the model are normalized to the range [-1,1], and the output image shows nothing but black.Does anyone know how to convert this image to [0, 255]?
Post
Replies
Boosts
Views
Activity
I've trained my GAN pytorch model, converted into onnx, and then mlmodel.but the mlmodel's input/output type is MLMultiarray.How can I change this into Image?Which stage (pytorch or onnx or mlmodel) should I retouch?
I've built an app with ARKit 3.5 previously.
With [configuration.sceneReconstruction = .mesh],
I put all meshAnchors to 3D models.
Do I able to filter this meshAnchors by confidence,
and add color data from camera feed?
Or with demo code with MetalKit, how could I able to convert point cloud into 3d model?
Like answered on previous question.
[If you want to color the mesh based on the camera feed, you could do so manually, for example by unprojecting the pixels of the camera image into 3D space and color the according mesh face with the pixel's color.]
Could you be able to give some hint how to solve this?
I'm not really familiar with this concept.
In your video and demo codes, I get that detect hand landmarks with VNImageRequestHandler in captureOutput functions.
With what method, could I able to draw hand skeleton as shown in video?
On Arkit project, in funtion -
func session(_ session: ARSession, didUpdate frame: ARFrame)
I tried to get -
guard let observation = handPoseRequest.results?.first as? VNRecognizedPointsObservation else { return }
and get -
let thumb = try! observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb)
segmentation fault 11 pop up.
Is this a bug ? or did I made any mistake?
I tried to use handposerequest on arkit.
It does get result on VNRecognizedPointsObservation.
But, when I tried to get information of detail,
like : let thumbPoint = try!observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb)
[Segmentation fault: 11 ] keep comes up.
Is this a bug ? or do I making some mistakes?
Please give me some answer...
On arkit, I put these codes...
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let handler = VNImageRequestHandler(cvPixelBuffer: frame.capturedImage, orientation: .up, options: [:])
do {
try? handler.perform([handPoseRequest])
guard let observation = handPoseRequest.results?.first as?VNRecognizedPointsObservation else {return}
let thumb = try! observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb)
} catch {}
}
At last part with let thumb.. segmentation fault 11 comes up.
I saw a twit of video running this, but I tried this for week in any possible ways, I couldn't figure this out.
Please, please give me some answer 😭
In WWDC21 Lecture video, he said's only iphone with neural engine support captureTextFromCamera Function.
As I tried, it does work on iPhone, but not on ipad pro 5th. gen.
During beta, does it planed to support iPad-os?
I tried to add com.apple.developer.kernel.increased-memory-limit entitlements, but don't know which capability to add.
I found out that it works well in this new api on beta.
But I want to make more customizable settings, so want to make one with AVFoundation.
I want to know the AVCaptureSession and AVCapturePhotoSettings that is applied on this new API.
With code below, I added color and depth image from RealityKit ARView, and ran Photogrammetry on iOS device, the mesh looks fine, but the scale of the mesh is quit different with real world scale.
let color = arView.session.currentFrame!.capturedImage
let depth = arView.session.currentFrame!.sceneDepth!.depthMap
//😀 Color
let colorCIImage = CIImage(cvPixelBuffer: color)
let colorUIImage = UIImage(ciImage: colorCIImage)
let depthCIImage = CIImage(cvPixelBuffer: depth)
let heicData = colorUIImage.heicData()!
let fileURL = imageDirectory!.appendingPathComponent("\(scanCount).heic")
do {
try heicData.write(to: fileURL)
print("Successfully wrote image to \(fileURL)")
} catch {
print("Failed to write image to \(fileURL): \(error)")
}
//😀 Depth
let context = CIContext()
let colorSpace = CGColorSpace(name: CGColorSpace.linearGray)!
let depthData = context.tiffRepresentation(of: depthCIImage,
format: .Lf,
colorSpace: colorSpace,
options: [.disparityImage: depthCIImage])
let depth_dir = imageDirectory!.appendingPathComponent("IMG_\(scanCount)_depth.TIF")
try! depthData!.write(to: depth_dir, options: [.atomic])
print("depth saved")
And also tried this.
let colorSpace = CGColorSpace(name: CGColorSpace.linearGray)
let depthCIImage = CIImage(cvImageBuffer: depth,
options: [.auxiliaryDepth : true])
let context = CIContext()
let linearColorSpace = CGColorSpace(name: CGColorSpace.linearSRGB)
guard let heicData = context.heifRepresentation(of: colorCIImage,
format: .RGBA16,
colorSpace: linearColorSpace!,
options: [.depthImage : depthCIImage]) else {
print("Failed to convert combined image into HEIC format")
return
}
Does Anyone know why and how to fix this?
If I make custom point cloud, how can I send this to photogrammetry session? Does it seperately saved to directory? Or does it saved into heic image?
With AVFoundation's builtInLiDARDepthCamera,
if I save photo.fileDataRepresentation to heic, it only has Exif and TIFF metadata.
But, RealityKit's object capture's heic image has not only Exif and TIFF, but also has HEIC metadata including camera calibration data.
What should I do for AVFoundation's exported image has same meta data?
In WWDC 2021, It saids 'we also offer an interface for advanced workflows to provide a sequence of custom samples.
A PhotogrammetrySample includes the image plus other optional data such as a depth map, gravity vector, or custom segmentation mask.'
But in code, PhotogrammetrySession initialize with data saved directory.
How can I give input of PhotogrammetrySamples to PhotogrammetrySession?