Post not yet marked as solved
Hi,
I'm trying to display an STL model file in visionOS. I import the STL file using SceneKit's ModelIO extension, add it to an empty scene USDA and then export the finished scene into a temporary USDZ file. From there I load the USDZ file as an Entity and add it onto the content.
However, the model in the resulting USDZ file has no lighting and appears as an unlit solid. Please see the screenshot below:
Top one is created from directly importing a USDA scene with the model already added using Reality Composer through in an Entity and works as expected.
Middle one is created from importing the STL model as an MDLAsset using ModelIO, adding onto the empty scene, exporting as USDZ. Then importing USDZ into an Entity. This is what I want to be able to do and is broken.
Bottom one is just for me to debug the USDZ import/export. It was added to the empty scene using Reality Composer and works as expected, therefore the USDZ export/import is not broken as far as I can tell.
Full code:
import SwiftUI
import ARKit
import SceneKit.ModelIO
import RealityKit
import RealityKitContent
struct ContentView: View {
@State private var enlarge = false
@State private var showImmersiveSpace = false
@State private var immersiveSpaceIsShown = false
@Environment(\.openImmersiveSpace) var openImmersiveSpace
@Environment(\.dismissImmersiveSpace) var dismissImmersiveSpace
var modelUrl: URL? = {
if let url = Bundle.main.url(forResource: "Trent 900 STL", withExtension: "stl") {
let asset = MDLAsset(url: url)
asset.loadTextures()
let object = asset.object(at: 0) as! MDLMesh
let emptyScene = SCNScene(named: "EmptyScene.usda")!
let scene = SCNScene(mdlAsset: asset)
// Position node in scene and scale
let node = SCNNode(mdlObject: object)
node.position = SCNVector3(0.0, 0.1, 0.0)
node.scale = SCNVector3(0.02, 0.02, 0.02)
// Copy materials from the test model in the empty scene to our new object (doesn't really change anything)
node.geometry?.materials = emptyScene.rootNode.childNodes[0].childNodes[0].childNodes[0].childNodes[0].geometry!.materials
// Add new node to our empty scene
emptyScene.rootNode.addChildNode(node)
let fileManager = FileManager.default
let appSupportDirectory = try! fileManager.url(for: .applicationSupportDirectory, in: .userDomainMask, appropriateFor: nil, create: true)
let permanentUrl = appSupportDirectory.appendingPathComponent("converted.usdz")
if emptyScene.write(to: permanentUrl, delegate: nil) {
// We exported, now load and display
return permanentUrl
}
}
return nil
}()
var body: some View {
VStack {
RealityView { content in
// Add the initial RealityKit content
if let scene = try? await Entity(contentsOf: modelUrl!) {
// Displays middle and bottom models
content.add(scene)
}
if let scene2 = try? await Entity(named: "JetScene", in: realityKitContentBundle) {
// Displays top model using premade scene and exported as USDA.
content.add(scene2)
}
} update: { content in
// Update the RealityKit content when SwiftUI state changes
if let scene = content.entities.first {
let uniformScale: Float = enlarge ? 1.4 : 1.0
scene.transform.scale = [uniformScale, uniformScale, uniformScale]
}
}
.gesture(TapGesture().targetedToAnyEntity().onEnded { _ in
enlarge.toggle()
})
VStack (spacing: 12) {
Toggle("Enlarge RealityView Content", isOn: $enlarge)
.font(.title)
Toggle("Show ImmersiveSpace", isOn: $showImmersiveSpace)
.font(.title)
}
.frame(width: 360)
.padding(36)
.glassBackgroundEffect()
}
.onChange(of: showImmersiveSpace) { _, newValue in
Task {
if newValue {
switch await openImmersiveSpace(id: "ImmersiveSpace") {
case .opened:
immersiveSpaceIsShown = true
case .error, .userCancelled:
fallthrough
@unknown default:
immersiveSpaceIsShown = false
showImmersiveSpace = false
}
} else if immersiveSpaceIsShown {
await dismissImmersiveSpace()
immersiveSpaceIsShown = false
}
}
}
}
}
#Preview(windowStyle: .volumetric) {
ContentView()
}
To test this even further, I exported the generated USDZ and opened in Reality Composer. The added model was still broken while the test model in the scene was fine. This also further proved that import/export is fine and RealityKit is not doing something weird with the imported model.
I am convinced this has to be something with the way I'm using ModelIO to import the STL file.
Any help is appreciated. Thank you
Post not yet marked as solved
I got a vision pro. I need a recording function, but when I looked it up in the developer documentation, I'm a little worried because it's only in the developer simulator. I'm making a recording app, but I'm asking other developers if they have any information because I don't want to make it so that Apple doesn't suggest or allow guidelines.
Post not yet marked as solved
Hi everyone
I'm making a small private app for my one of my engineering projects, a part of this app shows a 3D model of what it looks like in real life based on a position value of a joint that needs to be updated in real time.
I was able import a USDZ of the exact model of the project, and make the proper nodes that can rotate, however I run into a problem where SceneKit takes forever to update the node, I'm not sure if my code just needs optimizing or SceneKit is just not the framework to use when needing things in a 3D model to be updated in real time
I've confirmed that the device receives the values in realtime, it is just SceneKit that doesn't update the model in time
I'm not very good at explaining things so I put as much detail as I possibly can and hope my problem is clear, I'm also pretty new to swift and iOS development.
Here is the code I'm using
import SwiftUI
import SceneKit
struct ModelView2: UIViewRepresentable {
@State private var eulerAngle: Float = 0.0
@StateObject var service = BluetoothService()
let sceneView = SCNView()
func makeUIView(context: Context) -> SCNView {
if let scene = SCNScene(named: "V4.usdz") {
sceneView.scene = scene
if let meshInstanceNode = scene.rootNode.childNode(withName: "MeshInstance", recursively: true),
let meshInstance1Node = scene.rootNode.childNode(withName: "MeshInstance_1", recursively: true),
let meshInstance562Node = scene.rootNode.childNode(withName: "MeshInstance_562", recursively: true) {
// Rotate mesh instance around its own axis
/*
meshInstance562Node.eulerAngles = SCNVector3(x: 0, y: -0.01745329 * service.posititonValue, z: 0)
*/
print(meshInstance562Node.eulerAngles)
}
}
sceneView.allowsCameraControl = true
sceneView.autoenablesDefaultLighting = true
return sceneView
}
func updateUIView(_ uiView: SCNView, context: Context) {
if let scene = SCNScene(named: "V4.usdz") {
sceneView.scene = scene
if let meshInstanceNode = scene.rootNode.childNode(withName: "MeshInstance", recursively: true),
let meshInstance1Node = scene.rootNode.childNode(withName: "MeshInstance_1", recursively: true),
let meshInstance562Node = scene.rootNode.childNode(withName: "MeshInstance_562", recursively: true) {
let boundingBox = meshInstance562Node.boundingBox
let pivot = SCNMatrix4MakeTranslation(
boundingBox.min.x + (boundingBox.max.x - boundingBox.min.x) / 2,
boundingBox.min.y + (boundingBox.max.y - boundingBox.min.y) / 2,
boundingBox.min.z + (boundingBox.max.z - boundingBox.min.z) / 2
)
meshInstance562Node.pivot = pivot
meshInstance562Node.addChildNode(meshInstanceNode)
meshInstance562Node.addChildNode(meshInstance1Node)
var original = SCNMatrix4Identity
original = SCNMatrix4Translate(original, 182.85785, 123.54999, 17.857864) // Translate along the Y-axis
meshInstance562Node.transform = original
print(service.posititonValue)
var buffer: Float = 0.0
if service.posititonValue != buffer {
meshInstance562Node.eulerAngles = SCNVector3(x: 0, y: -0.01745329 * service.posititonValue, z: 0)
buffer = service.posititonValue
}
}
}
}
func rotateNodeInPlace(node: SCNNode, duration: TimeInterval, angle: Float) {
// Create a rotation action
let rotationAction = SCNAction.rotateBy(x: 0, y: CGFloat(angle), z: 0, duration: duration)
// Repeat the rotation action indefinitely
// let repeatAction = SCNAction.repeatForever(rotationAction)
// Run the action on the node
node.runAction(rotationAction)
print(node.transform)
}
func rotate(node: SCNNode, angle: Float) {
node.eulerAngles = SCNVector3(x: 0, y: -0.01745329 * angle, z: 0)
}
}
#Preview {
ModelView2()
}
Post not yet marked as solved
In our app we are having feature in which we scan our qr and shows an ARScene with floating images or texts(based on the qr data) in our app.From this screen we can move to the Home page of the app with a button click as well.
This was working fine till iOS16.After we updated to iOS 17 what we observe is that once the ARScene is dismissed and the next screen are loaded, there will not be any touches recognised in the positions where there used to be the ARScene.
For floating images the ARScene is the entire screen and so in this case the next screen wont have any touch at all
For floating texts the ARScene is the top part of screen and so in this case the next screen wont have any touch in the top half only.
This issue seems to go away once we minimise the app and open again.
Example: Consider a navigation stack with 3 viewcontrollers A, B and C. The AR content is present in B.
If we load the AR content in B and move to A or C those screens are unresponsive at the positions where the ARScene was loaded
But If we dont load the AR content in B i.e just open B and move to A or C those screens have no issues.
Post not yet marked as solved
Is there a SceneKit equivalent of the HoverEffectComponent used in RealityKit to highlight an entity as the user looks around a scene in a VisionOS app?
Post not yet marked as solved
We scan the room using the RoomPlan API, and after the scan, we obtain objects with a white color along with shadows and shading. However, upon updating the color of these objects, we experience a loss of shadows and shading.
RoomPlan scan
After Update
Post not yet marked as solved
Does anyone have a working example on how to play OGG files with swift?
I've been trying for over a year now. I was able to wrap the C Vorbis library in swift. I then used it to parse an OGG file successfully. Then I was required to use Obj-C\++ to fill the PCM because this method seems to only be available in C\++ and that part hangs my app for a good 40 seconds to several minutes depending on the audio file, it then plays for about 2 seconds and then crashes.
I can't get the examples on the Vorbis site to work in objective-c and i tried every example on github I could find (most of which are for iOS - I want to play the files on mac)
I also tried using Cricket Audio framework below.
https://github.com/sjmerel/ck
It has a swift example and it can play their proprietary soundbank format but it is also supposed to play OGG and it just doesn't do anything when trying to play OGG as you can see in the posted issue
https://github.com/sjmerel/ck/issues/3
Right now I believe every player that can play OGGs on mac is written in Objective-C or C++.
Anyway, any help/advice is appreciated. OGG format is very prevalent in the gaming community. I could use unity, which I believe plays oggs through the mono framework but I really really want to stay in swift.
Post not yet marked as solved
Greetings!
I have made use of Apple ARKit documentations to create a simple ARKit application which utilizes SceneKit (Tried Metal too)
I am currently unsure of how to make use of SmoothedSceneDepth(SceneDepth) in general to acquire the DepthData from the DataMap acquired in the View.
is there any particular method or way that I can access this data for displaying the depth.
would be grateful with any inputs or suggestions.
Thanks in advance
Post not yet marked as solved
I found Scenekit crash on iOS 17 very frequently for all device on iOS 17
here is crash trace
Crashed: com.apple.scenekit.renderingQueue.SCNView0x15878c630
0 SceneKit 0x3eee4 C3DMatrix4x4GetAffineTransforms + 344
1 SceneKit 0x30208 C3DAdjustZRangeOfProjectionInfos + 140
2 SceneKit 0x2c0a90 C3DCullingContextSetupPointOfViewMatrices + 700
the attachment have the whole log
Crash Log
have anybody know how fo fix it
Post not yet marked as solved
I am trying to use my animated model in XCode with SceneKit. I exported my model from Maya with Animation Data in .usd format, then converted it to .usdz with Reality Converter. When I open it in XCode viewer it is animated and everything is fine. However when I try to use it in my app it doesn't animate. On the other hand, when I try with the robot_walk_idle model from Apple's example models, it is animated. Maybe I am missing a option in export settings. Thanks for any help.
import SwiftUI
import SceneKit
struct ModelView: View {
var body: some View{
VStack{
SceneView(scene: SCNScene(named: "robot_walk_idle.usdz"))
}
}
}
Post not yet marked as solved
The touch input stutter issue that exists since iOS 16 on devices with Pro Motion Displays has not been fixed yet. I filed a bug report in July but there isn't any progress since months.
I see the problem in all games I tried. My game is fast paced so the stutters are quite obvious and I receive a lot of complaining emails.
My game did run smoothly on Pro Motion devices with iOS 15. Is there a known workaround? I am seeing other developers having the same issue but I can't find any solutions.
Other threads about this issue:
IPhone 14 Pro stuttering in most games when using touch controls
FPS drops when tapping the screen on iPhone 13 Pro Max
Post not yet marked as solved
I am trying to control the orientation of a box in Scene Kit (iOS) using gestures. I am using the translation in x and y to update the x and y rotation of the SCNNode.
After a long search I have realised that x and y rotation will always lead to z rotation, thanks to this excellent post: [https://gamedev.stackexchange.com/questions/136174/im-rotating-an-object-on-two-axes-so-why-does-it-keep-twisting-around-the-thir?newreg=130c66c673f848a7be2873bf675573a9)
So I am trying to get the z rotation causes, and then remove this from my object by applying the inverse quaternion however when I rotate the object 90 deg around x, and then 90 deg around Y it behaves VERY weirdly.
It is almost behaving as it is in gimbal lock, but I did not think that using quaternion in the way that I am would cause gimbal lock in this way.
I am sure it is something I am missing, or perhaps I am not able to remove the z rotation in this way.
Thanks!
I have added a video of the strange behaviour here [https://github.com/marcusraty/RotationExample/blob/main/Example.MP4)
And the code example is here [https://github.com/marcusraty/RotationExample)
Post not yet marked as solved
Hi everyone, I am trying on iOS 17.1.1 the Nearby Interaction framework and SceneKit. I am testing it on iPhone 15 Pro Max and iPhone 12 Pro Max.
if NISession.deviceCapabilities.supportsDirectionMeasurement {
print("Interact using device distance and direction.")
} else if NISession.deviceCapabilities.supportsPreciseDistanceMeasurement {
print("Interact using distance only.")
}
iPhone 12 Pro Max is working normally, but supportsDirectionMeasurement property in iPhone 15 Pro Max returns false and i cant use the SceneKit.
Is anyone experiencing the same issue?
Regards,
Shin
Post not yet marked as solved
In an Xcode MultiPlatorm App, the following works as expected (detecting tap gestures) in iOS simulator, but not when compiled and run on "My Mac" -- on Mac OS the view doesn't seem to get any clicks.
Does anyone know a way to get this to work?
One of the reasons
import SwiftUI
import SceneKit
class RenderDelegate: NSObject, SCNSceneRendererDelegate {
var lastRenderer: SCNSceneRenderer!
func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
lastRenderer = renderer
}
}
class Model: ObservableObject {
let scene = SCNScene()
let renderDelegate = RenderDelegate()
}
struct ContentView : View {
@ObservedObject var model = Model()
@State private var pointOfView = "distantCamera"
init() {
let sphereGeometry = SCNSphere(radius: 0.05)
#if os(iOS) || os(watchOS) || os(tvOS)
sphereGeometry.firstMaterial?.diffuse.contents = UIColor.red
#else
sphereGeometry.firstMaterial?.diffuse.contents = NSColor.red
#endif
let sphereNode = SCNNode(geometry: sphereGeometry)
sphereNode.position = SCNVector3Make(0.0, 0.0, 0.0)
model.scene.rootNode.addChildNode(sphereNode)
}
var body: some View {
ZStack {
SceneView(
scene: model.scene,
options: [ .allowsCameraControl ],
delegate: model.renderDelegate
).onTapGesture {print("tap")}
}
.padding()
}
}
struct ContentView_Previews: PreviewProvider {
static var previews: some View {
ContentView()
}
}
Post not yet marked as solved
I have a spherical HDR image that is being used for environment lighting in a SceneKit scene. I want to rotate the environment image.
To set the environment lighting, I use the lightingEnvironment SCNMaterialProperty. This works fine, and my scene is lit using the IBL.
As with all SCNMaterialProperty, I expect that I can use the contentsTransform property to rotate or transform the HDR. So I set it as follows:
lightingEnvironment.contentsTransform = SCNMatrix4MakeRotation((45.0).degreesAsRadians, 0.0, 1.0, 0.0)
My expectation is that the lighting environment would rotate 45 degrees in Y, but it doesn't change at all. Even if I throw in a completely random transform on all axis, there is no apparent change.
To test if there is a change, I added a chrome ball and a diffuse ball to my scene and I'm comparing reflections on the chrome ball, and lighting on the diffuse ball. There is no change on either.
It doesn't matter where I set the contentsTransform, it doesn't work. I had intended to set it from the renderer(_:updateAtTime:) method on the SCNRendererDelegate, so that I can rotate the IBL to match the point of view of the scene, but even if I transform the environment immediately after it is set, there is never a change.
Is this a bug? Or am I doing something entirely wrong? Has anyone on here ever managed to get this to work?
Post not yet marked as solved
After the iOS 17 update, objects rendered in SceneKit that have both a normal map and morph targets do not render correctly. The shading and lighting appear dark and without reflections. Using a normal map without morph targets or having morph targets on an object without using a normal map works fine. However, the combination of using both breaks the rendering.
Using diffuse, normal map and a morpher:
Diffuse and normal, NO morpher:
Post not yet marked as solved
I am exploring ARKit and SceneKit, but I am not sure if what I want to do is possible.
In App1:
Run an ARKit session using configuration.sceneReconstruction = .mesh
I am rending the mesh in func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) and func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) and the mesh appears correct
I have set up a button that does the following:
Capture the mesh to an .obj file (based off the excellent answer in here)
Capture a snapshot (SCNView snapshot)
Store the AR Camera Transform
Store the AR SCN View Point Of View Projection Transform
Storing the transforms in a CSV, and taking care to ensure I restore them in column major order.
In App2:
Load the geometry from the .obj file into SCNNode, do not apply any transform to it, apply a wireframe to it so I can visualise it
Add a camera and apply the 2 saved transforms (AR Camera Transform and then AR SCN View Point Of View Projection Transform
Set the background of the scene to be the image from the snapshot.
I was expecting that the mesh, as visualised by the wireframe in App2 would match the rendering as captured in a point in time from App1 - however I cannot get it to match.
I have two specific questions:
Have I missed something fundamental in my understanding of what is possible? Should this be possible and could I be missing some step? The codebase is large but I have put the basic outline here
What is the difference between ScnCamera Projection Transform and AR Camera Project Matrix - they appear different in App1 which is not expected to me.
Thanks very much
Post not yet marked as solved
I am in need to create a 2D floor plan with the dimensions mentioned, from the generated 3D result of RoomPlan. Is there a way to create it in a little easy-to-understand manner? Or will it require manual elaborate coding?
Post not yet marked as solved
I've watched this issue for a long time but it seems this hasn't been fixed yet.
My use case is to assign a UIView to the 'contents' variable of SCNMaterialProperty.
It works without problem in terms of rendering, but when I assign 'nil' to the variable the allocated memory of IOSurface by SceneKit does not being destroyed
I've searched about this and many other developers have been suffered by this issue. I did a 'Game Memory' profiling of my toy example and the allocated memory (134MB) by SceneKit hadn't been released after I've assigned nil.
I'm sure I released every relavant UIViews and view controllers used for the 'contents'.
Post not yet marked as solved
Dear Apple Team and everyone who has experience with MapKit.
I am building an app where I need to hide some 3D models and replace them with my custom 3D meshes using SceneKit.
Up until now I was using Mapbox it allows to get mesh row data to reconstruct all maps 3D.
Is there something like this possible with MapKit?
Use cases
Say you navigated to Kennedy Space Center Launch Complex 39 and there is no 3D model of actual building. I would like to be able to hide simple massing and replace it with my model.
In 3D Satellite VIew some areas have detailed meshes. Say London The Queen's Walk. I would like to make specific area flat so I can place my 3D model on top of Satellite 3D View to illustrate new structure or building.
Last one. Is it possible to change existing buildings colours? I know it is possible transparency
Thank you
@apple
zer0x.
