Hi, just got an Apple M3 Pro to try it out on some Jax operations. I see the development is actively ongoing so maybe this error can help. This is the environment: Metal device set to: Apple M3 Pro systemMemory: 18.00 GB maxCacheSize: 6.00 GB jax: 0.4.26 jaxlib: 0.4.23 numpy: 1.26.4 python: 3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:49:36) [Clang 16.0.6 ] jax.devices (1 total, 1 local): [METAL(id=0)] process_count: 1 platform: uname_result(system='Darwin', node='MKFL96VR9YT', release='23.4.0', version='Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:54 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6030', machine='arm64') This is a minimal example which produces an error, I think due to the fft part: from jax import numpy as np array = np.ones((16, 16)) np.fft.fft2(array) This is the full traceback: Traceback (most recent call last): File "/Users/user/Downloads/wow.py", line 5, in <module> np.fft.fft2(array) File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/numpy/fft.py", line 216, in fft2 return _fft_core_2d('fft2', xla_client.FftType.FFT, a, s=s, axes=axes, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/numpy/fft.py", line 210, in _fft_core_2d return _fft_core(func_name, fft_type, a, s, axes, norm) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/numpy/fft.py", line 102, in _fft_core transformed = lax.fft(arr, fft_type, tuple(s)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/traceback_util.py", line 179, in reraise_with_filtered_traceback return fun(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/pjit.py", line 298, in cache_miss outs, out_flat, out_tree, args_flat, jaxpr, attrs_tracked = _python_pjit_helper( ^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/pjit.py", line 176, in _python_pjit_helper out_flat = pjit_p.bind(*args_flat, **params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/core.py", line 2788, in bind return self.bind_with_trace(top_trace, args, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/core.py", line 425, in bind_with_trace out = trace.process_primitive(self, map(trace.full_raise, args), params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/core.py", line 913, in process_primitive return primitive.impl(*tracers, **params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/pjit.py", line 1494, in _pjit_call_impl return xc._xla.pjit(name, f, call_impl_cache_miss, [], [], donated_argnums, # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/pjit.py", line 1471, in call_impl_cache_miss out_flat, compiled = _pjit_call_impl_python( ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/pjit.py", line 1406, in _pjit_call_impl_python lowering_parameters=mlir.LoweringParameters()).compile() ^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/interpreters/pxla.py", line 2369, in compile executable = UnloadedMeshExecutable.from_hlo( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/interpreters/pxla.py", line 2908, in from_hlo xla_executable, compile_options = _cached_compilation( ^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/interpreters/pxla.py", line 2718, in _cached_compilation xla_executable = compiler.compile_or_get_cached( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/compiler.py", line 266, in compile_or_get_cached return backend_compile(backend, computation, compile_options, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/profiler.py", line 335, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/anaconda3/envs/jaxmetal/lib/python3.11/site-packages/jax/_src/compiler.py", line 238, in backend_compile return backend.compile(built_c, compile_options=options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ jaxlib.xla_extension.XlaRuntimeError: UNKNOWN: <unknown>:0: error: 'func.func' op One or more function input/output data types are not supported. <unknown>:0: note: see current operation: "func.func"() <{arg_attrs = [{mhlo.layout_mode = "default", mhlo.sharding = "{replicated}"}], function_type = (tensor<16x16xf32>) -> tensor<16x16xcomplex<f32>>, res_attrs = [{jax.result_info = "", mhlo.layout_mode = "default"}], sym_name = "main", sym_visibility = "public"}> ({ ^bb0(%arg0: tensor<16x16xf32>): %0 = "mhlo.convert"(%arg0) : (tensor<16x16xf32>) -> tensor<16x16xcomplex<f32>> %1 = "mhlo.fft"(%0) {fft_length = dense<16> : tensor<2xi64>, fft_type = #mhlo<fft_type FFT>} : (tensor<16x16xcomplex<f32>>) -> tensor<16x16xcomplex<f32>> "func.return"(%1) : (tensor<16x16xcomplex<f32>>) -> () }) : () -> () <unknown>:0: error: failed to legalize operation 'func.func' <unknown>:0: note: see current operation: "func.func"() <{arg_attrs = [{mhlo.layout_mode = "default", mhlo.sharding = "{replicated}"}], function_type = (tensor<16x16xf32>) -> tensor<16x16xcomplex<f32>>, res_attrs = [{jax.result_info = "", mhlo.layout_mode = "default"}], sym_name = "main", sym_visibility = "public"}> ({ ^bb0(%arg0: tensor<16x16xf32>): %0 = "mhlo.convert"(%arg0) : (tensor<16x16xf32>) -> tensor<16x16xcomplex<f32>> %1 = "mhlo.fft"(%0) {fft_length = dense<16> : tensor<2xi64>, fft_type = #mhlo<fft_type FFT>} : (tensor<16x16xcomplex<f32>>) -> tensor<16x16xcomplex<f32>> "func.return"(%1) : (tensor<16x16xcomplex<f32>>) -> () }) : () -> () I'd be happy running more tests should you need them, I'm new to this, so not sure which just yet. Many thanks!!

(Copied from https://github.com/google/jax/issues/20835) I am attempting to use JAX on Metal (on a M1 Pro chip) to model discrete (count) data. I've installed the latest version jax-metal 0.0.6 using pip. The installation seems to have worked overall as I can perform basic Jax array operations on GPU. However, when I try to compute the (log-)PMFs/PDFs of random variables which are defined in terms of the (log-)Gamma function I get errors like the one below which seems to indicate that the lax.lgamma function is not supported under the hood on M1 metal. This is essential functionality for a wide class of probabilistic machine learning models. Note that following functions (among others) are broken as a result: jax.scipy.stats.binom.logpmf jax.scipy.stats.nbinom.logpmf jax.scipy.stats.poisson.logpmf jax.scipy.stats.dirichlet.logpdf jax.scipy.stats.beta.logpdf jax.scipy.stats.gamma.logpdf ... >>> jax.scipy.stats.binom.logpmf(1, n=2, p=0.5) jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/ljb80/.virtualenvs/jax-metal/lib/python3.10/site-packages/jax/_src/scipy/stats/binom.py", line 31, in logpmf gammaln(n + 1), File "/Users/ljb80/.virtualenvs/jax-metal/lib/python3.10/site-packages/jax/_src/scipy/special.py", line 44, in gammaln return lax.lgamma(x) File "/Users/ljb80/.virtualenvs/jax-metal/lib/python3.10/site-packages/jax/_src/lax/special.py", line 46, in lgamma return lgamma_p.bind(x) File "/Users/ljb80/.virtualenvs/jax-metal/lib/python3.10/site-packages/jax/_src/core.py", line 422, in bind return self.bind_with_trace(find_top_trace(args), args, params) File "/Users/ljb80/.virtualenvs/jax-metal/lib/python3.10/site-packages/jax/_src/core.py", line 425, in bind_with_trace out = trace.process_primitive(self, map(trace.full_raise, args), params) File "/Users/ljb80/.virtualenvs/jax-metal/lib/python3.10/site-packages/jax/_src/core.py", line 913, in process_primitive return primitive.impl(*tracers, **params) File "/Users/ljb80/.virtualenvs/jax-metal/lib/python3.10/site-packages/jax/_src/dispatch.py", line 87, in apply_primitive outs = fun(*args) jaxlib.xla_extension.XlaRuntimeError: UNKNOWN: <stdin>:1:0: error: failed to legalize operation 'chlo.lgamma' <stdin>:1:0: note: see current operation: %0 = "chlo.lgamma"(%arg0) : (tensor<f32>) -> tensor<f32> System info (python version, jaxlib version, accelerator, etc.) jax: 0.4.26 jaxlib: 0.4.23 numpy: 1.26.4 python: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:38:29) [Clang 13.0.1 ] jax.devices (1 total, 1 local): [METAL(id=0)] process_count: 1 platform: uname_result(system='Darwin', node='PHS027794', release='23.4.0', version='Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000', machine='arm64')

Hi there, I have some existing metal rendering / shader views that I would like to use to present stereoscopic content on the Vision Pro. Is there a metal shader function / variable that lets me know which eye we're currently rendering to inside my shader? Something like Unity's unity_StereoEyeIndex? I know RealityKit has GeometrySwitchCameraIndex, so I want something similar (but outside of a RealityKit context). Many thanks, Rich

What is the most efficient way to use a MTLTexture (created procedurally at run-time) as a RealityKit TextureResource? I update the MTLTexture per-frame using regular Metal rendering, so it’s not something I can do offline. Is there a way to wrap it without doing a copy? A specific example would be great. Thank you!

Help!!!

When I distribute a app that includes Apple Unity plugin, I got an error of asset validation failed. The bundle 'Payload/{app name}/Frameworks/AppleCoreNative.framework' is missing plist key. The Info.plist file is missing the required key: CFBundleShortVersionString. Does anyone get this?

The Drawing fully immersive content using Metal guide describes how to use Metal for visionOS immersive experiences, but seemingly requires swift to bring up the required CompositorLayer. @main struct MyApp: App { var body: some Scene { ImmersiveSpace(id: "MyContent") { CompositorLayer { layerRenderer in let renderThread = Thread { let engine = myEngineCreate(layerRenderer) myEngineRenderLoop(engine) } renderThread.name = "Render Thread" renderThread.start() } } } The ImmersiveSpace scene can presumably be replaced with a call to [UIApplication.sharedApplication activateSceneSessionForRequest:[UISceneSessionActivationRequest requestWithRole:UISceneSessionRoleImmersiveSpaceApplication] errorHandler:nil] But is there a replacement for CompositorLayer? Or some other way to produce a cp_layer_renderer? Perhaps it would be possible to write a small swift helper for this, but given the swift interface for CompositorLayer how would that be tied to an existing UIScene as created above? @available(visionOS 1.0, *) public struct CompositorLayer : SwiftUI.ImmersiveSpaceContent { public init(configuration: any _CompositorServices_SwiftUI.CompositorLayerConfiguration = .default, renderer: @escaping (CompositorServices.LayerRenderer) -> Swift.Void) public var body: Swift.Never { get } public typealias Body = Swift.Never }

Is it possible to set the header include paths when compiling a metal library from source at runtime? I'm dynamically generating kernel source that I'm compiling at runtime. I want to include some functions I have defined in a header. If I was doing online computation, I can specify include paths using -I /path/to/include using the command. xcrun metal -c -I /path/to/include example.metal -o example.air However when I'm online, it doesn't appear possible to define a header search path. MTLCompileOptions *options = [MTLCompileOptions new]; options.fastMathEnabled = NO; library = [device newLibraryWithSource:kernel_source options:options error:&error]; Is there a way to specify a header search path using newLibraryWithSource?

I wanted to show a progress of a certain part of the game using an entity that looks like a "pie chart", basically cylinder with a cut-out. And as progress is changed (0-100) the entity would be fuller. Is there a way to create this kind of model entity? I know there are ways to animated entities, warp them between meshes, but I was wondering if somebody knows how to achieve it in a simplest way possible? Maybe some kind of custom shader that would just change how the material is rendered? I do not need its physics body, just to show it. I know how to do it in UIKit and classic 2d UI Apple frameworks but here working with model entities it gets a bit tricky for me. Here is example of how it would look, examples are in 2d but you can imagine it being 3d cylinders with a cut-out. Thank you!

Where do I start with this error? I am using the Metal Debugger and have.a bunch of stuck command buffers. how do I look at the command buffers to see the errors? My suspicion is that the cause is some sort of memory leak. Not having access to the source for Metal leaves me stuck. The following message shows up in the logging pane of the execution. Execution of the command buffer was aborted due to an error during execution. Ignored (for causing prior/excessive GPU errors) (00000004:kIOGPUCommandBufferCallbackErrorSubmissionsIgnored) Type: Error | Timestamp: 2024-04-11 14:16:13.464336-05:00 | Library: Metal | Subsystem: Metal | Category: Default | TID: 0x2a0b8c I just need some guidance

I'm brand new to Metal. I've googled, but can't get the right answer to come up. (Thanks, unhelpful ChatGPT generated answers polluting everything, but I digress...) Ultimately, I'm trying to figure out how to use Metal to render 3D DICOM data on iOS specifically. If you're not familiar with DICOM, let's just say I've got a whole stack of CT image slices. Or to get really simple, I've got a cube of voxel values with differing values at each voxel coordinate. Where do I even start in Metal to render something like this? (I was trying to get the VTK toolkit compiled for iOS, which uses OpenGL, but that appears to be a dead end. And besides, Metal is supposed to be so much better.) Thanks for any tips/leads/suggestions/general pointers.

Hi there - Where would a dev go these days to get an initial understanding of SceneKit? The WWDC videos linked in various places seem to be gone?! For example, the SceneKit page at developer.apple.com lists features a session videos link that comes up without any result, https://developer.apple.com/scenekit/ Any advice..? Cheers, Jan

In the WWDC talk "Enhance your spatial computing app with RealityKit." we see how to create a portal effect with RealityKit. In the "Encounter Dinosaurs" experience on Vision Pro there is a similar portal, except this portal allows entities to stick out of the portal. Using the provided example code, I have been unable to replicate this effect. With the example code, anything that sticks out of the portal gets clipped. How do I get entities to stick out of the portal in a way similar to the "Encounter Dinosaurs" experience? I am familiar with the old way of using OcclusionMaterial to create portals, but if the camera gets between the OcclusionMaterial and the entity (such as walking behind the portal), this can break the effect, and I was unable to break the effect in the "Encounter Dinosaurs" experience. If it helps at all: I have noticed that if you look from the edge of the portal very closely, the rocks will not stick out the way that the dinosaurs do; The rocks get clipped. Therefore, the dinosaurs are somehow being rendered differently.

I am trying to implement a way to rotate a 3D model around its y axis, but this doesn't seem to work. What am I missing? The scene only contains one model entity. @State private var rotateBy:Double = 0.0 RealityView { content in do { let entity = try await Entity.init(named: "VinylScene", in: realityKitContentBundle) entity.scale = SIMD3<Float>(repeating: 0.6) content.add(entity) } catch { ProgressView() } } .gesture( DragGesture(minimumDistance: 0.0) .targetedToAnyEntity() .onChanged { value in let location3d = value.convert(value.location3D, from: .local, to: .scene) let startLocation = value.convert(value.startLocation3D, from: .local, to: .scene) let delta = location3d - startLocation rotateBy = Double(atan(delta.x * 200)) } )

Hi, I'm struggling a little with the Shader Graph in Reality Composer Pro. Doing something simple like linearly interpolating between two colors has me searching through nodes without any luck. Is there a node comparable to the lerp node in Unity? And if so, what is it called?

App Store Connect says that the name for my app is taken, but when I search on the App Store, there aren't any apps with that name. Is this a bug? Is there a way to contact Apple so that we can claim the name we'd like? Thank you!

I have a fbo generated by last frame, which contain a color texture and a depth texture. Then i want blit that fbo to current MTKView, but failed. Thanks a lot for any suggestions. Here is the code: last frame: id <MTLTexture> src_color; id <MTLTexture> src_depth; drawTo(src_color, src_depth); current frame: first, init a depth texture if null: id<MTLTexture> depth_texture_; if(depth_texture_) { depth_texture_ = [device_ newTextureWithDescriptor:desc]; } second， create desc for encoder: current_desc = metal_view_.currentRenderPassDescriptor; current_desc.depthAttachment.texture = depth_texture_; current_desc.depthAttachment.loadAction = MTLLoadActionClear; current_desc.depthAttachment.clearDepth = 1.0; current_desc.stencilAttachment.texture = depth_texture_; current_desc.stencilAttachment.loadAction = MTLLoadActionClear; current_desc.stencilAttachment.clearStencil = 0; current_desc.colorAttachments[0].loadAction = MTLLoadActionClear; current_desc.colorAttachments[0].clearColor = MTLClearColorMake(1.0, 1, 1, 1.0); third, blit to current MTKView: auto dst_color = current_desc.colorAttachments[0].texture; auto dst_depth = current_desc.depthAttachment.texture; id<MTLCommandBuffer> commandBuffer = [command_queue_ commandBuffer]; commandBuffer.label = @"blit"; id <MTLBlitCommandEncoder> blitEncoder = [commandBuffer blitCommandEncoder]; [blitEncoder copyFromTexture:src_color sourceSlice:0 sourceLevel:0 sourceOrigin:MTLOriginMake(srcRect.x, srcRect.y, 0) sourceSize:MTLSizeMake(srcRect.width, srcRect.height, 1) toTexture:dst_color destinationSlice:0 destinationLevel:0 destinationOrigin:MTLOriginMake(dstRect.x, dstRect.y, 0)]; [blitEncoder copyFromTexture:src_depth sourceSlice:0 sourceLevel:0 sourceOrigin:MTLOriginMake(srcRect.x, srcRect.y, 0) sourceSize:MTLSizeMake(srcRect.width, srcRect.height, 1) toTexture:dst_depth destinationSlice:0 destinationLevel:0 destinationOrigin:MTLOriginMake(dstRect.x, dstRect.y, 0)]; [blitEncoder endEncoding]; [commandBuffer commit]; [commandBuffer waitUntilCompleted];

I'm trying to follow the metal-cpp tutorials I've found at https://developer.apple.com/metal/sample-code/?q=learn The program seems to be launching correctly (I can see the menu bar and interact with it), but nothing is rendered inside the window. I suppose the culprit is somewhere in the following function (I see it binds the device, the view and the window with the object in charge of drawing stuff in the view) void core::Application::applicationDidFinishLaunching(NS::Notification *pNotification) { CGRect frame = (CGRect){{100.0, 100.0}, {512.0, 512.0}}; m_Window->init(frame, NS::WindowStyleMaskClosable | NS::WindowStyleMaskTitled, NS::BackingStoreBuffered, false); m_Device = MTL::CreateSystemDefaultDevice(); m_View = MTK::View::alloc()->init(frame, m_Device); m_View->setColorPixelFormat(MTL::PixelFormat::PixelFormatBGRA8Unorm); m_View->setClearColor(MTL::ClearColor::Make(1.0, 0.0, 0.0, 1.0)); m_ViewDelegate = new graphics::ViewDelegate(m_Device); m_View->setDelegate(m_ViewDelegate); m_Window->setContentView(m_View); m_Window->setTitle(NS::String::string("Template 1", NS::StringEncoding::UTF8StringEncoding)); m_Window->makeKeyAndOrderFront(nullptr); NS::Application* nsApp = reinterpret_cast<NS::Application*>(pNotification->object()); nsApp->activateIgnoringOtherApps(true); } but, as you can infer from the fact that I'm failing at the very first tutorial of the bunch, I'm quite lost. I've tried debugging the app with the Xcode debugger and I saw that it never enters in this function. void ViewDelegate::drawInMTKView(MTK::View *pView) { m_Renderer->Draw(pView); } Can it be a symptom of some call missing from my code? Thank you in advance for your help

Hi, I have a CUDA program that I want to convert to Metal Compute so that we can support Apple hardware. When I wrote the CUDA version, I was able to write efficient code because I learned first about the Cuda-core architecture. The way the cores can access memory for instance is very important information so that I could write code that efficiently access the memory. Now I want to do the same for the Metal Compute software. But I can not find any information about the low level architecture and especially the things you should know to be able to write efficient code. Do I miss something? Is there some guide giving hints for the most efficient way to access memory for instance?

I am working on an application where we are planning to use Metal for directly rendering custom content. When user looks at something on the rendered image, I want to get the position or ray of cursor (the point where the user is currently looking at) to render something else like a crosshair. Is it possible to get the cursor position information on VisionOS to accomplish this? How can I know if something is being hovered on by the eyes?