This restriction causes me to be unable to use Metal to create images and simultaneously use Swift to add UI controls or RealityKit content (without using a window) in immersive mode.
Metal
RSS for tagRender advanced 3D graphics and perform data-parallel computations using graphics processors using Metal.
Posts under Metal tag
200 Posts
Sort by:
Post
Replies
Boosts
Views
Activity
Since I haven't heard so much as a peep from Apple on this, I thought I'd take a poll here on how long I could expect an easily reproducible (albeit possibly obscure) kernel panic to be fixed.
I was under the impression that kernel panics were a big deal but it's been almost 2 months since I updated from macOS 14 to macOS 15.0 dev beta 7 / public beta 5 when I originally came across and reported a panic triggered while playing StarCraft II.
I've been able to consistently trigger panics playing certain (maybe all) Co-op maps in SC2 and since my first report Aug 22, I've filed 8 additional bug reports, each automatically generated after hitting yet another panic. (I'm not sure exactly who is able to view these but for what it's worth, these are the reports I've filed so far: FB14886510, FB14905773, FB14960435, FB15304609, FB15391195, FB15467943, FB15468127, FB15491485, FB15491684.)
A few other people have reported the issue to SC2's developer, Blizzard, and apparently Blizzard has acknowledged they're aware of the problem so it's safe to rule out the possibility of a hardware defect or other issue specific only to my computer.
The logs point the blame at the AppleDCP driver, although I suppose the problem could technically be in the DCP firmware instead. Regardless, Apple's code is clearly at fault here.
I'll admit the importance of a video game isn't exactly like keeping the power on at a hospital but I don't know why it would be deemed particularly unimportant either.
At 53 days in, am I wrong to expect this to have been fixed by now or is Apple really being that slow?
Given
I do not understand much at all about how to write shaders
I do not understand the math associated with page-curl effects
I am trying to:
implement a page-curl shader for use on SwiftUI views.
I've lifted a shader from HIROKI IKEUCHI that I believe they lifted from a non-metal shader resource online, and I'm trying to digest it.
One thing I want to do is to paint the "underside" of the view with a given color and maintain the transparency of rounded corners when they are flipped over.
So, if an underside pixel is "clear" then I want to sample the pixel at that position on the original layer instead of the "curl effect" pixel.
There are two comments in the shader below where I check the alpha, and underside flags, and paint the color red as a debug test.
The shader gives this result:
The outside of those rounded corners is appropriately red and the white border pixels are detected as "not-clear". But the "inner" portion of the border is... mistakingly red?
I don't get it. Any help would be appreciated. I feel tapped out and I don't have any IRL resources I can ask.
//
// PageCurl.metal
// ShaderDemo3
//
// Created by HIROKI IKEUCHI on 2023/10/17.
//
#include <metal_stdlib>
#include <SwiftUI/SwiftUI_Metal.h>
using namespace metal;
#define pi float(3.14159265359)
#define blue half4(0.0, 0.0, 1.0, 1.0)
#define red half4(1.0, 0.0, 0.0, 1.0)
#define radius float(0.4)
// そのピクセルの色を返す
[[ stitchable ]] half4 pageCurl
(
float2 _position,
SwiftUI::Layer layer,
float4 bounds,
float2 _clickedPoint,
float2 _mouseCursor
) {
half4 undersideColor = half4(0.5, 0.5, 1.0, 1.0);
float2 originalPosition = _position;
// y座標の補正
float2 position = float2(_position.x, bounds.w - _position.y);
float2 clickedPoint = float2(_clickedPoint.x, bounds.w - _clickedPoint.y);
float2 mouseCursor = float2(_mouseCursor.x, bounds.w - _mouseCursor.y);
float aspect = bounds.z / bounds.w;
float2 uv = position * float2(aspect, 1.) / bounds.zw;
float2 mouse = mouseCursor.xy * float2(aspect, 1.) / bounds.zw;
float2 mouseDir = normalize(abs(clickedPoint.xy) - mouseCursor.xy);
float2 origin = clamp(mouse - mouseDir * mouse.x / mouseDir.x, 0., 1.);
float mouseDist = clamp(length(mouse - origin)
+ (aspect - (abs(clickedPoint.x) / bounds.z) * aspect) / mouseDir.x, 0., aspect / mouseDir.x);
if (mouseDir.x < 0.)
{
mouseDist = distance(mouse, origin);
}
float proj = dot(uv - origin, mouseDir);
float dist = proj - mouseDist;
float2 linePoint = uv - dist * mouseDir;
half4 pixel = layer.sample(position);
if (dist > radius)
{
pixel = half4(0.0, 0.0, 0.0, 0.0); // background behind curling layer (note: 0.0 opacity)
pixel.rgb *= pow(clamp(dist - radius, 0., 1.) * 1.5, .2);
}
else if (dist >= 0.0)
{
// THIS PORTION HANDLES THE CURL SHADED PORTION OF THE RESULT
// map to cylinder point
float theta = asin(dist / radius);
float2 p2 = linePoint + mouseDir * (pi - theta) * radius;
float2 p1 = linePoint + mouseDir * theta * radius;
bool underside = (p2.x <= aspect && p2.y <= 1. && p2.x > 0. && p2.y > 0.);
uv = underside ? p2 : p1;
uv = float2(uv.x, 1.0 - uv.y); // invert y
pixel = layer.sample(uv * float2(1. / aspect, 1.) * float2(bounds[2], bounds[3])); // ME<----
if (underside && pixel.a == 0.0) { //<---- PIXEL.A IS 0.0 WHYYYYY
pixel = red;
}
// Commented out while debugging alpha issues
// if (underside && pixel.a == 0.0) {
// pixel = layer.sample(originalPosition);
// } else if (underside) {
// pixel = undersideColor; // underside
// }
// Shadow the pixel being returned
pixel.rgb *= pow(clamp((radius - dist) / radius, 0., 1.), .2);
}
else
{
// THIS PORTION HANDLES THE NON-CURL-SHADED PORTION OF THE SAMPLING.
float2 p = linePoint + mouseDir * (abs(dist) + pi * radius);
bool underside = (p.x <= aspect && p.y <= 1. && p.x > 0. && p.y > 0.);
uv = underside ? p : uv;
uv = float2(uv.x, 1.0 - uv.y); // invert y
pixel = layer.sample(uv * float2(1. / aspect, 1.) * float2(bounds[2], bounds[3])); // ME
if (underside && pixel.a == 0.0) { //<---- PIXEL.A IS 0.0 WHYYYYY
pixel = red;
}
// Commented out while debugging alpha issues
// if (underside && pixel.a == 0.0) {
// // If the new underside pixel is clear, we should sample the original image's pixel.
// pixel = layer.sample(originalPosition);
// } else if (underside) {
// pixel = undersideColor;
// }
}
return pixel;
}
I'm wondering if it's possible to do Parallax Occlusion Mapping in RealityKit? Does RK's metal shader API provide enough?
I think it would need to be able to discard fragments and thus can't be run as a deferred pass. Not sure though!
I have a neural network model for segmentation, I successfully integrated it and am getting a grayscale image. Next, I need to apply the segmentation mask in RealityKit to achieve the occlusion effect (like person segmentation). I tried doing it through post-processing and other methods, but none of them worked. Is there any example of how this can be done in RealityKit?
I'm writing a RealityKit/ARKit app that runs on iOS.
Starting with Xcode 16.0 beta 1, at least through Xcode 16.1 beta 2 (16B5014f), in the iOS 18 simulator, my app randomly crashes in about 20% of app sessions the first time it attempts to present an ARView.
The crashes seem to occur at multiple points within RealityKit and Metal.
Below, I've included screenshots of the call stacks of the crashes, which occur as a result of both EXC_BAD_ACCESS and assertion failures within RealityKit.
The app only crashes in the iOS 18 simulator, and does not crash in the iOS 17 simulator or earlier.
The app only crashes in the simulator, and does not crash on a device running iOS 18.
Before I investigate further, I'd appreciate it if an Apple engineer could give me a sense of if these crashes are most likely the result of known issues within RealityKit and/or the simulator, or if your opinion is that there are probably bugs in my app's code.
I've submitted several feedback issues in the past, and I'd love to submit this issue too, but I expect that I would spend many hours attempting to create a repro case in a sample app. Understandably, I'd rather not spend this time if an Apple engineer could tell me this is a known issue, for example.
Thank you.
I created an empty binary archive and try to save it on disk, but it returned false without giving any error message.
The code is from
https://developer.apple.com/documentation/metal/shader_libraries/metal_binary_archives/creating_binary_archives_from_device-built_pipeline_state_objects?language=objc
Explanation:
I am working on an application and had trouble diagnosing the warnings below. I pulled the search bar code out into a separate project, and the warnings still show up. The entire code for the sample project is shown below. The "AddInstanceForFactory", "LoudnessManager", and "NSBundle" warning all show up immediately after tapping the search bar. The "RTIInputSystemClient" warning arises after tapping, dismissing, and re-tapping the search bar.
Any help with resolving these warnings is appreciated!
Warnings:
"-[RTIInputSystemClient remoteTextInputSessionWithID:performInputOperation:] perform input operation requires a valid sessionID. inputModality = Keyboard, inputOperation = , customInfoType = UIEmojiSearchOperations"
"NSBundle file:///Library/Developer/CoreSimulator/Volumes/iOS_22A3351/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS%2018.0.simruntime/Contents/Resources/RuntimeRoot/System/Library/PrivateFrameworks/MetalTools.framework/ principal class is nil because all fallbacks have failed"
"AddInstanceForFactory: No factory registered for id <CFUUID --------------> -----------------"
"LoudnessManager.mm:709 unable to open stream for LoudnessManager plist"
Code:
import UIKit
class SearchView: UIView {
private lazy var searchBar: UISearchBar = {
let searchBar = UISearchBar()
searchBar.delegate = self
searchBar.translatesAutoresizingMaskIntoConstraints = false
searchBar.placeholder = "Search"
return searchBar
}()
required init?(coder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
init() {
super.init(frame: .zero)
translatesAutoresizingMaskIntoConstraints = false
backgroundColor = .yellow
addSubview(searchBar)
NSLayoutConstraint.activate([
searchBar.topAnchor.constraint(equalTo: topAnchor, constant: 12),
searchBar.leadingAnchor.constraint(equalTo: leadingAnchor, constant: 12),
searchBar.trailingAnchor.constraint(equalTo: trailingAnchor, constant: -12),
searchBar.bottomAnchor.constraint(equalTo: bottomAnchor, constant: -12),
])
}
}
extension SearchView: UISearchBarDelegate {
func searchBar(_ searchBar: UISearchBar, textDidChange searchText: String) {
print(searchText)
}
}
class ViewController: UIViewController {
private lazy var searchView = SearchView()
override func viewDidLoad() {
super.viewDidLoad()
view.backgroundColor = .red
let tap = UITapGestureRecognizer(target: self, action: #selector(dismissKeyboard))
view.addGestureRecognizer(tap)
view.addSubview(searchView)
NSLayoutConstraint.activate([
searchView.leadingAnchor.constraint(equalTo: view.leadingAnchor),
searchView.topAnchor.constraint(equalTo: view.safeAreaLayoutGuide.topAnchor, constant: 24),
searchView.centerXAnchor.constraint(equalTo: view.centerXAnchor),
])
}
@objc func dismissKeyboard() {
view.endEditing(true)
}
}
Hi folks,
I'm working on a Tile based Deferred renderer, similar to this Apple example. I'm wondering how to add MSAA to the renderer, and I see two choices:
Copy the single-sampled texture at the end of the GBuffer/Lighting render pass to a multi-sampled texture and resolve from that
Make all render targets (GBuffer) multi-sampled and deal with sampling/resolving all intermediate textures as well as the final, combined texture.
Which is the proper approach, and are there any examples of how to implement it?
Thanks!
Hi, I am recently writing metal shader language to parallelize the algorithms to accelerate the speed of it.
I created a simple example to show the acceleration result of it. Since Rust is used in our algorithm, so I used metal-rs as the wrapper to execute the MSL kernels from rust side.
In this example, I am calculating the result of two arrays, and kernel looks like:
kernel void two_array_addition_2(
constant uint* a [[buffer(0)]],
constant uint* b [[buffer(1)]],
device uint* c [[buffer(2)]],
uint idx [[thread_position_in_grid]]
) {
c[idx] = a[idx] + b[idx];
}
in the main.rs, you can see a function called execute_kernel() , this function has all it needs to execute the kernel in MSL (such as commandEncoder, piplelineState, etc).
use core::mem;
use metal::{Buffer, MTLSize};
use objc::rc::autoreleasepool;
use std::time::Instant;
use two_array_addition::abstractions::state::MetalState;
fn execute_kernel(
name: &str,
state: &MetalState,
input_a: &Buffer,
input_b: &Buffer,
output_c: &Buffer,
) -> Vec<u32> {
// assert!(input_a.len() == input_b.len() && input_a.len() == output_c.len());
// let len = input_a.len() as u64;
let len = input_a.length() as u64 / mem::size_of::<u32>() as u64;
// 1. Init the MetalState
// - we inited it
// 2. Set up Pipeline State
let pipeline = state.setup_pipeline(name).unwrap();
// 3. Allocate the buffers for A, B, and C
// - we allocated outside of this function
let mut result: &[u32] = &[];
autoreleasepool(|| {
// 4. Create the command buffer & command encoder
let (command_buffer, command_encoder) = state.setup_command(
&pipeline,
Some(&[(0, input_a), (1, input_b), (2, output_c)]),
);
// 5. command encoder dispatch the threadgroup size and num of threads per threadgroup
let threadgroup_count = MTLSize::new((len + 256 - 1) / 256, 1, 1);
let thread_per_threadgroup = MTLSize::new(256, 1, 1);
// let grid_size = MTLSize::new(len, 1, 1);
// let threadgroup_count = MTLSize::new(pipeline.max_total_threads_per_threadgroup(), 1, 1);
command_encoder.dispatch_thread_groups(threadgroup_count, thread_per_threadgroup);
command_encoder.end_encoding();
command_buffer.commit();
command_buffer.wait_until_completed();
// 6. Copy the result back to the host
let start = Instant::now();
result = MetalState::retrieve_contents::<u32>(output_c);
let duration = start.elapsed();
println!("Duration for copying result back to host: {:?}", duration);
});
result.to_vec()
}
The performance of the result is kinda interesting to me.
This is the result:
$ cargo run -r
This is expected to run for a while... please wait...
Generating input arrays...
Generating input arrays...
Generating output array...
Generating expected output...
Duration for allocating buffers: 2.015258s
Executing 1st kernel (1)...
Duration for copying result back to host: 5.75µs
Executing 1st kernel (2)...
Duration for copying result back to host: 542ns
Executing 2nd kernel (1)...
Duration for copying result back to host: 1µs
Executing 2nd kernel (2)...
Duration for copying result back to host: 458ns
Duration expected: 183.406167ms
Duration for 1st kernel (1): 1.894994875s
Duration for 1st kernel (2): 537.318208ms
Duration for 2nd kernel (1): 501.33275ms
Duration for 2nd kernel (2): 497.339916ms
You have successfully run the kernels!
The speed is slower when executing in the MSL kernel, while I reckon of the dataset is quite big ($2^{29}$)
The first kernel execution takes more time to launch.
Is there any way to optimize the MSL in this case? And in most case, when you design the algorithm into parallelism, what would be the concerns?
The machine I am using is M1 Pro with 14-core GPU and 16 GB memory.
Does anyone have idea / explanation for why these happen?
Thank you
In my project I need to do the following:
In runtime create metal Dynamic library from source.
In runtime create metal Executable library from source and Link it with my previous created Dynamic library.
Create compute pipeline using those two libraries created above.
But I get the following error at the third step:
Error Domain=AGXMetalG15X_M1 Code=2 "Undefined symbols:
_Z5noisev, referenced from: OnTheFlyKernel
" UserInfo={NSLocalizedDescription=Undefined symbols:
_Z5noisev, referenced from: OnTheFlyKernel
}
import Foundation
import Metal
class MetalShaderCompiler {
let device = MTLCreateSystemDefaultDevice()!
var pipeline: MTLComputePipelineState!
func compileDylib() -> MTLDynamicLibrary {
let source = """
#include <metal_stdlib>
using namespace metal;
half3 noise() {
return half3(1, 0, 1);
}
"""
let option = MTLCompileOptions()
option.libraryType = .dynamic
option.installName = "@executable_path/libFoundation.metallib"
let library = try! device.makeLibrary(source: source, options: option)
let dylib = try! device.makeDynamicLibrary(library: library)
return dylib
}
func compileExlib(dylib: MTLDynamicLibrary) -> MTLLibrary {
let source = """
#include <metal_stdlib>
using namespace metal;
extern half3 noise();
kernel void OnTheFlyKernel(texture2d<half, access::read> src [[texture(0)]],
texture2d<half, access::write> dst [[texture(1)]],
ushort2 gid [[thread_position_in_grid]]) {
half4 rgba = src.read(gid);
rgba.rgb += noise();
dst.write(rgba, gid);
}
"""
let option = MTLCompileOptions()
option.libraryType = .executable
option.libraries = [dylib]
let library = try! self.device.makeLibrary(source: source, options: option)
return library
}
func runtime() {
let dylib = self.compileDylib()
let exlib = self.compileExlib(dylib: dylib)
let pipelineDescriptor = MTLComputePipelineDescriptor()
pipelineDescriptor.computeFunction = exlib.makeFunction(name: "OnTheFlyKernel")
pipelineDescriptor.preloadedLibraries = [dylib]
pipeline = try! device.makeComputePipelineState(descriptor: pipelineDescriptor, options: .bindingInfo, reflection: nil)
}
}
I encountered a crash with my SwiftUI app for macOS.
NSInvalidArgumentException: -[MTLIGAccelRenderCommandEncoder setVertexBuffer:offset:attributeStride:atIndex:]: unrecognized selector sent to instance
struct ContentView: View {
@State var text = ""
var body: some View {
TextEditor(text: $text)
}
}
When I run macOS app, console print
NSBundle file:///System/Library/PrivateFrameworks/MetalTools.framework/ principal class is nil because all fallbacks have failed
and I found it crash with TextEditor.
Can somebody help me?
SwiftUI [[stichable]] metal shader & CIFilter written in metal extern"C" can't work at the same time
In my project, I used two metal shaders in two ways.
One is link to SwiftUI's modifier .colorEffect(ShaderLibrary.myShader()), which metal shader marked as [[stichable]].
Another one is a custom CIFilter, which kernel been written in external "C" closure.
Because custom CIFilter must add build rules so Xcode can compile it, so I added -fcikernel to Metal Compiler and -cikernel to Metal Linker from Build Settings, just like Apple's document told me that.
But the result is weird, if I add rules, custom CIFilter works, [[stichable]] shader doesn't work. if I delete rules, and comment out code of CIFilter(for avoiding warning), [[stichable]] shader works, but now I can't use my custom CIFilter.
Actually, once these two shaders works well in my project, but when I updated Xcode from 15 to 16, it became weird, the 2 shaders can't exist at same time. Even though I go back to Xcode 15, I can't save them.
I have no idea, please help, thank you.
XCode 16 / iOS 18 on iPhone 14 Pro
The following code runs fine when compiled with Swift 5, but crashes when compiled with Swift 6 (stack trace below). In the draw method, commenting out the addCompletedHandler line fixes the problem. I'm testing on iOS 18.0 and see the same behavior in both the simulator and on a device. What's going on here?
import Metal
import MetalKit
import UIKit
class ViewController: UIViewController {
@IBOutlet var metalView: MTKView!
private var commandQueue: MTLCommandQueue?
override func viewDidLoad() {
super.viewDidLoad()
guard let device = MTLCreateSystemDefaultDevice() else {
fatalError("expected a Metal device")
}
self.commandQueue = device.makeCommandQueue()
metalView.device = device
metalView.enableSetNeedsDisplay = true
metalView.isPaused = true
metalView.delegate = self
}
}
extension ViewController: MTKViewDelegate {
func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {}
func draw(in view: MTKView) {
guard let commandQueue,
let commandBuffer = commandQueue.makeCommandBuffer()
else { return }
commandBuffer.addCompletedHandler { _ in } // works with Swift 5, crashes with Swift 6
commandBuffer.commit()
}
}
Here's the stack trace:
Thread 10 Queue : connection Queue (serial)
#0 0x000000010581c3f8 in _dispatch_assert_queue_fail ()
#1 0x000000010581c384 in dispatch_assert_queue ()
#2 0x00000002444c63e0 in swift_task_isCurrentExecutorImpl ()
#3 0x0000000104d71ec4 in closure #1 in ViewController.draw(in:) ()
#4 0x0000000104d71f58 in thunk for @escaping @callee_guaranteed (@guaranteed MTLCommandBuffer) -> () ()
#5 0x0000000105ef1950 in __47-[CaptureMTLCommandBuffer _preCommitWithIndex:]_block_invoke_2 ()
#6 0x00000001c50b35b0 in -[MTLToolsCommandBuffer invokeCompletedHandlers] ()
#7 0x000000019e94d444 in MTLDispatchListApply ()
#8 0x000000019e94f558 in -[_MTLCommandBuffer didCompleteWithStartTime:endTime:error:] ()
#9 0x000000019e95352c in -[_MTLCommandQueue commandBufferDidComplete:startTime:completionTime:error:] ()
#10 0x0000000226ef50b0 in handleMainConnectionReplies ()
#11 0x00000001800c9690 in _xpc_connection_call_event_handler ()
#12 0x00000001800cad90 in _xpc_connection_mach_event ()
#13 0x000000010581a86c in _dispatch_client_callout4 ()
#14 0x0000000105837950 in _dispatch_mach_msg_invoke ()
#15 0x0000000105822870 in _dispatch_lane_serial_drain ()
#16 0x0000000105838c10 in _dispatch_mach_invoke ()
#17 0x0000000105822870 in _dispatch_lane_serial_drain ()
#18 0x00000001058237b0 in _dispatch_lane_invoke ()
#19 0x00000001058301f0 in _dispatch_root_queue_drain_deferred_wlh ()
#20 0x000000010582f75c in _dispatch_workloop_worker_thread ()
#21 0x00000001050abb74 in _pthread_wqthread ()
https://developer.apple.com/documentation/compositorservices/drawing_fully_immersive_content_using_metal
I'm following this doc to use metal in vision os.
I noticed that the tangent is being deprecated which is being used in the sample
https://developer.apple.com/documentation/compositorservices/layerrenderer/drawable/view
will the sample code be updated?
Hey,
Wondering how other developers have been able to determine the location of a mouse event or tap (ie NSEvent) when using MetalView (MKTView) with SKRenderer with a SpriteKit scene (.sks scene) for a 2D game.
In the original scenario with SpriteKit, we could use SKViews convertPoint(fromView:) to determine where in the scene the user tapped. But with the SKRenderer we can no longer use convertPoint(fromView:) as its reliant on SKView being used and thus its making it difficult to determine.
What I do have is:
locationInWindow: NSPoint for showing me where in the MKTView which was touched
Any ideas, would be great
Many thanks
I wanted to try the new logging feature for Metal but could not get it to work.
I modified the PerformingCalculationsOnAGPU example by adding os_log_default.log_debug("Hello thread: %d", index); to log the current thread id. But never saw any messages neither in the console nor in Xcode.
I also added the -fmetal-enable-logging flag. I am running the Sequoia release candidate 15.0 (24A335) on M1 Max and Xcode 16.0 (16A242).
What am I missing?
When I load some usdz file , it crash 100%, Why ?
It crash in simulate , but not crash in Vision Pro
-[MTLDebugDevice newBufferWithBytes:length:options:]:723: failed assertion `Buffer Validation
newBufferWith*:length 0x100fff80 must not exceed 256 MB.
specifically using the newer colorEffect, layerEffect, etc routes. the below does not seem to work. do i have to use the old MTK stuff?
import SwiftUI
import MetalKit
struct HoloPreview: View {
let startDate = Date()
@State private var noiseTexture: MTLTexture?
var body: some View {
TimelineView(.animation) { context in
RoundedRectangle(cornerRadius: 20)
.fill(Color.red)
.layerEffect(ShaderLibrary.iridescentEffect(
.float(startDate.timeIntervalSinceNow),
.texture(noiseTexture)
), maxSampleOffset: .zero)
}
.onAppear {
noiseTexture = loadTexture(named: "perlinNoiseMap")
}
}
func loadTexture(named imageName: String) -> MTLTexture? {
guard let device = MTLCreateSystemDefaultDevice(),
let url = Bundle.main.url(forResource: imageName, withExtension: "png") else {
return nil
}
let textureLoader = MTKTextureLoader(device: device)
let texture = try? textureLoader.newTexture(URL: url, options: nil)
return texture
}
}
#Preview {
HoloPreview()
}
I am trying to get a little game prototype up and running using Metal using the metal-cpp libraries where I run everything natively at 120Hz with a coupled renderer using Vsync turned on so that I have the absolute physically minimum input to photon latency possible.
// Create the metal view
SDL_MetalView metal_view = SDL_Metal_CreateView(window);
CA::MetalLayer *swap_chain = (CA::MetalLayer *)SDL_Metal_GetLayer(metal_view);
// Set up the Metal device
MTL::Device *device = MTL::CreateSystemDefaultDevice();
swap_chain->setDevice(device);
swap_chain->setPixelFormat(MTL::PixelFormat::PixelFormatBGRA8Unorm);
swap_chain->setDisplaySyncEnabled(true);
swap_chain->setMaximumDrawableCount(2);
I am using SDL3 just for creating the window. Now when I go through my game / render loop - I stall for a long time on getting the next drawable which is understandable - my app runs in about 2-3ms.
m_CurrentContext->m_Drawable = m_SwapChain->nextDrawable();
m_CurrentContext->m_CommandBuffer = m_CommandQueue->commandBuffer()->retain();
char frame_label[32];
snprintf(frame_label, sizeof(frame_label), "Frame %d", m_FrameIndex);
m_CurrentContext->m_CommandBuffer->setLabel(NS::String::string(frame_label, NS::UTF8StringEncoding));
m_CurrentContext->m_RenderPassDescriptor[ERenderPassTypeNormal] = MTL::RenderPassDescriptor::alloc()->init();
MTL::RenderPassColorAttachmentDescriptor* cd = m_CurrentContext->m_RenderPassDescriptor[ERenderPassTypeNormal]->colorAttachments()->object(0);
cd->setTexture(m_CurrentContext->m_Drawable->texture());
cd->setLoadAction(MTL::LoadActionClear);
cd->setClearColor(MTL::ClearColor( 0.53f, 0.81f, 0.98f, 1.0f ));
cd->setStoreAction(MTL::StoreActionStore);
However my ProMotion display does not reliably run at 120Hz when fullscreen and using the direct to display system - it seems to run faster when windowed in composite which is the opposite of what I would expect. The Metal HUD says 120Hz, but the delay to getting the next drawable and looking at what Instruments is saying tells otherwise.
When I profile it, the game loop has completed and is sitting there waiting for the next drawable, but the screen does not want to complete in 8.33ms, so the whole thing slows down for no discernible reason.
Also as a game developer it is very strange for the command buffer to actually need the drawable texture free to be allowed to encode commands - usually the command buffers and swapping the front and back render buffers are not directly dependent on each other. Usually you only actually need the render buffer texture free when you want to draw to it. I could give myself another drawable, but because I am completing in less than 3ms, all it would do would be to add another frame of latency.
I also looked at the FramePacing example and its behaviour is even worse at having high framerate with low latency - the direct to display is always rejected for some reason.
Is this just a flaw in the Metal API? Or am I missing something important? I hope someone can help - the behaviour of the display is baffling.