Post

Replies

Boosts

Views

Activity

How to find the camera transform (or view matrix) in the world coordinate from a camera frame
I'm trying to implement a prototype to render virtual objects in a mixed immersive space on the camer frames captured by CameraFrameProvider. Here are what I have done: Get camera's instrinsics from frame.primarySample.parameters.intrinsics Get camera's extrinsics from frame.primarySample.parameters.extrinsics Get the device anchor by worldTrackingProvider.queryDeviceAnchor(atTimestamp: CACurrentMediaTime()) Setup a RealityKit.RealityRenderer to render virtual objects on the captured camera frames let realityRenderer = try RealityKit.RealityRenderer() realityRenderer.cameraSettings.colorBackground = .outputTexture() let cameraEntity = PerspectiveCamera() // see https://developer.apple.com/forums/thread/770235 let cameraTransform = deviceAnchor.originFromAnchorTransform * extrinsics.inverse cameraEntity.setTransformMatrix(cameraTransform, relativeTo: nil) cameraEntity.camera.near = 0.01 cameraEntity.camera.far = 100 cameraEntity.camera.fieldOfViewOrientation = .horizontal // manually calculated based on camera intrinsics cameraEntity.camera.fieldOfViewInDegrees = 105 realityRenderer.entities.append(cameraEntity) realityRenderer.activeCamera = cameraEntity Virtual objects, which should be seen in the camera frames, are clipped out by the camera transform. If I use deviceAnchor.originFromAnchorTransform as the camera transform, virtual objects can be rendered on camera frames at wrong positions (I think it is because the camera extrinsics isn't used to adjust the camera to the correct position). My question is how to use the camera extrinsic matrix for this purpose? Does the camera extrinsics point to a similar orientation of the device anchor with some minor rotation and postion change? Here is an extrinsics from a camera frame. It seems that the direction of Y-axis and Z-axis are flipped by the extrinsics. So the camera is point to a wrong direction. simd_float4x4([[0.9914258, 0.012555369, -0.13006608, 0.0], // X-axis [-0.0009778949, -0.9946325, -0.10346654, 0.0], // Y-axis [-0.13066702, 0.10270659, -0.98609203, 0.0], // Z-axis [0.024519, -0.019568002, -0.058280986, 1.0]]) // translation
1
0
241
2w
Websocket might be disconnected on iOS 14 beta2
I try to connect to websocket server and send some messages, using NSURLSessionWebSocketTask. Here is my demo code. This code works well on iOS 13. The Xcode that I uses is Version 12.0 beta 2 (12A6163b) But the websocket is disconnected automatically (with the close code of NSURLSessionWebSocketCloseCodeProtocolError = 1002) a few seconds after the connection is created on iOS 14 beta2. If I send short messages (e.g the message length < 10), iOS 14 beta2 might have much better chance to keep the websocket connection @interface ViewController () &lt;NSURLSessionWebSocketDelegate, NSURLSessionDelegate&gt; @property (weak, nonatomic) IBOutlet UITextField *outgoing; @property (weak, nonatomic) IBOutlet UITextView *incoming; @property (nonatomic) NSURLSession* session; @property (nonatomic) NSURLSessionWebSocketTask* task; @property (nonatomic) dispatch_queue_t queue; @property (nonatomic) NSOperationQueue* operationQueue; @end @implementation ViewController (void)viewDidLoad {   [super viewDidLoad];   NSURLSessionConfiguration* sc = [NSURLSessionConfiguration defaultSessionConfiguration];       self.queue= dispatch_queue_create("WebSocket", DISPATCH_QUEUE_SERIAL);   self.operationQueue = [[NSOperationQueue alloc] init];   self.operationQueue.maxConcurrentOperationCount = 1;   self.operationQueue.underlyingQueue = self.queue;   self.session = [NSURLSession sessionWithConfiguration:sc delegate:self delegateQueue:self.operationQueue];       self.task = [self.session webSocketTaskWithURL:[NSURL URLWithString:@"wss://echo.websocket.org/"]];   [self.task resume];   [self _runReadLoop]; } (IBAction)onSend:(id)sender {   __block NSUInteger index = 0;   [NSTimer scheduledTimerWithTimeInterval:0.5 repeats:YES block:^(NSTimer * _Nonnull timer) {     NSUInteger i = index;     NSURLSessionWebSocketMessage* message = [[NSURLSessionWebSocketMessage alloc] initWithString:[NSString stringWithFormat:@"The first section of this page will let you do an HTML5 WebSocket test against the echo server. The second section walks you through creating a WebSocket application yourself - %@", @(i)]];     [self.task sendMessage:message completionHandler:^(NSError* error) {       if (error) {         NSLog(@"send error: %@, %@", @(i), error);       } else {         NSLog(@"send: %@, %@", @(i), @(message.string.length));       }     }];     index++;   }]; } (void) _runReadLoop {     [self.task receiveMessageWithCompletionHandler:^(NSURLSessionWebSocketMessage* message, NSError* error) {       if (error) {         NSLog(@"recv error: %@", error);         return;       }       dispatch_async(dispatch_get_main_queue(), ^{         self.incoming.text = [NSString stringWithFormat:@"%@\n%@",self.incoming.text, message.string];         [self _runReadLoop];       });             }]; } #pragma mark - NSURLSessionDelegate (void)URLSession:(NSURLSession*)session didBecomeInvalidWithError:(NSError*)error {   NSLog(@"Session Invalidated: %@", error); } (void)URLSession:(NSURLSession*)session task:(NSURLSessionTask*)task didCompleteWithError:(NSError*)error {   NSLog(@"Task Completed: %@, %@", task, error); } #pragma mark - NSURLSessionStreamDelegate (void)URLSession:(NSURLSession*)session webSocketTask:(NSURLSessionWebSocketTask*)task didOpenWithProtocol:(NSString*)protocol {   NSLog(@"WebSocket Closed: %@, %@", task, protocol); } (void)URLSession:(NSURLSession*)session webSocketTask:(NSURLSessionWebSocketTask*)task didCloseWithCode:(NSURLSessionWebSocketCloseCode)closeCode reason:(NSData*)reason {   NSLog(@"WebSocket Closed: %@, %ld, %@", task, (long)closeCode, reason); } @end Here is the log in the console 2020-07-08 23:35:56.597508+0800 WebSocketDemo[420:32598] libMobileGestalt MobileGestaltCache.c:166: Cache loaded with 4563 pre-cached in CacheData and 53 items in CacheExtra. 2020-07-08 23:36:02.645074+0800 WebSocketDemo[420:32600] WebSocket Closed: LocalWebSocketTask &lt;07DE69E4-CDEA-4DD3-AC0A-40EE33A60C37&gt;.&lt;1&gt;, (null) 2020-07-08 23:36:11.609142+0800 WebSocketDemo[420:32594] send: 0, 178 2020-07-08 23:36:12.831746+0800 WebSocketDemo[420:32594] [connection] nw_read_request_report [C1] Receive failed with error "Socket is not connected" 2020-07-08 23:36:12.837520+0800 WebSocketDemo[420:32594] Connection 1: received failure notification 2020-07-08 23:36:12.838900+0800 WebSocketDemo[420:32597] recv error: Error Domain=NSPOSIXErrorDomain Code=57 "Socket is not connected" UserInfo={NSErrorFailingURLStringKey=https://echo.websocket.org/, NSErrorFailingURLKey=https://echo.websocket.org/} 2020-07-08 23:36:12.839745+0800 WebSocketDemo[420:32753] [connection] nw_connection_copy_connected_local_endpoint [C1] Connection has no connected path 2020-07-08 23:36:12.839931+0800 WebSocketDemo[420:32753] [connection] nw_connection_copy_connected_remote_endpoint [C1] Connection has no connected path 2020-07-08 23:36:12.884086+0800 WebSocketDemo[420:32594] WebSocket Closed: LocalWebSocketTask &lt;07DE69E4-CDEA-4DD3-AC0A-40EE33A60C37&gt;.&lt;1&gt;, 1002, (null) 2020-07-08 23:36:12.884914+0800 WebSocketDemo[420:32594] Task Completed: LocalWebSocketTask &lt;07DE69E4-CDEA-4DD3-AC0A-40EE33A60C37&gt;.&lt;1&gt;, (null)
8
0
6.6k
Jul ’20
Can we use socket streams in App Clips?
The following code can connect to the host successfully on iOS Simulator. But the code fails to connect on an iOS device. I has checked the WLAN and Cellular Data are both allowed for the app clip. I also try NSURLSession to access to a web site. NSURLSession is successful to get the data on both iOS simulator and iOS devices. Do we allow to use socket streams in App Clips? Xcode Version 12.0 beta (12A6159) iOS 14 Beta: 14.0 But the code fails to connect on   CFReadStreamRef readStream;   CFWriteStreamRef writeStream;       //create socket connection   CFStreamCreatePairWithSocketToHost(NULL, (__bridge CFStringRef)@"www.baidu.com", 443, &amp;readStream, &amp;writeStream);   NSInputStream* inputStream = (__bridge_transfer NSInputStream*)readStream;   NSOutputStream* outputStream = (__bridge_transfer NSOutputStream*)writeStream;       [inputStream setProperty:NSStreamSocketSecurityLevelNegotiatedSSL            forKey:NSStreamSocketSecurityLevelKey];   // and open the streams   [inputStream open];   [outputStream open]; Console output: [connection] nw_socket_connect [C1.1:1] connectx(5, [srcif=0, srcaddr=&lt;NULL&gt;, dstaddr=61.135.169.121:443], SAE_ASSOCID_ANY, 0, NULL, 0, NULL, SAE_CONNID_ANY) failed: [65: No route to host] [connection] nw_socket_connect [C1.1:1] connectx failed (fd 5) [65: No route to host] [] nw_socket_connect connectx failed [65: No route to host] [connection] nw_socket_connect [C1.2:1] connectx(5, [srcif=0, srcaddr=&lt;NULL&gt;, dstaddr=61.135.169.125:443], SAE_ASSOCID_ANY, 0, NULL, 0, NULL, SAE_CONNID_ANY) failed: [65: No route to host] [connection] nw_socket_connect [C1.2:1] connectx failed (fd 5) [65: No route to host] [] nw_socket_connect connectx failed [65: No route to host] [connection] nw_connection_get_connected_socket [C1] Client called nw_connection_get_connected_socket on unconnected nw_connection TCP Conn 0x280200370 Failed : error 0:65 [65]
6
0
4.1k
Jun ’20