I'm trying to implement a prototype to render virtual objects in a mixed immersive space on the camer frames captured by CameraFrameProvider.
Here are what I have done:
Get camera's instrinsics from frame.primarySample.parameters.intrinsics
Get camera's extrinsics from frame.primarySample.parameters.extrinsics
Get the device anchor by worldTrackingProvider.queryDeviceAnchor(atTimestamp: CACurrentMediaTime())
Setup a RealityKit.RealityRenderer to render virtual objects on the captured camera frames
let realityRenderer = try RealityKit.RealityRenderer()
realityRenderer.cameraSettings.colorBackground = .outputTexture()
let cameraEntity = PerspectiveCamera()
// see https://developer.apple.com/forums/thread/770235
let cameraTransform = deviceAnchor.originFromAnchorTransform * extrinsics.inverse
cameraEntity.setTransformMatrix(cameraTransform, relativeTo: nil)
cameraEntity.camera.near = 0.01
cameraEntity.camera.far = 100
cameraEntity.camera.fieldOfViewOrientation = .horizontal
// manually calculated based on camera intrinsics
cameraEntity.camera.fieldOfViewInDegrees = 105
realityRenderer.entities.append(cameraEntity)
realityRenderer.activeCamera = cameraEntity
Virtual objects, which should be seen in the camera frames, are clipped out by the camera transform.
If I use deviceAnchor.originFromAnchorTransform as the camera transform, virtual objects can be rendered on camera frames at wrong positions (I think it is because the camera extrinsics isn't used to adjust the camera to the correct position).
My question is how to use the camera extrinsic matrix for this purpose?
Does the camera extrinsics point to a similar orientation of the device anchor with some minor rotation and postion change? Here is an extrinsics from a camera frame. It seems that the direction of Y-axis and Z-axis are flipped by the extrinsics. So the camera is point to a wrong direction.
simd_float4x4([[0.9914258, 0.012555369, -0.13006608, 0.0], // X-axis
[-0.0009778949, -0.9946325, -0.10346654, 0.0], // Y-axis
[-0.13066702, 0.10270659, -0.98609203, 0.0], // Z-axis
[0.024519, -0.019568002, -0.058280986, 1.0]]) // translation
Post
Replies
Boosts
Views
Activity
We have an existing app and we would like to add an additional privacy policy URL. So, there will be two policy URLs for the app. Is that possible? If yes, how to add the second privacy policy URL?
I try to connect to websocket server and send some messages, using NSURLSessionWebSocketTask.
Here is my demo code. This code works well on iOS 13. The Xcode that I uses is Version 12.0 beta 2 (12A6163b)
But the websocket is disconnected automatically (with the close code of NSURLSessionWebSocketCloseCodeProtocolError = 1002) a few seconds after the connection is created on iOS 14 beta2.
If I send short messages (e.g the message length < 10), iOS 14 beta2 might have much better chance to keep the websocket connection
@interface ViewController () <NSURLSessionWebSocketDelegate, NSURLSessionDelegate>
@property (weak, nonatomic) IBOutlet UITextField *outgoing;
@property (weak, nonatomic) IBOutlet UITextView *incoming;
@property (nonatomic) NSURLSession* session;
@property (nonatomic) NSURLSessionWebSocketTask* task;
@property (nonatomic) dispatch_queue_t queue;
@property (nonatomic) NSOperationQueue* operationQueue;
@end
@implementation ViewController
(void)viewDidLoad {
[super viewDidLoad];
NSURLSessionConfiguration* sc = [NSURLSessionConfiguration defaultSessionConfiguration];
self.queue= dispatch_queue_create("WebSocket", DISPATCH_QUEUE_SERIAL);
self.operationQueue = [[NSOperationQueue alloc] init];
self.operationQueue.maxConcurrentOperationCount = 1;
self.operationQueue.underlyingQueue = self.queue;
self.session = [NSURLSession sessionWithConfiguration:sc delegate:self delegateQueue:self.operationQueue];
self.task = [self.session webSocketTaskWithURL:[NSURL URLWithString:@"wss://echo.websocket.org/"]];
[self.task resume];
[self _runReadLoop];
}
(IBAction)onSend:(id)sender {
__block NSUInteger index = 0;
[NSTimer scheduledTimerWithTimeInterval:0.5 repeats:YES block:^(NSTimer * _Nonnull timer) {
NSUInteger i = index;
NSURLSessionWebSocketMessage* message = [[NSURLSessionWebSocketMessage alloc] initWithString:[NSString stringWithFormat:@"The first section of this page will let you do an HTML5 WebSocket test against the echo server. The second section walks you through creating a WebSocket application yourself - %@", @(i)]];
[self.task sendMessage:message completionHandler:^(NSError* error) {
if (error) {
NSLog(@"send error: %@, %@", @(i), error);
} else {
NSLog(@"send: %@, %@", @(i), @(message.string.length));
}
}];
index++;
}];
}
(void) _runReadLoop {
[self.task receiveMessageWithCompletionHandler:^(NSURLSessionWebSocketMessage* message, NSError* error) {
if (error) {
NSLog(@"recv error: %@", error);
return;
}
dispatch_async(dispatch_get_main_queue(), ^{
self.incoming.text = [NSString stringWithFormat:@"%@\n%@",self.incoming.text, message.string];
[self _runReadLoop];
});
}];
}
#pragma mark - NSURLSessionDelegate (void)URLSession:(NSURLSession*)session didBecomeInvalidWithError:(NSError*)error {
NSLog(@"Session Invalidated: %@", error);
}
(void)URLSession:(NSURLSession*)session task:(NSURLSessionTask*)task didCompleteWithError:(NSError*)error {
NSLog(@"Task Completed: %@, %@", task, error);
}
#pragma mark - NSURLSessionStreamDelegate (void)URLSession:(NSURLSession*)session webSocketTask:(NSURLSessionWebSocketTask*)task didOpenWithProtocol:(NSString*)protocol {
NSLog(@"WebSocket Closed: %@, %@", task, protocol);
} (void)URLSession:(NSURLSession*)session webSocketTask:(NSURLSessionWebSocketTask*)task didCloseWithCode:(NSURLSessionWebSocketCloseCode)closeCode reason:(NSData*)reason {
NSLog(@"WebSocket Closed: %@, %ld, %@", task, (long)closeCode, reason);
}
@end
Here is the log in the console
2020-07-08 23:35:56.597508+0800 WebSocketDemo[420:32598] libMobileGestalt MobileGestaltCache.c:166: Cache loaded with 4563 pre-cached in CacheData and 53 items in CacheExtra.
2020-07-08 23:36:02.645074+0800 WebSocketDemo[420:32600] WebSocket Closed: LocalWebSocketTask <07DE69E4-CDEA-4DD3-AC0A-40EE33A60C37>.<1>, (null)
2020-07-08 23:36:11.609142+0800 WebSocketDemo[420:32594] send: 0, 178
2020-07-08 23:36:12.831746+0800 WebSocketDemo[420:32594] [connection] nw_read_request_report [C1] Receive failed with error "Socket is not connected"
2020-07-08 23:36:12.837520+0800 WebSocketDemo[420:32594] Connection 1: received failure notification
2020-07-08 23:36:12.838900+0800 WebSocketDemo[420:32597] recv error: Error Domain=NSPOSIXErrorDomain Code=57 "Socket is not connected" UserInfo={NSErrorFailingURLStringKey=https://echo.websocket.org/, NSErrorFailingURLKey=https://echo.websocket.org/}
2020-07-08 23:36:12.839745+0800 WebSocketDemo[420:32753] [connection] nw_connection_copy_connected_local_endpoint [C1] Connection has no connected path
2020-07-08 23:36:12.839931+0800 WebSocketDemo[420:32753] [connection] nw_connection_copy_connected_remote_endpoint [C1] Connection has no connected path
2020-07-08 23:36:12.884086+0800 WebSocketDemo[420:32594] WebSocket Closed: LocalWebSocketTask <07DE69E4-CDEA-4DD3-AC0A-40EE33A60C37>.<1>, 1002, (null)
2020-07-08 23:36:12.884914+0800 WebSocketDemo[420:32594] Task Completed: LocalWebSocketTask <07DE69E4-CDEA-4DD3-AC0A-40EE33A60C37>.<1>, (null)
The app clip is limited to 10MB. During the development, how I could know if my bundle is bigger than this limit?
The following code can connect to the host successfully on iOS Simulator. But the code fails to connect on an iOS device. I has checked the WLAN and Cellular Data are both allowed for the app clip.
I also try NSURLSession to access to a web site. NSURLSession is successful to get the data on both iOS simulator and iOS devices.
Do we allow to use socket streams in App Clips?
Xcode Version 12.0 beta (12A6159)
iOS 14 Beta: 14.0
But the code fails to connect on
CFReadStreamRef readStream;
CFWriteStreamRef writeStream;
//create socket connection
CFStreamCreatePairWithSocketToHost(NULL, (__bridge CFStringRef)@"www.baidu.com", 443, &readStream, &writeStream);
NSInputStream* inputStream = (__bridge_transfer NSInputStream*)readStream;
NSOutputStream* outputStream = (__bridge_transfer NSOutputStream*)writeStream;
[inputStream setProperty:NSStreamSocketSecurityLevelNegotiatedSSL
forKey:NSStreamSocketSecurityLevelKey];
// and open the streams
[inputStream open];
[outputStream open];
Console output:
[connection] nw_socket_connect [C1.1:1] connectx(5, [srcif=0, srcaddr=<NULL>, dstaddr=61.135.169.121:443], SAE_ASSOCID_ANY, 0, NULL, 0, NULL, SAE_CONNID_ANY) failed: [65: No route to host]
[connection] nw_socket_connect [C1.1:1] connectx failed (fd 5) [65: No route to host]
[] nw_socket_connect connectx failed [65: No route to host]
[connection] nw_socket_connect [C1.2:1] connectx(5, [srcif=0, srcaddr=<NULL>, dstaddr=61.135.169.125:443], SAE_ASSOCID_ANY, 0, NULL, 0, NULL, SAE_CONNID_ANY) failed: [65: No route to host]
[connection] nw_socket_connect [C1.2:1] connectx failed (fd 5) [65: No route to host]
[] nw_socket_connect connectx failed [65: No route to host]
[connection] nw_connection_get_connected_socket [C1] Client called nw_connection_get_connected_socket on unconnected nw_connection
TCP Conn 0x280200370 Failed : error 0:65 [65]