Hi all, been working with visionOS for a bit so far and am trying to develop a feature that allows users to shareplay and interact with a 3D model pulled from the cloud (icloud in this case, but may use a regular backend service in the future). Ideally, I would want to be able to click a custom button on a regular window that starts the group activity/shareplay with another person in the facetime call and opens the volumetric window with the model and can switch to an immersive space freely. TLDR/questions at the very end for reference point.
I was able to get this working when only working with a single window group (i.e. a volumetric window group scene and an immersive space scene). However I am running into trouble getting shareplay to correctly grab the desired scene (or any scene at all) when I have multiple window group scenes defined.
I have been referencing the following documentation in my attempts to implement this:
- https://developer.apple.com/documentation/groupactivities/sceneassociationbehavior
- https://developer.apple.com/documentation/groupactivities/adding-spatial-persona-support-to-an-activity
- https://developer.apple.com/documentation/groupactivities/defining-your-apps-shareplay-activities
- https://developer.apple.com/documentation/groupactivities/joining-and-managing-a-shared-activity
No luck so far however. Additionally, here is a quick breakdown of what I've done so far to attempt implementation:
-
Define group activity that contains
static var activityIdentifier: Stringandvar metadata: GroupActivityMetadataas well as conforms toGroupActivity. -
Provide methods to start shareplay via a
startShareplay()method that instantiates the above group activity and switch awaits onactivity.prepareForActivation()to activate the activity ifcase .activationPreferred. I have also provided a separate group activity registration method to start shareplay via airdrop as mentioned in the Building spatial SharePlay experiences developer video (timestamped), which does expose a group activity to the share context menu/ornament but does not indicate being shared afterwards. -
On app start, trigger a method to configure group sessions and provide listeners for sessions (including subscribers for active participants, session state, messages of the corresponding state type - in my case it's
ModelState.self, journal attachments for potentially providing models that the other user may not have as we are getting models from cloud/backend, local participant states, etc). At the very end, callgroupSession.join(). -
Add external activation handlers to the corresponding scenes in the scene declaration (as per this documentation on SceneAssociationBehavior using the
handlesExternalEvents(matching:)scene modifier to open the scene when shareplay starts). I have also attempted using the view modifierhandlesExternalEvents(preferring:allowing:)on views but also no luck. Both are being used with the correspondingactivityIdentifierfrom the group activity and I've also tried passing a specific identifier while using the.content(_)sceneAssociationBehavior as well but no luck there either.
I have noted that in this answer regarding shareplay in visionOS, the VP engineer notes that when the app receives the session, it should setup any necessary UI then join the session, but I would expect even if the UI isn't being set up via the other person's session that the person who started shareplay should still see the sharing ornament turn green on the corresponding window which doesn't seem to occur. In fact, none of the windows that are open even get the green sharing ornament (just keeps showing "Not shared").
TLDR: Added external events handling and standard group activity stuff to multi-window app. When using shareplay, no windows are indicated as being shared.
My questions thus are:
-
Am I incorrect in my usage of the scene/view modifiers for
handlesExternalEventsto open and associate a specific windowgroup/scene with the group activity? -
In regards to opening a specific window when the group activity is activated, how do we pass any values if the window group requires it? i.e. if it's something like
WindowGroup(id: ..., for: URL.self) { url in ... } -
Do I still need to provide UI setup in the session listener (
for await session in MyActivity.sessions())? Is this just a simpleopenWindow? -
Past the initializing shareplay stuff above, what are the best practices for sharing 3d models that not all users in the session might have? Is it adding it as an attachment to
GroupSessionJournal? Or should I pass the remote URL to everyone to download the model locally instead?
Thanks for any help and apologies for the long post. Please let me know if there's any additional information I can provide to help resolve this.
Thanks for the detailed question. You're doing the right thing and hitting a couple known limitations.
Do I still need to provide UI setup in the session listener (for await session in MyActivity.sessions())? Is this just a simple openWindow?
-
Scene preference is only applied to windows opened before the session is received. Simply put, all participants must have the window open before the initiator clicks the "Start [Activity]" button (that activates the activity) for SharePlay to associate them. It appears you are opening the recipient window after the session is received, but before the session is joined. Note: received is distinct from joined.
-
An app's initial window style (defined in the plist as
Preferred Default Scene Session Role) needs to match the window style of the window you wish to SharePlay (in this case volumetric) for window association to work when the app is initially open (via a SharePlay link).
You can work around the limitations by adjusting the content of the initial volumetric window based on the presence of a session and the session's activity's content.
More details
Define an app whose initial window is volumetric.
struct SceneAssociationApp: App {
var appModel:AppModel = AppModel()
var body: some SwiftUI.Scene {
WindowGroup {
ContentView().environment(appModel)
}
.windowStyle(.volumetric)
}
}
Define a model to hold shared state related to the SharePlay activity.
@Observable
class GroupActivitiesModel {
var session:GroupSession<TestActivity>?
}
Define an activity with the data you need to load the desired model when a session is received. In this case I've used modelId.
struct TestActivity: GroupActivity, Transferable, Sendable {
static let activityIdentifier = "com.yourdomain.TestActivity"
let modelId:String
var metadata: GroupActivityMetadata = {
var metadata = GroupActivityMetadata()
metadata.type = .generic
metadata.title = "Model sharing app"
metadata.sceneAssociationBehavior = .content(TestActivity.activityIdentifier)
return metadata
}()
}
This is the heart of the work around. Define a view that displays HomeView when there is no session and displays ModelView when there is one.
struct ContentView: View {
@State var groupActivitiesModel = GroupActivitiesModel()
var body: some View {
if let session = groupActivitiesModel.session {
ModelView(modelId: session.activity.modelId)
}
else {
HomeView(groupActivitiesModel: groupActivitiesModel)
}
}
}
Define ModelView to display the model for the modelId associated with the incoming session.
struct ModelView: View {
@State var modelId:String
var body: some View {
RealityView { content in
let mesh = MeshResource.generateText(modelId,
extrusionDepth: 0.01,
font: UIFont.systemFont(ofSize: 0.07))
let textEntity = ModelEntity(mesh: mesh, materials: [SimpleMaterial(color: .systemPink, isMetallic: true)])
textEntity.position.x = -mesh.bounds.extents.x / 2
content.add(textEntity)
}
}
}
Finally define HomeView to present the user with a button to start the activity while listening for incomming sessions. Note the use of the frame and glassBackgroundEffect modifiers on HomeView; both of those help the window “feel" more like a traditional 2D home view. That said, consider taking advantage of the volumetric window to enhance the home view with 3D content.
struct HomeView: View {
@State var groupActivitiesModel = GroupActivitiesModel()
var body: some View {
VStack {
Text("Welcome to model share").font(.headline)
if groupActivitiesModel.session == nil {
Button("Share model", systemImage: "shareplay") {
Task {
let _ = try? await TestActivity(modelId: "some id").activate()
}
}
}
}
.task {
for await session in TestActivity.sessions() {
groupActivitiesModel.session = session
session.join()
}
}
.frame(width: 800, height: 600)
.glassBackgroundEffect()
}
}
Once a session is established you can use GroupSessionMessenger to send and receive messages containing the data needed to update the model(s) displayed in the volume.
