"Simultaneous accesses, but modification requires exclusive access" from CoreData

I'm a bit stumped on this one. I have two core data entities: Person and Memory. Memory has a people field that is a "to many" relationship to Person, and Person has a memories field that is a "to many" relationship to Memory.

I created a PeopleForm view to manage the people associated with a memory. The memory object gets passed in as an @ObservedObject to the view. I want to list the people, but people is an NSSet, so I created a computed property called peopleArray that converts the NSSet to [Person] (and sorts it for good measure).

I use the ForEach view with peopleArray to create a list of people. I use the onDelete view modifier to let the user remove a person from the list of people for that memory. The onDelete calls the removePerson function, which looks up the person object in peopleArray by the index, and then passes it to the memory object's removeFromPeople() method.

The removeFromPeople() method appears to be what triggers the runtime error. I've ready about exclusivity enforcement, but I can't figure out where I've gone wrong here.

The following is a pared down version of my code:

struct PeopleForm: View {
    @ObservedObject var memory: Memory
    @Environment(\.managedObjectContext) var viewContext
    @Environment(\.dismiss) var dismiss

    private var peopleArray: [Person] {
        get {
            Array(memory.people as! Set<Person>).sorted {
                $0.displayName < $1.displayName
            }
        }
    }

    var body: some View {
        NavigationView {
            List {
                ForEach(peopleArray) { person in
                    Text(person.displayName)
                }
                .onDelete(perform: removePerson)
            }
            .toolbar {
                ToolbarItem(placement: .confirmationAction) {
                    Button("Save") {
                        dismiss()
                    }
                }
                ToolbarItem(placement: .cancellationAction) {
                    Button("Cancel") {
                        dismiss()
                    }
                }
            }
        }
    }

    func removePerson(at offsets: IndexSet) {
        offsets.forEach { index in
            let person = peopleArray[index]

            viewContext.perform {
                memory.removeFromPeople(person)
            }
        }
    }
}

Editing to note that the stack trace points at the App.main() call. The error does not occur when stepping through the code line by line in the debugger, but I also can't step into the removeFromPeople method that is generated from my data model. If I step through to the removeFromPeople line and resume, the error occurs, so it seems to be something internal to CoreData that is going on here.