SwiftUI List with Core Data / SwiftData - memory usage while scrolling

I want to display a list of 'contacts' in my app, loaded from a local Core Data sqlite database. Currently, I use UIKit, and with UITableView's cell reuse, even with 5000+ rows, the memory usage is great ... it loads at about 80MB and stays around that no matter how much I scroll up or down the list.

I implemented the same list with SwiftUI List, and the memory usage while scrolling is very different ... the initial load is about the same, but each time I go down the whole list, it adds 20-30MB to the memory usage (according to Xcode).

Is this a side-effect of using SwiftUI's List, or am I doing something wrong here?

Here's the implementation:

struct CJContactsListView: View {
        
    @SectionedFetchRequest var sectionContacts: SectionedFetchResults<String, Person>
    
    init() {
    
        let fetchRequest = Person.allContactsFetchRequest()
        _sectionContacts = SectionedFetchRequest(fetchRequest: fetchRequest, sectionIdentifier: \.normalizedSectionLetter!, animation: .default)
    }

    var body: some View {
        
        List {
            ForEach(sectionContacts) { section in
                Section(header: Text(section.id)) {
                    
                    ForEach(section) { person in
                        
                        CJContactsListLabelRowView(person: person)
                    }
                }
            }
        }
        .listStyle(.plain)
    }
}


struct CJContactsListLabelRowView: View {
    @ObservedObject var person: Person
    
    var body: some View {
        HStack (alignment: .center, spacing: 8) {
            
            VStack (alignment: .leading){
                if let displayName = person.displayName {
                    Text(displayName).font(.headline)
                }
                if let companyName = person.companyName {
                    Text(companyName).font(.subheadline).foregroundColor(.secondary)
                }
            }
        }
    }
}

extension Person: Identifiable {
    public var id: String {
        return self.objectID.uriRepresentation().absoluteString
    }
    
    public static func allContactsFetchRequest() -> NSFetchRequest<Person> {
        let request = Person.fetchRequest()
        request.sortDescriptors = Person.makeSortDescriptors()
        request.predicate = NSPredicate(format: "(isContactArchived == nil || isContactArchived == 0)")
        request.fetchBatchSize = 100
        request.relationshipKeyPathsForPrefetching = ["tags"]
        return request
    }
}

There isn't a visible performance issue in my testing (i.e. I don't see a 'stutter' when scrolling really fast), but the memory profile growing does concern me, especially when this isn't a problem in UIKit.

I've tested the "Earthquakes" sample project from Apple and it seems to display the same issue (memory profile grows substantially as the user scrolls down the list).

Would love to know if there's a way to avoid this issue.