SwiftUI toolbar action with StateObject causes frame rate to drop

I have a fairly straight forward swift UI view, that allows you to drag a shape around.

I've noticed that whenever I have a toolbar action that has an action that captures my observed object grid, the frame rate drops significantly.

Here's the code below, with the state object currently in the action (grid.snapEnabled.toggle())

func makeItem() -> SGGridItem {
    SGGridItem(id: UUID(), rect: CGRect(origin: .init(x: 250, y: 250), size: .init(width: 50, height: 50)))
}

struct ContentView: View {
    @StateObject var grid = SGGridState(items: [
        makeItem(),
        makeItem(),
        makeItem(),
        makeItem(),
        makeItem(),
        makeItem()
    ])
    
    var body: some View {
        SGGridCanvas(grid:grid) {
            SGGridView(grid: grid)
        }.frame(maxWidth:.infinity, maxHeight: .infinity)
        .toolbar(content: {
            ToolbarItem(placement: .principal) {
                Button(action: { grid.snapEnabled.toggle() }) {
                    Image(systemName: "square.grid.3x3.topleft.fill").opacity(grid.snapEnabled ? 1 : 0.5)
                }
            }
        })
        .onAppear {
            grid.globalGridItemPadding = 5
            grid.globalGridItemCornerRadius = 5
        }
    }
}

In this GIF you can see a noticeable lag as the shape being dragged follows behind the mouse cursor.

However, if I now change the toolbar action to be an empty closure like so

ToolbarItem(placement: .principal) {
                Button(action: { }) {
                    Image(systemName: "square.grid.3x3.topleft.fill").opacity(grid.snapEnabled ? 1 : 0.5)
                }
            }

Then as you can see, the view now performs as expected (the shape follows the mouse exactly).

Is there something special I'm meant to be doing here when capturing a state object in a closure?

Is this a performance issue inherent in swift UI, or do I need to be changing my structure somehow?

Edit: This can be reproduced even with simple state variables.