Context menus in SwiftUI on macOS?

This is basic enough I doubt the versions matter, but I'm running macOS 12.5.1 and Xcode 13.4.1, with my build targeting macOS 11. I'm trying to add contextual menus to some views I'm drawing with SwiftUI, but the menus aren't activating how I expect. Here's a trivial reproduction of the issue:

struct ContentView: View {
    var body: some View {
        Text("Hello, world!")
            .overlay(
                Rectangle()
                    .strokeBorder(Color.primary, lineWidth: 0.5))
            .frame(width: 200, height: 200)
            .overlay(
                Rectangle()
                    .strokeBorder(Color.primary, lineWidth: 0.5))

            .contextMenu {
                Button {
                    print("Nothing to see here yet...")
                } label: {
                    Text("A Menu Item")
                }
            }
            .padding()
    }
}

The padding is just to provide some visual separation between the frame line and the edge of the window. When I right-click or control-click the text, I get the menu as expected. Anywhere outside the text's rectangle and inside the frame's rectangle, I get nothing. I expect to get the contextual menu.

If I add a background (e.g, .background(Rectangle().fill(Color.yellow))), the contextual menu works over the whole frame. If I change the background's color to Color.clear, the menu is back to working only on the text.

What is going on? Why does it work this way? The change in behavior from Color.yellow to Color.clear makes me think it's some kind of optimization where SwiftUI decides it doesn't need to do work there, but it clearly still knows about the frame as a whole.

The specific use case relates to one of my earlier threads. I'm drawing a table with variable-height rows depending on the number of items in the row's fields. Sometimes, a row has only one value in one field and many values in a second field, so it is drawn taller, with empty space in the first field. I want the contextual menu to work in the whole field, not just when clicking exactly on an object.

I also want to have a contextual menu when I right-click exactly on an object with some additional entries. For example, "Copy this object's UUID", and "Remove this object" when I click on an object, and "Add an object" menu item when I click on an object or when I click in the empty space in the field. Since SwiftUI is composable, I'm able to build the general menu items once and reference them in a section of a particular object's contextual menu.

Any ideas on how to get the contextual menu working across the whole frame rather than just the text? I could implement one version of the general menu items as an NSMenu with selectors and all, then another version of the general menu items in SwiftUI, but this seems like duplication of effort. Surely that can't be the right way to solve the problem.