How to avoid .searchable on a macOS to cause the focus to always jump back to the search field

I asked this on Stack Overflow, but I'll ask here as well since there is no answers there yet.

I'm trying to move away from having a TextField in the toolbar as my search field by using the new .searchable. But there seems to be a problem I can't solve.

When you type the text you want to search, I can filter the list with that text, but when I place the mouse cursor on the first item and try to move down the list with the arrow key, with each arrow key press, the focus goes back to the search field, making it impossible to navigate up and down the list with the keyboard.

Maybe I'm not implementing it right, or maybe it doesn't work yet with macOS, either way, this is the code I'm using. This is part of a three panel view, with this one being the middle view (right next to the side bar):

struct AllNotes: View {
   @EnvironmentObject private var data: DataModel
   @State var selectedNoteId: UUID?
   @State var searchText: String = ""

   var body: some View {
      NavigationView {
            List(data.notes.filter { searchText.isEmpty ? true : $0.text.localizedCaseInsensitiveContains(searchText) }) { note in
                NavigationLink(
                    destination: NoteView(note: note, text: note.text),
                    tag: note.id,
                    selection: $selectedNoteId
                ) {
                    VStack(alignment: .leading) {
                        Text(getFirstLine(noteText: note.text)).font(.body).fontWeight(.bold)
                    }
                }
            }
            .searchable(
                text: $searchText,
                placement: .toolbar,
                prompt: "Search..."
            )
            .toolbar {
                // a few other buttons
            }
      }
   }
}

The DataModel is simple a struct of NoteItem:

struct NoteItem: Codable, Hashable, Identifiable {
    let id: UUID
    var text: String
    var changed: Bool = false
}

Am I missing anything? Am I implementing this right?