All right, I have figured it out. Apparently, unlike iOS, macOS FileProviderUI extension does not support NSExtensionMainStoryboard Info.plist key. Using NSExtensionPrincipalClass worked like a charm.
Beware of a few things though:
Don't forget to load view for principal class (e.g. by having corresponding .xib)
On macOS prepareForActionWithIdentifier:itemIdentifiers: and prepareForError: are called before viewDidLoad (on iOS it's vice versa)