You can have as many different views (or kinds of views) in the same window as you want. A window has a single content view, but you can add any number of subviews to the content view, and you can arrange them any way you want. Sprite Kit or Scene Kit views are no different in this regard, the only restriction being that it's not desirable to have any other view overlap a Sprite or Scene Kit view, since those views use OpenGL or Metal and compositing them with regular views can be a performance killer. Adjacent views should be fine, though.
If your app is document-based, your NSDocument subclass instance has an associated window, but there's no particular relationship between the document's data and which subviews of the window display a representation of that data.
Note that if you represent document data in a specific subview, you probably want a view controller for that subview (for document-UI-specific logic, since it no longer really belongs to the top level content view). Because of the way storyboards work, you would have to use a view controller hierarchy as well as (in parallel to, sort of) the view hierarchy, and that will introduce an extra level into the view hierarchy. Let me know if you want me to spell out how to achieve that in your storyboard. (The alternative is to have the top level view controller manage all of the views in the hierarchy.)
You will have to come up with a strategy for laying out the subviews of the content view to fit the window. Autolayout constraints are probably what you want here. However, if the subviews are to be user resizable (by dragging the borders between the views) then you would likely use a NSSplitViewController to structure your window instead of simply using subviews.
That's the overview, as briefly as I can tell it. If you want me to expand on any part of this, just say so.