Here is what I’ve been able to find/gather/learn from others. A Help Book appears to run on a separate app/process called “HelpViewer”. Any Apple macOS app displaying a help makes use of a DDMViewerController that isn’t public.
There is an “app.css” and an “app.js” being used by the Apple macOS app “index.html” of the Apple Help Book. The Javascript one manipulates the DOM to create the “show-hide” link that toggles the Sidebar. Haven’t been able to find how to instruct HelpViewer to use a sidebar.
There is a WWDC talk from back in 2014, “Introducing the Modern WebKit API” that talks about “User Scripts” and “Script Messages” which allow communication between a Webview and Cocoa. https://developer.apple.com/videos/play/wwdc2014/206/
AFAICS, there is no way to have HelpViewer display a custom view or have a sidebar. My *guess* is that you would have to implement everything yourself. That is an NSSplitViewController, NSToolbar, NSOutlineView, any Javascript alongside the “app.css” to get the look and feel.