What is the designated way to do custom background drawing in TextKit 2 when using UITextView/NSTextView?

I'm partially answering my own question here. It seems a second option is to do background drawing by overriding NSTextLayoutFragment.draw(at:in:) in a custom subclass of NSTextLayoutFragment and use the NSTextLayoutManagerDelegate.textLayoutManager(:textLayoutFragmentFor:in:) API to return that custom layout fragment to the system.

This works fine for custom background drawing, but what if I want some custom interaction using UIView as the rendering surface? This may overlap with another question that I posted relating to the use of configureRenderingSurface with UITextView / NSTextView. Is using configureRenderingSurface kosher when using UITextView / NSTextView?

Hopefully someone can give some guidance or clarity on this?

Overriding NSTextLayoutFragment.draw(at:in:) in an NSTextLayoutFragment subclass provided by the NSTextLayoutManagerDelegate is indeed the designated way of doing custom drawing now. However, note that it still doesn't have quite the same effect as NSLayoutManager.drawBackground(forGlyphRange:at:). While you can draw behind the text by doing your custom drawing before calling super.draw(at: point, in: context), all custom drawing in NSTextLayoutFragment is still done above text text view's selection (in NSTextView anyway). In NSLayoutManager, drawing done in drawBackground(forGlyphRange:at:) was done beneath the text selection, which could be useful for some elements. There seems to be no way of doing the same in TextKit 2 just yet. (I filed this as #FB10159592 but received a fairly standard dismissive reply).

You can also do custom drawing in NSTextView's drawBackround(in:), but that's more awkward because calculating the visible ranges is expensive.

For custom interaction, I suppose it depends on what you're trying to do. You could do it in a text view subclass and use NSTextLayoutManager's various methods to get the frames of the text you need. For instance, I allow live image resizing in an NSTextView, and draw resizing handles over image text attachments. For that I can use NSTextLayoutManager's layoutFragment(for: point) in mouseDown(_:), converting the point from text view to text container coordinates, and I can then get the image frame using NSTextLayoutFragment.frameForTextAttachment(at:), and add a new sublayer to the text view to draw my resizing handles. There are a couple of gotchas here, though:

1. Because NSTextView (and presumably UITextView) in TextKit 2 does all of its drawing using layers, you have to set the z position of any new sublayer you add to something high (e.g.FLT_MAX).

2. The coordinate system used for TextKit 2 is so far undocumented, and seems pretty strange. NSTextLayoutFragment.layoutFragmentFrame is straightforward, being in text container coordinates, and NSTextLayoutFragment.renderingSurfaceBounds is relative to that (although the docs say its coordinates are flipped compared to layoutFragmentFrame, when they don't seem to be). However, both NSLayoutFragment.frameForTextAttachment(at:) and NSTextLineFragment.typographicBounds seem to be relative to NSTextLayoutFragment.layoutFragmentFrame vertically, but relative to the text container horizontally. (And there are also some issues with the frames being narrower than you'd expect when using overlay scroll bars.)

I'm not sure how helpful that is to what you want to do.

I ran into the same issue. I'm doing something different – writing my own custom Text View instead of using NSTextView – but you may be able to make use of similar techniques. My goal was to:

1. Fix Text Kit 2's lackluster background color rendering (comparison screenshots of TextEdit in Ventura and Monterey attached; FB10541687).
2. Be able to draw the selection in front of the background color but behind the text (FB10542426).

The steps are

1. Make a custom NSAttributeString.Key for "undrawnBackgroundColor"
2. Implement NSTextContentStorageDelegate.textContentStorage(_:textParagraphWith:) and return a custom NSTextParagraph with all occurrences of the .backgroundColor attribute replaced by .undrawnBackgroundColor. This prevents NSTextLayoutFragment.draw(at:in:) from drawing the background color.
3. In NSTextViewportLayoutControllerDelegate.textViewportLayoutController(_:configureRenderingSurfaceFor:) create a layer for the layout fragment, and then create N layers for each of the background color frames. You get the attributed string from the layout fragment's textElement (an NSTextParagraph), use NSAttributedString.enumerateAttribute(_:in:) to get all the runs that contain background colors, and then for each of those runs, use NSTextLayoutManager.enumerateTextSegments(in:type:options:using:) to enumerate the frames for each of your background color layers.

If you use .highlight or .selection (no idea what the difference is, they seem to have all the same drawing rules) as the type in enumerateTextSegments(in:type:options:using:), you'll get the nicer background color drawing behavior seen in Monterey's TextEdit (which still uses TextKit 1 for Rich Text).