Tile Rendering in Metal

Question

Created Mar ’23

Replies 6

Boosts 0

Views 1.1k

Participants 4

I developed a small drawing application for the iPad in Swift using Metal. It supports drawing Lines with the Apple Pencil, erase them, select them and so on.

Everything is drawn onto an offscreen texture so I don't need to redraw all of them. The App also supports zooming and panning. If you zoom you have to redraw everything. And there begins the problem. At the moment if I pan my canvas I simply draw the offscreen texture at zoom level 1.0 and pan it around. So it looks very pixelated because I zoom into the offscreen texture without redrawing the offscreen texture itself. The problem is I can't redraw the whole offscreen texture every frame while panning because for performance reasons. My idea was to implement a tile based progressive rendering technique. I thought I could split up the screen into tiles with size of for example 256x256 pixels and if I pan I can move those tiles and only the new visible ones I need to render. But I don't really know where to start. I do not know how to render the lines onto those tiles. every line has a single VertexBuffer at the moment storing the triangles. I thought maybe you can use multiple color Attachments for The Render Encoder to draw the lines. So every colorAttachements[n].texture would be a tile. And maybe through the help of Viewport Culling? I don't have much experience in this area so I have no idea.

I also found the Sparse Texture feature, which looked like it would solve my problem. But I need to support iPads that do not support this feature. Has someone else made something similar ? Or has any example code for this? Or has a complete different Idea that could help?

Thank you very much for your help!

Answered by endecotp in 747785022

> There are about 16651560 Vertices on the screen.

In that case, I suggest that you do some pre-processing on the CPU:

When you're zoomed out and the entire image is visible, use Douglas-Peucker line simplification, or similar, to reduce the number of vertices.
When you're zoomed in, use some sort of spatial index to send only the vertices that are on the screen. (I.e. "tile" your input vertices, not your output.)

You'll end up with some sort of multi-resolution data structure.

Note it's easy to over-engineer this (I've done that). The GPU is fast. You only need to do some crude CPU pre-processing to get the number of vertices down to something tractable.

Also: is the very large number of vertices because you want smooth curves made from very short straight segments? If so, consider storing fewer vertices and using the GPU to draw the curves; search for "GPU quadratic bezier".

Boost

Answer 1

Ceylo OP

Mar ’23

> The problem is I can't redraw the whole offscreen texture every frame while panning because for performance reasons. My idea was to implement a tile based progressive rendering technique.

Before trying to go this way I would look into these 2 options:

make your offscreen texture bigger, so that it is less blurry when zooming in (I guess you don't need very high zoom levels?). Pros: very simple to implement. Cons: uses more memory and rendering to that bigger texture will be slower.
directly render everything to the framebuffer. I'm skeptical about you having to go through an offscreen texture. Do you really have so much to render that it's impossible to render in 16ms? Did you profile with Instruments/GPU Frame Capture to see what part is slow and whether it can be improved? Is the bottleneck the rendering or scheduling part?

0

Answer 2

endecotp OP

Mar ’23

> The problem is I can't redraw the whole offscreen texture every frame while panning because for performance reasons

Redrawing everything every frame is how most 3D apps work. If you can do that, then you don’t need your offscreen texture.

How many triangles do you need to draw? Have you characterised where the performance bottleneck is?

0

Answer 3

Alecazam OP

Mar ’23

You have to tile it yourself, or use one of Apple's tiling classes. UIKit has some of these. But handling blurs across tiles isn't fun. The M1/iOS hw is TBDR, so it's further tiling up the image into 32x32 tiles, and only drawing elements within. So that is how you reduce bandwidth. Otherwise, you need a dirty rect system. The sparse texture is mostly for reading textures. Like in game, having a megatexture, and only pulling int tiles that were needed to display. It's also only on A13 and higher.

0

Answer 4

KimJongTiramisu OP

Mar ’23

Thanks your your reply guys!! Making the offscreen texture bigger will not work for me because I want to support high zoom levels. (Comparable to GoodNotes for example). First I tried rendering directly everything. But I included some profiler Images while panning and while drawing. There are about 16651560 Vertices on the screen. This might be overkill for a drawing / writing app but other Drawing apps can handle this many lines. But with this many Vertices the FPS starts to drop to 30. I don't really know what else of the rendering part could be improved and it does make sense to paste all my rendering code in here, but if you need a specific part I can of course provide it.

Profiler while panning

Bildschirmfoto 2023-03-15 um 12.04.04.png

Profiler while drawing

I draw to the offscreen so I only need to draw the new line onto the texture.

Bildschirmfoto 2023-03-15 um 12.04.33.png

Reference Image of the profiled State

Those were about 16651560 vertices.

0

Answer 5

endecotp OP

Mar ’23

Accepted Answer

> There are about 16651560 Vertices on the screen.

In that case, I suggest that you do some pre-processing on the CPU:

When you're zoomed out and the entire image is visible, use Douglas-Peucker line simplification, or similar, to reduce the number of vertices.
When you're zoomed in, use some sort of spatial index to send only the vertices that are on the screen. (I.e. "tile" your input vertices, not your output.)

You'll end up with some sort of multi-resolution data structure.

Note it's easy to over-engineer this (I've done that). The GPU is fast. You only need to do some crude CPU pre-processing to get the number of vertices down to something tractable.

Also: is the very large number of vertices because you want smooth curves made from very short straight segments? If so, consider storing fewer vertices and using the GPU to draw the curves; search for "GPU quadratic bezier".

1

Answer 6

endecotp OP

Mar ’23

Cache the buffers.

0