The culprits were actually different ones and not in the code I posted.
) The kernel_task:
The following code contains the memory leak:
let croppedImage = frame.image.cropped(to: CGRect(
		x: Int(Int(videoSize.width) / 2 - (self.RECOGNITION_WIDTH / 2)),
		y: Int(self.videoSize.height - CGFloat(self.RECOGNITION_HEIGHT)),
		width: self.RECOGNITION_WIDTH,
		height: self.RECOGNITION_HEIGHT
)
Apparently CIImage.cropped(to:) leads to the original CIImage not being cleaned up. According to my Google search I should have called CGImageRelease(:) but that is not contained in Swift 5.1 since Core Foundation Objects are now memory managed. (At least that is the Xcode error message.)
So I convert the image to a CGImage, crop that one and turn it into a CIImage. I know this is not efficient, but for me that's fine. So here is the new code:
import Foundation
import CoreImage
extension CIContext
{
		func cropWithoutMemoryLeak(_ image: CIImage, to rect: CGRect) -> CIImage
		{
				let newImage = self.createCGImage(image, from: rect)
				return CIImage(cgImage: newImage!)
		}
}
) The VTDecoderXPCService:
I was able to reliably remove the memory leak by commenting out this line, which obviously defeats the point of creating a new video:
self.frameInput!.append(newSampleBuffer)
After trying some more stuff I removed some of my debug print statements which I had throughout my code. And all the sudden the leak was gone...? 🤷♀️ So I don't really know what the problem was. Removing the debug lines was most certainly a correlation but no causation. I guess it triggered a recompile of previously cached files but admittedly I have no idea, what the problem was.