SwiftUI ScrollView w/ LazyStack stutters when children contain conditionals

I've been wrestling with the challenge of performant ScrollViews for quite some time now and I can't seem to figure it out.

With either a a LazyVStack or a LazyHStack within a ScrollView, when I drag beyond the top (or leading) bounds, it will jitter or flicker.

I removed all of the code from the subviews of the stacks and put it back line by line until I narrowed it down to using any sort of if let statements within the child.

Here's a simplified example:

Code Block 
HStack {
  if let duration = item.duration {
    Text(duration)
  }  
}

If I take out the if let and just put Text("blah"), it no longer stutters.

This is a mystery to me, and I'm curious if anyone has any knowledge on the inner working of SwiftUI and why this might be happening.