Curious if you ever figured it out.
I think what's happening is that with a dynamic sized layout. The layout doesn't know height of each cell until it is dequeued and rendered. Therefore, the content offset is meaningless because it will require every cell to be rendered to accurately scroll to the right place.
Instead of invalidating the layout, I find that updating a cell using DiffableDataSource works (https://betterprogramming.pub/new-in-ios-16-self-resizing-uicollectionview-cells-f02fbb7b45b1). I think that DIffableDataSource just calls UICollectionVIew methods under the hood, so I'm curious if there's a lower level way to handle it.