Lazy load data when scrolling to the top of LazyVStack

I'm trying to implement a lazy load for the bottom and top of a list. Everything works fine for the bottom list loading. When the scroll reaches the bottom, more data is loaded, and the user can continue scrolling.

However, when the scroll reaches the top, the currently visible view jumps out of the screen as new views are added to the top of the list, but the offset of the scroll view remains the same.

Any suggestions on how to achieve a smooth lazy load when scrolling to the top?

struct CalendarScheduleView: View {
   
  @StateObject var viewModel: CalendarScheduleViewModel
   
  var body: some View {
    ScrollView {
      LazyVStack {
        ForEach(viewModel.scheduleDays) { scheduleDay in
          CalendarScheduleDayView(scheduleDay)
            .onAppear {
              Task { await viewModel.fetchMoreIfNeeded() }
            }
        }
      }
    }
    .onAppear {
      Task { await viewModel.fetchMoreIfNeeded() }
    }
  }
   
}