Isolating position animation in SwiftUI?

Recently I've tried to make custom control in SwiftUI and faced a problem with animation. The problem is that implicit animation in the view overrides any external position animation of the view itself.

To show the problem I made simplified toggle control. I'm using implicit animation so that changes in that custom toggle would animate independent on any external animations.

struct CustomToggle: View {
  @Binding
  var isOn: Bool
  var size: CGFloat
  
  init(_ isOn: Binding<Bool>, size: CGFloat = 50) {
    _isOn = isOn
    self.size = size
  }
  
  var body: some View {
    let w = size
    let h = size / 1.7
    let d = h / 10
    ZStack {
      RoundedRectangle(cornerSize: .init(width: h / 2, height: h / 2))
        .fill(Color(isOn ? UIColor.systemGreen : .systemGray5))
      Circle()
        .fill(.white)
        .frame(width: h - d, height: h - d)
        .shadow(radius: d / 2)
        .position(x: isOn ? w / 4 + d : 3 * w / 4 - d, y: h / 2)
    }
    .animation(.spring(response: 0.4), value: isOn)
    .frame(width: w, height: h)
    .onTapGesture {
      isOn.toggle()
    }
  }
}

Then, in containing view, when toggle switches it's also changes its y position with animation (circle gets bigger and pushes toggle up):

struct ContentView: View {
  @State
  var expanded = false
  
  var body: some View {
    VStack(alignment: .center) {
      let animated = $expanded
        .animation(.spring(response: 1, dampingFraction: 0.3))
      let size: CGFloat = expanded ? 300 : 100
      HStack {
        VStack {
          Text("Native:")
          Toggle(isOn: animated, label: EmptyView.init)
            .fixedSize()
        }
        VStack {
          Text("Custom:")
          CustomToggle(animated)
        }
      }
      Circle().fill(.mint).frame(width: size, height: size)
    }
  }
}

Unfortunately, result looks like this:

As you can see, position of custom Toggle animates with .spring(response: 0.4) animation, not with desired .spring(response: 1, dampingFraction: 0.3).

Native Toggle control behaves as expected, but that is probably because it is just a wrapper around UISwitch.

As I understand, SwiftUI animates position of every leaf view somewhat independently, and implicit animations in that views override any outer animations. That differs from UIKit where you can animate some view's frame without affecting its child views.

I've tried things like compositingGroup, drawingGroup and so on with no luck.

Is there any trick to make it work?