Inconsistent DragGesture translation?

I feel like I must be missing something dumb, but I can't figure it out. I'm trying to create a modifier to make items resizable by dragging on the corner (I haven't actually implemented the corner part yet though so dragging anywhere on the object resizes it). However the rate that I'm dragging at is different from the rate that the object is resizing. It's also different for horizontal and vertical translation (the horizontal change is smaller than the rate that I'm dragging while the vertical change is larger).

Any help would be greatly appreciated!

Here's my code for the modifier:

struct Resizable: ViewModifier {
    @State var size: CGSize = CGSize(width: 500, height: 500)
    @State var activeSize: CGSize = .zero
    func body(content: Content) -> some View {
        content
            .frame(width: abs(size.width + activeSize.width), height: abs(size.height + activeSize.height))
            // offset is so the top right corner doesn't move
            .offset(x: -abs(size.width + activeSize.width) / 2, y: abs(size.height + activeSize.height) / 2)
            .gesture(
                DragGesture()
                    .onChanged { gesture in
                        activeSize.width = -gesture.translation.width
                        activeSize.height = gesture.translation.height
                    }
                    .onEnded { _ in
                        size.width += activeSize.width
                        size.height += activeSize.height
                        activeSize = .zero
                    }
            )
        
    }
}

extension View {
    func resizable(maxSize: CGSize = .zero) -> some View {
        modifier(Resizable())
    }
}

And it is used like so:

struct ContentView: View {
    var body: some View {
        Rectangle()
            .fill(Color.blue)
            .resizable()
    }
}