animateWithDuration not animating

I have a view that is say 100x100 nested into my view controller's view. Inside that 100x100 view I add a AVPlayerViewController. On focus change I am doing this:

if context.nextFocusedView == self.playerController.view {
                self.view.layoutIfNeeded()
                UIView.animateWithDuration(4.0, animations: ({ () -> Void in
                    self.view.removeConstraint(self.videoToViewAspectConstraint)
                    self.videoViewLeadingConstraint.constant = 0
                    self.videoViewTopConstraint.constant = 0
                    self.videoToViewAspectConstraint = NSLayoutConstraint(item: self.view, attribute: NSLayoutAttribute.Height, relatedBy: NSLayoutRelation.Equal, toItem: self.playerView, attribute: NSLayoutAttribute.Width, multiplier: 9.0/8.0, constant: 0.0)
                    self.view.addConstraint(self.videoToViewAspectConstraint)
                    self.view.layoutIfNeeded()
                    /
                    /
                }));
            }

The constraint change happens as expected, however it happens immediately instead of taking the expected 4 seconds to complete.

I have been beating my head on this for like 5 hours now and I am at a loss. I think the problem is actually a bug with AVPlayer because if I have playercontrols enabled and I change constraints on that view I get constraint errors for the AVPlayer in my logs. However they could be unrelated.