I encountered this problem of excessive memory usage of Swift-frontend when I declared a variable without an explicit type.
let defaultMargin = 16.0
When I set it with CGFloat, I don't have the problem any more.
let defaultMargin: CGFloat = 16.0
I hope this can help.