Although not a direct solution for the problem, I can suggest using AVAudioUnitVarispeed instead of AVAudioUnitTimePitch. While working on a real-time sampler, I also encountered the same latency issue of the AVAudioUnitTimePitch, and I found out that AVAudioUnitVarispeed produces similar results (in my case resulting sound was even more consistent) while introducing no (perceivable) additional latency.