Not sure if you've found a fix for it yet, but I was having the same issue.
What worked for me is booting into recovery mode and disabling system integrity protection (csrutil disable) and rebooting. Then I used terminal to force empty the trash containing Safarifamily (sudo rm -R <path to file in trash>)
just remember to boot back into recovery mode afterwards and re-enable system integrity protection (csrutil enable)
Excellent advice. Thank you.
One observation. .The terminal message made it seem that 'disable' worked and reboot was a verification step, so I decided not to. I was able to remain in recovery mode, and removed that folder and contents, use the enable command, then I just restarted my imac.
Things appeared normal and that pesky Directory and contents were gone.
But what do I know? % csrutil status
System Integrity Protection status: enabled.
But what do I know? %