HandBrake with H.264 10 Bit
HandBrake with H.265 10 Bit / 12 Bit
I did not compile this App all credit goes to mattgadient at https://mattgadient.com
He compiled a 10-bit version (both x264 and x265). Just download it, unzip, throw it in your Applications folder if you want, and run it (the same way you'd run your current version).
A couple notes on it though:
It's the easiest/cleanest option. If you decide to get rid of it, uninstalling is as simple as dragging to the trash.
It's ONLY 10-bit. So don't go un-installing your current version.
Don't “check for updates” or anything, because if it actually finds an update, it'll be from the HandBrake website (and for the standard 8-bit version most likely).
It was built on “El Capitan” (OS X 10.11.3)
To download directly from his website the link is below: