It's situational. One of those things you learn after being busted many times for many reasons. There's no magic number. I'll even go so far as to say that there are some setups where a saddle isn't the best choice.
For example, on the rare occasion that I sit a food plot, I go as high as I can. I may take my hand climber and go until I hit the first limb of a tall southern pine. I expect multiple deer lingering for long periods, and extreme height keeps me away from multiple eyes and ears. If there's an established stand or shoot house, I stay as far away from it as possible. Deer down here watch those things constantly.
If I'm hunting thick, young woods I hunt lower because the canopy is lower. Deer are usually comfortable in these areas, and you will hear them before you see them, allowing you to set up. In these cases, I usually pivot and draw as soon as I sense a deer. Shots are close and deer are usually moving. I still like to get 20ft if possible.
If I'm hunting a marsh, I usually am doing good to hit 10ft. Just high enough to shoot a runway. Deer/pigs are VERY comfortable in this environment, and may very well be bedded 20 yards from my setup. Even if a high tree is available, I will not get high up in the middle of a marsh, because I believe it is very possible to be spotted by deer bedded nearby. Scent is not a concern because wind is very consistent and fairly strong in these areas. Much like it behaves over open water. I like a hang on stand and usually climb with one lone Wolf stick and an Aider. Right at 10ft, quick, quiet, climb. I usually hunt standing, and can easily pivot to shoot with minimal movement, so concealment isn't an issue.
Rifle vs bow is a factor too. With a rifle I like to hunt high and cover long shooting lanes (fire breaks, utility right-of-ways, etc). I hunt somewhat lower with a bow.
On my lease (120 acres) I try to only hunt presets and set them as high as possible. I am anal about how I access them, and only hunt perfect conditions. Sometimes I will go through in the summer and use a leaf blower to make a trail that I can walk on a bit quieter. If I blow a spot, it's over for the season. I DO NOT want to screw up if I can't relocate.
On public land, which is the majority of what I hunt, I am a little more fast and loose. I have more spots than I have time to hunt. Get in, hunt, move out. I have had good luck with this plan and rarely hunt the same tree twice. Don't usually preset, just take a pair of hand clippers and climb around what I can't cut. Don't like shooting lanes because other hunters key in on them.