That's another good point too.I prefer to take mine off in the tree and have it set to pull another arrow if needed as others have said. I generally practice with and without it but not purposefully, more because it depends on time and what range I’m at.
Side note, and this may not be true. But I’ve always thought the quiver on and having my bow between the deer and myself helps to break up my profile. That may or may not be true, but it makes me feel more hidden when I’m on the ground and a deer is looking right at me haha.