For the past several years, I've used a sleeping bag stuff-sack to haul my outer ware, boots, pack, grunt tube, rattling bag, RC climbing harness, etc. That way I am carrying only a couple things to the Jeep......my bow case and that sack. This summer, I bought a camo nylon duffle with carry handles that is working even better. Staying organized keeps me from leaving stuff on the Jeep roof or laying forgotten in the grass as I drive off.....both of which have happened in over 40 years of bowhunting. I still chafe at memory of the Double Bull blind chair I left in the weeds in a public spot and never saw again.
So when I get home from a hunt, I set the duffle down where I keep my current hunting clothing in the basement.....and take care of my bow first. Once it is wiped down or simply hung back on the rack, I take off the layers I am wearing and hang them on hangers to go outdoors to air.....followed by the outer layers in the duffle. Like the OP, I rarely launder the other layers unless wet and/or bloody. Mid-layers every couple weeks and base layers weekly.....untless I sweated in them setting up or hauling out a deer. Boots come out of the duffle to air out and every once in a while, a shot of baking soda to prevent odor. Since our MI weather fluctuates greatly, I will hang out the set I plan to hunt in prior to a hunt to "freshen" them up.....including base layers. Rinse and repeat.....