Different hunting styles have different needs. My go-to rifle hunting pack(s) don't work for carrying a 1-stick. I tried and couldn't make it work without more modifications than I cared to do. It gets cold during rifle season and I like to carry extra layers and a thermos of something hot to drink. I like to carry the thermos in an outside pocket and the layers inside. I also like to carry a paperback, a lunch, and some scent products; and I like to have them accessible without digging to the bottom of my bag. Early season, no layers, I can get by with a Camelbak and turkey vest. Late season I find myself adding more snivel gear at the end of a miserable day, and consequently need a bigger pack. I hate having zippers on hunting packs and prefer drawstrings and buckles. I dislike having any more bulk on my shoulders than I have to, as layers of straps can interfere with shouldering a rifle. In MN, boning out the deer to pack out the meat isn't an option as I can only leave entrails in the field. So my particular preferences eliminate the vast majority of packs on the market.
Different body sizes require different pack sizes. Larger budgets can result in quieter, lighter, stronger, etc. materials. Fit is absolutely an issue, but unfortunately fit is something you'll have to budget for. The good news is that the used pack market is pretty amazing, especially if you don't need a camo pack made expressly for a particular style of hunting. A '90s-era earth-tone backpacking pack can be had on an auction site for less than a new Alps pack, and you can (probably) adjust the fit to suit your needs. A youth or women's model will be a bit smaller than a men's model, which might be a more attractive option. It's easier to carry 20 liters of gear in a 60L pack than vice versa. And if it's cheap and a bit ratty you won't feel bad adding lashing loops or whatever to accommodate whatever you want to attach to the outside.