My solution was to waterproof fleece with camp-dri or whatever spray-on waterproofer I could find, and leave the Gortex in my pack. I sweat so much in Goretex I might as well not be wearing rain gear. I've mostly transitioned to USGI ponchos, but, honestly, if it's raining so hard I'm worried about losing my blood trail, I'm going to take a break from hunting. Maybe I'll rig a shelter and read a book while I wait for it to pass, or ride it out in my truck, or hike until I find a good place to set up when the weather breaks.
If you're set on wearing your Goretex, maybe try a light fleece vest over it? I bet a turkey vest would serve the same purpose as well. Fleece chaps would probably get torn up pretty quickly, but maybe some cordura-type material, or retired chainsaw chaps?