Well, I have both cotton fleece and what I think is a synthetic fleece. I suppose I should dig a little deeper and give fleece more of a shot as a mid-layer. The one I'm basing it on is by the brand "Polartec". I'm pretty sure it's sythetic but not 100% sure. It is an older garment, they may have improved versions.
The Polartec does insulate pretty well even when wet. But it really gets wet, we're talking dripping. Yeah, you could wring it out, and I do, but when a clothing layer always seems to need wringed out when none of my other garments are overly wet, it doesn't make me think super highly of it.
That said, wool is probably taking on every bit as much water, but it seems to be able to absorb it and doesn't "feel" so saturated like the fleece. Ultimately, though, if you're willing to put up with some wringing, wringed-out fleece is going to stay lighter than wet wool.
But yeah, the middle layer is where I've always struggled. I feel like I've perfected the base and outer layers and struggle with that middle layer. Often, I just pack it in, so that you can stay cool and sweat less while hiking. Once you get where your going and slow down (on stand, start fishing, etc.), then break out a dry mid-layer and button up, and it doesn't so much matter the material, even cotton is fine as long as its dry.
Jay, yeah, underarmour gets kind of rank after a long day. In deer camp, we each have like 1 set, there's no washing machine, and we're out like 5 or 6 days straight, which is a pretty disturbing thought! Washing it in the sink with hand soap or dish soap is like a nightly ritual, it does take the rankness away. When camping, I do the same by taking it to a stream and soak/wring/soak/wring. Overnight is MORE than enough time to dry it completely. Really, I think it'd be wearable in an hour or two.
The only issues I've had with that routine is when camping below freezing. It doesn't dry then!!!!!