I figured I'd follow up with a report, so that hopefully it can be of use to anyone who makes their way down there in the future. Thank you to everyone who offered any advice, tips, or information for me, I truly appreciate it. Hopefully this recap can help someone else on this site in the future. This will be a long post, so consider this your warning.
I was down there from last Friday until this past Wednesday, fishing Friday-Tuesday, and staying in Highlands (which is not an ideal home base for what its worth; beautiful town, but a little far removed from the best fishing).
Friday night and all day Saturday I fished the Lower Raven Fork in the C&R Trophy Section. This area is stocked and maintained by the Cherokee Indians and holds very large rainbow trout that are incredibly colored. Everything I read before I went indicated the fish holdover very well there, and the water is very cold and fast-moving with good habitat, so I believe that is true. The water was very cold both days, did not get a stream temp, but fish were actively feeding when I could see them. As someone mentioned above, it rained almost every day there so the water was up and off-color, but it was just enough for the fishing to still be good. I had no issues wading and didn't see any indication that a flash flood would put me in danger. I hooked up with 3-5 fish in the 20"+ range, and got one to the net. I also got a couple in the 15"-18" range as well. No pictures because of a poor net job and not wanting to get my phone soaked but they were incredible fish nonetheless. If you want an idea of how colorful these fish are I recommend you google them and you'll see what I mean. Golden stones, big black streamers, small black wooly buggers, and other assorted streamers were getting strikes and hook-ups. I think they were pretty much eating anything big. They ignored anything smaller than a sz14 hook, but it could have been my presentation. Summary- this stream was incredible, if you make it anywhere near Cherokee, NC you owe it to yourself to fish here. You have to pay for a C&R permit (roughly $30 for 3 days) but I felt it was worth it. They may be stocked but these were hard-fighting and massive trout. As a side-note, the Blue Ridge Parkway (which will drop you off 2 minutes from the river) is worth driving on while down there, it is very scenic and a very fun drive. I took it almost all the way from Asheville, and while it wasn't faster, I enjoyed it thoroughly.
Sunday I fished the Upper Nantahala River (above, which is south of, the Power Plant where they do whitewater releases for the rafting in the Lower portion). This river is designated as wild trout waters and is known to hold wild bows and browns. This was the most blown out water I fished all trip. It poured all day, and I was essentially fishing blown out pocket water in the pouring rain, but at least I had the place to myself. I was moving some fish (all on nymphs in sz16-22), hooking some, but got none to net. I lasted maybe 2 hours before I left. This was an easily accessible river and I wish I could've fished it in better conditions.
Monday I went brookie fishing in Panthertown Valley, and finally had a beautiful sunny day. My mom even came along to enjoy the hiking portion. We were at roughly 3800' of elevation while there, and it was much different than the brookie fishing I've done in PA. Less boulder hopping, more of an actual mountain hike. There were much larger waterfalls there than I'm used to here, which dumped into swimming holes for hikers (affecting the fishing as you can imagine). Water was still a bit up from all the rain, but not too high to fish. The brookies were not eating as often as I'm used to but I managed to get 5-10 in about an hour or two of fishing. This was easy hiking and very accessible, which is probably why there so many people there.
Tuesday was spent on the Chatooga River (where they filmed Deliverance) and the Whitewater River (home to some of the largest falls east of the Rockies). The Chatooga had one of the craziest hikes into a stream I've ever experienced. I took a less travelled trail which involved scaling a mountain and it took about 40 minutes to get to the water. I moved a couple fish there in an hour or so of fishing, but the water was extremely warm and it was very hot out, so I called it early. On the way back up the mountain I got lost and came across a pretty good size rattlesnake. Fortunately, my dog who was with me was able to sniff out the trail, and got us out of there after a long detour. On the way to the Whitewater River I stopped and checked out the falls, these are a site to see. They're almost right on the SC/NC border but still in NC. I fished the Whitewater upstream of the falls and came into the same conditions- hot and warm water. I decided to call it a day and head back.
All in all it was a great trip. If I had to do it over again I would stay further west and spend my time on the Raven Fork and Nantahala. There are more brookie streams out that way as well and there are some other rivers for trout I would check out.
Hope this is useful to somebody, and didn't put anybody to sleep!