Unless I'm in a spot where I am certain there are not going to be hunters, or it's a Sunday, I wear orange from mid-October through the New Year. Most of the time I'm fishing in a SF or SGL though, so I'm almost always wearing orange during this time frame. I do think fish spot me easier, and spook easier when wearing the orange versus camo or drab colors, but it’s better to spook a Trout than inadvertently have an accident.
Personally, I’m not a big fan of fishing period during rifle season, unless it’s a Sunday or it’s somewhere where I know hunting isn’t allowed. Just too many guys in the woods for my taste then, and the likelihood of an accident I’m sure is greatest during that time period.
It's actually a fairly normal occurrence for me year-round to hear gunshots while in the woods fishing in PA. Fortunately, they’re usually well off in the distance, and someone target shooting on a property adjacent to the public land I’m on.