Brown, imo.
Brown/copper/amber/rose all create contrast, which helps when identifying structure and seeing fish. Gray keeps colors and contrast natural, and some like that, especially for everyday use and driving. For fishing, my personal taste is that I want all the contrast I can get, and I'm fine with it elsewhere too.
Note that if there's any sort of mirror, the color of the lens may not match the outward appearance of said lens. Lots of people have no idea what their real lens color is!
Yellow lenses are generally for low light and tend to have more light transmission. With decent lenses you can generally look up the transmission %. Higher transmission allows you to wear them earlier in the morning and later in the evening, under a dense forest canopy, etc, but they may be too bright on a bluebird day in the open. Many fisherman have 2 pair, dark and light, and rotate accordingly. With only 1, I'd opt for a tweener. Light brown or rose.
Even just a polarizer, no tint, will block about 60% of light, so there's just no way to get a lens that'll be wearable right through the witching hour. Polarization loses effectiveness at low sun angles anyway.