There isn't a 100% accurate way to tell without killing the fish, but IMO Brookies are the easiest to tell stockers from wild fish, and in most cases you can make a pretty good guess. I'd bet most of the Brookie guys on this site could get it right 90%+ of the time.
Wild Brookies (especially freestone Brookies) will generally have rather streamlined bodies and usually are not very "fat." Their fins will generally be clean, and sharp and have straight black and white edges on them. Their dorsal fin is often a brilliant orange color with black spots. Their overall coloration can vary a lot, but there is usually a distinct contrast between the spots and background colors, regardless of the overall color tone...meaning the colors don't run together.
Stocker Brookies are usually wider bodied. Their fins are often rough or tattered on the edges, or completely deformed in some cases. The edges on the fins lack the straight, sharp black/white lines too...the black and white often run together. The dorsal fin often lacks the brilliant orange color and is a duller orange/brown in many cases. They can have pretty decent color patterns depending on what they were being fed, but they almost always lack contrast between the spots and the background colors...the colors seem to blend together more.
All that said...There will be some stockers that will look wild (especially if they were fingerling stocked or escaped a hatchery, etc), and some wild fish that will look like stockers (worn fins after spawning, apparent lack of contrast between colors/spots based on diet, etc).
Location trumps all though IMO...Does the stream you're catching them in have a hatchery on it, or get a stocking of fingerling Brookies from somewhere (club/kids program/etc)? If not, just based on the numbers of sublegal fish, I suspect you're probably catching wild fish. Some "short" stockers sometimes get mixed in with the state's stockings, but not enough in most cases that you'd be regularly catching them.
Take and post a few pics if you can...we love stocked/wild trout debates around here!