I've never fished Cape May harbor but, generally, harbors are often not considered the best places to fish due to high boat activity. There are exceptions, mainly in Florida (that I can think of).
It might depend on the nature of the harbor - how enclosed it is, how much boat traffic is dispersed, etc. Most harbor areas have deeper water and this can make FFing more difficult for a host of reasons.
Back bay areas on the NJ shore can fish very well and April/May are prime time for a variety of species. If it's possible to do so, however, I'd try to get away from harbors, marinas, etc and find shallow grass flats. This will mostly require a boat.