What you have discovered is that you have high spots and low spots in your finish. If you are an extreme perfectionist you could use a long board and gently sand until it is uniformly dull, in which case you'd end up with an extremely flat surface that will look more like an automotive finish. If this is what you're after, go ahead and sand more.

I would think that you've probably flattened it about as much as you would want to or you would have done it the first time you varnished. If that's the case I would recommend NOT sanding it any more, so that you don't risk sanding into the fiberglass. Instead, take a fine Scotch Brite pad and scuff the low spots that you didn't get during sanding. Then varnish away. You're good to go.

m