Lindsay, If you dont mind seeing the swirls leave em.
If you do mind I would step through the grits and feather everything in, in that area. I would then do a final sanding over the entire hull.
FWIW. If you use a plane on your projects it pretty much eliminates the need for the agressive grits that cause these problems. Also I would get an ROS. They really cut down on the visible gouge marks when going across the grain, which really shouldnt be a factor in canoe and kayak building.

Keep building, Elliott