A pic of the sanding pad, and one of the sandpaper that left the swirls could be useful.
You can always check the conditions of the pad (whats under the sandpaper) it should be a soft surface in good conditions, no worn out corners no pieces of wood or other stuff embedded in it. if its just a piece of metal someone peeled the spongy thing it off.
To fix that hunt for a thick not too coarse grained piece of cork (i use round cork that Ikea sells to put under hot pots, they are the right stuff and cheap) glue it in place and go to work.
Another reason its the condition of your sandpaper, i overuse my pieces and once they start not cutting as much they load easily, if you put a lot of pressure on the piece the dust loads the paper faster too, some softwood with coarse paper leave piece of fibers that get stuck and make the swirls too. So if your paper has even a couple clogs swirls are going to show. If you get a crepe block to refresh the paper you can get more use out of it.
Try to move the sander back and forth with the grain, it doesnt avoid the but at least they are a little less visible.
Vaccum or brush away the dust frequently, see the post about the dust free ROS, so theres less material that can pack up.