Question: Do you intend to use the boat?

If you ever put the kayak down on the beach, touch it with a paddle, touch it with your greasy hands, allow saltwater to dry, wipe the boat off with a rag, put it on a car or look at it funny, you will end up leaving marks on your varnish.

I brush with foam brushes, in my work shop after halfhearted efforts to clean up, I mask off at the sheer line, and I do end up with dust in the finish. I think the boats look great when I'm done with the varnish, and a year later after they have been scratched up and abused, they still look great.

If you are careful with your varnish application you will get a beautiful finish even if there is dust. I see very few first time varnishers who do a good enough job brushing on the finish that the dust detracts from the results. While still good looking, the finish usually has enough holidays, and sags, that the dust is not a big deal.

A well laid line of masking tape will leave a barely noticeable line along the sheer, it is usually camouflaged by the fact that there is at the very least the edge of a strip running along the same line. If the varnish is applied well, the minor ridge left by the masking tape will virtually disappear.

I would spend my time making sure I did a very good job prepping the surface of the kayak prior to varnish as it is the most important aspect of a nice finish. Follow this by practicing laying a nice even layer of varnish. I use foam brushes and can get a flawless finish by careful, systematic application.

Well done, this should make a beautiful finish, where after the boat has been used for a day or two will not reveal whatever dust ends up in the finish.

If dust really bothers you and the edge at the sheer bothers you, the only solution is to buff. Wait at least a week before proceeding, longer is better. It takes a lot of work to get the finish back to the gloss you had fresh off the brush. If you spend the time buffing, know that anything you do to the boat from now on will immediately detract from the finish.

If you intend to use the kayak, your best bet is to relax and just be happy with your beautiful boat. You probably made enough other mistakes that eliminating a few flecks of dust isn't going to make it look significantly better.

BTW, On my older boats that are in dire need of a new coat of varnish, they all look spectacular when they are wet. A fresh coat of sea water looks great and is generally dust free. Unfortunately, it does not provide the UV protection which is really the only reason to bother with varnish at all.