It is possible to trim your kayak to eliminate weathercocking by moving the CG back. On the fly the easiest method is to carry an empty water bag and fill it with sea water and put it under and clip it to your aft deck rigging. If you then leecock pour out some water or move the bag forward. For a permanent solution first move your seat back until the kayak is wind neutral. If your rear coaming gets too close to your back before curing the weathercocking you can 1) enlarge the cockpit opening to the rear to move your seat further back, 2) add permanent aft ballast, 3) add a fixed skeg 4) add a deployable skeg, 5) add a rudder or 6) remove deadwood at the bow.
Photo by Duane Strosaker.