I've encountered two different conditions that fit that description.

Type 1: the boat turns into (or away from) the wind. Weathercocking or leecocking.

Type 2: in the absence of wind the boat tries to turn sometimes to the left and sometimes to the right when gliding. Not sure of the name. I had an extreme case with a rental boat once that could not be paddled straight for any distance. I found out it came with a rudder and it had been removed for rental.

Both problems can be fixed with a skeg or rudder, Type 1 can also be fixed by adding lateral area above the waterline either at the bow or the stern. Type 2 is caused by weight too far astern; as soon as the boat deviates from a straight course centrifugal force pushes the center of mass outwards and the curve gets tighter. Normally this gets corrected automatically during paddling but shows up when gliding.