I had already read that advice somewhere here but it didn't work for me. After adjusting the fence so that it would end with the blade exit definitely helped but it didn't solve the problem as i wasn't able to push the boards exactly straight forward without any sideway movement.
I wish it had worked since that was the reason I bought the table saw module for my handsaw…

By the way, I also tried using a fence attachment for the saw, Björn Thomasson recommends that in his building manual. Didn't work for me either.