Our group here in Michigan has been holding winter pool sessions with our kayaks for the last 15 years.
I have never had a problem, or know anyone that has had any issues. if you bring them in and then change into your swim trunks the boats will be acclimated to the pool temp. We empty any water from the hull and dry the boats prior to taking them back out into the cold and have not had any issues.
Walt