Polar bear swimming will help to acclimatize you to ice water greatly reducing the risk of death caused by accidental cold water immersion emergencies during wilderness travels and it has amazing health benefits that strongly outweigh the risks.
I really do recommend it, it has healed me of many physical ailments and has increased my circulation, improved my skin, digestion, and overall health so much not to mention I don't have to worry about the cold anymore.
I do not experience cold shock (you know that cold water reflex that has you gasping and choking and unable to breathe when you plunge suddenly into cold water?). This reflex can instantly kill some people, but someone who is acclimatized to the water temperature will not experience it at all.
In glacier water even though I do not feel the cold, it takes less than 5 minutes before my hands begin to go numb and freeze and that is always the first sign that you should get out right away because not long after, the rest of your muscles will lose mobility too. However, I swim for a very long time after that in shallow areas where I can touch bottom where I don't have a risk of drowning, even when my limbs are frozen. It might be good to practice swimming in shallow areas when your muscles are so frozen that they can barely move, but make sure to always have a spotter nearby because there are considerable risks LoL. It can get to the point where you can barely walk. The ice immersion is very good for healing injuries, especially long ice immersion. And it is also good for helping depression. No more Seasonal Affective Disorder for me.
I think the most important tool to have to survive cold water immersion is hand fins. LOL it sounds funny but people worry so much about covering the rest of their body, but they forget their hands and hands are the most important things you need to swim and guess where all your blood is going to? It's all rushing to the internal organs, far away from your hands, which get colder and colder by the second, so you need to worry most about your hands! The rest of the body should have enough body fat and muscle to not freeze as fast but you need your hands the most to swim as fast as you can to reach safety before the rest of your muscles do freeze up and you drown, and when submersed in ice water your hands claw up into fists and no longer move, in very cold water this can happen in less than 5 mins! Those hand fins are good for when your hands aren't able to open up anymore! They will help you swim faster.
If I were kayaking in cold places, I would make sure to have hand fins close by at all times. It's not like you'd wear them all the time but having them easily accesible would be good. Just wearing gloves wouldn't do the trick because your hands will still get so cold in ice water through any material that they will eventually claw into fists and you won't be able to move your fingers anymore. Try and swim with fists, you don't get very far!
You can continue swimming for quite awhile but you will lose all mobility in your hands and wrists and feet and ankles, next your arms and legs. Eventually you will just have to float if you can't move. Which is probably a good thing in majority of cases because the more people can move, the more they panic and sink. But when you relax and don't move, your body naturally floats...
Anyway, just a bunch of rambling but I am just really passionate about cold water immersion haha it is probably the best thing that has ever happened to me and man is the rush ever amazing. As long as you take precautions and are generally fit, it can actually be very healthy.
