...I have been considering doing this route for years, and I admit that the closer you get to the core, the more challenging it will become, due to the prevalence of commercial et al boat traffic. So I'd be content with completing the journey well before the mouth of the river....
My wife and I have gone from Fort Langley to the mouth of the Fraser several times and never had a problem. Big tugs or freighters powering up river can generate some large waves (several feet). Conditions do vary quite a bit depending on wind, tides, currents and day of week (for traffic) so we enjoy the variety.
I've done portions of the upper end and it can have some squirly currents depending on water levels, locations of sandbars, confluence of side channels etc. Not sure which is more challenging but certainly different.