Thanks for all your input. Eoin, your post helped me understand this problem greatly.
I will at this point try them out and may just find myself buying a 35 1.4L if it suits, you can always stop it down if the DOF is too narrow, then it will only be sharper as well.
I love my 16-35L the copy I have is extremely sharp. I also upgrade from a 17-40L. If you find a sharp copy, you'll love it. There is also a noticable difference between the 17mm and 16mm. My galleries have examples if you live
After much soul searching, I bought the 16-35 for the extra stop. I sometimes use wide angle because I can use a slower shutter speed in low light, so the speed of the lens is important to me. If you are shooting outdoors there is no point. I think the F4 is generally a tad sharper.
No matter what - the primes will always be sharper.