I have the original 16-35/2.8, two of the f/2.8 II, and the 16-35/4 IS.
The only f/2.8 that is sharp is the III. The first version is especially bad in the corners at 16mm.
The f/4 IS is quite good and the only one I use. At the time I opted for it over the f/2.8 III due to the IS.
I recently was able to get a near mint copy of the version III lens at my local camera shop for about $950, which appears to be a pretty decent price at retail, and I’ve been quite pleased with it. It’s very good throughout the range. It’s probably the best overall DSLR ultrawide I’ve used. There are some newer mirrorless ultra wides that are a little better, but this one still holds up. The earlier versions, though, are probably not worth your time. The f/4 IS is quite good too though if you don’t need f/2.8.
Like with all lenses, the latest version is the sharpest and best contrast. You can get around this by closing down a stop or two which defeats the purpose for most of us buying fast lenses. I own the 16-35/2.8LIII and am happy with it, it's also the only zoom that I own.