A lot of the refurbished lenses are nothing more than open box returns. Because they have been sold and returned they have to go through the refurbished process. I've bought them before and had excellent luck with them. In fact, the last one I bought came in the original box, was packaged like a new one and had the full warranty. There was nothing to indicate that it had ever been opened and wasn't a new one. I think it depends too on who you are buying the lens from. Some disreputable vendors sell used or gray market lenses as refurbished.
The bottom line to me is how much the savings is. In my case I saved plenty. But I've seen refurbished ones that didn't save you that much and in those cases I would just rather pay a little bit more and get a new one.