I tried the Fuji (first the x10 to get a feel for the layout) system for a short spell, and had it along side the Oly. However, its so slow to manage their system compared to the OM-D, that I sent it back.
X-E1 and 18-55.
Love that lens, faster and better than the Oly, but limited by the camera and its functionality IMO. Of course, that is just me and I know many great shooters love the Fuji system.
Give it a spin, but if you have used the OLY OM-D, you will find it a bit on the sluggish side I dare say.