As a new-to-Fuji user, I bought the X-E5. One big differentiator for me is the IS which the X-Pro 3 does not have as well as the 40MP sensor.
The X-Pro 3 has a higher resolution LCD, higher resolution EVF, faster mechanical shutter, weather sealing, two card slots and longer battery life compared to the X-E5. All nice to haves but the IS and sensor were more important to me.
It would interesting to see an X-Pro 4 inherit similar capabilities as the X-E5/X100VI in regards to IS, Sensor, Subject detection, the X100's ND filter, etc. while keeping (or improving) the features above.