I have a Q-P, which is a wonderful camera, a CL with all the lenses but the 60mm macro, and an M10-P. The Q-P is really pleasant to use if you can live with the 28mm - which I can if I carry the M10 or one of my other ILCs.
However, I still believe the CL is underrated, as are its lenses. The 23mm is outstanding and small. Also, don't pass on the 11-23 because it's a zoom - optical quality is great. Autofocus is more than fast enough (and similar to the Q). If pressed I think I would buy the 23mm for the CL, but either way you'll be happy I think.
I had the m43 15/1.7 - that is a terrific lens, definitely my favourite when I shot on that system.
I also had the Q, sold it - mainly because I didnt like being limited to just 28mm. Also it wasnt very comfortable to hold 1 handed, and the EVF would smear if you weren't looking directly through it.
Currently have a TL2 as my secondary, 'fun' kit. The TL lenses are actually all very good, definitely is an underrated system. Their lenses are all very high quality and basically set the path for the SL full frame lenses.
If I didn't have the 18mm + 35mm, I would get the 23mm. That is a nice combination of compact/lightweight/fast aperture and great image quality.
I think it makes more sense to add lenses to your CL system, that is part of the fun of being interchangable.
If you buy the 23mm used, you can likely resell that for similar cost. Camera bodies like the Q depreciate fast. And even faster when the QP/ Q2 get superceeded further.
Although an 11-23 + 35 + 55-135 would be a great 3 lens combo.
Although I've not used the 11-23, I have read many people saying that (despite being slow aperture) it is amazingly good optically.