I keep one hand on the camera to help take the load off my neck. The bag with the lenses is being carried by my neck, so I try to use my hand as much as possible to carry the camera weight.
i went through this last summer. we went on a five day hiking trip into the yosemite backcountry and i wanted to bring my d300. after a lot of inquiry, i settled on a thinktank holster in front with their auxiliary straps which hooked onto the front straps of my backpack. i also used a small bungee cord on the bottom of the holster (again strapped to my backpack somewhere) to keep the entire thing from moving around. the setup worked perfectly. it took me literally 3 seconds to access the camera and it was very well balanced and comfortable. i saw others on the trip who kept their cameras inside their backpack and they hardly used them since it was such a pain to access. i took almost 600 images on the 5 days and never even had to stop to get the camera out! all i ended up taking was my 18-200vr and sigma 10-20. this worked out well as we were at altitude the entire time and didn't see much wildlife. it was much more about the scenery.
I used to hike with a 1D, 2 or 3 lenses and a monopod, but the weight was killing me so I gave up and now carry a Panasonic LX3. The images really arent that much different in decent light. Of course, there isnt the same flexibility and if you want to catch birds or wildlife, the Lx3 wont be long enough, but for wider landscapes it's great.
The new Panasonic GF1 looks very interesting too...