The adapter is also my number one guess at the culprit... second, each lens is differently assembled, and infinity can vary a little due to this. I have 3 copies of the 55/3.5, two pre-ai and one Ai. All are very slightly different, however they are all close enough to curate that I can't be bothered with it even on the D850. It's been a while to remember whether the infinity can be adjusted but I'm sure to some small degree it can be adjusted with a little disassembly.
The 55/3.5 is like my Zeiss Makro-Planar 100/2. The focus throw is too short for precise normal-distance focusing, so the tiniest of focus ring movement can change focus by quite a bit. And that these lenses are so ridiculously high resolving, this can really translate badly with cameras like a D850.
Also, some camera generations are different. Focus point on all D850 cameras I've tried is slightly different to my D810 and other cameras of that era. The flange distance behaves as if it's a tiny amount shorter, which changes the infinity focus to focus slightly behind infinity at the hard stop, which happens to be spot on with my D810.
The Z6ii with FTZ is even more so. However, not quite as much as what you're seeing.