Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | Sony Forum | Join Upload & Sell

1
       2       end
  

Archive 2017 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures

  
 
navmannz
Offline
• • •
Upload & Sell: On
p.1 #1 · p.1 #1 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


A recent post on SonyAlphaRumours available here, reports that the latest version of the A7RII and A7SII firmware (3.3) applies some form of median filtering of the raw sensor data on all exposures longer than 3.2 seconds, with these changes then locked irrevocably into the raw file. A number of sources provided in that post point to different demonstrations of this effect, which in astrophotography circles is generally referred to as a 'star-eating algorithm'. The reported significance of this change is that it reduces substantially the luminance of small stars, leading to significant loss of night-sky detail in any exposure longer than 3.4 seconds.

As someone with an occasional interest in astrophotography, I decided that I should check out the magnitude of this, taking two sets of images, controlling as carefully as I could to eliminate any other sources of variability. I took two sets of images of part of the Milky Way with my ZM Distagon 35mm at F/2.8. I shot thirty frames for each set, with one set using exposures of 3.2 seconds (i.e., below the exposure threshold at which filtering is applied) and a second at 6 seconds (above the filtering threshold described in the Sony Forum post). I used Sony's timelapse app with an ISO of 6400, silent shutter and with a 1 second gap between each exposure. The only difference in setting between these two sets of images was the exposure time.

I processed these images in three steps as follows:

1. I initially processed the images in Lightroom applying very minimal settings, i.e., manually setting the colour temperature, setting the black slider to -80 to compensate for the light (urban-lit) sky background, and setting the exposure slider on the 6-second exposure set to -0.91 to equalise its exposure relative to that of the 3.2 second set. All other sliders were set to zero except for the saturation slider, which was set to 5.

2. I exported a 2000 by 2000 pixel subset from both sets of images, and aligned each set in DeepSkyStacker, using default star-detection settings, and saving the aligned images. Interestingly at this stage DeepSkyStacker detected around 21,000 stars per image on the 3.2 second set, but only 6100 stars per image on the 6 second set.

3. Finally, I separately imported the two sets of aligned images into Photoshop using the script 'load files into stack'. I combined all images in each set by changing the opacity of the 2nd to 28th images in the stack to 4%, but the top two images were both set to screen mode to brighten the image. I then flattened each and saved the images as single tiffs. In effect this largely emulates the process applied in Sony's smooth reflection app, where multiple images are combined by averaging to create a single image with greatly improved signal to noise ratio.

While the pair of final images superficially appear similar as in the first image below, zooming in to a higher resolution (300%) as in the second image reveals a quite different picture. IMHO the second image pair clearly demonstrates a loss of fine-scale detail occurring presumably as a consequence of this additional 'filtering' step implemented by Sony in their latest firmware for exposures longer than 3.2 seconds.

Several of us have tried to raise this issue with Sony on their community forum, but so far haven't managed to generate any great engagement beyond a generic 'yes we value your opinions'. My own view is that I would like Sony to either reverse this imposition of an arbitrary data filter step on longer-exposure data, or to provide an option to save the pure raw format data in a strict sense.

However, I would be interested in the views of others as to (i) the robustness of the comparative test procedure that I've used, (ii) the significance of the differences I've demonstrated, (iii) whether you see this as having implications in your practice of photography with an A7 series camera, which might include use of long exposures for subjects other than stars, and (iv) whether you have any suggestions on how to encourage Sony to engage positively around this question to restore access to the full sensor functionality that we enjoyed prior to this latest firmware 'update'.

Any other practical suggestions welcome...

-John





Thirty stacked and aligned images - left at 3.2 seconds/image - right at 6 seconds/image







Same image pair at 300%



Edited on Apr 29, 2017 at 02:59 AM · View previous versions



Apr 19, 2017 at 07:51 PM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.1 #2 · p.1 #2 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


Are you sure this started with firmware v3.3?

I reported this issue with the first firmware for the A7RII (From v1.0 to v1.10)
Here is the thread:
https://www.fredmiranda.com/forum/topic/1388331



Apr 28, 2017 at 09:58 PM
navmannz
Offline
• • •
Upload & Sell: On
p.1 #3 · p.1 #3 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


Hi Fred, thanks for your response. I scanned that thread, and it all seems to apply to long exposures shot under bulb. The crunch issue here is that in f/w 3.3 Sony extended their application of median filtering to all exposures of 4 seconds and longer, seriously degrading images shot at exposures of lengths most commonly used for astrophotography. I've summarised the problem with some examples in a blog post here.

However, I don't just see this as being an astro issue - it will affect anyone doing exposures of 4 seconds or longer, although most don't seem to have realised yet.

-John

ps. the ZM Distagon 35 with OptoSigma at F/2.8 absolutely rocks for astrophotography!



Apr 28, 2017 at 11:41 PM
trogdon
Offline
• • •
Upload & Sell: Off
p.1 #4 · p.1 #4 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


As an a7r owner, I appreciate the info. Looks like I'll be staying with the older firmware!


Apr 29, 2017 at 12:25 AM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.1 #5 · p.1 #5 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


navmannz wrote:
Hi Fred, thanks for your response. I scanned that thread, and it all seems to apply to long exposures shot under bulb. The crunch issue here is that in f/w 3.3 Sony extended their application of median filtering to all exposures of 4 seconds and longer, seriously degrading images shot at exposures of lengths most commonly used for astrophotography. I've summarised the problem with some examples in a blog post here.

However, I don't just see this as being an astro issue - it will affect anyone doing exposures of 4 seconds or longer, although most don't seem to have
...Show more

I didn't realize that and it's indeed bad news. I was not happy by the way they 'fixed' the hot pixel issue in bulb mode almost two years ago... They can't just smear the image hoping to hide artifacts..
It's something to keep in mind when shooting exposures longer than 3.2 seconds now.

Just to confirm: This also happens when LENR is set to "off". Correct?

Thanks for the info.



Apr 29, 2017 at 12:41 AM
Audii-Dudii
Offline
• • •
Upload & Sell: Off
p.1 #6 · p.1 #6 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


FYI, Firmware version 3.3 is applicable only to the A7RII, not the A7R, as the original post claimed. According to the Sony website, the latest firmware for the A7R is version 3.20.

Also, I photograph urban areas almost exclusively at night and for my purposes, I've found the A7R is actually a better performer than the A7RII. The files aren't quite as clean, but it resolves slightly more detail in the deepest shadow areas.



Apr 29, 2017 at 02:29 AM
navmannz
Offline
• • •
Upload & Sell: On
p.1 #7 · p.1 #7 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


Fred Miranda wrote:
I didn't realize that and it's indeed bad news. I was not happy by the way they 'fixed' the hot pixel issue in bulb mode almost two years ago... They can't just smear the image hoping to hide artifacts..
It's something to keep in mind when shooting exposures longer than 3.2 seconds now.

Just to confirm: This also happens when LENR is set to "off". Correct?

Thanks for the info.


Yes, Fred, it happens even with long exposure noise reduction turned off!

-John



Apr 29, 2017 at 02:59 AM
navmannz
Offline
• • •
Upload & Sell: On
p.1 #8 · p.1 #8 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


Audii-Dudii wrote:
FYI, Firmware version 3.3 is applicable only to the A7RII, not the A7R, as the original post claimed. According to the Sony website, the latest firmware for the A7R is version 3.20.

Also, I photograph urban areas almost exclusively at night and for my purposes, I've found the A7R is actually a better performer than the A7RII. The files aren't quite as clean, but it resolves slightly more detail in the deepest shadow areas.



Thanks for that pointer - I've corrected it in the op.



Apr 29, 2017 at 03:00 AM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.1 #9 · p.1 #9 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


Audii-Dudii wrote:
FYI, Firmware version 3.3 is applicable only to the A7RII, not the A7R, as the original post claimed. According to the Sony website, the latest firmware for the A7R is version 3.20.

Also, I photograph urban areas almost exclusively at night and for my purposes, I've found the A7R is actually a better performer than the A7RII. The files aren't quite as clean, but it resolves slightly more detail in the deepest shadow areas.


Even before this debacle, as soon as I purchased my A7RII, I tested it against the A7R for astro and the A7R was way better with less chroma and luma noise. Even for regular landscapes, I still believe the A7R has slightly better DR in comparison.



Apr 29, 2017 at 10:56 AM
snapsy
Offline
• • • • • •
Upload & Sell: On
p.1 #10 · p.1 #10 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


As a casual astro shooter I never found the A7-series cameras well suited - too much thermal noise if you don't like to use LENR/black frames/stacking.


Apr 29, 2017 at 11:10 AM
navmannz
Offline
• • •
Upload & Sell: On
p.1 #11 · p.1 #11 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


Fred Miranda wrote:
Are you sure this started with firmware v3.3?

I reported this issue with the first firmware for the A7RII (From v1.0 to v1.10)
Here is the thread:
https://www.fredmiranda.com/forum/topic/1388331


Fred, I was thinking more about this thread that you pointed me to, and thought it would be worth applying a similar test to that which you used there. Below are a pair of images shot at iso 6400 with the lens capped and wrapped in a thick, black lens bag to block all light. The left is at 3.2 seconds, and the right at 4 seconds, i.e., the exposure length identified by Jim Kasson and others at which Sony's median filtering kicks in as described here. All sliders in Lightroom were set to 0, except for exposure, which was set at +5 (left) and +4.6 (right) to produce equivalent images.

To me the differences in noise patterns are nowhere near as striking as in your much longer exposures, and I expected that, but the right hand image certainly shows a different structure than the more random noise of the left-hand image. I'm presuming that this reflects the spatial filtering effects identified by Jim.

Any suggestions on how to approach Sony to get them to give us a genuine raw, and not one that is massaged in camera before being written to the card?

-John







Apr 29, 2017 at 05:57 PM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.1 #12 · p.1 #12 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


I just did the same test and can confirm there is indeed some filtering from 3.2 to 4 seconds.
I wish Sony had used a different approach as there are ways to remove hot pixels without smearing the image...Better yet, leave it alone and let photographers edit their own RAW files.

Here is a screenshot. ISO 10,000 (LEFT 3.2s , RIGHT 4s)
Processed in lightroom with zero sharpness and exposure pushed 5EV.




ISO 10,000 (LEFT 3.2s , RIGHT 4s) pushed 5EV




Apr 29, 2017 at 08:22 PM
navmannz
Offline
• • •
Upload & Sell: On
p.1 #13 · p.1 #13 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


I was interested to see how Sony's spatial filtering would play out with long-exposures with a typical daylight landscape subject, shooting the view from my front lawn. I took these in raw with a 10-stop ND filter in front of my Loxia 85 set at F/5.6 with ISO at 100, and shooting one at 3.2 seconds, and the second at 4 seconds. There is a very slight loss of fine detail, e.g., along the back edge of the red-brown roofing panels, but what really gets taken out by the filtering is the reflection highlights of the sun on the foliage in the background - hardly surprising when you think about.

-John





Daylight image without (left) and with (right) Sony's spatial filtering




May 02, 2017 at 08:40 PM
nampramos
Offline
• • •
Upload & Sell: Off
p.1 #14 · p.1 #14 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


Another case of Sony not giving us the menu option to turn something ON/OFF. They really like doing that.


May 03, 2017 at 12:42 AM
navmannz
Offline
• • •
Upload & Sell: On
p.1 #15 · p.1 #15 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


nampramos wrote:
Another case of Sony not giving us the menu option to turn something ON/OFF. They really like doing that.


And the surprising thing is that they still call it 'raw', when clearly its significantly manipulated between coming off the sensor and being written to the card!

-John




May 03, 2017 at 12:45 AM
virtualrain
Offline
• • •
Upload & Sell: Off
p.1 #16 · p.1 #16 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


navmannz wrote:
And the surprising thing is that they still call it 'raw', when clearly its significantly manipulated between coming off the sensor and being written to the card!

-John



The RAW should remain untouched. They can do whatever they want with the JPEG but this is bullshit.



May 03, 2017 at 01:42 AM
johnahill
Offline
• • • •
Upload & Sell: On
p.1 #17 · p.1 #17 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


damn it, just upgraded to the latest fw to get the latest remote shooting app (wouldn't load on my old fw)




May 03, 2017 at 04:23 AM
johnahill
Offline
• • • •
Upload & Sell: On
p.1 #18 · p.1 #18 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


is it possible to downgrade the firmware to a previous release?



May 03, 2017 at 04:24 AM
navmannz
Offline
• • •
Upload & Sell: On
p.1 #19 · p.1 #19 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


johnahill wrote:
is it possible to downgrade the firmware to a previous release?


Not at present - a bit of a breach of trust from where I sit, given the lack of specific detail in the firmware docs, and the clear indication that it cannot be reversed. Some of us have been interacting over the last few days with Sony on it, and there is possibly an indication that they will make available a backwards regression to 3.2 for those wanting to restore true raws. However, it wouldn't hurt to ring your local support number and lodge a complaint - the more constructive voices speaking to them, the more chance we have of getting something done.

-John



May 03, 2017 at 04:49 AM
artur5
Offline
• • •
Upload & Sell: Off
p.1 #20 · p.1 #20 · Latest A7R firmware (3.3) no longer delivering 'raw' raw on long exposures


Sometimes I have the impression that clever undercover agents from Canikon have infiltrated Sony’s high ranked engineering staff. They seem to be successfully undermining the confidence of the more knowledgeable Sony customers with these absurd decisions. Instead of crippling the camera capabilities with any further iteration of the firmware why not offer, for instance, lossless compressed Raws? Or leave the user decide, in a menu option, if the lens has to stop down or not for acquiring focus ?.
I wrote the Canikon sabotage stuff as a lame joke but, on second thoughts, maybe it’s not ..



May 03, 2017 at 07:15 AM
1
       2       end




FM Forums | Sony Forum | Join Upload & Sell

1
       2       end
    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.