Home · Register · Search · View Winners · Software · Hosting · Software · Join Upload & Sell

Moderated by: Fred Miranda
Username   Password

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | Alternative Gear & Lenses | Join Upload & Sell

1       2      
3
       4       5       end
  

Lossless RAW files from the Sony A7/A7R?
  
 
snapsy
Offline
• • • •
Upload & Sell: On
p.3 #1 · p.3 #1 · Lossless RAW files from the Sony A7/A7R?


Ok, I was able to find a scene that shows Fred's artifacts on the A7r but not the D800. In the process I made a very interesting discovery - the artifacts seem to only occur when when the high-compressed tonal areas (from Sony's lossless compression) are not on all 4 channels (RGGB) in the 16x2 pixel block but instead only on some of the channels. I only have a sample of two so far to confirm this (Fred's image and mine from today) but it seems pretty clear.

First, here's a comparison showing artifacts on the A7r but not the D800:






And here is a different A7r image highlighting two areas of high-compression highlighted, one with compression across all 4 channels and the other where the compression is only on certain channels, demonstrating the artifacts on the latter:







Jan 20, 2014 at 11:30 PM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.3 #2 · p.3 #2 · Lossless RAW files from the Sony A7/A7R?


That's it. You found the same artifacts. Thanks for confirming what we already suspected. They are caused by Sony's RAW compression and it's no longer speculation.

I would never see these artifacts in landscape photos but as an example, it may be an issue for those shooting high contrast B&W and recovering the shadows in post.
Fred



Jan 20, 2014 at 11:54 PM
dennishh
Offline
• • •
Upload & Sell: On
p.3 #3 · p.3 #3 · Lossless RAW files from the Sony A7/A7R?


Thanks Fred I thought I was seeing things with my files. It's definitely an issue with HDR's. How do we tell Sony about this and other problems? I guess the question is will they actually do something about it and not just deny it.


Jan 21, 2014 at 01:05 AM
snapsy
Offline
• • • •
Upload & Sell: On
p.3 #4 · p.3 #4 · Lossless RAW files from the Sony A7/A7R?


dennishh wrote:
Thanks Fred I thought I was seeing things with my files. It's definitely an issue with HDR's. How do we tell Sony about this and other problems? I guess the question is will they actually do something about it and not just deny it.


So far these shadow artifacts are the only ones which have been conclusively correlated to the compression. Many are attributing other faults to the compression, like the orange peel pattern, stair stepping of detail, and posterization. Those will have to be examined on a case by case basis to find the root cause.

Fred, do you have any other artifacts in your A7r images beside the shadow ones?



Jan 21, 2014 at 01:14 AM
akuba
Offline

Upload & Sell: Off
p.3 #5 · p.3 #5 · Lossless RAW files from the Sony A7/A7R?


snapsy wrote:
Ok, I was able to find a scene that shows Fred's artifacts on the A7r but not the D800. In the process I made a very interesting discovery - the artifacts seem to only occur when when the high-compressed tonal areas (from Sony's lossless compression) are not on all 4 channels (RGGB) in the 16x2 pixel block but instead only on some of the channels.


This is great investigation. Are you highlighting as lossy the areas that require left-shift of the 7-bit delta during decoding? How are you determining whether a specific channel didn't need compression when the other one(s) did? Are you just looking at the difference between that channel's decoded max value in the block and the block-wide 11-bit min?



Jan 21, 2014 at 01:50 AM
snapsy
Offline
• • • •
Upload & Sell: On
p.3 #6 · p.3 #6 · Lossless RAW files from the Sony A7/A7R?


akuba wrote:
This is great investigation. Are you highlighting as lossy the areas that require left-shift of the 7-bit delta during decoding? How are you determining whether a specific channel didn't need compression when the other one(s) did? Are you just looking at the difference between that channel's decoded max value in the block and the block-wide 11-bit min?

I'm using the delta of the min/max values encoded in each 16x2 block of pixels and including blocks where the max-min requires the largest shift delta to decode (ie, coarsest representation, where the code here produces a shift value of 3 (sh == 3).



Jan 21, 2014 at 06:50 PM
snapsy
Offline
• • • •
Upload & Sell: On
p.3 #7 · p.3 #7 · Lossless RAW files from the Sony A7/A7R?


I have published the utility mentioned earlier in the thread. This utility will generate a PNG showing all the high-compression lossy areas from a Sony raw file, allowing you to overlay that PNG within Photoshop against your raw to see if any perceived artifacts are related to the compression. At the present time the only artifact I have positively correlated to Sony's compression are the shadow blobs that Fred had in his raw (reproduced here). This utility will work with all modern Sony raw files, including those from the A7(r), NEX, SLT, and RX1(r) bodies.

Here are the instructions for obtaining and setting up the utility:

Step #1 - Download rawshack, my raw analysis utiliity (Windows, Mac, or Linux). For Mac/Linux you'll need to set the execution privilege after downloading by performing a "chmod +x rawshack" in a terminal window

Step #2 - Download my PHP script that will convert the output of rawshack into a PNG. You can download it by clicking here and then saving it from your browser via right-click/ctrl-click - save it as createsonypng.php, in the same directory as rawshack.

Step #3 - Install PHP if you don't already have it on your system. PHP comes standard on the Mac/OSX. For Windows users, click here and download the ZIP of the latest PHP build available (5.5.8 as of this writing). Create the directory "C:\Program Files\PHP" and unzip the PHP files into (or whatever drive you wish to put PHP on). Add this PHP directory to your system path by running Control Panel -> System -> Advanced System Settings -> Environmental Variables. Double-click "Path" in the "System variables section" and add the path to your PHP files - be sure it's separated with a semicolon from the other portions of your path.

Step #4 - Create php.ini if you don't already have one. For windows users you do this by copying the php.ini-production file to php.ini in the PHP directory. For Mac users you do "sudo cp /private/etc/php.ini.default /private/etc/php.ini" in a terminal window

Step #5 - Modify php.ini to increase the amount of memory a script is allowed to use by increasing the "memory-limit" field from 128M (default) to at least 512M. Modify php.ini to enable the PNG library by uncommenting the ";extension=php_gd2.dll" line (you uncomment it by removing the leading semicolon).

Ok, that's all the messy setup...running it is easy:

Step #1 - Execute "rawshack --sonycompdata <rawfilename>". For example, "rawshack --sonycompdata DSC0001.arw". This will create two files, the first of which is rawshack's standard raw file report (DSC0001.txt for this example), the second will be a comma-delimited file containing all the pixel positions from your raw that have heavily compressed tones (DSC0001.sonycomp.csv).

Step #2 - Execute "php createsonypng.php <CSVfilename>". For the above example this will be "php createsonypng.php DSC0001.sonycomp.csv", which will generate a PNG file named "DSC0001.sonycomp.png". You're now ready to load this PNG into Photoshop and overlay it against your original image to see which parts are highly compressed.

Options:

Sony compresses each 16x2 block of pixels in one of 5 compression levels. The level selected is based on how diverse the tonal values are for the block - the more diverse, the more tonal compression applied...and the more likelihood for IQ to be affected. Level 0 is lossless and is used when the tonal values are rather close in luminosity. Levels 1-4 are lossy, with 1 being the least compressed and 4 being the highest compressed. By default rawshack will report only Levels 3-4 in the CSV file used to generate the PNG. You can override this default by passing the specific levels you wish to see as part of the --sonycompdata option. Examples:

rawshack --sonycompdata=4 (only include pixels that are the highest compressed [Level 4])
rawshack --sonycompdata=3 (only include pixels that are the 2nd most compressed [Level 3])
rawshack --sonycompdata=2,3,4 (include pixels compressed at levels 2,3,4)
rawshack --sonycompdata=0,1,2,3,4 (include all pixel compression levels)

The PNG generated by createsonypng.php is color coded for each 16x2 pixel block based on the compression level of that block. The colors are as follows: Level 4 - White, Level 3 - Red, Level 2 = Yellow, Level 1 = Blue, Level 0 = Green. Here are some samples of DNGs generated from Fred's raw at different compression detection levels:

Level 4
Levels 3 and 4
All Levels (0, 1, 2, 3, and 4)

And here's what it looks like with the DNG overlaid with the original image in CS6, using a layer opacity of 20% on the DNG and Fred's artifacts circled: Overlay

For full documentation of rawshack, including source code, click here.



Jan 23, 2014 at 04:07 AM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.3 #8 · p.3 #8 · Lossless RAW files from the Sony A7/A7R?


snapsy wrote:
I have published the utility mentioned earlier in the thread. This utility will generate a PNG showing all the high-compression lossy areas from a Sony raw file, allowing you to overlay that PNG within Photoshop against your raw to see if any perceived artifacts are related to the compression. At the present time the only artifact I have positively correlated to Sony's compression are the shadow blobs that Fred had in his raw (reproduced here). This utility will work with all modern Sony raw files, including those from the A7(r), NEX, SLT, and RX1(r) bodies.

Here are the instructions for obtaining
...Show more

That is great stuff Adam!
I will try this with a couple images I have been editing. Thanks for the tutorial.
Fred



Jan 23, 2014 at 05:56 AM
RCicala
Offline
• • • •
Upload & Sell: On
p.3 #9 · p.3 #9 · Lossless RAW files from the Sony A7/A7R?


Fred, can this thread be made a sticky or something? You guys have done some amazing work here. Thank you!!!


Jan 23, 2014 at 12:05 PM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.3 #10 · p.3 #10 · Lossless RAW files from the Sony A7/A7R?


RCicala wrote:
Fred, can this thread be made a sticky or something? You guys have done some amazing work here. Thank you!!!


This issue does not seem to get much interest because the worse artifacts like posterization are more pronounced when using the camera's JPEG output.
As we demonstrated here, under certain light conditions, compression artifacts became visible in RAW files after pushing the shadows in post.
Fred



Jan 23, 2014 at 08:27 PM
 

Search in Used Dept. 



Nicolas_Goulet
Offline
• •
Upload & Sell: Off
p.3 #11 · p.3 #11 · Lossless RAW files from the Sony A7/A7R?


Is this something that could be fixed with a firmware update?


Jan 23, 2014 at 08:27 PM
slungu
Offline
• • •
Upload & Sell: Off
p.3 #12 · p.3 #12 · Lossless RAW files from the Sony A7/A7R?


Fred, no offence, but you would see this only if you have differences of more than can be represented by 11 bits and it would be visible on any current viewing device only if you would be pushing it, and in that case you will always get a hit sooner or later. So, while it is not nice, it should seldom be of real practical interest ( pushing 3 stops in PP, that is ).


Jan 23, 2014 at 08:54 PM
BrianVS
Offline
• •
Upload & Sell: Off
p.3 #13 · p.3 #13 · Lossless RAW files from the Sony A7/A7R?


Until Sony puts in a true RAW option, or someone unlocks it as they did on the M8- hard to know what you are missing. I suspect the main advantage is for high-ISO shots. The Sony falls short in that regard, too bad no one can do a real comparison. It's much easier to store a true RAW file than a compressed one. Are they that worried about write-Speed? That is the reason why Leica left it out of the M8. The High-ISO performance of the M8 suffered greatly because of that decision, as uncompressed Raw files show.


Jan 23, 2014 at 10:00 PM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.3 #14 · p.3 #14 · Lossless RAW files from the Sony A7/A7R?


slungu wrote:
Fred, no offence, but you would see this only if you have differences of more than can be represented by 11 bits and it would be visible on any current viewing device only if you would be pushing it, and in that case you will always get a hit sooner or later. So, while it is not nice, it should seldom be of real practical interest ( pushing 3 stops in PP, that is ).


I agree. Aside from test shots under high contrast lighting conditions, I have not encountered IQ issues even after pushing my images in post. I don't consider this to be a practical issue and I am very impressed with the A7R's IQ.

My point is that the lossy compression is visible when pushing the files under certain conditions. These artifacts are not present in uncompressed D800E NEF files demonstrating it's indeed the result of the lossy compression.
Fred



Jan 23, 2014 at 10:17 PM
naturephoto1
Offline
• • •
Upload & Sell: On
p.3 #15 · p.3 #15 · Lossless RAW files from the Sony A7/A7R?


Fred Miranda wrote:
I agree. Aside from test shots under high contrast lighting conditions, I have not encountered IQ issues even after pushing my images in post. I don't consider this to be a practical issue and I am very impressed with the A7R's IQ.

My point is that the lossy compression is visible when pushing the files under certain conditions. These artifacts are not present in uncompressed D800E NEF files demonstrating it's indeed the result of the lossy compression.
Fred


Fred,

But have any of seen these artifacts in a printed image?

Rich



Jan 23, 2014 at 10:19 PM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.3 #16 · p.3 #16 · Lossless RAW files from the Sony A7/A7R?


naturephoto1 wrote:
Fred,

But have any of seen these artifacts in a printed image?

Rich


Rich,
Here is the 100% crops showing compression artifacts. Before and After.
You may print it simulating any paper size. The artifacts shows up in Lightroom, Capture One or Sony's Image Data Converter.
The only post-processing was pushing the "Shadows" in Lightroom. (no change to exposure or any other setting)






Before with Lightroom default settings







After with the Shadows slider being increased




Jan 23, 2014 at 11:23 PM
philip_pj
Offline
• • • •
Upload & Sell: Off
p.3 #17 · p.3 #17 · Lossless RAW files from the Sony A7/A7R?


'I suspect the main advantage is for high-ISO shots. The Sony falls short in that regard, too bad no one can do a real comparison.'

Would you care to elaborate, Brian as I may have misunderstood?
If this is a reference to Sony's high ISO performance generally in the a7r, we must disagree as it rather stridently appears to 'fall long'.

DxO see it this way:

http://www.dxomark.com/Cameras/Compare/Side-by-side/Sony-A7R-versus-Canon-EOS-6D-versus-Nikon-D800E___917_836_814

So it goes very well indeed up to ISo 25600 against the two best high Mp DSLRs from C/N, I certainly need no better, 8 bits or 11 bits or the full monty 14 bits. Maybe they saw it that way too?





Jan 24, 2014 at 12:56 AM
douglasf13
Offline
• • • • •
Upload & Sell: Off
p.3 #18 · p.3 #18 · Lossless RAW files from the Sony A7/A7R?


The good news is that pushed shadows on my RX1 still look better than they did with my A900's uncompressed raws, and they're worlds better than a Canon sensor being pushed.

This is probably more academic than anything, but interesting findings, so far.



Jan 24, 2014 at 01:09 AM
Paul Gardner
Offline
• • •
Upload & Sell: On
p.3 #19 · p.3 #19 · Lossless RAW files from the Sony A7/A7R?


On principle while it may not matter today, but as raw processing improves over the years I do not want ANY degradation of my files. Storage is cheap these days and getting cheaper by the month. 1TB SATAIII drives are under $100.00USD so what is the problem. Personally I don't care if the files are 500MB. If it is internal write speed use the A7 and leave the A7r for those of us whom shoot at maybe 1 frame per minute and wish for max IQ and preferably 16bits. I was just on my way to purchase an A7r when I came across this discussion. So now I have to rethink my purchase. I also have never seen the problem discussed in my D800E.


Jan 24, 2014 at 02:30 AM
Tariq Gibran
Offline
• • • • • •
Upload & Sell: On
p.3 #20 · p.3 #20 · Lossless RAW files from the Sony A7/A7R?


Beyond write speed, I wonder if writing larger raw files more often would increase heat issues, which might be a much bigger issue with a compact full frame camera. Then again, Sony is using some computational processing power to compress the raws so is that producing more heat or less than simply writing the raw file? I don't know.


Jan 24, 2014 at 03:00 AM
1       2      
3
       4       5       end




FM Forums | Alternative Gear & Lenses | Join Upload & Sell

1       2      
3
       4       5       end
    
 

You are not logged in. Login or Register

Username   Password    Retrive password