Upload & Sell: On
I have published the utility mentioned earlier in the thread. This utility will generate a PNG showing all the high-compression lossy areas from a Sony raw file, allowing you to overlay that PNG within Photoshop against your raw to see if any perceived artifacts are related to the compression. At the present time the only artifact I have positively correlated to Sony's compression are the shadow blobs that Fred had in his raw (reproduced here). This utility will work with all modern Sony raw files, including those from the A7(r), NEX, SLT, and RX1(r) bodies.
Here are the instructions for obtaining and setting up the utility:
Step #1 - Download rawshack, my raw analysis utiliity (Windows, Mac, or Linux). For Mac/Linux you'll need to set the execution privilege after downloading by performing a "chmod +x rawshack" in a terminal window
Step #2 - Download my PHP script that will convert the output of rawshack into a PNG. You can download it by clicking here and then saving it from your browser via right-click/ctrl-click - save it as createsonypng.php, in the same directory as rawshack.
Step #3 - Install PHP if you don't already have it on your system. PHP comes standard on the Mac/OSX. For Windows users, click here and download the ZIP of the latest PHP build available (5.5.8 as of this writing). Create the directory "C:\Program Files\PHP" and unzip the PHP files into (or whatever drive you wish to put PHP on). Add this PHP directory to your system path by running Control Panel -> System -> Advanced System Settings -> Environmental Variables. Double-click "Path" in the "System variables section" and add the path to your PHP files - be sure it's separated with a semicolon from the other portions of your path.
Step #4 - Create php.ini if you don't already have one. For windows users you do this by copying the php.ini-production file to php.ini in the PHP directory. For Mac users you do "sudo cp /private/etc/php.ini.default /private/etc/php.ini" in a terminal window
Step #5 - Modify php.ini to increase the amount of memory a script is allowed to use by increasing the "memory-limit" field from 128M (default) to at least 512M. Modify php.ini to enable the PNG library by uncommenting the ";extension=php_gd2.dll" line (you uncomment it by removing the leading semicolon).
Ok, that's all the messy setup...running it is easy:
Step #1 - Execute "rawshack --sonycompdata <rawfilename>". For example, "rawshack --sonycompdata DSC0001.arw". This will create two files, the first of which is rawshack's standard raw file report (DSC0001.txt for this example), the second will be a comma-delimited file containing all the pixel positions from your raw that have heavily compressed tones (DSC0001.sonycomp.csv).
Step #2 - Execute "php createsonypng.php <CSVfilename>". For the above example this will be "php createsonypng.php DSC0001.sonycomp.csv", which will generate a PNG file named "DSC0001.sonycomp.png". You're now ready to load this PNG into Photoshop and overlay it against your original image to see which parts are highly compressed.
Sony compresses each 16x2 block of pixels in one of 5 compression levels. The level selected is based on how diverse the tonal values are for the block - the more diverse, the more tonal compression applied...and the more likelihood for IQ to be affected. Level 0 is lossless and is used when the tonal values are rather close in luminosity. Levels 1-4 are lossy, with 1 being the least compressed and 4 being the highest compressed. By default rawshack will report only Levels 3-4 in the CSV file used to generate the PNG. You can override this default by passing the specific levels you wish to see as part of the --sonycompdata option. Examples:
rawshack --sonycompdata=4 (only include pixels that are the highest compressed [Level 4])
rawshack --sonycompdata=3 (only include pixels that are the 2nd most compressed [Level 3])
rawshack --sonycompdata=2,3,4 (include pixels compressed at levels 2,3,4)
rawshack --sonycompdata=0,1,2,3,4 (include all pixel compression levels)
The PNG generated by createsonypng.php is color coded for each 16x2 pixel block based on the compression level of that block. The colors are as follows: Level 4 - White, Level 3 - Red, Level 2 = Yellow, Level 1 = Blue, Level 0 = Green. Here are some samples of DNGs generated from Fred's raw at different compression detection levels:
Levels 3 and 4
All Levels (0, 1, 2, 3, and 4)
And here's what it looks like with the DNG overlaid with the original image in CS6, using a layer opacity of 20% on the DNG and Fred's artifacts circled: Overlay
For full documentation of rawshack, including source code, click here.