fredmiranda.com Login
This topic is in the 2024 archive. You can read it and like its posts here. It is read only.

Archive 2024Sony A9III CFE-a Memory Card Performance Data

  
 
snapsy
Offline
Details
p.4 #1Sony A9III CFE-a Memory Card Performance Data
docusync wrote:
snapsy wroteThink about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate
I never thought about this from a "general purpose" computer perspective, but you're absolutely right, it must be a seriously powerful server to process 24Mpx 14bit images at 120 fps, especially considering the pipeline mentioned in RawTherapee.


Yep. The imaging ASICs have dedicated logic to accelerate the operations but it's still quite intensive and many can't be done in parallel, which means many round trips of the data over the internal bus and SDRAM.
May 14, 2024 at 04:51 AM
jhapeman
Offline
Details
p.4 #2Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
docusync wrote
dclark wrote
docusync wroteI'm wondering why no camera manufacturer offers RAID0-like card access? Of course not on the sector level but on the file level. I.e. odd files go on the card 1 and even files go on the card 2. Even if you lose a card - you'd still have half of your files. That would reduce the image dump time in half.
That would help image write time only if it is limited by card bandwidth, not camera electronic processing. It appears to be limited by camera electronics ability to process files not the card's bandwidth..
I'm not sure, but I thought the files are already processed (in the manufacturer-specific raw format) when they get into the buffer, no? If we're dealing with a slow system bus, then switching to the CFeA 4.0 cards won't make any difference
There is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:

https://rawpedia.rawtherapee.com/Toolchain_Pipeline

Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that
...Show more →

And also why "they can just add that in firmware" is a very naive answer to why a camera doesn't feature x that someone wants. There's so many dependencies in how the imaging pipeline has been developed, programmed and how the ASIC has been designed to compensate/enhance that.
May 14, 2024 at 06:01 AM
jhapeman
Offline
Details
p.4 #3Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
jhapeman wrote
UKBob wroteShuttermuse has an interesting article about the best card for an A9iii and he recommends one with the highest sustained write speed ie the ProDigital Gold Type A card with a sustained write speed of 600mb/secBest memory cards for the A9iii -I was wrong
Can I ask what capacity cards would be recommended for an exciting day of wildlife action and how do you configure slots 1 and 2
For the A1 I use 160gb Sony card in Slot 1 with compressed RAW and 80gb in slot 2 for JPEG as back up which gives about 2200 images for each slot However I sort through the images each day and have never run out of space on the 160gb card with my shooting style which is short bursts
This is why I have mentioned the Angelbird 1TB cards several times now when this topic comes up. They have a sustained write speed of 650MB/s. They have been discontinued but AngelBird did reply to my email request and indicated that a new version is in the works. I'd guess it will be PCIe 4.0 like the other new cards and hopefully have an even faster sustained write speed.
Can you measure the performance of the Angelbird 1TB card and post for comparison the same curves as are posted in this thread for the Lexar, Delkin and Sony cards. Those cards showed no difference in performance, even though they differ in their specs. As noted, that is even true for the Lexar Gold and Silver, and the Lexar Gold lists minimum write speed spec that is better than the Angelbird. Also, it has been rumored that the Angelbird was withdrawn because it failed to qualify VPG-200 certification. I don't have much confidence in any of the specs which is
...Show more →

Can you give me a link on how to extract that data? I'll give it a shot. But one thing about the Lexar cards--I don't see anywhere where they state what the sustained minimum write speed is or how they measure that. To be rated VPG400 all they have to meet is a 400MB/s sustained write, so without precise data it's hard to know how much better a card really is.
May 14, 2024 at 07:14 AM
dclark
Online
Details
p.4 #4Sony A9III CFE-a Memory Card Performance Data
jhapeman wrote:
dclark wrote
jhapeman wrote
UKBob wroteShuttermuse has an interesting article about the best card for an A9iii and he recommends one with the highest sustained write speed ie the ProDigital Gold Type A card with a sustained write speed of 600mb/secBest memory cards for the A9iii -I was wrong
Can I ask what capacity cards would be recommended for an exciting day of wildlife action and how do you configure slots 1 and 2
For the A1 I use 160gb Sony card in Slot 1 with compressed RAW and 80gb in slot 2 for JPEG as back up which gives about 2200 images for each slot However I sort through the images each day and have never run out of space on the 160gb card with my shooting style which is short bursts
This is why I have mentioned the Angelbird 1TB cards several times now when this topic comes up. They have a sustained write speed of 650MB/s. They have been discontinued but AngelBird did reply to my email request and indicated that a new version is in the works. I'd guess it will be PCIe 4.0 like the other new cards and hopefully have an even faster sustained write speed.
Can you measure the performance of the Angelbird 1TB card and post for comparison the same curves as are posted in this thread for the Lexar, Delkin and Sony cards. Those cards showed no difference in performance, even though they differ in their specs. As noted, that is even true for the Lexar Gold and Silver, and the Lexar Gold lists minimum write speed spec that is better than the Angelbird. Also, it has been rumored that the Angelbird was withdrawn because it failed to qualify VPG-200 certification. I don't have much confidence in any of the specs which is why I prefer to make measurements. A set of actual measurements for the Angelbird would be very interesting. I tried to get a card to test but could not buy, rent, or borrow one.
Can you give me a link on how to extract that data? I'll give it a shot. But one thing about the Lexar cards--I don't see anywhere where they state what the sustained minimum write speed is or how they measure that. To be rated VPG400 all they have to meet is a 400MB/s sustained write, so without precise data it's hard to know how much better a card really is.


I will make up a PDF describing the process and providing some details on how I collect the data and process it to make the charts. It's pretty simple. I will not be able to do that until late today or tomorrow (very occupied with Peregrines) and I will send you a PM.
Once done I can provide the PDF to anyone who may have a card they want to measure. If there is anyone else who wants a copy, send me a PM.

May 14, 2024 at 07:47 AM
Douglas L
Offline
Details
p.4 #5Sony A9III CFE-a Memory Card Performance Data
Sorry I am getting off the topic a bit. You folks are the experts. I have a quite a few 160 Sony Type A "G" cards and 160 GB ProGrade Type A Cobalt cards, they have similar speeds, read; 800 MB/S, write: Max. 700 MB/S. Min.:400 MB/S (VPG 400).

I see some of the newer cards with larger capacity have similar read speed and max. write speed, but the min. write speed is 200 MB/S (VPG 200), they are much cheaper per GB. If I don't shoot much video, and shoot still at higher frame rate in bursts, maybe 2-4 seconds, does the lower min. write speed of the newer card matter?
May 16, 2024 at 08:59 AM
dclark
Online
Details
p.4 #6Sony A9III CFE-a Memory Card Performance Data
Douglas L wrote:
Sorry I am getting off the topic a bit. You folks are the experts. I have a quite a few 160 Sony Type A "G" cards and 160 GB ProGrade Type A Cobalt cards, they have similar speeds, read; 800 MB/S, write: Max. 700 MB/S. Min.:400 MB/S (VPG 400).

I see some of the newer cards with larger capacity have similar read speed and max. write speed, but the min. write speed is 200 MB/S (VPG 200), they are much cheaper per GB. If I don't shoot much video, and shoot still at higher frame rate in bursts, maybe
...Show more →

I think the only reliable way to answer your question is to make the measurements.
May 16, 2024 at 09:24 AM
Daran
Offline
Details
p.4 #7Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
docusync wrote
dclark wrote
docusync wroteI'm wondering why no camera manufacturer offers RAID0-like card access? Of course not on the sector level but on the file level. I.e. odd files go on the card 1 and even files go on the card 2. Even if you lose a card - you'd still have half of your files. That would reduce the image dump time in half.
That would help image write time only if it is limited by card bandwidth, not camera electronic processing. It appears to be limited by camera electronics ability to process files not the card's bandwidth..
I'm not sure, but I thought the files are already processed (in the manufacturer-specific raw format) when they get into the buffer, no? If we're dealing with a slow system bus, then switching to the CFeA 4.0 cards won't make any difference
There is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:

https://rawpedia.rawtherapee.com/Toolchain_Pipeline

Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that
...Show more →

Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
May 16, 2024 at 11:45 AM
snapsy
Offline
Details
p.4 #8Sony A9III CFE-a Memory Card Performance Data
Daran wrote:
snapsy wrote
docusync wrote
dclark wrote
docusync wroteI'm wondering why no camera manufacturer offers RAID0-like card access? Of course not on the sector level but on the file level. I.e. odd files go on the card 1 and even files go on the card 2. Even if you lose a card - you'd still have half of your files. That would reduce the image dump time in half.
That would help image write time only if it is limited by card bandwidth, not camera electronic processing. It appears to be limited by camera electronics ability to process files not the card's bandwidth..
I'm not sure, but I thought the files are already processed (in the manufacturer-specific raw format) when they get into the buffer, no? If we're dealing with a slow system bus, then switching to the CFeA 4.0 cards won't make any difference
There is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:

https://rawpedia.rawtherapee.com/Toolchain_Pipeline

Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate and you'll get a sense for how computationally and data intensive the specialized imaging ASICs in cameras are, and why they are often the bottleneck in buffer clearing performance.
Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.


Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
May 16, 2024 at 12:10 PM
dclark
Online
Details
p.4 #9Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
Daran wrote
snapsy wrote
docusync wrote
dclark wrote
docusync wroteI'm wondering why no camera manufacturer offers RAID0-like card access? Of course not on the sector level but on the file level. I.e. odd files go on the card 1 and even files go on the card 2. Even if you lose a card - you'd still have half of your files. That would reduce the image dump time in half.
That would help image write time only if it is limited by card bandwidth, not camera electronic processing. It appears to be limited by camera electronics ability to process files not the card's bandwidth..
I'm not sure, but I thought the files are already processed (in the manufacturer-specific raw format) when they get into the buffer, no? If we're dealing with a slow system bus, then switching to the CFeA 4.0 cards won't make any difference
There is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:

https://rawpedia.rawtherapee.com/Toolchain_Pipeline

Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate and you'll get a sense for how computationally and data intensive the specialized imaging ASICs in cameras are, and why they are often the bottleneck in buffer clearing performance.
Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.


A1 readout is the time required to digitize all the sensor pixels and get the data into memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full.

We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread page 1, #1). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.


Edited on May 16, 2024 at 01:01 PM · View previous versions
May 16, 2024 at 12:56 PM
snapsy
Offline
Details
p.4 #10Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
snapsy wrote
Daran wrote
snapsy wrote
docusync wrote
dclark wroteThat would help image write time only if it is limited by card bandwidth, not camera electronic processing. It appears to be limited by camera electronics ability to process files not the card's bandwidth..
I'm not sure, but I thought the files are already processed (in the manufacturer-specific raw format) when they get into the buffer, no? If we're dealing with a slow system bus, then switching to the CFeA 4.0 cards won't make any difference
There is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:

https://rawpedia.rawtherapee.com/Toolchain_Pipeline

Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate and you'll get a sense for how computationally and data intensive the specialized imaging ASICs in cameras are, and why they are often the bottleneck in buffer clearing performance.
Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for
...Show more →

Fully agree, which is why I questioned how @Daran arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
May 16, 2024 at 12:59 PM
Daran
Offline
Details
p.4 #11Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
dclark wrote
snapsy wrote
Daran wrote
snapsy wrote
docusync wroteI'm not sure, but I thought the files are already processed (in the manufacturer-specific raw format) when they get into the buffer, no? If we're dealing with a slow system bus, then switching to the CFeA 4.0 cards won't make any difference
There is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:

https://rawpedia.rawtherapee.com/Toolchain_Pipeline

Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate and you'll get a sense for how computationally and data intensive the specialized imaging ASICs in cameras are, and why they are often the bottleneck in buffer clearing performance.
Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."

Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
May 17, 2024 at 06:58 AM
snapsy
Offline
Details
p.4 #12Sony A9III CFE-a Memory Card Performance Data
Daran wrote:
snapsy wrote
dclark wrote
snapsy wrote
Daran wrote
snapsy wroteThere is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:

https://rawpedia.rawtherapee.com/Toolchain_Pipeline

Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate and you'll get a sense for how computationally and data intensive the specialized imaging ASICs in cameras are, and why they are often the bottleneck in buffer clearing performance.
Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be
...Show more →

I don't quite follow what position you're taking on this. Do you believe all the processing is done the instant the data is deposited into a buffer from the sensor? Or are you arguing the opposite?
May 17, 2024 at 08:35 AM
j4nu
Offline
Details
p.4 #13Sony A9III CFE-a Memory Card Performance Data
Yes, it wouldn't make sense IMHO to compress the data into the buffer, then decompress it to do some processing and compress it again onto the card...

Daran wrote:
snapsy wrote
dclark wrote
snapsy wrote
Daran wrote
snapsy wroteThere is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:

https://rawpedia.rawtherapee.com/Toolchain_Pipeline

Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate and you'll get a sense for how computationally and data intensive the specialized imaging ASICs in cameras are, and why they are often the bottleneck in buffer clearing performance.
Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be
...Show more →
May 17, 2024 at 08:54 AM
dclark
Online
Details
p.4 #14Sony A9III CFE-a Memory Card Performance Data
Daran wrote:
snapsy wrote
dclark wrote
snapsy wrote
Daran wrote
snapsy wroteThere is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:

https://rawpedia.rawtherapee.com/Toolchain_Pipeline

Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate and you'll get a sense for how computationally and data intensive the specialized imaging ASICs in cameras are, and why they are often the bottleneck in buffer clearing performance.
Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be
...Show more →

---------------------------------------------

snapsy wrote:
Daran wrote
snapsy wrote
dclark wrote
snapsy wrote
Daran wroteBe that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
I don't quite follow what position you're taking on this. Do you believe all the processing is done the instant the data is deposited into a buffer from the sensor? Or are you arguing the opposite?


---------------------------------------------

j4nu wrote:
Yes, it wouldn't make sense IMHO to compress the data into the buffer, then decompress it to do some processing and compress it again onto the card...

Daran wrote
snapsy wrote
dclark wrote
snapsy wrote
Daran wroteBe that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.


Daran is quite clear that he believes that most of the processing has been completed when the frame is put into the buffer.

I added some data to the thread about the A1, A1 Performance, that is meant to provide a bit more information about the processing and JPEGs. It is not clear to me whether it is useful.

One point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it does not, even though in a simple model it should. The cameras seem to change modes of operation.
May 17, 2024 at 10:29 AM
snapsy
Offline
Details
p.4 #15Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
Daran wrote
snapsy wrote
dclark wrote
snapsy wrote
Daran wroteBe that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
---------------------------------------------

snapsy wrote
Daran wrote
snapsy wrote
dclark wrote
snapsy wroteCan you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
I don't quite follow what position you're taking on this. Do you believe all the processing is done the instant the data is deposited into a buffer from the sensor? Or are you arguing the opposite?
---------------------------------------------

j4nu wroteYes, it wouldn't make sense IMHO to compress the data into the buffer, then decompress it to do some processing and compress it again onto the card...

Daran wrote
snapsy wrote
dclark wrote
snapsy wroteCan you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
Daran is quite clear that he believes that most of the processing has been completed when the frame is put into the buffer.

I added some data to the thread about the A1, A1 Performance, that is meant to provide a bit more information about the processing and JPEGs. It is not clear to me whether it is useful.

One point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it
...Show more →

How can the processing be complete at the time it's read off the sensor and put into the buffer? That implies the long pipeline of image processing operations performed on the data are completed in real-time.
May 17, 2024 at 10:34 AM
snapsy
Offline
Details
p.4 #16Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
One point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it does not, even though in a simple model it should. The cameras seem to change modes of operation.


When the data doesn't fit the conclusion then it's best to reconsider the conclusion.
May 17, 2024 at 10:39 AM
dclark
Online
Details
p.4 #17Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
dclark wrote
Daran wrote
snapsy wrote
dclark wrote
snapsy wroteCan you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
---------------------------------------------

snapsy wrote
Daran wrote
snapsy wrote
dclark wroteA1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
I don't quite follow what position you're taking on this. Do you believe all the processing is done the instant the data is deposited into a buffer from the sensor? Or are you arguing the opposite?
---------------------------------------------

j4nu wroteYes, it wouldn't make sense IMHO to compress the data into the buffer, then decompress it to do some processing and compress it again onto the card...

Daran wrote
snapsy wrote
dclark wroteA1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
Daran is quite clear that he believes that most of the processing has been completed when the frame is put into the buffer.

I added some data to the thread about the A1, A1 Performance, that is meant to provide a bit more information about the processing and JPEGs. It is not clear to me whether it is useful.

One point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it does not, even though in a simple model it should. The cameras seem to change modes of operation.
How can the processing be complete at the time it's read off the sensor and put into the buffer? That implies the long pipeline of image processing operations performed on the data are completed in real-time.

It is clear there are a series of processing steps that need to be completed at some point along the path from the sensor to the buffer(s) to the memory card. The question is what is done before the buffer(s) and what is done after the buffers. All the processing and data movement is "real time".
May 17, 2024 at 10:43 AM
dclark
Online
Details
p.4 #18Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
dclark wroteOne point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it does not, even though in a simple model it should. The cameras seem to change modes of operation.
When the data doesn't fit the conclusion then it's best to reconsider the conclusion.

Conclusion? What conclusion is in conflict with what data?
May 17, 2024 at 10:44 AM
snapsy
Offline
Details
p.4 #19Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
snapsy wrote
dclark wrote
Daran wrote
snapsy wrote
dclark wroteA1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
Fully agree, which is why I questioned how @Daran@@@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
---------------------------------------------

snapsy wrote
Daran wrote
snapsy wroteFully agree, which is why I questioned how @Daran@@@@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
I don't quite follow what position you're taking on this. Do you believe all the processing is done the instant the data is deposited into a buffer from the sensor? Or are you arguing the opposite?
---------------------------------------------

j4nu wroteYes, it wouldn't make sense IMHO to compress the data into the buffer, then decompress it to do some processing and compress it again onto the card...

Daran wrote
snapsy wroteFully agree, which is why I questioned how @Daran@@@@@@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?

I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes

Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.

PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
Daran is quite clear that he believes that most of the processing has been completed when the frame is put into the buffer.

I added some data to the thread about the A1, A1 Performance, that is meant to provide a bit more information about the processing and JPEGs. It is not clear to me whether it is useful.

One point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it does not, even though in a simple model it should. The cameras seem to change modes of operation.
How can the processing be complete at the time it's read off the sensor and put into the buffer? That implies the long pipeline of image processing operations performed on the data are completed in real-time.
It is clear there are a series of processing steps that need to be completed at some point along the path from the sensor to the buffer(s) to the memory card. The question is what is done before the buffer(s) and what is done after the buffers. All the processing and data movement is "real time".

The data is moved off the sensor and into a buffer. The image processing steps are then performed on that buffer via the imaging ASIC. When those steps are complete the buffer is written to media and then recycled to be used for another frame.
May 17, 2024 at 10:46 AM
snapsy
Offline
Details
p.4 #20Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
snapsy wrote
dclark wroteOne point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it does not, even though in a simple model it should. The cameras seem to change modes of operation.
When the data doesn't fit the conclusion then it's best to reconsider the conclusion.
Conclusion? What conclusion is in conflict with what data?


You wrote the data indicates a "simple two sequential buffer model", then commented how it sometimes causes a break and sometimes does not, then commented how the model indicates it should. The data then does not fit the conclusion about the model.
May 17, 2024 at 10:48 AM
FM ForumsSony ForumJoin Upload & Sell
  
 

You are not logged inLog inRegister