p.4 #1 · Sony A9III CFE-a Memory Card Performance Data
docusync wrote:
I never thought about this from a "general purpose" computer perspective, but you're absolutely right, it must be a seriously powerful server to process 24Mpx 14bit images at 120 fps, especially considering the pipeline mentioned in RawTherapee.
Yep. The imaging ASICs have dedicated logic to accelerate the operations but it's still quite intensive and many can't be done in parallel, which means many round trips of the data over the internal bus and SDRAM.
p.4 #2 · Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
There is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:
Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate and you'll get a sense for how computationally and data intensive the specialized imaging ASICs in cameras are, and why they are often the bottleneck in buffer clearing performance....Show more →
And also why "they can just add that in firmware" is a very naive answer to why a camera doesn't feature x that someone wants. There's so many dependencies in how the imaging pipeline has been developed, programmed and how the ASIC has been designed to compensate/enhance that.
p.4 #3 · Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
Can you measure the performance of the Angelbird 1TB card and post for comparison the same curves as are posted in this thread for the Lexar, Delkin and Sony cards. Those cards showed no difference in performance, even though they differ in their specs. As noted, that is even true for the Lexar Gold and Silver, and the Lexar Gold lists minimum write speed spec that is better than the Angelbird. Also, it has been rumored that the Angelbird was withdrawn because it failed to qualify VPG-200 certification. I don't have much confidence in any of the specs which is why I prefer to make measurements. A set of actual measurements for the Angelbird would be very interesting. I tried to get a card to test but could not buy, rent, or borrow one. ...Show more →
Can you give me a link on how to extract that data? I'll give it a shot. But one thing about the Lexar cards--I don't see anywhere where they state what the sustained minimum write speed is or how they measure that. To be rated VPG400 all they have to meet is a 400MB/s sustained write, so without precise data it's hard to know how much better a card really is.
p.4 #4 · Sony A9III CFE-a Memory Card Performance Data
jhapeman wrote:
Can you give me a link on how to extract that data? I'll give it a shot. But one thing about the Lexar cards--I don't see anywhere where they state what the sustained minimum write speed is or how they measure that. To be rated VPG400 all they have to meet is a 400MB/s sustained write, so without precise data it's hard to know how much better a card really is.
I will make up a PDF describing the process and providing some details on how I collect the data and process it to make the charts. It's pretty simple. I will not be able to do that until late today or tomorrow (very occupied with Peregrines) and I will send you a PM.
Once done I can provide the PDF to anyone who may have a card they want to measure. If there is anyone else who wants a copy, send me a PM.
p.4 #5 · Sony A9III CFE-a Memory Card Performance Data
Sorry I am getting off the topic a bit. You folks are the experts. I have a quite a few 160 Sony Type A "G" cards and 160 GB ProGrade Type A Cobalt cards, they have similar speeds, read; 800 MB/S, write: Max. 700 MB/S. Min.:400 MB/S (VPG 400).
I see some of the newer cards with larger capacity have similar read speed and max. write speed, but the min. write speed is 200 MB/S (VPG 200), they are much cheaper per GB. If I don't shoot much video, and shoot still at higher frame rate in bursts, maybe 2-4 seconds, does the lower min. write speed of the newer card matter?
p.4 #6 · Sony A9III CFE-a Memory Card Performance Data
Douglas L wrote:
Sorry I am getting off the topic a bit. You folks are the experts. I have a quite a few 160 Sony Type A "G" cards and 160 GB ProGrade Type A Cobalt cards, they have similar speeds, read; 800 MB/S, write: Max. 700 MB/S. Min.:400 MB/S (VPG 400).
I see some of the newer cards with larger capacity have similar read speed and max. write speed, but the min. write speed is 200 MB/S (VPG 200), they are much cheaper per GB. If I don't shoot much video, and shoot still at higher frame rate in bursts, maybe 2-4 seconds, does the lower min. write speed of the newer card matter? ...Show more →
I think the only reliable way to answer your question is to make the measurements.
p.4 #7 · Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
There is a long image pipeline the raw sensor data has to undergo before it becomes a fully-formed image read for writing. This includes the various flavors of noise reduction (spatial filtering, impulse noise removal), WB scaling, PDAF pixel interpolation, pixel remapping interpolation, etc... and that's just the raw bayered data. Even raws then undergo the full debayered pipeline since they all included embedded jpgs. For a general idea of what this pipeline entails here's the documentation of RawTherapee's pipeline:
Think about how long it takes for a computer-based raw processor to render a single raw image and then multiply that by a camera's continuous frame rate and you'll get a sense for how computationally and data intensive the specialized imaging ASICs in cameras are, and why they are often the bottleneck in buffer clearing performance....Show more →
Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
p.4 #8 · Sony A9III CFE-a Memory Card Performance Data
Daran wrote:
Be that as it may, it seems to me that for lossy compressed images the A1 is pretty much done processing after pushing the image into the buffer. So buffer clearing speed should benefit. And like docusync suggested it would provide some level of data loss protection for weirdos like me who shoot practically everything in bursts, without requiring extra hardware nor consuming extra space.
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
p.4 #9 · Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
Can you describe what observation can be made to establish that the A1 is done processing after pushing the sensor data into the buffer? For reference the A1 reads out at 1/256, so fully processing an image in 3.9ms with all the image processing steps that requires doesn't seem plausible.
A1 readout is the time required to digitize all the sensor pixels and get the data into memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full.
We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread page 1, #1). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed.
p.4 #10 · Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
A1 readout is the time required to digitize all the sensor pixels and get the data into a memory that is in the stacked electronics, it is not the time required to move the data to the buffer which is much longer. If it could move the data to the buffer in 1/256 sec it could run at 256 fps until the buffer was full. We have pretty strong evidence that the A1 stores the frame data in the buffer after it has been compressed since the buffer capacity in greater in the lossy compressed mode than it is for uncompressed frames (see the linked thread above). That does not mean all the other stuff that needs to be done has been completed by the time the frame is in the buffer but the lossy compression has been completed. ...Show more →
Fully agree, which is why I questioned how @Daran arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
p.4 #11 · Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
Fully agree, which is why I questioned how @Daran@ arrived at his conclusion of "A1 is pretty much done processing after pushing the image into the buffer."
Which processing do you think is not done right away when shooting lossy compressed?
I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes
Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.
PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer.
p.4 #12 · Sony A9III CFE-a Memory Card Performance Data
Daran wrote:
Which processing do you think is not done right away when shooting lossy compressed?
I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes
Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.
PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer....Show more →
I don't quite follow what position you're taking on this. Do you believe all the processing is done the instant the data is deposited into a buffer from the sensor? Or are you arguing the opposite?
p.4 #13 · Sony A9III CFE-a Memory Card Performance Data
Yes, it wouldn't make sense IMHO to compress the data into the buffer, then decompress it to do some processing and compress it again onto the card...
Daran wrote:
Which processing do you think is not done right away when shooting lossy compressed?
I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes
Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.
PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer....Show more →
p.4 #14 · Sony A9III CFE-a Memory Card Performance Data
Daran wrote:
Which processing do you think is not done right away when shooting lossy compressed?
I'll note that:
- the buffer size in terms of fitting image count is specified to be smaller when shooting compressed RAW+JPEG then when just shooting compressed RAW, so the JPEG data is not simply generated from the RAW at the output writing stage
- even compressed RAW image capture requires a JPEG version (be that the embedded preview or a separate JPEG)
- most JPEG affecting processing is also required by live view
- live view processing may be done on a lower resolution, but it certainly can not be delayed by buffering
- storing uncompressed data has identical computational requirements, except the lossy encoding, which we know to happen in hardware when feeding into the buffer
- storing uncompressed data from the buffer happens at about half the speed of lossy compressed data, which is a decent fit to the relative file sizes
Hence my guess that for lossy RAWs very little happens to the data while the buffer is cleared.
PS: One opposing data point is that shooting JPEG only does not result in an effectively endless buffer....Show more →
---------------------------------------------
snapsy wrote:
I don't quite follow what position you're taking on this. Do you believe all the processing is done the instant the data is deposited into a buffer from the sensor? Or are you arguing the opposite?
---------------------------------------------
j4nu wrote:
Yes, it wouldn't make sense IMHO to compress the data into the buffer, then decompress it to do some processing and compress it again onto the card...
Daran is quite clear that he believes that most of the processing has been completed when the frame is put into the buffer.
I added some data to the thread about the A1, A1 Performance, that is meant to provide a bit more information about the processing and JPEGs. It is not clear to me whether it is useful.
One point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it does not, even though in a simple model it should. The cameras seem to change modes of operation.
Daran is quite clear that he believes that most of the processing has been completed when the frame is put into the buffer.
I added some data to the thread about the A1, A1 Performance, that is meant to provide a bit more information about the processing and JPEGs. It is not clear to me whether it is useful.
One point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it does not, even though in a simple model it should. The cameras seem to change modes of operation. ...Show more →
How can the processing be complete at the time it's read off the sensor and put into the buffer? That implies the long pipeline of image processing operations performed on the data are completed in real-time.
p.4 #16 · Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
One point that I think is ignored is that some of the curves have two break points, which would be characteristic of a simple two sequential buffer model. It is also clear that sometimes the second buffer causes a break and sometimes it does not, even though in a simple model it should. The cameras seem to change modes of operation.
When the data doesn't fit the conclusion then it's best to reconsider the conclusion.
p.4 #17 · Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
How can the processing be complete at the time it's read off the sensor and put into the buffer? That implies the long pipeline of image processing operations performed on the data are completed in real-time.
It is clear there are a series of processing steps that need to be completed at some point along the path from the sensor to the buffer(s) to the memory card. The question is what is done before the buffer(s) and what is done after the buffers. All the processing and data movement is "real time".
p.4 #18 · Sony A9III CFE-a Memory Card Performance Data
snapsy wrote:
When the data doesn't fit the conclusion then it's best to reconsider the conclusion.
Conclusion? What conclusion is in conflict with what data?
p.4 #19 · Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
It is clear there are a series of processing steps that need to be completed at some point along the path from the sensor to the buffer(s) to the memory card. The question is what is done before the buffer(s) and what is done after the buffers. All the processing and data movement is "real time".
The data is moved off the sensor and into a buffer. The image processing steps are then performed on that buffer via the imaging ASIC. When those steps are complete the buffer is written to media and then recycled to be used for another frame.
p.4 #20 · Sony A9III CFE-a Memory Card Performance Data
dclark wrote:
Conclusion? What conclusion is in conflict with what data?
You wrote the data indicates a "simple two sequential buffer model", then commented how it sometimes causes a break and sometimes does not, then commented how the model indicates it should. The data then does not fit the conclusion about the model.
