Philip Peter wrote:.
Hi,i'm currently poking around with the jrawio raw support (http://jrawio.tidalwave.it/ ).And from what I have seen so far, I suspect that some Raw Processordo some kind of tone mapping by default..
I don't know about Canons Zoombrowser, but Lightroom does a lot of things by default, including bending the tone curve. There is an option to zero everything and it makes for an ugly picture..
I have only used Canons ZoomBrowser and Adobe Lightroom / ACR.But from some examples I looked at, I can't seem to get the samedynamic range by just selecting a 256 sample range from the raw file..
Huh?.
The examples from my Canon 400d usually have around 12 Bits ofinformation and if I set the upper and lower bound so that 3% of thehighlights are clipped and 3% of the shadows are dropped, I usuallystill get a 10 Bit range, which I then have to compress into 8 Bitfor further processing..
Huh?.
The processed images from Lightroom / ZoomBrowser have a shadow andhighlight detail which in my eyes is just not possible with just a 8Bit range..
That's because they operate in a 16bit environment..
Can anyone enlighten me, how theses Raw processors work?Or is there something in canon raw files that i'm missing?.
Bye,Philip.
I'm probably just being dense.But why would you want to compress to an 8 bit range?.
Typical raw workflow. Import into Lightroom and develop. Lightroom keeps the entire process in a 16bit environment (this avoids roundoff errors even though the data is typically 12bit, some cameras now 14bit)If necessary, I export a 16bit tiff to the photoeditor. My editor of choice also does all edits in a 16bit environment. The only time I compress to 8bit is when I make a jpeg for the web..
I Probably don't actually understand your question.A member of the rabble in good standing...
LM1 wrote:.
Philip Peter wrote:.
Hi,i'm currently poking around with the jrawio raw support (http://jrawio.tidalwave.it/ ).And from what I have seen so far, I suspect that some Raw Processordo some kind of tone mapping by default..
I don't know about Canons Zoombrowser, but Lightroom does a lot ofthings by default, including bending the tone curve. There is anoption to zero everything and it makes for an ugly picture..
Ok, that seems to be my problem. Currently i'm doing a linear conversion, which gets a very flat result with lots of lost information..
I have only used Canons ZoomBrowser and Adobe Lightroom / ACR.But from some examples I looked at, I can't seem to get the samedynamic range by just selecting a 256 sample range from the raw file..
Huh?.
I thought that the information the sensor captures would already be "spaced" correctly, in other words, that you could select an 8 Bit range from the 14 Bit data and get a usable result. For example just use the range from 800 - 1056, everything below becomes 0 and everything above becomes 255. Than you can adjust the result by sliding the range back and forth.I'm now pretty much convinced that this isn't the case..
The examples from my Canon 400d usually have around 12 Bits ofinformation and if I set the upper and lower bound so that 3% of thehighlights are clipped and 3% of the shadows are dropped, I usuallystill get a 10 Bit range, which I then have to compress into 8 Bitfor further processing..
Huh?.
When I analyze the raw data I get significant pixel values ranging from ~300 up to 4000..
To achieve a reasonable exposure I set one value as the absolute black and one as white, this values are selected so that less than 3% of the total pixels fall below / above the respective value. As a nice bonus this produces a pretty good white balance as well.The data inside this range is then compressed to 8 Bit withf(x) = x * (256 / (max - min)) - min.
The processed images from Lightroom / ZoomBrowser have a shadow andhighlight detail which in my eyes is just not possible with just a 8Bit range..
That's because they operate in a 16bit environment..
Yes, but the output they produce is 8 Bit..
As far as I understand, they do a non linear conversion to 8 Bit, which preserves the details..
Do you know if they just use a non linear conversion? This would mean that to achieve the additional detail in the highlight and shadow areas, some detail in the midrange would get lost..
Can anyone enlighten me, how theses Raw processors work?Or is there something in canon raw files that i'm missing?.
Bye,Philip.
I'm probably just being dense.But why would you want to compress to an 8 bit range?.
Typical raw workflow. Import into Lightroom and develop. Lightroomkeeps the entire process in a 16bit environment (this avoids roundofferrors even though the data is typically 12bit, some cameras now14bit)If necessary, I export a 16bit tiff to the photoeditor. Myeditor of choice also does all edits in a 16bit environment. The onlytime I compress to 8bit is when I make a jpeg for the web..
I Probably don't actually understand your question..
Basically i'm trying to code something similiar to Lightroom. (With less features of course).
My workflow so far is:- Import the raw data as 16 Bit..
- Do Noise Reduction on the original raw data, using the unused pixels at the edge of the sensor for reference on noise levels (still 16 Bit).
- Adjust white balance, blacklevel, absolute white (16 Bit but the resulting information usually is 10 Bit)- Adjust exposure, gamma (16 Bit)- Compress information to 8 Bit- Demosiac (8 Bit)- Adjust saturation (8 Bit)- Display Result ( 8 Bit).
I want to compress the data to 8 Bit after the exposure and gamma correction, because I have to either keep multiple copies of the data, or run through the whole development again, if I just change the saturation. Even with 8 Bit data the application already uses ~300 MB RAM, and i'm concerned, that this will increase drastically, once I have implemented the more advanced noise reduction methods..
Even though the only step that would benefit from 16 Bit details after the exposure would be the saturation adjustment, which can't be done before the demosaicing..
I might try to change the whole process to 16 Bit, but that would still leave me with the problem on how the reduce the data once I want to display it or save it as JPEG..
Thanks for the help so far,Bye,Philip..
Do you know that there is a free raw development module that you can incorporate into your software? Several existing software packages use it.I'm not sure but I think this is it.http://www.photo-freeware.net/raw-shooter-essentials.phpA member of the rabble in good standing...
What you are asking about is called a Gamma Curve. It is a characteristic of the color space. Both sRGB and Adobe RGB use a Gamma Curve of 2.2 (okay, the sRGB curve isn't precisely 2.2, but it's close)..
Raw data is almost always linear. sRGB and Adobe RGB JPEG always has a gamma curve..
Here's one place to read about it:http://en.wikipedia.org/wiki/Gamma_correction..
LM1 wrote:.
Do you know that there is a free raw development module that you canincorporate into your software? Several existing software packagesuse it.I'm not sure but I think this is it.http://www.photo-freeware.net/raw-shooter-essentials.phpA member of the rabble in good standing..
Thank you, but rawshooter has been bought by Adobe. Adobe stopped releasing it after the launch of Lightroom..
There are a few attemps to make an open source development module, but those that I have found were either very tightly integrated into the respective application, or lacking in some features..
I would like to help improve those open source projects, but I feal that I should first try to get a better feel of the problems. Therefore I decided to create a little raw development program by myself first..
Bye,Philip..
Doug Pardee wrote:.
What you are asking about is called a Gamma Curve. It is acharacteristic of the color space. Both sRGB and Adobe RGB use aGamma Curve of 2.2 (okay, the sRGB curve isn't precisely 2.2, butit's close)..
Raw data is almost always linear. sRGB and Adobe RGB JPEG always hasa gamma curve..
Here's one place to read about it:http://en.wikipedia.org/wiki/Gamma_correction.
Thank you!.
That seems to be exactly what I was missing, I have toyed around with some different curves, but that seems to be spot on.I heard of gamma 2.2 before, but I didn't even think of it before..
Am I correct in thinking that I can select any range of data from the raw file (9 Bit, 10 Bit, ...) and compress it to 8 Bit using the 1/2.2 curve?.
Bye,Philip..
... but the human eye-brain system (nor many things in our environment) is not. Thus you need to adapt the (mostly) linear info from the sensor to the human, and that doesn't just mean "gamma". If you only pick out any 8bit segment from the 12 or 14 bit raw you likely will be leaving out information that an observer would have found useful..
BTW, this is way beyond "beginner" stuff... you may find you get more replies in the "Open" forum..
Have you looked at the available dcraw source code?.
-gt..
Greentoe wrote:.
... but the human eye-brain system (nor many things in ourenvironment) is not. Thus you need to adapt the (mostly) linearinfo from the sensor to the human, and that doesn't just mean"gamma". If you only pick out any 8bit segment from the 12 or 14bit raw you likely will be leaving out information that an observerwould have found useful..
Now that I have a better understanding of the representation this makes a lot of sense..
BTW, this is way beyond "beginner" stuff... you may find you get morereplies in the "Open" forum..
Thanks, i'm pretty sure that I will have a few more questions, I will post those in the "open" forum..
Have you looked at the available dcraw source code?.
I looked at it when I started, and unfortunately the source code looks pretty much like my c projects, lots of neat little tricks, but little documentation..
If you have a understanding of the basics, the code is easy to read, but I would consider it a more advanced example, rather than documentation..
Once you know about gamma correction and the ITU-R BT.709 function this part:.
Val = 256 * ( !use_gamma ? r : r <= 0.018 ? r*4.5 : pow(r,0.45)*1.099-0.099 );.
Makes a lot more sense. But I only understood it, once I remembered the 4.5 slope from some diagram, as it's only mentioned on very few webpages..
Thanks for the help,Philip..
You say you are working with RAW data?.
Can this code give you the values of the sensor before any demosaicing? I do some digital image processing as a hobby, and it could be useful..
Lately I've been working on long exposure dark-current noise reduction..
Image control:Zoom outZoom 100%Zoom inExpand AllOpen in new window..
Although dcraw does a lot of things that you might not need, you can figure out the basics without reading anywhere near all of the code. I don't understand all of it either, but I was able to make some modifications for my own use..
The RAW data is linear, at least approximately. To put it on the scale of EVs or "stops" that is more familiar to photographers, just take the logarithm and divide by log 2..
The 8-bit sRGB values you probably want to output use a nonlinear coding. It's more complicated, but you can convert them to EVs too. I posted information about this a while ago:http://forums.dpreview.com/...forums/read.asp?forum=1018&message=21321487.
Tone curves are easier to understand when the axes are labeled in stops. The scales are closer to perceptually uniform, and the slope of the curve measures contrast..
Color makes things more complicated. If you tone-map each channel separately, darker colors will tend to come out wrong. But if you tone-map the luminance instead, then it can be tricky to deal with chromaticity in the highlights..
Good luck with your project... this is certainly more than the average "beginner" would attempt! .
Alan Martin..
Clint Sanders wrote:.
You say you are working with RAW data?.
Can this code give you the values of the sensor before anydemosaicing? I do some digital image processing as a hobby, and itcould be useful..
Lately I've been working on long exposure dark-current noise reduction..
Yepp,thats the reason why I want to process the raw data..
The raw data is the the direct digital version of what each individual photosite captures..
This is before the demosaicing and usually gives you a wider range or more detailed values. Usually you will get a 16 Bit image, in which each pixel only has one colour value, with both the others at zero..
One of my ideas is, that it should be a lot easier to do noise reduction, when you still have the indivdual source of noise isolated and not mashed up due to demosaicing..
There are currently 2.5 ways to get the raw data with open source programs. I use the java project jrawio, since java enables me to create a direct feedback more easily..
The other 1.5 options are to use dcraw. dcraw is written in c and on it's own it is a standalone programm to convert raw camera files. You can either use dcraw to output the 16 Bit unprocessed sensor data into a file, or you can take the source code and use parts of it in your own program..
I think the main decision you would have to make is wether you want to use java, or c / c++ for your program, after that there isn't that much choice left..
Your noise reduction looks pretty good so far! Are you using dark frame reduction, or is there any other reason, why you focus on long exposures?.
Bye,Philip..
Philip Peter wrote:.
Your noise reduction looks pretty good so far! Are you using darkframe reduction, or is there any other reason, why you focus on longexposures?.
Yep, I'm using dark frame reduction. I focused on long exposure noise as it is 'predictable' in a sense...
Clint Sanders wrote:.
Yep, I'm using dark frame reduction. I focused on long exposure noiseas it is 'predictable' in a sense..
Than it might be usefull for you to use the raw data, since you can eliminate single pixels, you might also find better replacement pixels, since you only have to eliminate a single colour value and not all three..
Bye,Philip..
Philip Peter wrote:.
Than it might be usefull for you to use the raw data, since you caneliminate single pixels, you might also find better replacementpixels, since you only have to eliminate a single colour value andnot all three..
I'm kind of already doing that. I've managed to de-demosaic the data from a tiff file, but obviously, getting the real data would be better...
Oh yeah, the RAW data works a LOT better. More so than I would have expected..
Thanks so much for this algorithm! A lot more options have opened up for me..
Val = 256 * ( !use_gamma ? r : r <= 0.018 ? r*4.5 : pow(r,0.45)*1.099-0.099 );.
If I'm not mistaken, before you correct for gamma (r*2) is the algorithm for 1 stop increase in exposure, or maybe it it 2 stops. I'll have to experiment..
Image control:Zoom outZoom 100%Zoom inExpand AllOpen in new window..
Wow, that defiantely a huge improvement, I think I will have to do a few trials at which time dark frame substraction becomes usefull..
Did you use dcraw and then demosaic it yourself?.
Clint Sanders wrote:.
Thanks so much for this algorithm! A lot more options have opened upfor me.val = 256 * ( !use_gamma ? r : r <= 0.018 ? r*4.5 :pow(r,0.45)*1.099-0.099 );.
If I'm not mistaken, before you correct for gamma (r*2) is thealgorithm for 1 stop increase in exposure, or maybe it it 2 stops.I'll have to experiment..
I assume that you just have to do r*(2^nr_stops) to adjust the the exposure, but after all i've read I wouldn't be suprised if it is more complicated..
Bye,Philip..
Philip Peter wrote:.
Did you use dcraw and then demosaic it yourself?.
Yep...
Philip Peter wrote:.
Val = 256 * ( !use_gamma ? r : r <= 0.018 ? r*4.5 : pow(r,0.45)*1.099-0.099 );.
Where did this formula come from? It looks like the sRGB gamma curve, but with slightly different coefficients..
SRGB: r <= 0.0031038 ? r*12.92 : pow(r,(1/2.4))*1.055-1.055Adobe RGB 1998: pow(r,(1/2.19921875))..
Val = 256 * ( !use_gamma ? r :#ifdef SRGB_GAMMAr <= 0.00304 ? r*12.92 : pow(r,2.5/6)*1.055-0.055 );#elser <= 0.018 ? r*4.5 : pow(r,0.45)*1.099-0.099 );#endif.
So the formula appears to be the gamma formula DCRAW uses if SRGB is not defined...
Doug Pardee wrote:.
Philip Peter wrote:.
Val = 256 * ( !use_gamma ? r : r <= 0.018 ? r*4.5 : pow(r,0.45)*1.099-0.099 );.
Where did this formula come from? It looks like the sRGB gamma curve,but with slightly different coefficients..
SRGB: r <= 0.0031038 ? r*12.92 : pow(r,(1/2.4))*1.055-1.055Adobe RGB 1998: pow(r,(1/2.19921875)).
That one is from the dcraw source code, the full source for this is:.
Void CLASS gamma_lut (uchar lut[0x10000]){int perc, c, val, total, i;float white=0, r;.
Perc = width * height * 0.01; /* 99th percentile white point */if (fuji_width) perc /= 2;if (highlight && highlight != 2) perc = -1;FORCC {for (val=0x2000, total=0; val > 32; )if ((total += histogram[c][val]) > perc) break;if (white < val) white = val;}white *= 8 / bright;for (i=0; I < 0x10000; i++) {r = I / white;val = 256 * ( !use_gamma ? r :#ifdef SRGB_GAMMAr <= 0.00304 ? r*12.92 : pow(r,2.5/6)*1.055-0.055 );#elser <= 0.018 ? r*4.5 : pow(r,0.45)*1.099-0.099 );#endifif (val > 255) val = 255;lut[i] = val;}}.
Which as far as I can tell creates a gamma lookup table based on the seting that 1% of the brightest channel is clipped..
The formel looks also very much like the Rec 709 transfer function in this article:http://www.poynton.com/...tes/colour_and_gamma/GammaFAQ.html#gamma_correction.
As far as I can tell it is a modification of the normal gamma correction adjusted to the requirements of electronic sensors..
Bye,Philip..
ITU-R BT.709 is for driving CRT monitors, not for generating image files. While the difference is probably somewhere between trivial and infinitesimal, I think that it would be better to use the appropriate gamma curve for the output colorspace...
Void CLASS gamma_lut (uchar lut[0x10000]){int perc, c, val, total, i;float white=0, r;.
Perc = width * height * 0.01; /* 99th percentile white point */if (fuji_width) perc /= 2;if (highlight && highlight != 2) perc = -1;FORCC {for (val=0x2000, total=0; val > 32; )if ((total += histogram[c][val]) > perc) break;if (white < val) white = val;}white *= 8 / bright;for (i=0; I < 0x10000; i++) {r = I / white;val = 256 * ( !use_gamma ? r :#ifdef SRGB_GAMMAr <= 0.00304 ? r*12.92 : pow(r,2.5/6)*1.055-0.055 );#elser <= 0.018 ? r*4.5 : pow(r,0.45)*1.099-0.099 );#endifif (val > 255) val = 255;lut[i] = val;}}.
He uses 'c' with out initializing it to some value!.
Or my eyes are going bad..
That's "FORCC" not "FOURCC"..
C is initialized in the defined value of FORCC..
Doug Pardee wrote:.
ITU-R BT.709 is for driving CRT monitors, not for generating imagefiles. While the difference is probably somewhere between trivial andinfinitesimal, I think that it would be better to use the appropriategamma curve for the output colorspace..
The difference is actually quite obvious. Without SRGB_GAMMA, the output of dcraw is similar to many cameras' JPEGs. With SRGB_GAMMA, the shadows are a stop and a half brighter and the colors are more accurate..
The sRGB gamma function is really an "encoding transfer function", which means it is supposed to be canceled out on display, producing a net linear response. The Rec. 709 gamma function, on the other hand, produces a net nonlinear response by design. It darkens the shadows to hide noise, and adds contrast and midtone saturation..
But applying a nonlinear transfer function to each channel separately is less than ideal. In particular, it can cause serious hue shifts in the midtones, e.g., golden yellow becomes orangy-brown..
More subjectively, I find that using a linear transfer function produces images that are somehow more "photo-realistic". Less contrast, nice sky blues, consistent shadows... I can identify some of the reasons, but I don't know which ones are essential. But it's enough to convince me to use the SRGB_GAMMA version of dcraw..
Alan Martin..