I had a quick idea I wanted to jot down, and I haven’t found anything on it. I feel like someone out there must have thought up something similar already, or it’s already in the works at some black lab of a sensor company or something.

The idea I have is an image sensor that measures light intensity based on time-to-saturate – the time it takes for a particular photowell (representing a pixel) to saturate to its maximum capacity. The concept I’ve come up with has some interesting theoretical advantages in dynamic range over conventional photon-counting designs used today.

Imaging today – photon counting

First, a layman’s overview of how the conventional photon-counting design works in today’s sensors:

The sensor is a light sensitive device, and whenever photons come into contact with it, they are absorbed and a proportional number of photoelectrons are “knocked out” by the photon energy and collected in a photowell. From this photowell, a voltage measurement is taken, and this ultimately translates to a brightness value in the resulting image. In essence: Image brightness ∝ voltage reading ∝ electrons collected ∝ photons collected.

When taking an image, there is a set exposure duration, often referred to as a “shutter speed” in photography terms. This defines the time when a sensor is exposed to and detecting light – the exposure starts, light hits the sensors, exposure stops, and then we count the photons.

A limiting factor in this design is the photowell capacity. The number of electrons that can be stored in a well is finite, and once the photowell capacity is saturated, any additional electrons are not collected and hence the photons they correlate to are not counted. On the flipside, there is also a noise floor, where enough electrons must be gathered to produce a signal that is discernible from the random signal variation due to various forms of dark (thermal), electronics (read), and shot (photon) noise.

These two attributes lead to a problem of dynamic range – in scenes where light intensity differs greatly between the darkest and brightest areas, the sensor is simply unable to measure the full range of brightnesses and must cap measurements above and/or below a certain threshold.  This leads to the “blown highlights” and “crushed shadows” attribute often found in photos of large dynamic range scenes.

Time-to-saturate

The idea behind a time-to-saturate sensor is fairly simple. What we aim to measure in an image is light intensity – the flux of photons per time per area. The area is cancelled out of the equation by the photosite corresponding to a pixel being a certain area, so the measure we are really after is photons per time, for each pixel.

With photon counting, we fix a shutter speed (time duration), and then count the number of photons (via voltage measurement of photoelectrons) captured in that span, and use both to derive the intensity:

Intensity = photons / time = photons recorded / shutter speed

In time-to-saturate, the photon count is fixed at the capacity of the photowell, and the variable we measure is the time it takes for an individual well to saturate fully to the capacity.

Intensity = photons / time = max photon capacity / time-to-reach max-photon-capacity

How would the system work exactly? With a time-to-saturate sensor, we use as long a shutter speed as needed to fully saturate all photowells (in a conventional sensor, this is the minimum shutter speed to generate an all-white (max brightness) image). At the moment a photowell reaches capacity, it records a timestamp which will indicate how long it took to reach capacity. Once the exposure is finished, we are then left with a two-dimensional array of saturation times, rather than photon counts. Rather than recording 100k photons at one photosite, and 50k photons at a neighboring photosite where light was half as intense, the readings we get from this sensor would be along the lines of 1 millisecond time-to-saturate for the first photosite, and 2 millisecond time-to-saturate for the second, half-intensity photosite.

Key Advantages

There are two key advantages in our ability to take light intensity readings, both ultimately advancing dynamic range:

• There is virtually no limitation to the range of highlights we can capture, unlike the limitation imposed by the photowell capacity with photon-counting sensors. In our example, if there was a third photosite which had double the intensity of the first 100k photosite, and was exposed to 200k photons, it would only end up recording 100k photons since this is the capacity of the photosite, and thus both pixels would record the same white (max brightness) value, even though the 200k photosite pixel clearly represents a brighter area in the scene than the 100k photosite. A time-to-saturate measurement, by contrast, would simply produce a shorter time measurement: the 200k photosite saturates in 0.5 milliseconds, which we can compare to the 1 millisecond measurement for the first photosite and clearly conclude that the 200k photosite is twice as bright.
• Noise levels are reduced to the level of a maximally-saturated photowell. In a photon-counting sensor, any photosite that does not record a max white value by definition recorded a fewer number of photons, and thus produces a sub-optimal signal-to-noise ratio (SNR). Photon or “shot” noise has a standard deviation of the square root of the signal – thus for 100k photons we have √(100,000) = 316.2 photons of standard deviation, and a SNR of N/√(N) = √(N) = 316.2. For 50k photons, however, we have an SNR of √(50,000) = 223.6. In contrast, all photosites in a time-to-saturate sensor reach the max well capacity, and will thus all have the max SNR. This ensures that all photosites record values well above the noise floor, and additionally reduces photon noise for all pixels to the level of a maximally saturated photosite (the 100k photon, 316.2 SNR in this example).

In theory, such a sensor would have an infinite dynamic range – the brightest intensities are simply recorded as short time-to-saturate durations, and enough samples are recorded from the darkest areas to place the measurement well above the noise floor.  This would have huge implications for large dynamic range photography and imaging in general, to be able to record the entire dynamic range of a scene in a single exposure, without having to resort to processing tricks like selective shadow/highlight adjustment or high dynamic range (HDR) blending.

Potential Feasibility Issues

I’m not aware of any sources that have thought of this idea before, but if there are then there must be some large feasibility (or perhaps cost) issues that have prevented its development thus far. The few issues that I can imagine, none of which seem like dealbreakers and none of which would place performance any worse than that of photon-counting methods, in theory:

• Timing accuracy/precision of photowell saturation. While photon-counting relies on accurate and precise voltage readings from the photowells, a time-to-saturate sensor would need good accuracy and precision in recording time when a photowell reaches saturation. How precise does the time need to be, to equal the theoretical precision of today’s cameras? Taking the contemporary example of a 100k photon capacity photowell, hooked up to a sensor/imaging pipeline with a 14-bit analog-to-digital converter (found on most high-end cameras today), we would need to quantize measurable photon counts into 2^14 = 16,384 steps. 100,000 / 16,384 = ~6 photons, which is the precision we need to be able to measure time-to-saturation by. Most high-end cameras today operate with a minimum shutter speed of 1/8000 second (125 microseconds) – a 100k photowell that fully saturates in this time (this is the maximum light intensity the photon-counting sensor is able to record, under any settings) is thus 100,000 photons/125 microseconds = 800,000,000 (0.8 billion) photons / second.  Finally, we use this intensity along with our 6 photon steps to arrive at 6 photons / (0.8 billion photons/second) = 7.6 nanoseconds. This is the precision with which a time-to-saturate sensor needs to record time by. Of course, depending on the application the numbers can vary – with fewer bits per pixel, we would need less precision (an 8-bit jpeg in this example would need just ~0.5 microseconds of precision), with lower photowell capacity we would need greater precision, and with a larger minimum exposure time we would need less precision.
• To take advantage of the greater dynamic range capabilities of a time-to-saturate sensor, the exposure duration must be longer than a conventional photon-counting sensor, to capture more light. For static scenes, this is unlikely to be an issue, but for dynamic scenes (e.g. moving subjects), the exposure duration can only be stretched so far before issues such as motion blur or camera shake blur are introduced. At worst, however, the exposure can simply stop after a defined maximum exposure time – at this point any photowells which have not reached capacity simply output a voltage reading like in a conventional sensor – this reading is then used to extrapolate a time-to-saturate which can then be compared with the other photosites. In the worst case, the maximum exposure time is the same as the exposure time in a conventional photon-counting sensor, and would produce an noise level and at least the same dynamic range, if not a greater dynamic range captured in the highlights. For any exposure duration exceeding that of the conventional sensor however, noise levels will be reduced and a greater dynamic range in the shadow regions will be achieved as well.

What do you think?  Any potential pitfalls or feasibility issues I might have missed? I’m especially interested if anyone has come across a source with similar ideas before. Feel free to post links in the comments!

tl;dr: Fujifilm’s EXR sensor is extraordinary, mostly for its dynamic range. If you’re after the best non-DSLR image quality around, your choices start at the Fuji F200EXR, F70EXR, S200EXR, and end there.

Fujifilm has long been a leader in revolutionary sensor technology, particularly at the smaller scale sensor market where the majority of manufacturers have long been content pumping out traditional, vanilla CCD sensors with square grid-based Bayer Filter Arrays.

In September of 2008, announced plans for their latest sensor: the Super CCD EXR, which combines the unique color filter array (CFA) and pixel binning features of various previous sensors into a single “switchable” sensor that can be optimized in one of several areas (which are typically mutually exclusive when designing a sensor): high resolution, high dynamic range, and low noise.

High resolution

High resolution mode is the default mode, which utilizes the full set of photosites on the sensor and produces an image with a corresponding pixel on each photosite – nothing too special here, though Fuji claims the diagonal layout of photosites (as opposed to simple square grid) helps to improve resolution.

High sensitivity

A comparison of a typical Bayer CFA (left) and the CFA on Fujifilm's new EXR sensor (right)

The second mode of operation for the EXR sensor is a high-sensitivity mode which Fuji calls “Pixel Fusion Technology”, which is fancy marketspeak for pixel-binning (combining reading from adjacent pixels together to produce a better signal). With the EXR’s pair-based CFA layout, Fujifilm claims that interpolation (and thus color resolution) will be more accurate because the binned pixels are closer together (e.g. the pair blue pixels are pretty much in the same location, while they’re separated by two pixel lengths in a standard square-grid Bayer array. I don’t know that I buy this argument particularly well – it’s true that same-color pixel values will be more accurate since they’re closer, but you can’t get something for nothing: for example, the average distance from red-to-blue is going to be increased, which lowers accuracy for interpolating blue values at red pixels.

Regardless of whether their CFA and photosite layout nets them better interpolation, the key element here is the combination of pixel readings to generate a stronger signal, thus decreasing the proportion of noise. Using microlenses to patch up the fill factor (area of the sensor which is actually responsive to light) and various optimizations to lower read noise will get the high sensitivity mode EXR sensor closer to the noise level of a natively lower resolution sensor.

Wide Dynamic Range

A diagram detailing the two exposures captured by the EXR sensor when operating in large dynamic range mode

The third mode of operation for the EXR sensor uses variable photosite sensitivity to greatly extend dynamic range.  The concept is taken from some of Fuji’s older generation SuperCCD SR sensors – at a given pixel location there are in fact two photosites, one operating at a lower sensitivity and one operating at a higher sensitivity. This essentially produces two images for any particular shot, one at low sensitivity that is underexposed (capturing highlight detail, such as a bright sky), and one at high sensitivity that is overexposed (capturing shadow detail, such as a shaded building face). These images are combined, much like HDR combination is done, to create a single image which captures a much larger dynamic range than a single exposure could.

Edit: dpreview seems to report that the EXR sensor actually achieves this by operating one image at a shorter exposure time (shutter speed) than the other, rather than actually varying the sensitivity. If so this would be even better, as you’d have lower noise due to operating both sets of photosites at the same lower sensitivity.

As with pixel binning for greater sensitivity, the pixel count in the resulting image will have to halve as well.

There are some notable improvements compared to Fuji’s older SR sensors. For starters, the low and high sensitivity photosites are now of equal size, which Fuji claims will allow for a greater dynamic range extension (the SR sensors consisted of mostly “regular” photosites with tiny “low sensitivity” photosites sandwiched in). Furthermore, based on most of the image samples that can be found, the recombination method used for EXR is a bit closer to HDR blending, which doesn’t map values linearly on the same tone curve – this produces a punchier photo with better contrast that still looks natural upon viewing (due to the way human vision judges brightness in relative terms rather than absolute), even if its not quite pixel-accurate. This seems to address one of the complaints about Fuji’s older SR sensors, which provided a large dynamic range but ended up squashing it linearly to the same 12-bit RAWs or 8-bit JPEG images that all other cameras provide – the results were images that did have more highlight detail but looked “flat” and lacked contrast (because that 0.5-1 stop of highlight detail at the top is squashed into a small 250-255 pixel value range).

The EXR sensor has a big advantage over conventional HDR as well (i.e. taking multiple exposures and blending them): it captures an extended range image in a single instance, making it usable for moving subjects (HDR sports photos, yay!).

The Results

The first EXR sensor, the Fujifilm F200EXR, debuted in February 2009, and was followed up not long afterwards by the S200EXR bridge camera and the ultracompact ultrazoom F70EXR, giving us a chance to see some hard results.

Imaging-resource, as always, has perhaps the most comprehensive test bed of images, and samples from their express review of the F200EXR can be found here: http://www.imaging-resource.com/PRODS/F200EXR/F200EXRA7.HTM

Their site isn’t the most comparison-friendly however (though you can give their comparator a shot) so I’ll link to dcresource’s reviews of the F200EXR and F70EXR as well and reference these.

The first thing to note is that Fujifilm hasn’t lost a step in the noise race – in both the standard high resolution (no binning) and high sensitivity (binning, lower resolution) modes, the EXR sensor simply wipes the floor with every camera on the market this side of a full-fledged DSLR.  In the F200EXR review there is a side-by-side comparison (search for the text “Again, things look great through ISO 400″ – it’s right above this) between the 6MP high-sensitivity mode image, and a 12MP high-resolution mode image that is downsized to 6MP – essentially doing the same as pixel binning but off-camera, and digitally, rather than in-camera and analog.  The result is a slightly crisper image but noticeably more noise, though the effect isn’t dramatic.

What’s interesting is that the side-by-side comparison in the F70EXR review shows that the high-resolution mode, downsized to the same resolution as the high-sensitivity mode, actually produces better results – the same amount of noise but much crisper detail. This seems to punch a hole in the effectiveness of the EXR’s in-camera pixel-binning: if the digital data (full of rounding errors, and compressed to 8-bit jpeg) can be averaged and produce more effective results than binning the analog data (the raw readings from the sensor), then we can surmise that having more accurate data on the location of brightness values (i.e. more pixels) helps us produce more accurate images overall than having slightly more accuracy on the actual brightness values.

Further down on the F200EXR review (search “so the two would be the same (6MP) resolution” – right below this), you’ll see a direct comparison using the camera’s wide dynamic range mode. As opposed to the high sensitivity mode, here we can see real, significant benefits – highlight detail that is hopelessly blown out in the left image is very much visible in the wide dynamic range image. For those of you too lazy to navigate the admittedly long and cumbersome dcresource review pages, here’s a marketing image from Fujifilm that gives you the general idea:

Standard dynamic range (left) and wide dynamic range (right) - probably exaggerated a bit

This image gives you a general idea of the difference, though I wouldn’t take it at face value. The image on the right is probably a fair representation of what you’ll get using the wide DR mode (and you can compare this with shaded interior/sunlit exterior/sky photos you’ve probably taken), but the image on the left has way more contrast (and less DR) than any typical camera would, on its default settings at least.

The Triumph of EXR – Dynamic Range

So is Fuji’s EXR sensor a success? It depends on what you’re after. Many diehard Fujifilm Super CCD fans fell in love with the low-resolution F10/11 and F30/31 ultracompacts, both of which came in at just 6MP and absolutely wiped the floor with the competition in terms of noise performance.  And while subsequent SuperCCD iterations have maintained a clear advantage over competitors in this area (and this newest EXR sensor does to it better than its predecessor), the fact is that the high 12MP or so resolutions found on today’s sensors still compromise noise performance, despite any fancy “Pixel Fusion Technology” that Fujifilm tries to market.

The true triumph of the EXR sensor is in its dynamic range capability, and its separate pixel design (it essentially operates two sensors) works not only better than any of its competitors, but far better than even a natively lower resolution sensor.  While a larger photosite does afford more highlight headroom, halving the pixels (doubling the area) affords at most 1 stop. The EXR’s method, which essentially captures two independent exposures, is in theory capable of capturing dynamic range that is infinitely far apart, though for most scenes they’ll likely need to overlap to avoid gaps in coverage, which based on the settings allowed on current cameras is 3 stops.

According to dpreview’s dynamic range test of the F200EXR, the EXR can deliver nearly 11 EV (stops) of dynamic range.  Not only does that far outclass any compact (or even the bulky SLR-like bridges that use the same small sensors) on the market, but exceeds the dynamic range of DSLRs like the Canon 7D, Nikon D300, et. al, which all range around 8 EV for their jpegs. With a bit of tweaking with RAW files in Adobe Camera Raw, the DSLRs just about manage 10EV.

It’s simply remarkable that jpegs from a camera with a pint-sized sensor can beat out RAW images from the highest-end DSLRs, but that’s what innovative technology can do for you over hammering away with sheer physical size and trying small refinements from there (which is how most of the rest of the sensor industry has been operating for years). I can’t begin to fathom how much the Super CCD would change the landscape of photography if Fujifilm ever scaled up the sensor to DSLR size.

So over the past couple of weeks I’ve been working on a little project called Demosaic.  It’s a little online demo that interpolates image data from (simulated) raw sensor output, similar to what almost every digital camera used today has to do.

http://www.thedailynathan.com/demosaic/

The core of the problem is that most digital cameras today make use of a color filter array, or CFA, in order to differentiate intensities based on wavelength, and thereby derive color.  CFAs basically layer different color filters over each photodetector, only allowing that color light through and thereby isolating that intensity.  By far the most common type of CFA used today is the Bayer filter, although many other variants exist.

Diagram of a Bayer pattern sensor

One of the downsides of using a CFA is that each photodetector, which normally corresponds to a pixel location in the final image, only detects light intensity for one wavelength range.  So a blue-filtered detector will record exactly how much blue light was at the location, but has no idea how much red or green light there might be, for instance.

Digital cameras must therefore interpolate the missing data based on values recorded at nearby photodetectors, in a process known as “demosaicing”.  Lots of different algorithms exist, with different tradeoffs in speed, quality, and suitability for different subject matter.

I developed the Demosaic demo as a sort of off-shoot of a broader research report I did on color detection for electronic image sensors for my Electrical Engineering 119 (EE119) optical engineering course at UC Berkeley. (That’ll get posted soon as well, probably in a few installments and after I get it reviewed and edited a few times.)  Demosaic takes a test Bayer-pattern image (containing only the red, green, or blue values recorded at each location) and renders it into a final image using a variety of different algorithms.

As an example, here’s an original test image (images magnified 3x for clarity):

And here’s the raw Bayer data for the image. Every pixel is encoded with just a green, red, or blue value denoting the intensity of that color.  To get a final image, we have to guess blue and red values at every green pixel, green and blue values at every red pixel, and so on.

One of the most basic ways to interpolate the information is to simply take an average (arithmetic mean) of the surrounding pixels.  This is known as bilinear interpolatin, and produces a result like this:

Not terrible, especially as bilinear interpolation is just about the fastest algorithm out there (after nearest neighbor interpolation, which simply picks a single nearby pixel).  The demosaicing artifacts are readily noticeable, however – color artifacting at white/black transitions, and “zipper” pattern artifacting along edges.  Needless to say, this really isn’t all that acceptable for serious imaging applications.

Here’s another example with a more advanced algorithm known as Edge-sensing bilinear.  This is an adaptive algorithm, which actually analyzes the image for local spatial features (in this case, harsh intensity transitions, or “edges”) and interpolates the data based on this context.

While not perfect, you can see this algorithm gets rid of many of the artifacts present in simple non-adaptive bilinear interpoltion.  The zipper pattern artifacting has mostly disappeared, and the color artifacts at black/white transitions is also less apparent.  As a tradeoff, however, this edge-sensing algorithm comes at a greater computational cost compared to simple bilinear interpolation.

To learn more about the different algorithms and run side-by-side comparisons, check out the Demosaic site.  Also feel free to upload your own custom images to see how the algorithms perform on different types of image content.