Noise
There are many sources of noise in images. When there is no light input on the sensor, there is still readout noise and thermal noise: Small fluctuations in the measured values even if no photons are hitting the sensor. This is due to electrons moving around randomly and imperfections in the conversion from an analog measurements to digital values.
Shot noiseΒΆ
The process of measuring photons on the sensor is inherently noisy. The photons are not arriving one by one with a fixed interval. They arrive on the sensor in a Poisson process.
The number of photons detected by a pixel during an exposure follows a Poisson-distribution. This means that if the average number of photons is $\lambda$ over many exposures, then the probability of measuring $k$ photons in a given exposure is
$$P(N = k) = \frac{\lambda^k e^{-\lambda}}{k!}$$
This means that our measurement is fundamentally noisy. It cannot be improved with electronics.
As the number of photons become large, the Poisson distribution tends toward a normal distribution with mean value $\lambda$ and variance $\lambda$. The noise is the standard deviation, which is $\sqrt{\lambda}$.
The signal-to-noise value is thus
$$\frac{\lambda}{\sqrt{\lambda}} = \sqrt{\lambda}$$
which means that bright regions will have a higher SNR than dark regions, and thus less noise.
It is important to note that the noise is related to the number of photons. This means that images captured in the dark, with few photons, are noisier than images captured in sunlight, with many orders of magnitude more photons.
However, an image captured in the dark can be made to appear as bright on a screen as a well-lit image. This is done by applying gain. Either as part of capturing the image on the sensor or after it has been captured. We will get back to how gain works on an image sensor, but for now, we can assume the gain we apply is perfect and see the effect it has on our images:
import halide as hl
from halide_book.interactive import Slider, interact
from halide_book.paths import Paths
import halide.imageio
from halide_book.math import random_normal
x = hl.Var("x")
y = hl.Var("y")
c = hl.Var("c")
well_depth = hl.Param(hl.Float(32), "well_depth", 500.0)
well_depth_slider = Slider(well_depth, min_value=1, max_value=1000, step=1)
image = hl.Func("image")
image[x, y] = hl.select(
x > 80,
hl.f32(0.8) * well_depth,
hl.f32(0.2) * well_depth,
)
size = [160, 90, 3]
def to_8_bit(func: hl.Func, max_value: float):
func_8_bit = hl.Func(f"{func.name()}_8_bit")
func_8_bit[x, y, c] = hl.u8(
hl.clamp(hl.u32(func[x, y]) / max_value * 255.0, 0.0, 255.0)
)
return func_8_bit
interact(to_8_bit(image, well_depth), size=size, controls=[well_depth_slider])
Think of the above image as what we would get with a perfect camera without any noise.
Now, we can introduce the noise based on the Poisson distribution. Use the seed slider to generate different images. You can also use the well depth slider to see the effect of the number of photons involved in producing the image.
seed = hl.Param(hl.Int(32), "seed", 1)
seed_slider = Slider(seed, min_value=1, max_value=20, step=1)
noise = hl.Func("noise")
noise[x, y] = hl.sqrt(image[x, y]) * random_normal(seed)
noisy_image = hl.Func("result")
noisy_image[x, y] = hl.clamp(image[x, y] + noise[x, y], 0.0, well_depth)
interact(
to_8_bit(noisy_image, well_depth),
size=size,
controls=[seed_slider, well_depth_slider],
)
It is a bit hard to compare the amount of noise in the dark and bright region. While the absolute noise is larger in the bright region of the image, the relative noise is largest in the dark region. To visualize this, we can implement gain and study what happens with the two halves with different gain values:
gain = hl.Param(hl.Float(32), "gain", 1.0)
gain_slider = Slider(gain, min_value=1.0, max_value=10.0, step=0.01)
gained = hl.Func("gained")
gained[x, y] = noisy_image[x, y] * gain
interact(
to_8_bit(gained, well_depth),
size=size,
controls=[
seed_slider,
well_depth_slider,
gain_slider,
],
)
If you increase the gain, the dark region becomes as bright as the bright region was initially, but with more noise. The bright region also becomes fully saturated.
Since this is a simulation and we know the expected average pixel value, we can also illustrate the relative noise by normalizing the two halves.
normalized = hl.Func("normalized")
normalized[x, y] = noisy_image[x, y] / image[x, y] * 0.5 * well_depth
interact(
to_8_bit(normalized, well_depth),
size=size,
controls=[seed_slider, well_depth_slider],
)
We can now see that while the average values are the same, the relative noise was clearly larger in the dark (left) region of the image.