Difference between revisions of "JP4"

From ElphelWiki
Jump to: navigation, search
(JP4 to DNG image conversion)
(Example file)
 
(49 intermediate revisions by 6 users not shown)
Line 1: Line 1:
Note: the JP4 mode described here is referred as "JP46" in current 8.0 firwmare
+
==<font color="blue">JP4 Format</font>==
  
== JP4 Format ==
+
The goal of the JP4 format is to preserve as much as possible of the information from the image sensor (ideally "raw") while providing reasonable compression. Based on JPEG compression, the format was originally developed at Elphel for accurate representation of the scanned book pages and later used in other applications that involve post-processing.
  
We have added a special JP4 mode that bypasses the Demosaic in the FPGA and provides an image with pixels in each 16x16 macroblock that are rearranged to separate Bayer colors in individual 8x8 blocks, then encoded as monochrome. [[Demosaic_on_client_side|Demosaic]] will be applied during post-processing on the host PC. This section describe different algorithms and implementations used to provide this functionality.
+
The standard JPEG was developed to compress images so they visually look almost the same as uncompressed but are significantly smaller in size. The most damaging part of the color JPEG compression is the demosaic part of it. "Lossy" part of the compression (quantization after DCT) can be eliminated with setting compression quality to 100% that effectively disables quantization. Dynamic range compression ("gamma conversion") is usually designed to be lower than the sensor shot noise - significant for most modern small-pixel (and so low full-well capacity) sensors. That leaves the demosaic ("guessing" the missing colors in Bayer mosaic sensor outputs) a single most damaging operation. In the camera this is usually done by processing data in 3x3 or 5x5 pixel blocks and it is difficult to distinguish between variations (especially sharp) in intensity and the color tone. There are much more advanced algorithms that can be used during post-processing that will provide much better results, but they require "raw" sensor data. Or - the data saved in JP4 format.
  
Main goals:
+
Here is a [http://community.elphel.com/jp4/jp4demo.php JP4 Demo] that illustrates this algorithm and how it is related to standard JPEG.
- compression speed improvement
 
- possibility to obtain more high quality image (near to RAW)
 
- drasticaly lowering data size
 
== Different JP4 Modes in 8.X Software ==
 
only modes 0-2 can be processed with standard libjpeg:               
 
*0 - mono6, monochrome (color YCbCr 4:2:0 with zeroed out color componets)               
 
*1 - color, YCbCr 4:2:0, 3x3 pixels               
 
*2 - jp46 - original JP4 (from 7.X software), encoded as 4:2:0 with zeroed color components               
 
*3 - jp46dc, modified jp46 so each color component uses individual DC diffenential encoding               
 
*4 - reserved for color with 5x5 conversion (not yet implemented)
 
*5 - jp4 with ommitted color components (4:0:0)
 
*6 - jp4dc, similar to jp46dc encoded as 4:0:0
 
*7 - jp4diff, differential where (R-G), G, (G2-G) and (B-G) components are encoded as 4:0:0
 
*8 - jp4hdr, (R-G), G, G2,(B-G) are encoded so G2 can be used with high gain               
 
*9 - jp4fiff2, (R-G)/2, G,(G2-G)/2, (B-G)/2 to avoid possible overflow in compressed values               
 
*10 - jp4hdr2, (R-G)/2, G,G2,(B-G)/2               
 
*14 - mono,  monochrome with ommitted color components (4:0:0)
 
  
 +
JP4 mode bypasses the demosaic/color conversion in the FPGA and provides an image with pixels in each 16x16 macroblock that are rearranged to separate Bayer colors in individual 8x8 blocks, then encoded as monochrome. [[Demosaic_on_client_side|Demosaic]] will be applied during post-processing on the host PC. This section describe different algorithms and implementations used to provide this functionality.
  
=== [[JP4 HDR]] ===
+
==<font color="blue">Features</font>==
Bayer pattern look like this
+
Compared to JPEG:
{| class="wikitable"  
+
* Higher compression speed
|-
+
* Better image quality (close to RAW)
|
+
* Pixel data is not modified by demosaicing algorithm
{| class="wikitable"align="center"
+
* Smaller file size
  |+RGGB
+
 
  |-
+
==<font color="blue">Format variations</font>==
  | bgcolor="red"|R    || bgcolor="green"|G1 || bgcolor="red"|R    || bgcolor="green"|G1
+
JP4 family:
  |-
+
{| class="wikitable"
  | bgcolor="green"|G2 || bgcolor="blue"|B  || bgcolor="green"|G2 || bgcolor="blue"|B
+
|-
  |-
+
! Name
  | bgcolor="red"|R    || bgcolor="green"|G1 || bgcolor="red"|R    || bgcolor="green"|G1
+
! Code
  |-
+
! Description
  | bgcolor="green"|G2 || bgcolor="blue"|B  || bgcolor="green"|G2 || bgcolor="blue"|B
+
|-
  |}
+
| align='center' | '''jp46'''
|
+
| align='center' | 2
{| class="wikitable"align="center"
+
| original JP4 (from 7.X software in 10353s), encoded as 4:2:0 with zeroed color components
  |+BGGR
+
|-
  |-
+
| align='center' | '''jp46dc'''
  | bgcolor="blue"|R    || bgcolor="green"|G1 || bgcolor="blue"|R    || bgcolor="green"|G1
+
| align='center' | 3
  |-
+
| modified jp46 so each color component uses individual DC differential encoding
  | bgcolor="green"|G2 || bgcolor="red"|B  || bgcolor="green"|G2 || bgcolor="red"|B
+
|-
  |-
+
| align='center' | -
  | bgcolor="blue"|R    || bgcolor="green"|G1 || bgcolor="blue"|R    || bgcolor="green"|G1
+
| align='center' | 4
  |-
+
| reserved for color with 5x5 conversion (not yet implemented)
  | bgcolor="green"|G2 || bgcolor="red"|B  || bgcolor="green"|G2 || bgcolor="red"|B
+
|-
  |}
+
| align='center' | '''jp4'''
|
+
| align='center' | 5
{| class="wikitable"align="center"
+
| 4:0:0, omitted color components
  |+GBRG
+
|-
  |-
+
| align='center' | '''jp4dc'''
  | bgcolor="green"|G2 || bgcolor="blue"|B   || bgcolor="green"|G2 || bgcolor="blue"|B
+
| align='center' | 6
  |-
+
| similar to jp46dc encoded as 4:0:0
  | bgcolor="red"|R    || bgcolor="green"|G1 || bgcolor="red"|R   || bgcolor="green"|G1
+
|-
  |-
+
| align='center' | '''jp4diff'''
  | bgcolor="green"|G2 || bgcolor="blue"|B  || bgcolor="green"|G2 || bgcolor="blue"|B
+
| align='center' | 7
  |-
+
| differential where (R-G), G, (G2-G) and (B-G) components are encoded as 4:0:0
  | bgcolor="red"|R    || bgcolor="green"|G1 || bgcolor="red"|R   || bgcolor="green"|G1
+
|-
  |}
+
| align='center' | '''jp4hdr'''
|
+
| align='center' | 8
{| class="wikitable"align="center"
+
| (R-G), G, G2,(B-G) are encoded so G2 can be used with high gain
  |+GRBG
+
|-
  |-
+
| align='center' | '''jp4fiff2'''
  | bgcolor="green"|G2 || bgcolor="red"|B  || bgcolor="green"|G2 || bgcolor="red"|B
+
| align='center' | 9
  |-
+
| (R-G)/2, G,(G2-G)/2, (B-G)/2 to avoid possible overflow in compressed values
  | bgcolor="blue"|R    || bgcolor="green"|G1 || bgcolor="blue"|R    || bgcolor="green"|G1
+
|-
  |-
+
| align='center' | '''jp4hdr2'''
  | bgcolor="green"|G2 || bgcolor="red"|B  || bgcolor="green"|G2 || bgcolor="red"|B
+
| align='center' | 10
  |-
+
| (R-G)/2, G,G2,(B-G)/2
  | bgcolor="blue"|R    || bgcolor="green"|G1 || bgcolor="blue"|R    || bgcolor="green"|G1
+
|}
  |}
+
Other supported formats:
|}
+
{| class="wikitable"
 +
|-
 +
! Name
 +
! Code
 +
! Description
 +
|-
 +
| align='center' | '''color'''
 +
| align='center' | 0
 +
| YCbCr 4:2:0, 3x3 pixels
 +
|-
 +
| align='center' | '''mono6'''
 +
| align='center' | 1
 +
| monochrome - color YCbCr 4:2:0 with zeroed out color components
 +
|-
 +
| align='center' | '''mono'''
 +
| align='center' | 14
 +
| monochrome - color YCbCr 4:0:0 with omitted color components
 +
|}
 +
 
 +
'''Notes:'''
 +
* The format code is written (along with other info) to the '''MakerNote''' in EXIF data
  
The remark: all kinds of bayer patterns can be received from initial RGGB by flipping on X and/or Y.
+
==<font color="blue">JP4</font>==
 +
===Example file===
 +
Download [https://community.elphel.com/pictures/test.jp4 sample]
 +
{|
 +
|[[File:Jp4 example rotated asjpeg.jpeg|200px|thumb|JP4 opened as JPEG, rotated 90&deg;]]
 +
|[[File:Jp4 example rotated processed.jpeg|200px|thumb|JP4 processed, rotated 90&deg;]]
 +
|}
 +
* If opened as a normal image the macro blocks will be displayed as 8x32.
  
Some sensors have possibility to set independed scale to G1 and G2.
+
===Online===
Considering that the accessible optics does not give the full permission of a sensor resolution, it [[JP4_HDR|can be]] used for increase in a dynamic range of a image sensor.
+
* [https://community.elphel.com/jp4viewer/?width=800&quality=1 Online JP4 viewer]
 +
** click & select or drag & drop
 +
** it's a quick view - the result is scaled down
 +
** image format is read from exif (using EXIF.js)
 +
** can view jp46
 +
** can view regular jpegs
 +
** images are decoded in the browser, no uploading, no communication with the server
  
== JP4 processing on the host ==
+
===PC===
 +
====ImageJ plugin====
 +
* [[Elphel_Software_Kit_for_Ubuntu#ImageJ_and_Elphel_plugins_for_imageJ|Install ImageJ plugins]]
 +
* run: Plugins > JP46 reader camera
 +
** can display JP4/JP46 as monochrome w/o demosaicing
 +
** can split color channels
 +
====html canvas + javascript====
 +
* [https://git.elphel.com/Elphel/elphel-web-393/blob/master/src/jp4-canvas/elphel.js reorderJP4Blocks()]
  
=== JP4 image decoding in MATLAB ===
+
====python====
JP4 format can be easy manipulated by [http://www.mathworks.com/matlabcentral/fileexchange/22144 MATLAB] [[Image:Fruits_jp4.jpg|thumb|JP4 image]]
+
* 0.5s for 2592x1936 GRBG image
 +
<font size='1'>
 +
import numpy as np
 +
from PIL import Image
 +
import scipy.misc
 +
 +
# first open image in grayscale
 +
I = scipy.misc.imread(filename, flatten=False, mode='L')
 +
 +
H,W = I.shape
 +
 +
# 16x16 block
 +
block  = np.zeros((16,16))
 +
# 16xW
 +
stripe = np.zeros((16,W))
 +
 +
I = np.reshape(I,(H/16,16,W))
 +
 +
# print(I.shape)
 +
# (1936, 2592) -> (121, 16, 2592)
 +
 +
for i in range(I.shape[0]):
 +
  # stripe 16xW
 +
  stripe = np.copy(I[i])
 +
  for j in range(0,W,16):
 +
    if j<W/2:
 +
      k = 0
 +
      l = 2*j
 +
    else:
 +
      k = 8
 +
      l = 2*(j-W/2)
 +
 +
      # gr r b gb
 +
      block[0::2,0::2] = stripe[k:k+8,l+ 0:l+ 0+8]
 +
      block[0::2,1::2] = stripe[k:k+8,l+ 8:l+ 8+8]
 +
      block[1::2,0::2] = stripe[k:k+8,l+16:l+16+8]
 +
      block[1::2,1::2] = stripe[k:k+8,l+24:l+24+8]
 +
 +
      I[i,0:16,j:j+16] = block
 +
 +
I = np.reshape(I,(H,W))
 +
# pixels ordered as in bayer pattern, this needs demosaicing
 +
cv2.imwrite("result.png",I)
 +
</font>
  
1. Read image
+
==<font color="blue">JP46</font>==
  I=imread('hdr02.jp4'); %read JP4 file like JPEG
+
{|
 +
|[[Image:Fruits_jp4.jpg|thumb|JP46 opened as JPEG]]
 +
|[[Image:Fruits_jp4_deblocked.jpg|thumb|Bayer CFA encoded, deblocked]]
 +
|[[Image:Fruits_jp4_debayered.jpg|thumb|Demosaiced]]
 +
|}
 +
===PC===
 +
====ImageJ plugin====
 +
* same as for [[JP4#ImageJ_plugin|JP4]]
 +
====Decoding in MATLAB====
 +
JP46 format can be easy manipulated in [http://www.mathworks.com/matlabcentral/fileexchange/22144 MATLAB]
 +
* Read image:
 +
  I=imread('hdr02.jp46'); %read JP46 file like JPEG
 
   ,or online grab image from http like this:
 
   ,or online grab image from http like this:
  I=imread('http://community.elphel.com/pictures/jp4.jpg');
+
  I=imread('http://community.elphel.com/pictures/jp46.jpg');
 
   ,or cam:
 
   ,or cam:
 
  I=imread('http://cam_ip/bimg'); %get online buffered image from cam
 
  I=imread('http://cam_ip/bimg'); %get online buffered image from cam
  
 
  I=I(:,:,1);            %strip color data
 
  I=I(:,:,1);            %strip color data
2. Remove block grouping[[Image:Fruits_jp4_deblocked.jpg|thumb|Bayer CFA encoded image]]
+
* Remove block grouping:
 
<code matlab>
 
<code matlab>
 
  II=deblock16x16(I);    %deblock image
 
  II=deblock16x16(I);    %deblock image
Line 119: Line 197:
 
  y=y0;
 
  y=y0;
 
</code>
 
</code>
2. Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)[[Image:Fruits_jp4_debayered.jpg|thumb|Decoded image]]
+
* Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)
 
  J=demosaic(II,'gbrg');
 
  J=demosaic(II,'gbrg');
3. Show image
+
* Display image:
 
  imshow(J);
 
  imshow(J);
  
=== JP4 to DNG image conversion ===
+
====Convert to DNG====
 
+
=====Movie2dng=====
 +
See [[Movie2dng]] for conversion of JP4/JP46 files or *.movs to DNG format.
 +
=====LibTIFF=====
 
Credits: Dave Coffin
 
Credits: Dave Coffin
  
This Linux command line tool allows conversion of JP4 files into a DNGs that dcraw and Adobe Photoshop can open.
+
This Linux command line tool allows conversion of JP46 files into a DNGs that dcraw and Adobe Photoshop can open.
  
 
Download [http://community.elphel.com/files/jp4/tiff-3.8.2.tar.gz LibTIFF v3.8.2]
 
Download [http://community.elphel.com/files/jp4/tiff-3.8.2.tar.gz LibTIFF v3.8.2]
Line 159: Line 239:
 
  Example: ./elphel_dng 100 example_JP4.jpeg example.dng
 
  Example: ./elphel_dng 100 example_JP4.jpeg example.dng
  
=== JP4 video stream decoding using MPlayer ===
+
====JP46 video stream decoding using MPlayer====
JP4 stream can be decoded by MPlayer.
+
JP46 stream can be decoded by MPlayer.
 
Use this [[http://community.elphel.com/files/mplayer/debayer.diff this patch]]
 
Use this [[http://community.elphel.com/files/mplayer/debayer.diff this patch]]
 
   patch . -p0 < debayer.diff in the mplayer source dir
 
   patch . -p0 < debayer.diff in the mplayer source dir
Line 193: Line 273:
 
  Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]
 
  Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]
  
=== Avisynth plugin for JP4 processing ===
+
====Avisynth plugin for JP46 processing====
 
[http://avisynth.org/mediawiki/Main_Page Avisynth plugin] also available
 
[http://avisynth.org/mediawiki/Main_Page Avisynth plugin] also available
  
Line 201: Line 281:
 
  JP4("AHD","RGGB")
 
  JP4("AHD","RGGB")
  
=== GStreamer plugins for Elphel JP4 image and video processing ===
+
==== GStreamer plugins for Elphel JP4 image and video processing ====
 
[http://code.google.com/p/gst-plugins-elphel/ This project] supported by http://ubicast.eu hosts Elphel related gstreamer components, so far:  
 
[http://code.google.com/p/gst-plugins-elphel/ This project] supported by http://ubicast.eu hosts Elphel related gstreamer components, so far:  
  
Line 207: Line 287:
 
* bayer2rgb2 converts raw Bayer streams to RGB images  
 
* bayer2rgb2 converts raw Bayer streams to RGB images  
  
==== jp462bayer: JP4 to Bayer ====
+
===== jp462bayer: JP46 to Bayer =====
  
 
After jpegdec, re-arranges the pixels in Bayer format.
 
After jpegdec, re-arranges the pixels in Bayer format.
  
==== bayer2rgb2: debayer ====
+
===== bayer2rgb2: debayer =====
  
 
It offers the same features as the legacy bayer2rgb, but by wrapping Libdc1394's debayering algorithms you can choose the interpoloation algorithm between : simple, bilinear, hqlinear, downsample, edgesense, vng, ahd, nearest  
 
It offers the same features as the legacy bayer2rgb, but by wrapping Libdc1394's debayering algorithms you can choose the interpoloation algorithm between : simple, bilinear, hqlinear, downsample, edgesense, vng, ahd, nearest  
  
==== Example pipelines ====
+
===== Example pipelines=====
 
 
 
  gst-launch-0.10 rtspsrc location=rtsp://elphel:554 protocols=0x00000001 ! rtpjpegdepay ! jpegdec ! \
 
  gst-launch-0.10 rtspsrc location=rtsp://elphel:554 protocols=0x00000001 ! rtpjpegdepay ! jpegdec ! \
 
  queue ! jp462bayer ! queue ! bayer2rgb2 ! queue ! ffmpegcolorspace ! videorate ! "video/x-raw-yuv, \
 
  queue ! jp462bayer ! queue ! bayer2rgb2 ! queue ! ffmpegcolorspace ! videorate ! "video/x-raw-yuv, \
 
  format=(fourcc)I420, width=(int)1920, height=(int)1088, framerate=(fraction)25/1" ! xvimagesink sync=false max-lateness=-1
 
  format=(fourcc)I420, width=(int)1920, height=(int)1088, framerate=(fraction)25/1" ! xvimagesink sync=false max-lateness=-1
  
== Demosaicing/debayering links ==
+
==<font color="blue">JP4 HDR</font>==
 +
* [[JP4 HDR|'''More info''']]
 +
* The most common Bayer patterns:
 +
{| class="wikitable"
 +
|-
 +
|
 +
{| class="wikitable"align="center"
 +
  |+RGGB
 +
  |-
 +
  | bgcolor="red"|R    || bgcolor="green"|G1 || bgcolor="red"|R    || bgcolor="green"|G1
 +
  |-
 +
  | bgcolor="green"|G2 || bgcolor="blue"|B  || bgcolor="green"|G2 || bgcolor="blue"|B
 +
  |-
 +
  | bgcolor="red"|R    || bgcolor="green"|G1 || bgcolor="red"|R    || bgcolor="green"|G1
 +
  |-
 +
  | bgcolor="green"|G2 || bgcolor="blue"|B  || bgcolor="green"|G2 || bgcolor="blue"|B
 +
  |}
 +
|
 +
{| class="wikitable"align="center"
 +
  |+BGGR
 +
  |-
 +
  | bgcolor="blue"|R    || bgcolor="green"|G1 || bgcolor="blue"|R    || bgcolor="green"|G1
 +
  |-
 +
  | bgcolor="green"|G2 || bgcolor="red"|B  || bgcolor="green"|G2 || bgcolor="red"|B
 +
  |-
 +
  | bgcolor="blue"|R    || bgcolor="green"|G1 || bgcolor="blue"|R    || bgcolor="green"|G1
 +
  |-
 +
  | bgcolor="green"|G2 || bgcolor="red"|B  || bgcolor="green"|G2 || bgcolor="red"|B
 +
  |}
 +
|
 +
{| class="wikitable"align="center"
 +
  |+GBRG
 +
  |-
 +
  | bgcolor="green"|G2 || bgcolor="blue"|B  || bgcolor="green"|G2 || bgcolor="blue"|B
 +
  |-
 +
  | bgcolor="red"|R    || bgcolor="green"|G1 || bgcolor="red"|R    || bgcolor="green"|G1
 +
  |-
 +
  | bgcolor="green"|G2 || bgcolor="blue"|B  || bgcolor="green"|G2 || bgcolor="blue"|B
 +
  |-
 +
  | bgcolor="red"|R    || bgcolor="green"|G1 || bgcolor="red"|R    || bgcolor="green"|G1
 +
  |}
 +
|
 +
{| class="wikitable"align="center"
 +
  |+GRBG
 +
  |-
 +
  | bgcolor="green"|G2 || bgcolor="red"|B  || bgcolor="green"|G2 || bgcolor="red"|B
 +
  |-
 +
  | bgcolor="blue"|R    || bgcolor="green"|G1 || bgcolor="blue"|R    || bgcolor="green"|G1
 +
  |-
 +
  | bgcolor="green"|G2 || bgcolor="red"|B  || bgcolor="green"|G2 || bgcolor="red"|B
 +
  |-
 +
  | bgcolor="blue"|R    || bgcolor="green"|G1 || bgcolor="blue"|R    || bgcolor="green"|G1
 +
  |}
 +
|}
 +
 
 +
'''Notes:'''
 +
* All kinds of Bayer patterns can be derived from initial RGGB by flipping on X and/or Y.
 +
* Some sensors have possibility to set independent gains for G1 and G2. For small optical formats (example: 1/2.5") the accessible optics has lower resolution than an image sensor - this [[JP4_HDR|can be]] used to increase the dynamic range of a image sensor without losing sharpness.
 +
 
 +
==<font color="blue">Demosaicing/debayering links</font>==
  
 
[http://scien.stanford.edu/class/psych221/projects/99/tingchen A Study of Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras. Ting Chen / Stanford University]
 
[http://scien.stanford.edu/class/psych221/projects/99/tingchen A Study of Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras. Ting Chen / Stanford University]
Line 232: Line 370:
  
 
Example files:
 
Example files:
* [http://community.elphel.com/files/jp4/example_JP4.jpeg Example JP4]
+
* [http://community.elphel.com/files/jp4/example_JP46.jpeg Example JP46]
 
* [http://community.elphel.com/files/jp4/example_flipped.dng Example DNG]
 
* [http://community.elphel.com/files/jp4/example_flipped.dng Example DNG]
 
* [http://community.elphel.com/files/jp4/example_converted.jpg Example JPG (converted)]
 
* [http://community.elphel.com/files/jp4/example_converted.jpg Example JPG (converted)]
 
  
 
See also:
 
See also:
Line 241: Line 378:
 
* [[Demosaic on client side]]
 
* [[Demosaic on client side]]
 
* [http://linuxdevices.com/articles/AT4187053130.html Elphel camera under the hood: from Verilog to PHP - on LinuxDevices.com]
 
* [http://linuxdevices.com/articles/AT4187053130.html Elphel camera under the hood: from Verilog to PHP - on LinuxDevices.com]
 +
* [http://pillow.readthedocs.io/en/3.4.x/handbook/writing-your-own-file-decoder.html Python PIL Writing Your Own File Decoder]
 +
* [[JP4 Compatible Software]]

Latest revision as of 18:00, 9 February 2018

JP4 Format

The goal of the JP4 format is to preserve as much as possible of the information from the image sensor (ideally "raw") while providing reasonable compression. Based on JPEG compression, the format was originally developed at Elphel for accurate representation of the scanned book pages and later used in other applications that involve post-processing.

The standard JPEG was developed to compress images so they visually look almost the same as uncompressed but are significantly smaller in size. The most damaging part of the color JPEG compression is the demosaic part of it. "Lossy" part of the compression (quantization after DCT) can be eliminated with setting compression quality to 100% that effectively disables quantization. Dynamic range compression ("gamma conversion") is usually designed to be lower than the sensor shot noise - significant for most modern small-pixel (and so low full-well capacity) sensors. That leaves the demosaic ("guessing" the missing colors in Bayer mosaic sensor outputs) a single most damaging operation. In the camera this is usually done by processing data in 3x3 or 5x5 pixel blocks and it is difficult to distinguish between variations (especially sharp) in intensity and the color tone. There are much more advanced algorithms that can be used during post-processing that will provide much better results, but they require "raw" sensor data. Or - the data saved in JP4 format.

Here is a JP4 Demo that illustrates this algorithm and how it is related to standard JPEG.

JP4 mode bypasses the demosaic/color conversion in the FPGA and provides an image with pixels in each 16x16 macroblock that are rearranged to separate Bayer colors in individual 8x8 blocks, then encoded as monochrome. Demosaic will be applied during post-processing on the host PC. This section describe different algorithms and implementations used to provide this functionality.

Features

Compared to JPEG:

  • Higher compression speed
  • Better image quality (close to RAW)
  • Pixel data is not modified by demosaicing algorithm
  • Smaller file size

Format variations

JP4 family:

Name Code Description
jp46 2 original JP4 (from 7.X software in 10353s), encoded as 4:2:0 with zeroed color components
jp46dc 3 modified jp46 so each color component uses individual DC differential encoding
- 4 reserved for color with 5x5 conversion (not yet implemented)
jp4 5 4:0:0, omitted color components
jp4dc 6 similar to jp46dc encoded as 4:0:0
jp4diff 7 differential where (R-G), G, (G2-G) and (B-G) components are encoded as 4:0:0
jp4hdr 8 (R-G), G, G2,(B-G) are encoded so G2 can be used with high gain
jp4fiff2 9 (R-G)/2, G,(G2-G)/2, (B-G)/2 to avoid possible overflow in compressed values
jp4hdr2 10 (R-G)/2, G,G2,(B-G)/2

Other supported formats:

Name Code Description
color 0 YCbCr 4:2:0, 3x3 pixels
mono6 1 monochrome - color YCbCr 4:2:0 with zeroed out color components
mono 14 monochrome - color YCbCr 4:0:0 with omitted color components

Notes:

  • The format code is written (along with other info) to the MakerNote in EXIF data

JP4

Example file

Download sample
JP4 opened as JPEG, rotated 90°
JP4 processed, rotated 90°
  • If opened as a normal image the macro blocks will be displayed as 8x32.

Online

  • Online JP4 viewer
    • click & select or drag & drop
    • it's a quick view - the result is scaled down
    • image format is read from exif (using EXIF.js)
    • can view jp46
    • can view regular jpegs
    • images are decoded in the browser, no uploading, no communication with the server

PC

ImageJ plugin

  • Install ImageJ plugins
  • run: Plugins > JP46 reader camera
    • can display JP4/JP46 as monochrome w/o demosaicing
    • can split color channels

html canvas + javascript

python

  • 0.5s for 2592x1936 GRBG image

import numpy as np
from PIL import Image
import scipy.misc

# first open image in grayscale
I = scipy.misc.imread(filename, flatten=False, mode='L')

H,W = I.shape

# 16x16 block
block  = np.zeros((16,16))
# 16xW
stripe = np.zeros((16,W))

I = np.reshape(I,(H/16,16,W))

# print(I.shape)
# (1936, 2592) -> (121, 16, 2592)

for i in range(I.shape[0]):
  # stripe 16xW
  stripe = np.copy(I[i])
  for j in range(0,W,16):
    if j<W/2:
      k = 0
      l = 2*j
    else:
      k = 8
      l = 2*(j-W/2)

      # gr r b gb
      block[0::2,0::2] = stripe[k:k+8,l+ 0:l+ 0+8]
      block[0::2,1::2] = stripe[k:k+8,l+ 8:l+ 8+8]
      block[1::2,0::2] = stripe[k:k+8,l+16:l+16+8]
      block[1::2,1::2] = stripe[k:k+8,l+24:l+24+8]

      I[i,0:16,j:j+16] = block

I = np.reshape(I,(H,W))
# pixels ordered as in bayer pattern, this needs demosaicing
cv2.imwrite("result.png",I)

JP46

JP46 opened as JPEG
Bayer CFA encoded, deblocked
Demosaiced

PC

ImageJ plugin

  • same as for JP4

Decoding in MATLAB

JP46 format can be easy manipulated in MATLAB

  • Read image:
I=imread('hdr02.jp46'); %read JP46 file like JPEG
  ,or online grab image from http like this:
I=imread('http://community.elphel.com/pictures/jp46.jpg');
  ,or cam:
I=imread('http://cam_ip/bimg'); %get online buffered image from cam
I=I(:,:,1);            %strip color data
  • Remove block grouping:

II=deblock16x16(I);    %deblock image
%file deblock16x16.m
function y=deblock16x16(I)
y0=uint8(zeros(size(I)));
for x=1:16:size(I,1)
  for y=1:16:size(I,2)
    blk16=I(x:x+15,y:y+15);
      for dx=0:7
        for dy=0:7
          y0(x+2*dx  ,y+2*dy)   = blk16(dx+1,dy+1);
          y0(x+2*dx+1,y+2*dy)   = blk16(dx+9,dy+1);
          y0(x+2*dx  ,y+2*dy+1) = blk16(dx+1,dy+9);
          y0(x+2*dx+1,y+2*dy+1) = blk16(dx+9,dy+9);
        end
      end
    end
  end
y=y0;

  • Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)
J=demosaic(II,'gbrg');
  • Display image:
imshow(J);

Convert to DNG

Movie2dng

See Movie2dng for conversion of JP4/JP46 files or *.movs to DNG format.

LibTIFF

Credits: Dave Coffin

This Linux command line tool allows conversion of JP46 files into a DNGs that dcraw and Adobe Photoshop can open.

Download LibTIFF v3.8.2

  • extract the tar.gz (this should create a new folder called "tiff-3.8.2")

Apply this patch: in terminal (first cd to path of libtiff.patch):

patch -p0 < libtiff.patch  

build LibTIFF:

cd tiff-3.8.2
./configure
make
sudo make install

Then compile this C program with:

gcc -o elphel_dng elphel_dng.c -ltiff -Wl,--rpath=/usr/local/lib

With Ubuntu 9.04 (and later) its possible that the wrong libtiff is selected automatically which results in a error like this when using the compiled application:

TIFFSetField: test.dng: Unknown tag 33421.
TIFFSetField: test.dng: Unknown tag 33422.
Segmentation faul

To solve that problem compile with this line forcing a specific libtiff version

gcc -o elphel_dng elphel_dng.c -lm /usr/local/lib/libtiff.so.3.8.2 -Wl,--rpath=/usr/local/lib

Then use the created application:

Usage: ./elphel_dng "gamma" "input.jpg" "output.dng"
Example: ./elphel_dng 100 example_JP4.jpeg example.dng

JP46 video stream decoding using MPlayer

JP46 stream can be decoded by MPlayer. Use this [this patch]

 patch . -p0 < debayer.diff in the mplayer source dir

Or download win32 binaries from sourceforge.

usage example: mplayer.exe test.avi -vf demosaic=deblock=1:method=7:pattern=3  -vo gl
mencoder example: mencoder.exe test.avi -ovc lavc -lavcopts vcodec=mjpeg -o output.avi -vf demosaic=deblock=1:method=1,scale
Debayer (Demosaic) algorithm variants provided by libdc1394:
- Nearest Neighbor : OpenCV library
- Bilinear         : OpenCV library
- HQLinear         : High-Quality Linear Interpolation For Demosaicing Of Bayer-Patterned
                     Color Images, by Henrique S. Malvar, Li-wei He, and Ross Cutler,    
                        in Proceedings of the ICASSP'04 Conference.                      
- Edge Sense II    : Laroche, Claude A. "Apparatus and method for adaptively interpolating
                     a full color image utilizing chrominance gradients"                  
                        U.S. Patent 5,373,322. Based on the code found on the website     
                     http://www-ise.stanford.edu/~tingchen/ Converted to C and adapted to 
                     all four elementary patterns.                                        
- Downsample       : "Known to the Ancients"                                              
- Simple           : Implemented from the information found in the manual of Allied Vision
                     Technologies (AVT) cameras.                                          
- VNG              : Variable Number of Gradients, a method described in                  
                     http://www-ise.stanford.edu/~tingchen/algodep/vargra.html            
                     Sources import from DCRAW by Frederic Devernay. DCRAW is a RAW       
                     converter program by Dave Coffin. URL:                               
                     http://www.cybercom.net/~dcoffin/dcraw/                              
- AHD              : Adaptive Homogeneity-Directed Demosaicing Algorithm, by K. Hirakawa  
                     and T.W. Parks, IEEE Transactions on Image Processing, Vol. 14, Nr. 3,
                     March 2005, pp. 360 - 369.
Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]

Avisynth plugin for JP46 processing

Avisynth plugin also available

AVS script example:

LoadCPlugin("jp4.dll")
DirectShowSource("test.avi")
JP4("AHD","RGGB")

GStreamer plugins for Elphel JP4 image and video processing 

This project supported by http://ubicast.eu hosts Elphel related gstreamer components, so far:

  • the jp462bayer plugin converts color and monochrome JP46 Elphel bitstreams to Bayer raw format. In the future, it might support other JP4 modes (JP4, JP4-HDR, ...)
  • bayer2rgb2 converts raw Bayer streams to RGB images
jp462bayer: JP46 to Bayer

After jpegdec, re-arranges the pixels in Bayer format.

bayer2rgb2: debayer

It offers the same features as the legacy bayer2rgb, but by wrapping Libdc1394's debayering algorithms you can choose the interpoloation algorithm between : simple, bilinear, hqlinear, downsample, edgesense, vng, ahd, nearest

Example pipelines
gst-launch-0.10 rtspsrc location=rtsp://elphel:554 protocols=0x00000001 ! rtpjpegdepay ! jpegdec ! \
queue ! jp462bayer ! queue ! bayer2rgb2 ! queue ! ffmpegcolorspace ! videorate ! "video/x-raw-yuv, \
format=(fourcc)I420, width=(int)1920, height=(int)1088, framerate=(fraction)25/1" ! xvimagesink sync=false max-lateness=-1

JP4 HDR

RGGB
R G1 R G1
G2 B G2 B
R G1 R G1
G2 B G2 B
BGGR
R G1 R G1
G2 B G2 B
R G1 R G1
G2 B G2 B
GBRG
G2 B G2 B
R G1 R G1
G2 B G2 B
R G1 R G1
GRBG
G2 B G2 B
R G1 R G1
G2 B G2 B
R G1 R G1

Notes:

  • All kinds of Bayer patterns can be derived from initial RGGB by flipping on X and/or Y.
  • Some sensors have possibility to set independent gains for G1 and G2. For small optical formats (example: 1/2.5") the accessible optics has lower resolution than an image sensor - this can be used to increase the dynamic range of a image sensor without losing sharpness.

Demosaicing/debayering links

A Study of Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras. Ting Chen / Stanford University

Source code:

eLynx Image Processing SDK and Lab
libdc1394
Efficient, high-quality Bayer demosaic filtering on GPUs
http://svn2.assembla.com/svn/ge/libgedrawing/trunk/src/ImageBayer.cpp

Example files:

See also: