From ElphelWiki

Jump to: navigation, search


JP4 Format

JP4 format is based on JPEG compression, it was originally developed at Elphel for accurate representation of the scanned book pages and later used in other applications that involve post-processing. This format is supported in both NC353 series and current NC393 series cameras. The standard JPEG was developed to compress images so they visually are almost the same as uncompressed but are significantly smaller in size. The goal of JP4 is camera-centric, not human-centric. It tries to preserve as much as possible of the information from the camera sensor (ideally "raw") while providing reasonable compression. The most damaging part of the color JPEG compression is the de-mosaic part of it. "Lossy" part of the compression (quantization after DCT) can be eliminated with setting compression quality to 100% that effectively disables quantization. Dynamic range compression ("gamma conversion") is usually designed to be lower than the sensor shot noise - significant for most modern small-pixel (and so low full-well capacity) sensors. That leaves the demosaic ("guessing" the missing colors in Bayer mosaic sensor outputs) a single most damaging operation. In the camera this is usually done by processing data in 3x3 or 5x5 pixel blocks and it is difficult to distinguish between variations (especially sharp) in intensity and the color tone. There are much more advanced algorithms that can be used during post-processing that will provide much better results, but they require "raw" sensor data. Or - the data saved in JP4 format.

Here is a JP4 Demo that illustrates this algorithm and how it is related to standard JPEG.

JP4 mode bypasses the demosaic/color converiosn in the FPGA and provides an image with pixels in each 16x16 macroblock that are rearranged to separate Bayer colors in individual 8x8 blocks, then encoded as monochrome. Demosaic will be applied during post-processing on the host PC. This section describe different algorithms and implementations used to provide this functionality.

Main goals:

Different JP4 Modes in 8.X Software

only modes 0-2 can be processed with standard libjpeg:


Bayer pattern look like this

R G1 R G1
G2 B G2 B
R G1 R G1
G2 B G2 B
R G1 R G1
G2 B G2 B
R G1 R G1
G2 B G2 B
G2 B G2 B
R G1 R G1
G2 B G2 B
R G1 R G1
G2 B G2 B
R G1 R G1
G2 B G2 B
R G1 R G1

The remark: all kinds of bayer patterns can be received from initial RGGB by flipping on X and/or Y.

Some sensors have possibility to set independed scale to G1 and G2. Considering that the accessible optics does not give the full permission of a sensor resolution, it can be used for increase in a dynamic range of a image sensor.

JP46 processing on the host

JP46 image decoding in MATLAB

JP4 format can be easy manipulated by MATLAB
JP4 image

1. Read image

I=imread('hdr02.jp4'); %read JP4 file like JPEG
  ,or online grab image from http like this:
  ,or cam:
I=imread('http://cam_ip/bimg'); %get online buffered image from cam
I=I(:,:,1);            %strip color data
2. Remove block grouping
Bayer CFA encoded image

II=deblock16x16(I);    %deblock image
%file deblock16x16.m
function y=deblock16x16(I)
for x=1:16:size(I,1)
  for y=1:16:size(I,2)
      for dx=0:7
        for dy=0:7
          y0(x+2*dx  ,y+2*dy)   = blk16(dx+1,dy+1);
          y0(x+2*dx+1,y+2*dy)   = blk16(dx+9,dy+1);
          y0(x+2*dx  ,y+2*dy+1) = blk16(dx+1,dy+9);
          y0(x+2*dx+1,y+2*dy+1) = blk16(dx+9,dy+9);

2. Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)
Decoded image

3. Show image


JP46 to DNG image conversion


See Movie2dng for conversion of JP4 movies to DNG.

still frames

Credits: Dave Coffin

This Linux command line tool allows conversion of JP4 files into a DNGs that dcraw and Adobe Photoshop can open.

Download LibTIFF v3.8.2

Apply this patch: in terminal (first cd to path of libtiff.patch):

patch -p0 < libtiff.patch  

build LibTIFF:

cd tiff-3.8.2
sudo make install

Then compile this C program with:

gcc -o elphel_dng elphel_dng.c -ltiff -Wl,--rpath=/usr/local/lib

With Ubuntu 9.04 (and later) its possible that the wrong libtiff is selected automatically which results in a error like this when using the compiled application:

TIFFSetField: test.dng: Unknown tag 33421.
TIFFSetField: test.dng: Unknown tag 33422.
Segmentation faul

To solve that problem compile with this line forcing a specific libtiff version

gcc -o elphel_dng elphel_dng.c -lm /usr/local/lib/ -Wl,--rpath=/usr/local/lib

Then use the created application:

Usage: ./elphel_dng "gamma" "input.jpg" "output.dng"
Example: ./elphel_dng 100 example_JP4.jpeg example.dng

JP46 video stream decoding using MPlayer

JP46 stream can be decoded by MPlayer. Use this [this patch]

 patch . -p0 < debayer.diff in the mplayer source dir

Or download win32 binaries from sourceforge.

usage example: mplayer.exe test.avi -vf demosaic=deblock=1:method=7:pattern=3  -vo gl
mencoder example: mencoder.exe test.avi -ovc lavc -lavcopts vcodec=mjpeg -o output.avi -vf demosaic=deblock=1:method=1,scale
Debayer (Demosaic) algorithm variants provided by libdc1394:
- Nearest Neighbor : OpenCV library
- Bilinear         : OpenCV library
- HQLinear         : High-Quality Linear Interpolation For Demosaicing Of Bayer-Patterned
                     Color Images, by Henrique S. Malvar, Li-wei He, and Ross Cutler,    
                        in Proceedings of the ICASSP'04 Conference.                      
- Edge Sense II    : Laroche, Claude A. "Apparatus and method for adaptively interpolating
                     a full color image utilizing chrominance gradients"                  
                        U.S. Patent 5,373,322. Based on the code found on the website     
            Converted to C and adapted to 
                     all four elementary patterns.                                        
- Downsample       : "Known to the Ancients"                                              
- Simple           : Implemented from the information found in the manual of Allied Vision
                     Technologies (AVT) cameras.                                          
- VNG              : Variable Number of Gradients, a method described in                  
                     Sources import from DCRAW by Frederic Devernay. DCRAW is a RAW       
                     converter program by Dave Coffin. URL:                               
- AHD              : Adaptive Homogeneity-Directed Demosaicing Algorithm, by K. Hirakawa  
                     and T.W. Parks, IEEE Transactions on Image Processing, Vol. 14, Nr. 3,
                     March 2005, pp. 360 - 369.
Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]

Avisynth plugin for JP46 processing

Avisynth plugin also available

AVS script example:


GStreamer plugins for Elphel JP4 image and video processing 

This project supported by hosts Elphel related gstreamer components, so far:

jp462bayer: JP4 to Bayer

After jpegdec, re-arranges the pixels in Bayer format.

bayer2rgb2: debayer

It offers the same features as the legacy bayer2rgb, but by wrapping Libdc1394's debayering algorithms you can choose the interpoloation algorithm between : simple, bilinear, hqlinear, downsample, edgesense, vng, ahd, nearest

Example pipelines

gst-launch-0.10 rtspsrc location=rtsp://elphel:554 protocols=0x00000001 ! rtpjpegdepay ! jpegdec ! \
queue ! jp462bayer ! queue ! bayer2rgb2 ! queue ! ffmpegcolorspace ! videorate ! "video/x-raw-yuv, \
format=(fourcc)I420, width=(int)1920, height=(int)1088, framerate=(fraction)25/1" ! xvimagesink sync=false max-lateness=-1

Demosaicing/debayering links

A Study of Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras. Ting Chen / Stanford University

Source code:

eLynx Image Processing SDK and Lab
Efficient, high-quality Bayer demosaic filtering on GPUs

Example files:

For more colorful examples please visit

See also:

Personal tools