JP4
Contents
- 1 JP4 Format
- 2 Features
- 3 Format variations
- 4 JP4
- 5 JP46
- 6 JP4 HDR
- 7 Demosaicing/debayering links
JP4 Format
The goal of the JP4 format is to preserve as much as possible of the information from the image sensor (ideally "raw") while providing reasonable compression. Based on JPEG compression, the format was originally developed at Elphel for accurate representation of the scanned book pages and later used in other applications that involve post-processing.
The standard JPEG was developed to compress images so they visually look almost the same as uncompressed but are significantly smaller in size. The most damaging part of the color JPEG compression is the demosaic part of it. "Lossy" part of the compression (quantization after DCT) can be eliminated with setting compression quality to 100% that effectively disables quantization. Dynamic range compression ("gamma conversion") is usually designed to be lower than the sensor shot noise - significant for most modern small-pixel (and so low full-well capacity) sensors. That leaves the demosaic ("guessing" the missing colors in Bayer mosaic sensor outputs) a single most damaging operation. In the camera this is usually done by processing data in 3x3 or 5x5 pixel blocks and it is difficult to distinguish between variations (especially sharp) in intensity and the color tone. There are much more advanced algorithms that can be used during post-processing that will provide much better results, but they require "raw" sensor data. Or - the data saved in JP4 format.
Here is a JP4 Demo that illustrates this algorithm and how it is related to standard JPEG.
JP4 mode bypasses the demosaic/color conversion in the FPGA and provides an image with pixels in each 16x16 macroblock that are rearranged to separate Bayer colors in individual 8x8 blocks, then encoded as monochrome. Demosaic will be applied during post-processing on the host PC. This section describe different algorithms and implementations used to provide this functionality.
Features
Compared to JPEG:
- Higher compression speed
- Better image quality (close to RAW)
- Pixel data is not modified by demosaicing algorithm
- Smaller file size
Format variations
JP4 family:
| Name | Code | Description |
|---|---|---|
| jp46 | 2 | original JP4 (from 7.X software in 10353s), encoded as 4:2:0 with zeroed color components |
| jp46dc | 3 | modified jp46 so each color component uses individual DC differential encoding |
| - | 4 | reserved for color with 5x5 conversion (not yet implemented) |
| jp4 | 5 | 4:0:0, omitted color components |
| jp4dc | 6 | similar to jp46dc encoded as 4:0:0 |
| jp4diff | 7 | differential where (R-G), G, (G2-G) and (B-G) components are encoded as 4:0:0 |
| jp4hdr | 8 | (R-G), G, G2,(B-G) are encoded so G2 can be used with high gain |
| jp4fiff2 | 9 | (R-G)/2, G,(G2-G)/2, (B-G)/2 to avoid possible overflow in compressed values |
| jp4hdr2 | 10 | (R-G)/2, G,G2,(B-G)/2 |
Other supported formats:
| Name | Code | Description |
|---|---|---|
| color | 0 | YCbCr 4:2:0, 3x3 pixels |
| mono6 | 1 | monochrome - color YCbCr 4:2:0 with zeroed out color components |
| mono | 14 | monochrome - color YCbCr 4:0:0 with omitted color components |
Notes:
- The format code is written (along with other info) to the MakerNote in EXIF data
JP4
- If opened as a normal image the macro blocks will be displayed as 8x32.
PC
ImageJ plugin
- Install ImageJ plugins
- run: Plugins > JP46 reader camera
- can display JP4/JP46 as monochrome w/o demosaicing
- can split color channels
html canvas + javascript
python
...
# in JP4 format the 16x16 block is 8x32 (GRBG)
# the 1st line of 8x32 blocks is the left half of the image
# the 2nd line of 8x32 blocks is the right half
# vertical step = 16px
for y in range(0,self.h,16):
for x in range(0,self.w,16):
bx = x if x<self.w/2 else x-self.w/2
by = y if x<self.w/2 else y+8
block8x8_gr = self.px[by:by+8,2*bx+0 :2*bx+ 8]
block8x8_r = self.px[by:by+8,2*bx+8 :2*bx+16]
block8x8_b = self.px[by:by+8,2*bx+16:2*bx+24]
block8x8_gb = self.px[by:by+8,2*bx+24:2*bx+32]
for dy in range(8):
for dx in range(8):
#OpenCV uses BGR format
output[y+2*dy+0,x+2*dx+0]=[0,block8x8_gr[dy,dx][0],0]
output[y+2*dy+0,x+2*dx+1]=[0,0,block8x8_r[dy,dx][0]]
output[y+2*dy+1,x+2*dx+0]=[block8x8_b[dy,dx][0],0,0]
output[y+2*dy+1,x+2*dx+1]=[0,block8x8_gb[dy,dx][0],0]
...
JP46
PC
ImageJ plugin
- same as for JP4
Decoding in MATLAB
JP46 format can be easy manipulated in MATLAB
- Read image:
I=imread('hdr02.jp46'); %read JP46 file like JPEG
,or online grab image from http like this:
I=imread('http://community.elphel.com/pictures/jp46.jpg');
,or cam:
I=imread('http://cam_ip/bimg'); %get online buffered image from cam
I=I(:,:,1); %strip color data
- Remove block grouping:
II=deblock16x16(I); %deblock image
%file deblock16x16.m
function y=deblock16x16(I)
y0=uint8(zeros(size(I)));
for x=1:16:size(I,1)
for y=1:16:size(I,2)
blk16=I(x:x+15,y:y+15);
for dx=0:7
for dy=0:7
y0(x+2*dx ,y+2*dy) = blk16(dx+1,dy+1);
y0(x+2*dx+1,y+2*dy) = blk16(dx+9,dy+1);
y0(x+2*dx ,y+2*dy+1) = blk16(dx+1,dy+9);
y0(x+2*dx+1,y+2*dy+1) = blk16(dx+9,dy+9);
end
end
end
end
y=y0;
- Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)
J=demosaic(II,'gbrg');
- Display image:
imshow(J);
Convert to DNG
Movie2dng
See Movie2dng for conversion of JP4/JP46 files or *.movs to DNG format.
LibTIFF
Credits: Dave Coffin
This Linux command line tool allows conversion of JP46 files into a DNGs that dcraw and Adobe Photoshop can open.
Download LibTIFF v3.8.2
- extract the tar.gz (this should create a new folder called "tiff-3.8.2")
Apply this patch: in terminal (first cd to path of libtiff.patch):
patch -p0 < libtiff.patch
build LibTIFF:
cd tiff-3.8.2 ./configure make sudo make install
Then compile this C program with:
gcc -o elphel_dng elphel_dng.c -ltiff -Wl,--rpath=/usr/local/lib
With Ubuntu 9.04 (and later) its possible that the wrong libtiff is selected automatically which results in a error like this when using the compiled application:
TIFFSetField: test.dng: Unknown tag 33421. TIFFSetField: test.dng: Unknown tag 33422. Segmentation faul
To solve that problem compile with this line forcing a specific libtiff version
gcc -o elphel_dng elphel_dng.c -lm /usr/local/lib/libtiff.so.3.8.2 -Wl,--rpath=/usr/local/lib
Then use the created application:
Usage: ./elphel_dng "gamma" "input.jpg" "output.dng" Example: ./elphel_dng 100 example_JP4.jpeg example.dng
JP46 video stream decoding using MPlayer
JP46 stream can be decoded by MPlayer. Use this [this patch]
patch . -p0 < debayer.diff in the mplayer source dir
Or download win32 binaries from sourceforge.
usage example: mplayer.exe test.avi -vf demosaic=deblock=1:method=7:pattern=3 -vo gl mencoder example: mencoder.exe test.avi -ovc lavc -lavcopts vcodec=mjpeg -o output.avi -vf demosaic=deblock=1:method=1,scale
Debayer (Demosaic) algorithm variants provided by libdc1394: - Nearest Neighbor : OpenCV library - Bilinear : OpenCV library - HQLinear : High-Quality Linear Interpolation For Demosaicing Of Bayer-Patterned Color Images, by Henrique S. Malvar, Li-wei He, and Ross Cutler, in Proceedings of the ICASSP'04 Conference. - Edge Sense II : Laroche, Claude A. "Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients" U.S. Patent 5,373,322. Based on the code found on the website http://www-ise.stanford.edu/~tingchen/ Converted to C and adapted to all four elementary patterns. - Downsample : "Known to the Ancients" - Simple : Implemented from the information found in the manual of Allied Vision Technologies (AVT) cameras. - VNG : Variable Number of Gradients, a method described in http://www-ise.stanford.edu/~tingchen/algodep/vargra.html Sources import from DCRAW by Frederic Devernay. DCRAW is a RAW converter program by Dave Coffin. URL: http://www.cybercom.net/~dcoffin/dcraw/ - AHD : Adaptive Homogeneity-Directed Demosaicing Algorithm, by K. Hirakawa and T.W. Parks, IEEE Transactions on Image Processing, Vol. 14, Nr. 3, March 2005, pp. 360 - 369.
Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]
Avisynth plugin for JP46 processing
Avisynth plugin also available
AVS script example:
LoadCPlugin("jp4.dll")
DirectShowSource("test.avi")
JP4("AHD","RGGB")
GStreamer plugins for Elphel JP4 image and video processing
This project supported by http://ubicast.eu hosts Elphel related gstreamer components, so far:
- the jp462bayer plugin converts color and monochrome JP46 Elphel bitstreams to Bayer raw format. In the future, it might support other JP4 modes (JP4, JP4-HDR, ...)
- bayer2rgb2 converts raw Bayer streams to RGB images
jp462bayer: JP46 to Bayer
After jpegdec, re-arranges the pixels in Bayer format.
bayer2rgb2: debayer
It offers the same features as the legacy bayer2rgb, but by wrapping Libdc1394's debayering algorithms you can choose the interpoloation algorithm between : simple, bilinear, hqlinear, downsample, edgesense, vng, ahd, nearest
Example pipelines
gst-launch-0.10 rtspsrc location=rtsp://elphel:554 protocols=0x00000001 ! rtpjpegdepay ! jpegdec ! \ queue ! jp462bayer ! queue ! bayer2rgb2 ! queue ! ffmpegcolorspace ! videorate ! "video/x-raw-yuv, \ format=(fourcc)I420, width=(int)1920, height=(int)1088, framerate=(fraction)25/1" ! xvimagesink sync=false max-lateness=-1
JP4 HDR
- More info
- The most common Bayer patterns:
|
|
|
|
Notes:
- All kinds of Bayer patterns can be derived from initial RGGB by flipping on X and/or Y.
- Some sensors have possibility to set independent gains for G1 and G2. For small optical formats (example: 1/2.5") the accessible optics has lower resolution than an image sensor - this can be used to increase the dynamic range of a image sensor without losing sharpness.
Demosaicing/debayering links
Source code:
eLynx Image Processing SDK and Lab libdc1394 Efficient, high-quality Bayer demosaic filtering on GPUs http://svn2.assembla.com/svn/ge/libgedrawing/trunk/src/ImageBayer.cpp
Example files:
For more colorful examples please visit http://cinema.elphel.com/still-images
See also:
