Difference between revisions of "JP4"

From ElphelWiki
Jump to: navigation, search
(Stream decoding)
Line 9: Line 9:
 
  - possibility to obtain more high quality image (near to RAW)
 
  - possibility to obtain more high quality image (near to RAW)
 
  - drasticaly lowering data size
 
  - drasticaly lowering data size
 
 
== Different JP4 Modes in 8.X Software ==
 
== Different JP4 Modes in 8.X Software ==
 
only modes 0-2 can be processed with standard libjpeg:                 
 
only modes 0-2 can be processed with standard libjpeg:                 
Line 25: Line 24:
 
*14 - mono,  monochrome with ommitted color components (4:0:0)
 
*14 - mono,  monochrome with ommitted color components (4:0:0)
  
== Image decoding ==
 
JP4 format can be easy manipulated by [http://www.mathworks.com/matlabcentral/fileexchange/22144 MATLAB] [[Image:Fruits_jp4.jpg|thumb|JP4 image]]
 
 
1. Read image
 
I=imread('hdr02.jp4'); %read JP4 file like JPEG
 
  ,or online grab image from http like this:
 
I=imread('http://community.elphel.com/pictures/jp4.jpg');
 
  ,or cam:
 
I=imread('http://cam_ip/bimg'); %get online buffered image from cam
 
 
I=I(:,:,1);            %strip color data
 
2. Remove block grouping[[Image:Fruits_jp4_deblocked.jpg|thumb|Bayer CFA encoded image]]
 
<code matlab>
 
II=deblock16x16(I);    %deblock image
 
 
%file deblock16x16.m
 
function y=deblock16x16(I)
 
y0=uint8(zeros(size(I)));
 
for x=1:16:size(I,1)
 
  for y=1:16:size(I,2)
 
    blk16=I(x:x+15,y:y+15);
 
      for dx=0:7
 
        for dy=0:7
 
          y0(x+2*dx  ,y+2*dy)  = blk16(dx+1,dy+1);
 
          y0(x+2*dx+1,y+2*dy)  = blk16(dx+9,dy+1);
 
          y0(x+2*dx  ,y+2*dy+1) = blk16(dx+1,dy+9);
 
          y0(x+2*dx+1,y+2*dy+1) = blk16(dx+9,dy+9);
 
        end
 
      end
 
    end
 
  end
 
y=y0;
 
</code>
 
2. Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)[[Image:Fruits_jp4_debayered.jpg|thumb|Decoded image]]
 
J=demosaic(II,'gbrg');
 
3. Show image
 
imshow(J);
 
 
== Stream decoding ==
 
JP4 stream can be decoded by mplayer.
 
Use this [[http://community.elphel.com/files/mplayer/debayer.diff this patch]]
 
  patch . -p0 < debayer.diff in the mplayer source dir
 
 
Or download win32 binaries from sourceforge.
 
usage example: mplayer.exe test.avi -vf demosaic=deblock=1:method=7:pattern=3  -vo gl
 
mencoder example: mencoder.exe test.avi -ovc lavc -lavcopts vcodec=mjpeg -o output.avi -vf demosaic=deblock=1:method=1,scale
 
  
Debayer ([[Demosaic_on_client_side|Demosaic]]) algorithm variants provided by libdc1394:
+
=== [[JP4 HDR]] ===
- Nearest Neighbor : OpenCV library
 
- Bilinear        : OpenCV library
 
- HQLinear        : High-Quality Linear Interpolation For Demosaicing Of Bayer-Patterned
 
                      Color Images, by Henrique S. Malvar, Li-wei He, and Ross Cutler,   
 
                        in Proceedings of the ICASSP'04 Conference.                     
 
- Edge Sense II    : Laroche, Claude A. "Apparatus and method for adaptively interpolating
 
                      a full color image utilizing chrominance gradients"                 
 
                        U.S. Patent 5,373,322. Based on the code found on the website   
 
                      http://www-ise.stanford.edu/~tingchen/ Converted to C and adapted to
 
                      all four elementary patterns.                                       
 
- Downsample      : "Known to the Ancients"                                             
 
- Simple          : Implemented from the information found in the manual of Allied Vision
 
                      Technologies (AVT) cameras.                                         
 
- VNG              : Variable Number of Gradients, a method described in                 
 
                      http://www-ise.stanford.edu/~tingchen/algodep/vargra.html           
 
                      Sources import from DCRAW by Frederic Devernay. DCRAW is a RAW     
 
                      converter program by Dave Coffin. URL:                             
 
                      http://www.cybercom.net/~dcoffin/dcraw/                             
 
- AHD              : Adaptive Homogeneity-Directed Demosaicing Algorithm, by K. Hirakawa 
 
                      and T.W. Parks, IEEE Transactions on Image Processing, Vol. 14, Nr. 3,
 
                      March 2005, pp. 360 - 369.
 
 
 
Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]
 
 
 
== [[JP4 HDR]] ==
 
 
Bayer pattern look like this
 
Bayer pattern look like this
 
{| class="wikitable"  
 
{| class="wikitable"  
Line 156: Line 84:
 
Considering that the accessible optics does not give the full permission of a sensor resolution, it [[JP4_HDR|can be]] used for increase in a dynamic range of a image sensor.
 
Considering that the accessible optics does not give the full permission of a sensor resolution, it [[JP4_HDR|can be]] used for increase in a dynamic range of a image sensor.
  
== [http://avisynth.org/mediawiki/Main_Page Avisynth] plugin ==
+
== JP4 processing on the host ==
Avisynth plugin also available
 
  
AVS script example:
+
=== JP4 image decoding in MATLAB ===
LoadCPlugin("jp4.dll")
+
JP4 format can be easy manipulated by [http://www.mathworks.com/matlabcentral/fileexchange/22144 MATLAB] [[Image:Fruits_jp4.jpg|thumb|JP4 image]]
DirectShowSource("test.avi")
 
JP4("AHD","RGGB")
 
  
 +
1. Read image
 +
I=imread('hdr02.jp4'); %read JP4 file like JPEG
 +
  ,or online grab image from http like this:
 +
I=imread('http://community.elphel.com/pictures/jp4.jpg');
 +
  ,or cam:
 +
I=imread('http://cam_ip/bimg'); %get online buffered image from cam
  
== Demosaicing/debayering links ==
+
I=I(:,:,1);            %strip color data
 +
2. Remove block grouping[[Image:Fruits_jp4_deblocked.jpg|thumb|Bayer CFA encoded image]]
 +
<code matlab>
 +
II=deblock16x16(I);    %deblock image
  
  [http://scien.stanford.edu/class/psych221/projects/99/tingchen A Study of Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras. Ting Chen / Stanford University]
+
  %file deblock16x16.m
 
+
function y=deblock16x16(I)
Source code:
+
y0=uint8(zeros(size(I)));
[http://sourceforge.net/projects/elynx eLynx Image Processing SDK and Lab]
+
for x=1:16:size(I,1)
  [http://libdc1394.git.sourceforge.net/git/gitweb.cgi?p=libdc1394;a=blob;f=libdc1394/dc1394/bayer.c;hb=HEAD libdc1394]
+
  for y=1:16:size(I,2)
  [http://graphics.cs.williams.edu/papers/BayerJGT09 Efficient, high-quality Bayer demosaic filtering on GPUs]
+
    blk16=I(x:x+15,y:y+15);
  http://svn2.assembla.com/svn/ge/libgedrawing/trunk/src/ImageBayer.cpp
+
      for dx=0:7
 +
        for dy=0:7
 +
          y0(x+2*dx ,y+2*dy)  = blk16(dx+1,dy+1);
 +
          y0(x+2*dx+1,y+2*dy)  = blk16(dx+9,dy+1);
 +
          y0(x+2*dx  ,y+2*dy+1) = blk16(dx+1,dy+9);
 +
          y0(x+2*dx+1,y+2*dy+1) = blk16(dx+9,dy+9);
 +
        end
 +
      end
 +
    end
 +
  end
 +
  y=y0;
 +
</code>
 +
2. Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)[[Image:Fruits_jp4_debayered.jpg|thumb|Decoded image]]
 +
  J=demosaic(II,'gbrg');
 +
3. Show image
 +
imshow(J);
  
== JP4 to DNG conversion ==
+
=== JP4 to DNG image conversion ===
  
 
Credits: Dave Coffin
 
Credits: Dave Coffin
  
This Linux comanndline tool allows conversion of JP4 files into a DNGs that dcraw and Adobe Photoshop can open.
+
This Linux command line tool allows conversion of JP4 files into a DNGs that dcraw and Adobe Photoshop can open.
  
 
Download [http://community.elphel.com/files/jp4/tiff-3.8.2.tar.gz LibTIFF v3.8.2] a
 
Download [http://community.elphel.com/files/jp4/tiff-3.8.2.tar.gz LibTIFF v3.8.2] a
Line 201: Line 150:
 
  Usage: ./elphel_dng "gamma" "input.jpg" "output.dng"
 
  Usage: ./elphel_dng "gamma" "input.jpg" "output.dng"
 
  Example: ./elphel_dng 100 example_JP4.jpeg example.dng
 
  Example: ./elphel_dng 100 example_JP4.jpeg example.dng
 +
 +
=== JP4 video stream decoding using MPlayer ===
 +
JP4 stream can be decoded by MPlayer.
 +
Use this [[http://community.elphel.com/files/mplayer/debayer.diff this patch]]
 +
  patch . -p0 < debayer.diff in the mplayer source dir
 +
 +
Or download win32 binaries from sourceforge.
 +
usage example: mplayer.exe test.avi -vf demosaic=deblock=1:method=7:pattern=3  -vo gl
 +
mencoder example: mencoder.exe test.avi -ovc lavc -lavcopts vcodec=mjpeg -o output.avi -vf demosaic=deblock=1:method=1,scale
 +
 +
Debayer ([[Demosaic_on_client_side|Demosaic]]) algorithm variants provided by libdc1394:
 +
- Nearest Neighbor : OpenCV library
 +
- Bilinear        : OpenCV library
 +
- HQLinear        : High-Quality Linear Interpolation For Demosaicing Of Bayer-Patterned
 +
                      Color Images, by Henrique S. Malvar, Li-wei He, and Ross Cutler,   
 +
                        in Proceedings of the ICASSP'04 Conference.                     
 +
- Edge Sense II    : Laroche, Claude A. "Apparatus and method for adaptively interpolating
 +
                      a full color image utilizing chrominance gradients"                 
 +
                        U.S. Patent 5,373,322. Based on the code found on the website   
 +
                      http://www-ise.stanford.edu/~tingchen/ Converted to C and adapted to
 +
                      all four elementary patterns.                                       
 +
- Downsample      : "Known to the Ancients"                                             
 +
- Simple          : Implemented from the information found in the manual of Allied Vision
 +
                      Technologies (AVT) cameras.                                         
 +
- VNG              : Variable Number of Gradients, a method described in                 
 +
                      http://www-ise.stanford.edu/~tingchen/algodep/vargra.html           
 +
                      Sources import from DCRAW by Frederic Devernay. DCRAW is a RAW     
 +
                      converter program by Dave Coffin. URL:                             
 +
                      http://www.cybercom.net/~dcoffin/dcraw/                             
 +
- AHD              : Adaptive Homogeneity-Directed Demosaicing Algorithm, by K. Hirakawa 
 +
                      and T.W. Parks, IEEE Transactions on Image Processing, Vol. 14, Nr. 3,
 +
                      March 2005, pp. 360 - 369.
 +
 +
Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]
 +
 +
===  Avisynth plugin for JP4 processing ===
 +
[http://avisynth.org/mediawiki/Main_Page Avisynth plugin] also available
 +
 +
AVS script example:
 +
LoadCPlugin("jp4.dll")
 +
DirectShowSource("test.avi")
 +
JP4("AHD","RGGB")
 +
 +
 +
== Demosaicing/debayering links ==
 +
 +
[http://scien.stanford.edu/class/psych221/projects/99/tingchen A Study of Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras. Ting Chen / Stanford University]
 +
 +
Source code:
 +
[http://sourceforge.net/projects/elynx eLynx Image Processing SDK and Lab]
 +
[http://libdc1394.git.sourceforge.net/git/gitweb.cgi?p=libdc1394;a=blob;f=libdc1394/dc1394/bayer.c;hb=HEAD libdc1394]
 +
[http://graphics.cs.williams.edu/papers/BayerJGT09 Efficient, high-quality Bayer demosaic filtering on GPUs]
 +
http://svn2.assembla.com/svn/ge/libgedrawing/trunk/src/ImageBayer.cpp
  
 
Example files:
 
Example files:

Revision as of 04:54, 14 December 2009

Note: the JP4 mode described here is referred as "JP46" in current 8.0 firwmare

JP4 Format

We have added a special JP4 mode that bypasses the Demosaic in the FPGA and provides an image with pixels in each 16x16 macroblock that are rearranged to separate Bayer colors in individual 8x8 blocks, then encoded as monochrome. Demosaic will be applied during post-processing on the host PC. This section describe different algorithms and implementations used to provide this functionality.

Main goals:

- compression speed improvement
- possibility to obtain more high quality image (near to RAW)
- drasticaly lowering data size

Different JP4 Modes in 8.X Software

only modes 0-2 can be processed with standard libjpeg:

  • 0 - mono6, monochrome (color YCbCr 4:2:0 with zeroed out color componets)
  • 1 - color, YCbCr 4:2:0, 3x3 pixels
  • 2 - jp46 - original JP4 (from 7.X software), encoded as 4:2:0 with zeroed color components
  • 3 - jp46dc, modified jp46 so each color component uses individual DC diffenential encoding
  • 4 - reserved for color with 5x5 conversion (not yet implemented)
  • 5 - jp4 with ommitted color components (4:0:0)
  • 6 - jp4dc, similar to jp46dc encoded as 4:0:0
  • 7 - jp4diff, differential where (R-G), G, (G2-G) and (B-G) components are encoded as 4:0:0
  • 8 - jp4hdr, (R-G), G, G2,(B-G) are encoded so G2 can be used with high gain
  • 9 - jp4fiff2, (R-G)/2, G,(G2-G)/2, (B-G)/2 to avoid possible overflow in compressed values
  • 10 - jp4hdr2, (R-G)/2, G,G2,(B-G)/2
  • 14 - mono, monochrome with ommitted color components (4:0:0)


JP4 HDR

Bayer pattern look like this

RGGB
R G1 R G1
G2 B G2 B
R G1 R G1
G2 B G2 B
BGGR
R G1 R G1
G2 B G2 B
R G1 R G1
G2 B G2 B
GBRG
G2 B G2 B
R G1 R G1
G2 B G2 B
R G1 R G1
GRBG
G2 B G2 B
R G1 R G1
G2 B G2 B
R G1 R G1

The remark: all kinds of bayer patterns can be received from initial RGGB by flipping on X and/or Y.

Some sensors have possibility to set independed scale to G1 and G2. Considering that the accessible optics does not give the full permission of a sensor resolution, it can be used for increase in a dynamic range of a image sensor.

JP4 processing on the host

JP4 image decoding in MATLAB

JP4 format can be easy manipulated by MATLAB
JP4 image

1. Read image

I=imread('hdr02.jp4'); %read JP4 file like JPEG
  ,or online grab image from http like this:
I=imread('http://community.elphel.com/pictures/jp4.jpg');
  ,or cam:
I=imread('http://cam_ip/bimg'); %get online buffered image from cam
I=I(:,:,1);            %strip color data
2. Remove block grouping
Bayer CFA encoded image

II=deblock16x16(I);    %deblock image
%file deblock16x16.m
function y=deblock16x16(I)
y0=uint8(zeros(size(I)));
for x=1:16:size(I,1)
  for y=1:16:size(I,2)
    blk16=I(x:x+15,y:y+15);
      for dx=0:7
        for dy=0:7
          y0(x+2*dx  ,y+2*dy)   = blk16(dx+1,dy+1);
          y0(x+2*dx+1,y+2*dy)   = blk16(dx+9,dy+1);
          y0(x+2*dx  ,y+2*dy+1) = blk16(dx+1,dy+9);
          y0(x+2*dx+1,y+2*dy+1) = blk16(dx+9,dy+9);
        end
      end
    end
  end
y=y0;

2. Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)
Decoded image
J=demosaic(II,'gbrg');

3. Show image

imshow(J);

JP4 to DNG image conversion

Credits: Dave Coffin

This Linux command line tool allows conversion of JP4 files into a DNGs that dcraw and Adobe Photoshop can open.

Download LibTIFF v3.8.2 a

  • extract the tar.gz (this should create a new folder called "tiff-3.8.2")

Apply this patch: in terminal (first cd to path of libtiff.patch):

patch -p0 < libtiff.patch  

build LibTIFF:

cd tiff-3.8.2
./configure
make
sudo make install

Then compile this C program with:

gcc -o elphel_dng elphel_dng.c -ltiff

Then use the created application:

Usage: ./elphel_dng "gamma" "input.jpg" "output.dng"
Example: ./elphel_dng 100 example_JP4.jpeg example.dng

JP4 video stream decoding using MPlayer

JP4 stream can be decoded by MPlayer. Use this [this patch]

 patch . -p0 < debayer.diff in the mplayer source dir

Or download win32 binaries from sourceforge.

usage example: mplayer.exe test.avi -vf demosaic=deblock=1:method=7:pattern=3  -vo gl
mencoder example: mencoder.exe test.avi -ovc lavc -lavcopts vcodec=mjpeg -o output.avi -vf demosaic=deblock=1:method=1,scale
Debayer (Demosaic) algorithm variants provided by libdc1394:
- Nearest Neighbor : OpenCV library
- Bilinear         : OpenCV library
- HQLinear         : High-Quality Linear Interpolation For Demosaicing Of Bayer-Patterned
                     Color Images, by Henrique S. Malvar, Li-wei He, and Ross Cutler,    
                        in Proceedings of the ICASSP'04 Conference.                      
- Edge Sense II    : Laroche, Claude A. "Apparatus and method for adaptively interpolating
                     a full color image utilizing chrominance gradients"                  
                        U.S. Patent 5,373,322. Based on the code found on the website     
                     http://www-ise.stanford.edu/~tingchen/ Converted to C and adapted to 
                     all four elementary patterns.                                        
- Downsample       : "Known to the Ancients"                                              
- Simple           : Implemented from the information found in the manual of Allied Vision
                     Technologies (AVT) cameras.                                          
- VNG              : Variable Number of Gradients, a method described in                  
                     http://www-ise.stanford.edu/~tingchen/algodep/vargra.html            
                     Sources import from DCRAW by Frederic Devernay. DCRAW is a RAW       
                     converter program by Dave Coffin. URL:                               
                     http://www.cybercom.net/~dcoffin/dcraw/                              
- AHD              : Adaptive Homogeneity-Directed Demosaicing Algorithm, by K. Hirakawa  
                     and T.W. Parks, IEEE Transactions on Image Processing, Vol. 14, Nr. 3,
                     March 2005, pp. 360 - 369.
Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]

Avisynth plugin for JP4 processing

Avisynth plugin also available

AVS script example:

LoadCPlugin("jp4.dll")
DirectShowSource("test.avi")
JP4("AHD","RGGB")


Demosaicing/debayering links

A Study of Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras. Ting Chen / Stanford University

Source code:

eLynx Image Processing SDK and Lab
libdc1394
Efficient, high-quality Bayer demosaic filtering on GPUs
http://svn2.assembla.com/svn/ge/libgedrawing/trunk/src/ImageBayer.cpp

Example files:


See also: