ElphelWiki - User contributions [en]

Wishlist

2010-10-04T17:00:01Z

Fthiery: /* C API */

As user I noticed some things I wanted to have on my camera, I hope anyone that has ideas can add them here.
What features are you waiting for?
What software components are still missing that you cannot live without?

==Software==
If a project has been completed just add the developer that did it, or is busy on it.
=== Sensor ===
* [[Dead/Hot pixel correction]]

=== Compass ===
* '''DONE''' [[Implement calibration]]

=== Networking ===
* [[DHCP (timeouted) and DNS-SD]]
* [[GigaBit ethernet]]
* Zeroconf (Rendezvous) for LAN device announcing
* [[http://opensoundcontrol.org OpenSound Control]] [[http://en.wikipedia.org/wiki/OpenSound_Control OpenSound Control on wikipedia]], [[http://www.linuxjournal.com/content/introduction-osc introduction to OSC on linuxjournal]]
* CF Ethernet card support [[http://www.pocketpcfaq.com/peripherals/cfethernet.htm CF 10/100 Ethernet cards list]]
* CF WiFi card support [[http://www.google.ru/url?sa=t&source=web&ct=res&cd=1&url=http%3A%2F%2Fwww.linksys.com%2Fservlet%2FSatellite%3Fchildpagename%3DUS%252FLayout%26packedargs%3Dc%253DL_Product_C2%2526cid%253D1115416826419%26pagename%3DLinksys%252FCommon%252FVisitorWrapper&ei=2CkjSb2pB4PG0gXtx92CAg&usg=AFQjCNFGaaRKL7uwQ7UNQQsxbdf9mrsoeA&sig2=wYBsCu8pfOyTJt57UW57EA WCF54G]]
* CF USB card support [[http://www.twin-paradox.com/SEPDA.html SolarExpress]] [[http://www.ratocsystems.com/english/products/CFU2U.html CFU2]]
* External USB Display support [http://gizmodo.com/5116061/mimo-um+750-7+inch-usb-display-lightning-review Mimo UM-750 7-inch].

=== Fastest Boot ===
* [[Fast Boot]]

=== APIs ===
* [[C API]]
* [[SOAP API]]

=== UDMA support ===
* Enable [[UDMA]] in the Axis kernel. Needed to support most fast CF cards, will also boost speed on SATA.

=== Debayer Algorithmn ===
* Higher image quality debayer algorithmn FPGA implementation -> [http://scien.stanford.edu/class/psych221/projects/99/tingchen/algodep/vargra.html VNG]

==Hardware==
What will the next elphel camera most likely entitled '''Elphel 373''' be like?

===Power Supply===
* optional 5-36V supply over additional jack [http://focus.ti.com/docs/prod/folders/print/tps5430.html DC/DC] [http://focus.ti.com/docs/prod/folders/print/tps23750.html POE] [http://www.rlocman.ru/review/article.html?di=52247 TI Article]

===Sensor===
* Vignetting hardware correction
* Lens distortion hardware correction
* Fixed pattern noise (FPN) correction
* Flash mode for rolling shutter
* Make image thumbnails (binning 8x8 from DCT[0,0] data)
'''Position adjustment'''
*initial focus/pan/tilt sensor adjustment

'''Interface'''
*group VLDS pairs in flexible cable from sensor to board
*[[CCD table|CCD sensors]]

'''Sony IMX021'''
*designed for DSLRs (possibly the sensor used in Nikon D300 / D90)
*12.4 MPix
*5.5 µm pixel
*0.18 µm technology
*28mm x 22.2 mm APS-C sized 1.8" image area (28.4 mm diagonal)
*12 bit column ADC
*10.4 fps in 12.4 MPix mode
*on-chip noise reduction
*parallel A/D for every pixel coloumn

===CPU===
*USB2.0 Support

===External data sensors support===
*'''DONE''' GPS
*'''DONE''' Compass
*Dry contact (simply switch event)
*'''DONE''' RFID
*Proximity
*Environment sensors: Temperature/Humidity/Pressure/Illuminance/Loudness/.. [http://www.catnip.co.uk/wx Open Weather], [http://www.sciproj.com/FeaturedVendorProduct.aspx?uid=1208227 EL-USB-RT], [http://www.raphnet.net/electronique/usbtenki/index_en.php USBTenki], [http://www.sparkfun.com/commerce/product_info.php?products_id=8311 USB Weather Board], [http://www.farnell.com/datasheets/129494.pdf CY3271-EXP1 (PSoC Environmental Sensing Kit)]
*Futaba servo captute interface (for robots)

===External control===
*USB2.0 Support
*Futaba servo control interface (for robots) [http://www.parallax.com/Store/Accessories/MotorServoControllers/tabid/160/CategoryID/35/List/0/Level/a/ProductID/346/Default.aspx?SortField=ProductName%2CProductName PSC] [http://www.endurance-rc.com/pantilt.html Pan&Tilt] [http://www.openservo.com/ OpenServo] [http://www.embeddedrelated.com/usenet/embedded/show/46581-1.php Verilog solution] [http://www.sparkfun.com/commerce/product_info.php?products_id=8897 Polulu]
*Relay control
*CAN support [http://www.qprotos.com/quickcan.htm quickcan] [http://www.gridconnect.com/usbcanin.html USB CAN Adapter]
*DMX512 support [http://en.wikipedia.org/wiki/DMX-512 DMX512]
*Secure Digital support [http://www.delkin.com/products/adapters/sd-to-cf/sd-to-cf.html CF-SD]

Wishlist

2010-10-04T16:58:57Z

Fthiery: /* Networking */

As user I noticed some things I wanted to have on my camera, I hope anyone that has ideas can add them here.
What features are you waiting for?
What software components are still missing that you cannot live without?

==Software==
If a project has been completed just add the developer that did it, or is busy on it.
=== Sensor ===
* [[Dead/Hot pixel correction]]

=== Compass ===
* '''DONE''' [[Implement calibration]]

=== Networking ===
* [[DHCP (timeouted) and DNS-SD]]
* [[GigaBit ethernet]]
* Zeroconf (Rendezvous) for LAN device announcing
* [[http://opensoundcontrol.org OpenSound Control]] [[http://en.wikipedia.org/wiki/OpenSound_Control OpenSound Control on wikipedia]], [[http://www.linuxjournal.com/content/introduction-osc introduction to OSC on linuxjournal]]
* CF Ethernet card support [[http://www.pocketpcfaq.com/peripherals/cfethernet.htm CF 10/100 Ethernet cards list]]
* CF WiFi card support [[http://www.google.ru/url?sa=t&source=web&ct=res&cd=1&url=http%3A%2F%2Fwww.linksys.com%2Fservlet%2FSatellite%3Fchildpagename%3DUS%252FLayout%26packedargs%3Dc%253DL_Product_C2%2526cid%253D1115416826419%26pagename%3DLinksys%252FCommon%252FVisitorWrapper&ei=2CkjSb2pB4PG0gXtx92CAg&usg=AFQjCNFGaaRKL7uwQ7UNQQsxbdf9mrsoeA&sig2=wYBsCu8pfOyTJt57UW57EA WCF54G]]
* CF USB card support [[http://www.twin-paradox.com/SEPDA.html SolarExpress]] [[http://www.ratocsystems.com/english/products/CFU2U.html CFU2]]
* External USB Display support [http://gizmodo.com/5116061/mimo-um+750-7+inch-usb-display-lightning-review Mimo UM-750 7-inch].

=== Fastest Boot ===
* [[Fast Boot]]

=== C API ===
* [[C API]]

=== UDMA support ===
* Enable [[UDMA]] in the Axis kernel. Needed to support most fast CF cards, will also boost speed on SATA.

=== Debayer Algorithmn ===
* Higher image quality debayer algorithmn FPGA implementation -> [http://scien.stanford.edu/class/psych221/projects/99/tingchen/algodep/vargra.html VNG]

==Hardware==
What will the next elphel camera most likely entitled '''Elphel 373''' be like?

===Power Supply===
* optional 5-36V supply over additional jack [http://focus.ti.com/docs/prod/folders/print/tps5430.html DC/DC] [http://focus.ti.com/docs/prod/folders/print/tps23750.html POE] [http://www.rlocman.ru/review/article.html?di=52247 TI Article]

===Sensor===
* Vignetting hardware correction
* Lens distortion hardware correction
* Fixed pattern noise (FPN) correction
* Flash mode for rolling shutter
* Make image thumbnails (binning 8x8 from DCT[0,0] data)
'''Position adjustment'''
*initial focus/pan/tilt sensor adjustment

'''Interface'''
*group VLDS pairs in flexible cable from sensor to board
*[[CCD table|CCD sensors]]

'''Sony IMX021'''
*designed for DSLRs (possibly the sensor used in Nikon D300 / D90)
*12.4 MPix
*5.5 µm pixel
*0.18 µm technology
*28mm x 22.2 mm APS-C sized 1.8" image area (28.4 mm diagonal)
*12 bit column ADC
*10.4 fps in 12.4 MPix mode
*on-chip noise reduction
*parallel A/D for every pixel coloumn

===CPU===
*USB2.0 Support

===External data sensors support===
*'''DONE''' GPS
*'''DONE''' Compass
*Dry contact (simply switch event)
*'''DONE''' RFID
*Proximity
*Environment sensors: Temperature/Humidity/Pressure/Illuminance/Loudness/.. [http://www.catnip.co.uk/wx Open Weather], [http://www.sciproj.com/FeaturedVendorProduct.aspx?uid=1208227 EL-USB-RT], [http://www.raphnet.net/electronique/usbtenki/index_en.php USBTenki], [http://www.sparkfun.com/commerce/product_info.php?products_id=8311 USB Weather Board], [http://www.farnell.com/datasheets/129494.pdf CY3271-EXP1 (PSoC Environmental Sensing Kit)]
*Futaba servo captute interface (for robots)

===External control===
*USB2.0 Support
*Futaba servo control interface (for robots) [http://www.parallax.com/Store/Accessories/MotorServoControllers/tabid/160/CategoryID/35/List/0/Level/a/ProductID/346/Default.aspx?SortField=ProductName%2CProductName PSC] [http://www.endurance-rc.com/pantilt.html Pan&Tilt] [http://www.openservo.com/ OpenServo] [http://www.embeddedrelated.com/usenet/embedded/show/46581-1.php Verilog solution] [http://www.sparkfun.com/commerce/product_info.php?products_id=8897 Polulu]
*Relay control
*CAN support [http://www.qprotos.com/quickcan.htm quickcan] [http://www.gridconnect.com/usbcanin.html USB CAN Adapter]
*DMX512 support [http://en.wikipedia.org/wiki/DMX-512 DMX512]
*Secure Digital support [http://www.delkin.com/products/adapters/sd-to-cf/sd-to-cf.html CF-SD]

JP4

2009-12-31T14:37:11Z

Fthiery: /* GStreamer plugins for Elphel JP4 image and video processing */

Note: the JP4 mode described here is referred as "JP46" in current 8.0 firwmare

== JP4 Format ==

We have added a special JP4 mode that bypasses the Demosaic in the FPGA and provides an image with pixels in each 16x16 macroblock that are rearranged to separate Bayer colors in individual 8x8 blocks, then encoded as monochrome. [[Demosaic_on_client_side|Demosaic]] will be applied during post-processing on the host PC. This section describe different algorithms and implementations used to provide this functionality.

Main goals:
- compression speed improvement
- possibility to obtain more high quality image (near to RAW)
- drasticaly lowering data size
== Different JP4 Modes in 8.X Software ==
only modes 0-2 can be processed with standard libjpeg:
*0 - mono6, monochrome (color YCbCr 4:2:0 with zeroed out color componets)
*1 - color, YCbCr 4:2:0, 3x3 pixels
*2 - jp46 - original JP4 (from 7.X software), encoded as 4:2:0 with zeroed color components
*3 - jp46dc, modified jp46 so each color component uses individual DC diffenential encoding
*4 - reserved for color with 5x5 conversion (not yet implemented)
*5 - jp4 with ommitted color components (4:0:0)
*6 - jp4dc, similar to jp46dc encoded as 4:0:0
*7 - jp4diff, differential where (R-G), G, (G2-G) and (B-G) components are encoded as 4:0:0
*8 - jp4hdr, (R-G), G, G2,(B-G) are encoded so G2 can be used with high gain
*9 - jp4fiff2, (R-G)/2, G,(G2-G)/2, (B-G)/2 to avoid possible overflow in compressed values
*10 - jp4hdr2, (R-G)/2, G,G2,(B-G)/2
*14 - mono, monochrome with ommitted color components (4:0:0)

=== [[JP4 HDR]] ===
Bayer pattern look like this
{| class="wikitable"
|-
|
{| class="wikitable"align="center"
|+RGGB
|-
| bgcolor="red"|R || bgcolor="green"|G1 || bgcolor="red"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="blue"|B || bgcolor="green"|G2 || bgcolor="blue"|B
|-
| bgcolor="red"|R || bgcolor="green"|G1 || bgcolor="red"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="blue"|B || bgcolor="green"|G2 || bgcolor="blue"|B
|}
|
{| class="wikitable"align="center"
|+BGGR
|-
| bgcolor="blue"|R || bgcolor="green"|G1 || bgcolor="blue"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="red"|B || bgcolor="green"|G2 || bgcolor="red"|B
|-
| bgcolor="blue"|R || bgcolor="green"|G1 || bgcolor="blue"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="red"|B || bgcolor="green"|G2 || bgcolor="red"|B
|}
|
{| class="wikitable"align="center"
|+GBRG
|-
| bgcolor="green"|G2 || bgcolor="blue"|B || bgcolor="green"|G2 || bgcolor="blue"|B
|-
| bgcolor="red"|R || bgcolor="green"|G1 || bgcolor="red"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="blue"|B || bgcolor="green"|G2 || bgcolor="blue"|B
|-
| bgcolor="red"|R || bgcolor="green"|G1 || bgcolor="red"|R || bgcolor="green"|G1
|}
|
{| class="wikitable"align="center"
|+GRBG
|-
| bgcolor="green"|G2 || bgcolor="red"|B || bgcolor="green"|G2 || bgcolor="red"|B
|-
| bgcolor="blue"|R || bgcolor="green"|G1 || bgcolor="blue"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="red"|B || bgcolor="green"|G2 || bgcolor="red"|B
|-
| bgcolor="blue"|R || bgcolor="green"|G1 || bgcolor="blue"|R || bgcolor="green"|G1
|}
|}

The remark: all kinds of bayer patterns can be received from initial RGGB by flipping on X and/or Y.

Some sensors have possibility to set independed scale to G1 and G2.
Considering that the accessible optics does not give the full permission of a sensor resolution, it [[JP4_HDR|can be]] used for increase in a dynamic range of a image sensor.

== JP4 processing on the host ==

=== JP4 image decoding in MATLAB ===
JP4 format can be easy manipulated by [http://www.mathworks.com/matlabcentral/fileexchange/22144 MATLAB] [[Image:Fruits_jp4.jpg|thumb|JP4 image]]

1. Read image
I=imread('hdr02.jp4'); %read JP4 file like JPEG
,or online grab image from http like this:
I=imread('http://community.elphel.com/pictures/jp4.jpg');
,or cam:
I=imread('http://cam_ip/bimg'); %get online buffered image from cam

I=I(:,:,1); %strip color data
2. Remove block grouping[[Image:Fruits_jp4_deblocked.jpg|thumb|Bayer CFA encoded image]]
<code matlab>
II=deblock16x16(I); %deblock image

%file deblock16x16.m
function y=deblock16x16(I)
y0=uint8(zeros(size(I)));
for x=1:16:size(I,1)
for y=1:16:size(I,2)
blk16=I(x:x+15,y:y+15);
for dx=0:7
for dy=0:7
y0(x+2*dx ,y+2*dy) = blk16(dx+1,dy+1);
y0(x+2*dx+1,y+2*dy) = blk16(dx+9,dy+1);
y0(x+2*dx ,y+2*dy+1) = blk16(dx+1,dy+9);
y0(x+2*dx+1,y+2*dy+1) = blk16(dx+9,dy+9);
end
end
end
end
y=y0;
</code>
2. Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)[[Image:Fruits_jp4_debayered.jpg|thumb|Decoded image]]
J=demosaic(II,'gbrg');
3. Show image
imshow(J);

=== JP4 to DNG image conversion ===

Credits: Dave Coffin

This Linux command line tool allows conversion of JP4 files into a DNGs that dcraw and Adobe Photoshop can open.

Download [http://community.elphel.com/files/jp4/tiff-3.8.2.tar.gz LibTIFF v3.8.2] a
*extract the tar.gz (this should create a new folder called "tiff-3.8.2")

Apply [http://community.elphel.com/files/jp4/libtiff.patch this patch]: in terminal (first cd to path of libtiff.patch):
patch -p0 < libtiff.patch

build LibTIFF:

cd tiff-3.8.2
./configure
make
sudo make install

Then compile [http://community.elphel.com/files/jp4/elphel_dng.c this C program] with:
gcc -o elphel_dng elphel_dng.c -ltiff

Then use the created application:

Usage: ./elphel_dng "gamma" "input.jpg" "output.dng"
Example: ./elphel_dng 100 example_JP4.jpeg example.dng

=== JP4 video stream decoding using MPlayer ===
JP4 stream can be decoded by MPlayer.
Use this [[http://community.elphel.com/files/mplayer/debayer.diff this patch]]
patch . -p0 < debayer.diff in the mplayer source dir

Or download win32 binaries from sourceforge.
usage example: mplayer.exe test.avi -vf demosaic=deblock=1:method=7:pattern=3 -vo gl
mencoder example: mencoder.exe test.avi -ovc lavc -lavcopts vcodec=mjpeg -o output.avi -vf demosaic=deblock=1:method=1,scale

Debayer ([[Demosaic_on_client_side|Demosaic]]) algorithm variants provided by libdc1394:
- Nearest Neighbor : OpenCV library
- Bilinear : OpenCV library
- HQLinear : High-Quality Linear Interpolation For Demosaicing Of Bayer-Patterned
Color Images, by Henrique S. Malvar, Li-wei He, and Ross Cutler,
in Proceedings of the ICASSP'04 Conference.
- Edge Sense II : Laroche, Claude A. "Apparatus and method for adaptively interpolating
a full color image utilizing chrominance gradients"
U.S. Patent 5,373,322. Based on the code found on the website
http://www-ise.stanford.edu/~tingchen/ Converted to C and adapted to
all four elementary patterns.
- Downsample : "Known to the Ancients"
- Simple : Implemented from the information found in the manual of Allied Vision
Technologies (AVT) cameras.
- VNG : Variable Number of Gradients, a method described in
http://www-ise.stanford.edu/~tingchen/algodep/vargra.html
Sources import from DCRAW by Frederic Devernay. DCRAW is a RAW
converter program by Dave Coffin. URL:
http://www.cybercom.net/~dcoffin/dcraw/
- AHD : Adaptive Homogeneity-Directed Demosaicing Algorithm, by K. Hirakawa
and T.W. Parks, IEEE Transactions on Image Processing, Vol. 14, Nr. 3,
March 2005, pp. 360 - 369.

Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]

=== Avisynth plugin for JP4 processing ===
[http://avisynth.org/mediawiki/Main_Page Avisynth plugin] also available

AVS script example:
LoadCPlugin("jp4.dll")
DirectShowSource("test.avi")
JP4("AHD","RGGB")

=== GStreamer plugins for Elphel JP4 image and video processing ===
[http://code.google.com/p/gst-plugins-elphel/ This project] supported by http://ubicast.eu hosts Elphel related gstreamer components, so far:

* the jp462bayer plugin converts color and monochrome JP46 Elphel bitstreams to Bayer raw format. In the future, it might support other JP4 modes (JP4, JP4-HDR, ...)
* bayer2rgb2 converts raw Bayer streams to RGB images

==== jp462bayer: JP4 to Bayer ====

After jpegdec, re-arranges the pixels in Bayer format.

==== bayer2rgb2: debayer ====

It offers the same features as the legacy bayer2rgb, but by wrapping Libdc1394's debayering algorithms you can choose the interpoloation algorithm between : simple, bilinear, hqlinear, downsample, edgesense, vng, ahd, nearest

==== Example pipelines ====

gst-launch-0.10 rtspsrc location=rtsp://elphel:554 protocols=0x00000001 ! rtpjpegdepay ! jpegdec ! \
queue ! jp462bayer ! queue ! bayer2rgb2 ! queue ! ffmpegcolorspace ! videorate ! "video/x-raw-yuv, \
format=(fourcc)I420, width=(int)1920, height=(int)1088, framerate=(fraction)25/1" ! xvimagesink sync=false max-lateness=-1

== Demosaicing/debayering links ==

[http://scien.stanford.edu/class/psych221/projects/99/tingchen A Study of Spatial Color Interpolation Algorithms for Single-Detector Digital Cameras. Ting Chen / Stanford University]

Source code:
[http://sourceforge.net/projects/elynx eLynx Image Processing SDK and Lab]
[http://libdc1394.git.sourceforge.net/git/gitweb.cgi?p=libdc1394;a=blob;f=libdc1394/dc1394/bayer.c;hb=HEAD libdc1394]
[http://graphics.cs.williams.edu/papers/BayerJGT09 Efficient, high-quality Bayer demosaic filtering on GPUs]
http://svn2.assembla.com/svn/ge/libgedrawing/trunk/src/ImageBayer.cpp

Example files:
* [http://community.elphel.com/files/jp4/example_JP4.jpeg Example JP4]
* [http://community.elphel.com/files/jp4/example_flipped.dng Example DNG]
* [http://community.elphel.com/files/jp4/example_converted.jpg Example JPG (converted)]

See also:

* [[Demosaic on client side]]
* [http://linuxdevices.com/articles/AT4187053130.html Elphel camera under the hood: from Verilog to PHP - on LinuxDevices.com]

Demosaic on client side

2009-07-23T15:04:12Z

Fthiery: /* Gstreamer implementation */

= Background =
Our actual FPGA code on the camera use very simple algorithm to calculate YCbCr from the Bayer pixels. It use just 3x3 block of neighbors. The other point is that this algorithm is time consuming and with the 5MPix sensor the FPGA became the bottleneck. So we have added a special [[JP4]] mode what bypass the Demosaic in the FPGA and provide an image with pixels in each 16x16 macroblock what are rearranged to separate Bayer colors in individual 8x8 blocks, then encoded as monochrome. Demosaic will be applied during post-processing on the host PC. This page describe different algorithms and implementations used to provide this functionality.

Cf. wikipedia for more info on [http://en.wikipedia.org/wiki/Demosaicing Demosaicing].

= Working with RAW image data =
There actually already is a workflow to get RAW image data from the elphel camera and demosaic/postprocess on a computer.

* Step 1: Use the JP4 encoded jpg output on a camera
* Step 2: We need to build a modified dcraw under linux to read our image: For the unexperience linux user this can be quite a challenge (at least it had been for me) I used Ubuntu 7.10 and first had to acquire the "build-essential" package via synaptic package manager. From the dcraw website grab [http://www.cybercom.net/~dcoffin/dcraw/elphel_dng.c elphel_dng.c] and [http://dl.maptools.org/dl/libtiff/tiff-3.8.2.tar.gz LibTIFF v3.8.2] as well as a [http://www.cybercom.net/~dcoffin/dcraw/libtiff.patch patch]. Unpack the LibTiff tar.gz and we have all files together. Apply the patch (the command is "patch -p1 < libtiff.patch"). This will do some minor changes in the source code. Now go into the libtiff directory and execute "./configure". If successful do "make" and then "make install" (I had to do "sudo su" first to be able to install anything as root). I had special trouble because I am running a 64bit system and "make" complained about static and dynamic libraries and that I should compile tif_stream.o with -fPIC parameter. Nothing of the suggested worked. In the end I did it with a "make clean" (don't ask me why!). After libtiff was successfully built we need to compile the elphel_dng.c. Everything you need to know is already written into the header of that source file: "gcc -o elphel_dng elphel_dng.c -O4 -Wall -lm -ljpeg -ltiff" and you have an executable.
* Step 3: We use the executable we just built to convert our jp4 jpeg. Usage "./elphel_dng gamma infile outfile" for example "./elphel_dng 100 jp4.jpg test.dng"
* Done. You should now have a DNG that you can load in any raw editing software like photoshop, lightroom, aperture or GUIs for dcraw.

= Comparing different demosaicing algorithms =

There are many different ways to reconstruct (or at least try to) all pixels color channels from Bayer pattern data.

Here is a side by side comparison:
{| border="0" cellpadding="2"
|-
| [[Image:1080 jpg 99.jpeg|thumb|JPEG processed within a Elphel 353 at 99% JPEG quality - camera uses the Variable Number of Gradients algorithm]]||[[Image:1080 UFRaw Bilinear.jpg|thumb|UFRaw processed using the fast "Bilinear" algorithm]]||[[Image:1080 UFRaw VNG.jpg|thumb|UFRaw processed using the VNG (Variable Number of Gradients interpolation) algorithm]]||[[Image:1080 UFRaw PPG.jpg|thumb|UFRaw processed using the PPG (Patterned Pixel Grouping) algorithm]]||[[Image:1080 UFRaw AHD.jpg|thumb|UFRaw processed using the AHD (Adaptive Homogeneity-Directed interpolation) algorithm]]
|-
|}

All RAW file were generated by converting JP4 encoded JPEGs to DNG and where then processed in UFRaw. (sorry for the slight difference in brightness and saturation, it was the closest I could get in UFRaw to what the camera does internally)

Detail comparison:

[[Image:Detail01.jpg|300% Zoom]]

[[Image:Detail02.jpg|300% Zoom]]

'''Conclusion''':
For some reason the elphel internal algorithm which should actually be identical to the UFRaw VNG falls behind all UFRaw created methods. The elphel processed image lacks sharpness and richness in detail and is similar to the bilinear interpolation. AHD, VNG and PPG algorithms processed in UFRaw show very sharp results with clearly more visible details. Noise is a bit stronger in luminosity but less in chroma. VNG has less color artifacts than AHD and PPG.

= The goals =
==Video processing==
Next step is to be able to embed this algorithm into MPlayer, VLC, FFmpeg and GSTreamer for video processing. The goal is to get the maximum sensor FPS processed on the computer. So at 5MPix it's 15 FPS, at 1920x1088 it's 30 FPS.

= Algorithm =
== Existing ==
There are several algorithms that provide good results with less artifacts (see Wikipedia article) and these detailed descriptions:
* [http://scien.stanford.edu/class/psych221/projects/99/tingchen/algodep/vargra.html Variable Number of Gradients]
* [http://web.cecs.pdx.edu/~cklin/demosaic/ Pixel Grouping]
* [http://www.dei.unipd.it/~menond/pub/dfapd/dfapd.html Demosaicing with Directional Filtering and a Posteriori Decision]

== Implemented ==
We actually did implement the Variable Number of Gradients algorithm.

= Implementation =
Several implementation with different numbers of software & hardware dependencies are possible.

== OpenCV implementation ==

== libjpeg implementation ==

== Gstreamer implementation ==

Gstreamer already offers:
* a jpegdec element
* a [http://webcvs.freedesktop.org/gstreamer/gst-plugins-bad/gst/bayer/gstbayer2rgb.c?view=markup bayer2rgb] component, which decodes raw camera bayer (fourcc BA81) to RGB

An implementation based on bayer2rgb is currently in development, using [http://libdc1394.git.sourceforge.net/git/gitweb.cgi?p=libdc1394;a=blob;f=libdc1394/dc1394/bayer.c;hb=HEAD libdc1394] for the demosaicing

It will be used like this:
gst-launch rtspsrc ... ! rtpjpegdepay ! jpegdec ! bayer2rgb2 jp4=true ! xvimagesink

Demosaic on client side

2009-07-23T14:50:50Z

Fthiery: /* Gstreamer implementation */

JP4

2009-07-23T14:39:43Z

Fthiery:

Note: the JP4 mode described here is referred as "JP46" in current 8.0 firwmare

== JP4 format ==

So we have added a special JP4 mode that bypasses the Demosaic in the FPGA and provides an image with pixels in each 16x16 macroblock that are rearranged to separate Bayer colors in individual 8x8 blocks, then encoded as monochrome. [[Demosaic_on_client_side|Demosaic]] will be applied during post-processing on the host PC. This section describe different algorithms and implementations used to provide this functionality.

Main goals:
- compression speed improvement
- possibility to obtain more high quality image (near to RAW)
- drasticaly lowering data size

== Image decoding ==
JP4 format can be easy manipulated by [http://www.mathworks.com/matlabcentral/fileexchange/22144 MATLAB] [[Image:Fruits_jp4.jpg|thumb|JP4 image]]

1. Read image
I=imread('hdr02.jp4'); %read JP4 file like JPEG
,or online grab image from http like this:
I=imread('http://community.elphel.com/pictures/jp4.jpg');
,or cam:
I=imread('http://cam_ip/bimg'); %get online buffered image from cam

I=I(:,:,1); %strip color data
2. Remove block grouping[[Image:Fruits_jp4_deblocked.jpg|thumb|Bayer CFA encoded image]]
<code matlab>
II=deblock16x16(I); %deblock image

%file deblock16x16.m
function y=deblock16x16(I)
y0=uint8(zeros(size(I)));
for x=1:16:size(I,1)
for y=1:16:size(I,2)
blk16=I(x:x+15,y:y+15);
for dx=0:7
for dy=0:7
y0(x+2*dx ,y+2*dy) = blk16(dx+1,dy+1);
y0(x+2*dx+1,y+2*dy) = blk16(dx+9,dy+1);
y0(x+2*dx ,y+2*dy+1) = blk16(dx+1,dy+9);
y0(x+2*dx+1,y+2*dy+1) = blk16(dx+9,dy+9);
end
end
end
end
y=y0;
</code>
2. Demosaic image (Decode from Bayer CFA (Color Filter Array) encoded image)[[Image:Fruits_jp4_debayered.jpg|thumb|Decoded image]]
J=demosaic(II,'gbrg');
3. Show image
imshow(J);

== Stream decoding ==
JP4 stream can be decoded by mplayer.
Use [[http://community.elphel.com/files/mplayer/debayer.diff this patch]] for glue libdc1394 and MPlayer-1.0rc2 video filter frontend.
Or download win32 binaries from sourceforge.
usage example: mplayer.exe test.avi -vf demosaic=deblock=1:method=7:pattern=3 -vo gl
mencoder example: mencoder.exe test.avi -ovc lavc -lavcopts vcodec=mjpeg -o output.avi -vf demosaic=deblock=1:method=1,scale

Debayer ([[Demosaic_on_client_side|Demosaic]]) algorithm variants provided by libdc1394:
- Nearest Neighbor : OpenCV library
- Bilinear : OpenCV library
- HQLinear : High-Quality Linear Interpolation For Demosaicing Of Bayer-Patterned
Color Images, by Henrique S. Malvar, Li-wei He, and Ross Cutler,
in Proceedings of the ICASSP'04 Conference.
- Edge Sense II : Laroche, Claude A. "Apparatus and method for adaptively interpolating
a full color image utilizing chrominance gradients"
U.S. Patent 5,373,322. Based on the code found on the website
http://www-ise.stanford.edu/~tingchen/ Converted to C and adapted to
all four elementary patterns.
- Downsample : "Known to the Ancients"
- Simple : Implemented from the information found in the manual of Allied Vision
Technologies (AVT) cameras.
- VNG : Variable Number of Gradients, a method described in
http://www-ise.stanford.edu/~tingchen/algodep/vargra.html
Sources import from DCRAW by Frederic Devernay. DCRAW is a RAW
converter program by Dave Coffin. URL:
http://www.cybercom.net/~dcoffin/dcraw/
- AHD : Adaptive Homogeneity-Directed Demosaicing Algorithm, by K. Hirakawa
and T.W. Parks, IEEE Transactions on Image Processing, Vol. 14, Nr. 3,
March 2005, pp. 360 - 369.

Pattern codes: pattern=0..3 -> [RGGB, BGGR, GBRG, GRBG]

== [[JP4 HDR]] ==
Bayer pattern look like this
{| class="wikitable"
|-
|
{| class="wikitable"align="center"
|+RGGB
|-
| bgcolor="red"|R || bgcolor="green"|G1 || bgcolor="red"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="blue"|B || bgcolor="green"|G2 || bgcolor="blue"|B
|-
| bgcolor="red"|R || bgcolor="green"|G1 || bgcolor="red"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="blue"|B || bgcolor="green"|G2 || bgcolor="blue"|B
|}
|
{| class="wikitable"align="center"
|+BGGR
|-
| bgcolor="blue"|R || bgcolor="green"|G1 || bgcolor="blue"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="red"|B || bgcolor="green"|G2 || bgcolor="red"|B
|-
| bgcolor="blue"|R || bgcolor="green"|G1 || bgcolor="blue"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="red"|B || bgcolor="green"|G2 || bgcolor="red"|B
|}
|
{| class="wikitable"align="center"
|+GBRG
|-
| bgcolor="green"|G2 || bgcolor="blue"|B || bgcolor="green"|G2 || bgcolor="blue"|B
|-
| bgcolor="red"|R || bgcolor="green"|G1 || bgcolor="red"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="blue"|B || bgcolor="green"|G2 || bgcolor="blue"|B
|-
| bgcolor="red"|R || bgcolor="green"|G1 || bgcolor="red"|R || bgcolor="green"|G1
|}
|
{| class="wikitable"align="center"
|+GRBG
|-
| bgcolor="green"|G2 || bgcolor="red"|B || bgcolor="green"|G2 || bgcolor="red"|B
|-
| bgcolor="blue"|R || bgcolor="green"|G1 || bgcolor="blue"|R || bgcolor="green"|G1
|-
| bgcolor="green"|G2 || bgcolor="red"|B || bgcolor="green"|G2 || bgcolor="red"|B
|-
| bgcolor="blue"|R || bgcolor="green"|G1 || bgcolor="blue"|R || bgcolor="green"|G1
|}
|}

The remark: all kinds of bayer patterns can be received from initial RGGB by flipping on X and/or Y.

Some sensors have possibility to set independed scale to G1 and G2.
Considering that the accessible optics does not give the full permission of a sensor resolution, it [[JP4_HDR|can be]] used for increase in a dynamic range of a image sensor.

== [http://avisynth.org/mediawiki/Main_Page Avisynth] plugin ==
Avisynth plugin also available

AVS script example:
LoadCPlugin("jp4.dll")
DirectShowSource("test.avi")
JP4("AHD","RGGB")

== Demosaic links ==
http://sourceforge.net/projects/elynx/
http://libdc1394.git.sourceforge.net/git/gitweb.cgi?p=libdc1394;a=blob;f=libdc1394/dc1394/bayer.c;hb=HEAD
http://graphics.cs.williams.edu/papers/BayerJGT09/

== JP4 to DNG conversion ==

Credits: Dave Coffin

This Linux comanndline tool allows conversion of JP4 files into a DNGs that dcraw and Adobe Photoshop can open.

Download [http://community.elphel.com/files/jp4/tiff-3.8.2.tar.gz LibTIFF v3.8.2] a
*extract the tar.gz (this should create a new folder called "tiff-3.8.2")

Apply [http://community.elphel.com/files/jp4/libtiff.patch this patch]: in terminal (first cd to path of libtiff.patch):
patch -p0 < libtiff.patch

build LibTIFF:

cd tiff-3.8.2
./configure
make
sudo make install

Then compile [http://community.elphel.com/files/jp4/elphel_dng.c this C program] with:
gcc -o elphel_dng elphel_dng.c -ltiff

Then use the created application:

Usage: ./elphel_dng "gamma" "input.jpg" "output.dng"
Example: ./elphel_dng 100 example_JP4.jpeg example.dng

Example files:
* [http://community.elphel.com/files/jp4/example_JP4.jpeg Example JP4]
* [http://community.elphel.com/files/jp4/example_flipped.dng Example DNG]
* [http://community.elphel.com/files/jp4/example_converted.jpg Example JPG (converted)]

See also:

* [[Demosaic on client side]]

Using gstreamer

2009-04-22T15:37:43Z

Fthiery:

= Gstreamer and live video processing over the network =

Gstreamer is very suited to live video and audio processing, notably for live decoding/encoding, audio muxing.

= Requirements =

You will need
* a fairly recent gstreamer distribution, with the rtpjpegdepay plugin
* a relatively powerful computer to decode real time

= Limitations =

You will not (at this time) be able to decode the elphel 353 cameras on resolutions > 1920x1080 (which is the highest resolution you can reach with 25 fps anyway), because of the RTP payloading limits, with 25 fps anyway.

If you don't, you'll just have to build it yourself (see [http://www.pitivi.org/wiki/GStreamer_CVS_Setup_Page this guide from PiTiVi]).

= Tips =

You won't get 25 fps if autoexposure is on and local brightness not high enough: the camera will automatically lower framerate for keeping clear picture. Either lighten up, or play with image settings (notably, gain).

= Command line experiments =

(Note: replace width and height accordingly to your camera setup and your computer's horsepower :p).

== Displaying ==

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue ! xvimagesink sync=false

== Dumping ==
=== mjpeg dumping ===

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! videorate ! capsfilter caps = "image/jpeg, framerate=(fraction)25/1, width=1024, height=768" ! queue ! matroskamux ! filesink location=/tmp/test.mkv

=== YUV Dumping ===

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue ! videorate ! capsfilter caps="video/x-raw-yuv, format=(fourcc)I420, width=(int)1024, height=(int)768, framerate=(fraction)25/1" ! queue ! avimux ! filesink location=/tmp/test.avi

== Dump transcoding example ==

gst-launch filesrc location=test.mkv ! matroskademux ! queue ! jpegdec ! queue ! theoraenc bitrate=4000 ! queue ! oggmux ! filesink location=test.ogg

== Live encoding ==

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue ! videorate ! capsfilter caps="video/x-raw-yuv, format=(fourcc)I420, width=(int)1024, height=(int)768, framerate=(fraction)25/1" ! queue ! theoraenc bitrate=4000 ! queue ! oggmux ! filesink location=/tmp/test1024.ogg

I did some benchmarks; a Core 2 Quad Q6600 (2.4 Ghz) is not powerful enough for h264 encoding @fullHD resolution (using 4 treads).

Using gstreamer

2009-03-12T00:22:26Z

Fthiery: /* Command line */

Using gstreamer

2009-03-12T00:21:59Z

Fthiery: /* Command line */

= Gstreamer and live video processing over the network =

Gstreamer is very suited to live video and audio processing, notably for live decoding/encoding, audio muxing.

= Requirements =

You will need
* a fairly recent gstreamer distribution, with the rtpjpegdepay plugin
* a relatively powerful computer to decode real time

= Limitations =

You will not (at this time) be able to decode the elphel 353 cameras on resolutions > 1920x1080 (which is the highest resolution you can reach with 25 fps anyway), because of the RTP payloading limits, with 25 fps anyway.

If you don't, you'll just have to build it yourself (see [http://www.pitivi.org/wiki/GStreamer_CVS_Setup_Page this guide from PiTiVi]).

= Tips =

You won't get 25 fps if autoexposure is on and local brightness not high enough: the camera will automatically lower framerate for keeping clear picture. Either lighten up, or play with image settings (notably, gain).

= Command line =
== Displaying ==

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue ! xvimagesink sync=false

== Dumping ==
=== mjpeg dumping ===

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! videorate ! capsfilter caps = "image/jpeg, framerate=(fraction)25/1, width=1024, height=768" ! queue ! matroskamux ! filesink location=/tmp/test.mkv

=== YUV Dumping ===

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue ! videorate ! capsfilter caps="video/x-raw-yuv, format=(fourcc)I420, width=(int)1024, height=(int)768, framerate=(fraction)25/1" ! queue ! avimux ! filesink location=/tmp/test.avi

== Dump transcoding example ==

gst-launch filesrc location=test.mkv ! matroskademux ! queue ! jpegdec ! queue ! theoraenc bitrate=4000 ! queue ! oggmux ! filesink location=test.ogg

== Live encoding ==

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue ! videorate ! capsfilter caps="video/x-raw-yuv, format=(fourcc)I420, width=(int)1024, height=(int)768, framerate=(fraction)25/1" ! queue ! theoraenc bitrate=4000 ! queue ! oggmux ! filesink location=/tmp/test1024.ogg

(Note: replace width and height accordingly to your camera setup and your computer's horsepower :p).

Using gstreamer

2009-03-12T00:17:26Z

Fthiery: /* Command line */

= Gstreamer and live video processing over the network =

Gstreamer is very suited to live video and audio processing, notably for live decoding/encoding, audio muxing.

= Requirements =

You will need
* a fairly recent gstreamer distribution, with the rtpjpegdepay plugin
* a relatively powerful computer to decode real time

= Limitations =

You will not (at this time) be able to decode the elphel 353 cameras on resolutions > 1920x1080 (which is the highest resolution you can reach with 25 fps anyway), because of the RTP payloading limits, with 25 fps anyway.

If you don't, you'll just have to build it yourself (see [http://www.pitivi.org/wiki/GStreamer_CVS_Setup_Page this guide from PiTiVi]).

= Tips =

You won't get 25 fps if autoexposure is on and local brightness not high enough: the camera will automatically lower framerate for keeping clear picture. Either lighten up, or play with image settings (notably, gain).

= Command line =
== Displaying ==

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue ! xvimagesink sync=false

== Dumping ==
=== mjpeg dumping ===

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! videorate ! capsfilter caps = "image/jpeg, framerate=(fraction)25/1, width=1024, height=768" ! queue ! matroskamux ! filesink location=/tmp/test.mkv

=== YUV Dumping ===

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue ! videorate ! capsfilter caps="video/x-raw-yuv, format=(fourcc)I420, width=(int)1024, height=(int)768, framerate=(fraction)25/1" ! queue ! avimux ! filesink location=/tmp/test.avi

== Encoding ==

gst-launch -v rtspsrc location=rtsp://elphel:554 ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue ! videorate ! capsfilter caps="video/x-raw-yuv, format=(fourcc)I420, width=(int)1024, height=(int)768, framerate=(fraction)25/1" ! queue ! theoraenc bitrate=4000 ! queue ! oggmux ! filesink location=/tmp/test1024.ogg

(Note: replace width and height accordingly to your camera setup and your computer's horsepower :p).

Using gstreamer

2009-03-12T00:06:42Z

Fthiery:

Demosaic on client side

2008-08-14T13:26:36Z

Fthiery: