ElphelWiki - User contributions [en]

Direct DMA

2008-07-25T15:55:51Z

JavierRojas: /* Notes/bugs/etc */

= Direct DMA =
This is an implementation of DMA between the FPGA and the main memory of an Elphel 353 camera.
== Requirements/preparation ==
We used the 7.1.7.26 version of the Elphel software. To use this driver you will have to recompile the kernel
(and because of this, also reflash the camera), and create a proper device file.

Since we will be using a new device (file in /dev), we'll have to create it as well; but, since the /dev filesystem is read-only, this must be done when creating the camera image (and therefore requires reflashing the camera).

Needless to say, you can recompile the kernel and arrange for the creation of the device at the same time so you will only have to reflash the camera once to apply both changes.

=== Kernel recompilation ===
The original Elphel drivers are compiled inside of the kernel (i.e., not as modules). Since they use resources
like the interruption line, they must be removed from the kernel, because to do proper DMA we need to be able to handle the interruption issued when the FPGA finishes the data transfer.

To remove these parts, you'll have to edit the file <tt>elphel353/os/linux-2.6/arch/cris/arch-v32/drivers/elphel/Makefile</tt> so it looks like this
obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_common.o
obj-$(CONFIG_ETRAX_ELPHEL353) += fpgajtag.o
obj-$(CONFIG_ETRAX_ELPHEL353) += fpgaclocks.o
obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_sdram.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_io.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += fpgactrl.o

#obj-$(CONFIG_ETRAX_ELPHEL353) += cc3x3.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += cxi2c.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += cxsdram.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += cxdma.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += hist.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += ext353.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += circbuf.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += exif353.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += stream.o

#obj-$(CONFIG_ETRAX_ELPHEL_MT9X001) += mt9x001.o
#obj-$(CONFIG_ETRAX_ELPHEL347) += ???.o
#obj-$(CONFIG_ETRAX_ELPHEL_KAI11002) += ???.o
#obj-$(CONFIG_ETRAX_ELPHEL_KAI16000) += ???.o

=== Creation of a new device node ===
Edit the file <tt>elphel353-7.1.7.26/elphel353/packages/devices/elphel/Makefile</tt> and add the line
@$(MKNOD) -m 0666 $(DEV)/dma_test c 42 0
after the <tt>install</tt> target. Take into account that behind the character behind the <tt>@</tt> is a TAB. It is not spaces.

=== Reflashing camera ===
Once both of these modifications are made, it is possible to reflash the camera following the usual procedure ([[Firmware_upgrade]])
== Driver compilation ==
The tarball attached to this page has a Makefile. To use it you have to edit it to correct the path to the kernel Makefile of the elphel tree (<tt>elphel353-7.1.7.26/elphel353/...</tt>). You also have to make sure of putting gcc-cris into the PATH (by sourcing the <tt>init_env</tt> located in the elphel tree).

Once this is done, you can compile the driver by simply issuing the command <tt>make</tt>
== Testing DMA ==
Log into the camera, upload there both the driver (<tt>frame_reader.ko</tt>) and the FPGA image (<tt>fpga.bit</tt>). A good place to put these files is <tt>/var</tt>. Once there, issue the following commands
fpcf -X 0 100
cat fpga.bit > /dev/fpgaconfjtag
insmod frame_reader.ko
after this, it is possible to test the DMA transfer by reading from the file <tt>/dev/dma_test</tt>. Like this
cat /dev/dma_test > out_file
The FPGA is programmed to write a sequence of numbers into the DMA buffer, and the driver is programmed to read the DMA buffer (once the FPGA has filled it) and pass it to user space. If the transfer was successful, the resulting file will have a size of 307200 bytes, or 76800 DWORDs. Each DWORD contains a number. The first DWORD has the number 0, the second has a 1, and so on. Inside the tarball attached there is a python program that reads the file and says if the data is consistent or not.

== Notes/bugs/etc ==
* Every once in a while the reading of the <tt>/dev/dma_test</tt> fails. It is still not clear why this happens. The failure doesn't seem to be related to the amount of times the device is read. Tracking the driver execution with printk shows that the system hangs when writing to the FPGA (when passing the DMA address).
#define MEM_CSP4_START (0x20000000)
#define MEM_NON_CACHEABLE (0x80000000)
...
volatile unsigned long *fpga;
...
... (on module (driver) inititialization)
fpga = ioremap(MEM_CSP4_START|MEM_NON_CACHEABLE, 8192);
... (on device reading)
printk("Yadda Yadda\n");
fpga[0x84] = dmabuf_phys_addr;
The last thing heard from the kernel (on the serial port) is the "Yadda Yadda" message. No kernel dumps/oops/etc.
We have also made tests with and without interruptions from the FPGA, and the problem arises on both cases.
* There seem to be issues with the frequency used. So far a frequency of 100Mhz is needed for the data transfer to work properly. Higher frequencies (e.g. 160Mhz) lead to problems when the FPGA writes the data to the DMA buffer, and sometimes to problems when passing the address of the DMA buffer to the FPGA (which makes the FPGA fill some random location of the memory with a nice sequence). It is important to note that since the SDRAM memory works with at most 130Mhz, the FPGA uses a fake clock of half the frequency, so it works with the SDRAM memory at, at most, 80Mhz.

== Files ==
[[Media:Dma-driver.tar.gz]]

Direct DMA

2008-07-24T21:27:07Z

JavierRojas:

= Direct DMA =
This is an implementation of DMA between the FPGA and the main memory of an Elphel 353 camera.
== Requirements/preparation ==
We used the 7.1.7.26 version of the Elphel software. To use this driver you will have to recompile the kernel
(and because of this, also reflash the camera), and create a proper device file.

Since we will be using a new device (file in /dev), we'll have to create it as well; but, since the /dev filesystem is read-only, this must be done when creating the camera image (and therefore requires reflashing the camera).

Needless to say, you can recompile the kernel and arrange for the creation of the device at the same time so you will only have to reflash the camera once to apply both changes.

=== Kernel recompilation ===
The original Elphel drivers are compiled inside of the kernel (i.e., not as modules). Since they use resources
like the interruption line, they must be removed from the kernel, because to do proper DMA we need to be able to handle the interruption issued when the FPGA finishes the data transfer.

To remove these parts, you'll have to edit the file <tt>elphel353/os/linux-2.6/arch/cris/arch-v32/drivers/elphel/Makefile</tt> so it looks like this
obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_common.o
obj-$(CONFIG_ETRAX_ELPHEL353) += fpgajtag.o
obj-$(CONFIG_ETRAX_ELPHEL353) += fpgaclocks.o
obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_sdram.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_io.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += fpgactrl.o

#obj-$(CONFIG_ETRAX_ELPHEL353) += cc3x3.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += cxi2c.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += cxsdram.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += cxdma.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += hist.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += ext353.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += circbuf.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += exif353.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += stream.o

#obj-$(CONFIG_ETRAX_ELPHEL_MT9X001) += mt9x001.o
#obj-$(CONFIG_ETRAX_ELPHEL347) += ???.o
#obj-$(CONFIG_ETRAX_ELPHEL_KAI11002) += ???.o
#obj-$(CONFIG_ETRAX_ELPHEL_KAI16000) += ???.o

=== Creation of a new device node ===
Edit the file <tt>elphel353-7.1.7.26/elphel353/packages/devices/elphel/Makefile</tt> and add the line
@$(MKNOD) -m 0666 $(DEV)/dma_test c 42 0
after the <tt>install</tt> target. Take into account that behind the character behind the <tt>@</tt> is a TAB. It is not spaces.

=== Reflashing camera ===
Once both of these modifications are made, it is possible to reflash the camera following the usual procedure ([[Firmware_upgrade]])
== Driver compilation ==
The tarball attached to this page has a Makefile. To use it you have to edit it to correct the path to the kernel Makefile of the elphel tree (<tt>elphel353-7.1.7.26/elphel353/...</tt>). You also have to make sure of putting gcc-cris into the PATH (by sourcing the <tt>init_env</tt> located in the elphel tree).

Once this is done, you can compile the driver by simply issuing the command <tt>make</tt>
== Testing DMA ==
Log into the camera, upload there both the driver (<tt>frame_reader.ko</tt>) and the FPGA image (<tt>fpga.bit</tt>). A good place to put these files is <tt>/var</tt>. Once there, issue the following commands
fpcf -X 0 100
cat fpga.bit > /dev/fpgaconfjtag
insmod frame_reader.ko
after this, it is possible to test the DMA transfer by reading from the file <tt>/dev/dma_test</tt>. Like this
cat /dev/dma_test > out_file
The FPGA is programmed to write a sequence of numbers into the DMA buffer, and the driver is programmed to read the DMA buffer (once the FPGA has filled it) and pass it to user space. If the transfer was successful, the resulting file will have a size of 307200 bytes, or 76800 DWORDs. Each DWORD contains a number. The first DWORD has the number 0, the second has a 1, and so on. Inside the tarball attached there is a python program that reads the file and says if the data is consistent or not.

== Notes/bugs/etc ==
* Every once in a while the reading of the <tt>/dev/dma_test</tt> fails. It is still not clear why this happens. As far as what the driver can see, it is not receiving any interruption from the FPGA, and therefore it hangs. In other tries the kernel crashes.
* There seem to be issues with the frequency used. So far a frequency of 100Mhz is needed for the data transfer to work properly. Higher frequencies (e.g. 160Mhz) lead to problems when the FPGA writes the data to the DMA buffer, and sometimes to problems when passing the address of the DMA buffer to the FPGA (which makes the FPGA fill some random location of the memory with a nice sequence). It is important to note that since the SDRAM memory works with at most 130Mhz, the FPGA uses a fake clock of half the frequency, so it works with the SDRAM memory at, at most, 80Mhz.

== Files ==
[[Media:Dma-driver.tar.gz]]

File:Dma-driver.tar.gz

2008-07-24T21:25:04Z

JavierRojas:

Direct DMA

2008-07-24T21:03:51Z

JavierRojas:

= Direct DMA =
This is an implementation of DMA between the FPGA and the main memory of an Elphel 353 camera.
== Requirements/preparation ==
We used the 7.1.7.26 version of the Elphel software. To use this driver you will have to recompile the kernel
(and because of this, also reflash the camera), and create a proper device file.

Since we will be using a new device (file in /dev), we'll have to create it as well; but, since the /dev filesystem is read-only, this must be done when creating the camera image (and therefore requires reflashing the camera).

Needless to say, you can recompile the kernel and arrange for the creation of the device at the same time so you will only have to reflash the camera once to apply both changes.

=== Kernel recompilation ===
The original Elphel drivers are compiled inside of the kernel (i.e., not as modules). Since they use resources
like the interruption line, they must be removed from the kernel, because to do proper DMA we need to be able to handle the interruption issued when the FPGA finishes the data transfer.

To remove these parts, you'll have to edit the file <tt>elphel353/os/linux-2.6/arch/cris/arch-v32/drivers/elphel/Makefile</tt> so it looks like this
obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_common.o
obj-$(CONFIG_ETRAX_ELPHEL353) += fpgajtag.o
obj-$(CONFIG_ETRAX_ELPHEL353) += fpgaclocks.o
obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_sdram.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_io.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += fpgactrl.o

#obj-$(CONFIG_ETRAX_ELPHEL353) += cc3x3.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += cxi2c.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += cxsdram.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += cxdma.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += hist.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += ext353.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += circbuf.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += exif353.o
#obj-$(CONFIG_ETRAX_ELPHEL353) += stream.o

#obj-$(CONFIG_ETRAX_ELPHEL_MT9X001) += mt9x001.o
#obj-$(CONFIG_ETRAX_ELPHEL347) += ???.o
#obj-$(CONFIG_ETRAX_ELPHEL_KAI11002) += ???.o
#obj-$(CONFIG_ETRAX_ELPHEL_KAI16000) += ???.o

=== Creation of a new device node ===
Edit the file <tt>elphel353-7.1.7.26/elphel353/packages/devices/elphel/Makefile</tt> and add the line
@$(MKNOD) -m 0666 $(DEV)/dma_test c 42 0
after the <tt>install</tt> target. Take into account that behind the character behind the <tt>@</tt> is a TAB. It is not spaces.

=== Reflashing camera ===
Once both of these modifications are made, it is possible to reflash the camera following the usual procedure ([[Firmware_upgrade]])
== Driver compilation ==
The tarball attached to this page has a Makefile. To use it you have to edit it to correct the path to the kernel Makefile of the elphel tree (<tt>elphel353-7.1.7.26/elphel353/...</tt>). You also have to make sure of putting gcc-cris into the PATH (by sourcing the <tt>init_env</tt> located in the elphel tree).

Once this is done, you can compile the driver by simply issuing the command <tt>make</tt>
== Testing DMA ==
Log into the camera, upload there both the driver (<tt>frame_reader.ko</tt>) and the FPGA image (<tt>fpga.bit</tt>). A good place to put these files is <tt>/var</tt>. Once there, issue the following commands
fpcf -X 0 100
cat fpga.bit > /dev/fpgaconfjtag
insmod frame_reader.ko
after this, it is possible to test the DMA transfer by reading from the file <tt>/dev/dma_test</tt>. Like this
cat /dev/dma_test > out_file
The FPGA is programmed to write a sequence of numbers into the DMA buffer, and the driver is programmed to read the DMA buffer (once the FPGA has filled it) and pass it to user space. If the transfer was successful, the resulting file will have a size of 307200 bytes, or 76800 DWORDs. Each DWORD contains a number. The first DWORD has the number 0, the second has a 1, and so on. Inside the tarball attached there is a python program that reads the file and says if the data is consistent or not.

== Notes/bugs/etc ==
* Every once in a while the reading of the <tt>/dev/dma_test</tt> fails. It is still not clear why this happens. As far as what the driver can see, it is not receiving any interruption from the FPGA, and therefore it hangs. In other tries the kernel crashes.
* There seem to be issues with the frequency used. So far a frequency of 100Mhz is needed for the data transfer to work properly. Higher frequencies (e.g. 160Mhz) lead to problems when the FPGA writes the data to the DMA buffer, and sometimes to problems when passing the address of the DMA buffer to the FPGA (which makes the FPGA fill some random location of the memory with a nice sequence). It is important to note that since the SDRAM memory works with at most 130Mhz, the FPGA uses a fake clock of half the frequency, so it works with the SDRAM memory at, at most, 80Mhz.

Direct DMA

2008-07-24T19:35:51Z

JavierRojas: /* Kernel recompilation */

Direct DMA

2008-07-24T19:22:23Z

JavierRojas: /* Kernel recompilation */

Direct DMA

2008-07-24T19:20:52Z

JavierRojas:

Firmware upgrade

2008-07-11T21:02:30Z

JavierRojas: corrected mac address: it had Oes (the vowel) instead of 0 (zeroes)

== Firmware Installation and Upgrade ==
=== Background ===
We are now reworking the process of the initial installation and upgrade of the software in Elphel cameras (starting with 353/363).
So far it was a modified version of [http://www.axis.com Axis communications] e100boot and fsboot, that after receiving the compressed
flash image from the host over the network was erasing and writing the flash chip.

It was OK with the NOR flash, but even with it I had to rewrite the flashing software as Axis supported only AMD type of NOR flash, but not Intel/Micron we used in the camera. With the NAND flash and ETRAX FS things got more complicated - handling NAND flash and bad block needs more code. In 2006 I had to port MTD drivers to the bootloader myself because Axis SDK that supports NAND flash was not ready then. In spring 2007 that new SDK (2.10) was released and so we need to merge it with our hardware-specific modifications once again. And this merging among other things includes the bootloader too.
=== Problems with the old bootloader ===
* Need to overhaul the bootloader with each SDK change
* duplicate port of MTD drivers and mtd-utils in the bootloader - possible mismatch with those running with the booted system
* problems with handling of the flash larger than RAM - normally bootloader downloads the image to RAM first, then writes it to flash
=== New software installation process ===
All the time I was using Axis ETRAX and software it had "ktest" - ability to boot system from the network bypassing flash. And most of the time that code was broken - sometimes it did not work in SDK, sometimes - we broke it ourselves. Now we fixed it and the system boots without doing anything with flash.

How does it help?

We have running a "normal" system running, with the current mtd drivers and mtd-utils - same ones that will be used in the camera during normal operation. The actual installation of the flash images is implemented using MTD drivers and mdt-utils that know how to skip the bad blocks in the NAND flash. The data to write to flash is received by the camera from the NFS server using regular network protocols that handle lost packets much better than spartan network implementation in the bootloader, so multiple cameras can be programmed at the same time without interfering with each other.

After booting from the network (no flash used) cameras will receive IP addresses from the DHCP server using either earlier assigned MAC address (written in bootblock of the flash) or, if it is a brand new camera, temporary use random MAC address. Next step - running installation script from the NFS server that, among others will assign permanent serial number/MAC address to each new camera. Different scripts can be prepared for different tasks - upgrading a single camera, production programming of the multiple cameras, programming of the specialized multi-camera systems.

Such installation process can use either complete images for each JFFS2 partition or it can just install uncompressed files.

Similar procedure can be used to read flash from the camera for postmortem analysis of the data if camera does not boot for some reason.

=== TODO ===

1. It would be nice to make it possible for the camera to boot same kernel + root file system from flash, not only from the network. That will simplify remote firmware upgrade (that does not require pressing the button on the camera)

2. Pass parameter string to the kernel through the network bootloader (i.e. serial number fro the new camera) and during reboot (too instruct flash bootloader not to mount flash but use RAM instead)

--[[User:Andrey.filippov|Andrey.filippov]] 12:07, 19 June 2007 (CDT)

Most of the TODO is already done, below is the [http://elphel.cvs.sourceforge.net/*checkout*/elphel/elphel353-2.10/README.flash?revision=1.3 README.flash] included with the current software:

=== How to install from CVS. ===

<pre>
mkdir -p /nfs/elphel353-2.10/
chmod 777 -R /nfs/elphel353-2.10/
cvs -d:pserver:anonymous@elphel.cvs.sourceforge.net:/cvsroot/elphel login
cvs -z3 -d:pserver:anonymous@elphel.cvs.sourceforge.net:/cvsroot/elphel co -P elphel353-2.10
./install_elphel_from_cvs
cp -R ../353/* .
./install_elphel
</pre>

=== Install NFS server on your computer ===

To install NFS server on your Debian/Ubuntu GNU/Linux execute:

<pre>
sudo apt-get install nfs-kernel-server nfs-common portmap
</pre>

When configuring portmap do ***not*** bind loopback.

Now configure exported file systems:

<pre>
sudo vi /etc/exports
</pre>

and put this text inside:

<pre>
/nfs/elphel353-2.10 192.168.0.15/24(rw,no_root_squash)
</pre>

last stage:

<pre>
sudo /etc/init.d/nfs-kernel-server restart
sudo exportfs -a
</pre>

Your NFS server should be up and running.

=== How to reflash (update firmware) in the camera. ===

==== I. Reflash from camera: ====

This work only if you upgrade from 7.1.*, if you are upgrading from 7.0.* you need to use method II.

<pre>
1. login with telnet to camera with login "root" and password "pass"
2. edit file /etc/reflash.conf, if needed, and change NFS server address and directory, where placed firmware
3. type "reflash"
4. check flash log file on NFS server in directory with firmware, log file name is flash.log.MAC-ADDRESS_of_camera
</pre>

==== II. Reflash from computer with ktest: ====

Your current working directory should be the CVS tree. (normally it should be ~/elphel353-2.10

Do not forget to change the mac address, the camera's IP and the IP of your NFS server (your computer)

If you are performing an upgrade from 7.0.* you should use "cmd=all" in other case use "cmd=update"

Unlike the previous method for this one you need to press the green button on the back of the camera while booting.

<pre>
cd elphel353/
sudo su
. ./init_env
./ktest -d eth1 mac=000E64000000 ip=192.168.0.9 nfs=192.168.0.15:/nfs/elphel353-2.10 cmd=update
</pre>

ktest parameters & options:

* [-d ethX] - network interface to camera, by default, eth0
* mac=OOOE64000000 - MAC address for camera; if SERNO in camera boot block not present, this MAC address will be written in bootblock
* ip=xxx.xxx.xxx.xxx - ip for camera (by default, 192.168.0.9)
* nfs=192.168.0.15:/nfs/elphel353-2.10 - NFS server, where placed firmware.
* cmd=boot|update|all

** boot - just boot camera, don't touch flash memory, by default
** update - update firmware (results as in II)
** all - rewrite all partitions on camera (be careful with this option)

When the camera started from ktest, it boot, mount NFS share and write the firmware to the flash.

If the firmware was flashed successfully you should see a log file in /nfs/elphel353-2.10 finishing with "done".

Only after this you can unplug the Ethernet cable for 10 seconds and reconnect the camera. (those 10 seconds are due to a bug that will be corrected in near future)

==== Troubleshouting ====

If you see only "Starting boot..." and nothing after it mean what you forgot the ktest's "-d" parameter or did not press the green button on the camera when power up.

If you see something after "Starting boot...", but do not see any log files in your NFS share after one minute it probably mean what the camera was not able to mount the NFS share. The camera will remain booted from the network and accessible trouth telnet. You may telnet to the camera and manually try to execute one by one commands from /bin/reflash script. (just to see what's wrong)

RawImage

2008-06-12T20:20:17Z

JavierRojas:

The raw sensor data can be obtained on the 353 camera by reading from the /dev/ccam_img file.

However, I am having some problems doing this:
#. Trying to read from /dev/ccam_img right after booting the camera causes the reading process (I've tried dd and cat) to take all the CPU and block. Only after opening the web page of the camera (192.168.0.9) the proccess finishes (with error :( read error: invalid argument)

#. Once I get the raw sensor data and convert it to an image (a PNG), I get the expected picture... sometimes. See URL for the results

RawImage

2008-06-12T19:37:01Z

JavierRojas:

The raw sensor data can be obtained on the 353 camera by reading from the /dev/ccam_img file.

However, I am having