Direct DMA
Contents
Direct DMA
This is an implementation of DMA between the FPGA and the main memory of an Elphel 353 camera.
Requirements/preparation
We used the 7.1.7.26 version of the Elphel software. To use this driver you will have to recompile the kernel (and because of this, also reflash the camera), and create a proper device file.
Since we will be using a new device (file in /dev), we'll have to create it as well; but, since the /dev filesystem is read-only, this must be done when creating the camera image (and therefore requires reflashing the camera).
Needless to say, you can recompile the kernel and arrange for the creation of the device at the same time so you will only have to reflash the camera once to apply both changes.
Kernel recompilation
The original Elphel drivers are compiled inside of the kernel (i.e., not as modules). Since they use resources like the interruption line, they must be removed from the kernel, because to do proper DMA we need to be able to handle the interruption issued when the FPGA finishes the data transfer.
To remove these parts, you'll have to edit the file elphel353/os/linux-2.6/arch/cris/arch-v32/drivers/elphel/Makefile so it looks like this
obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_common.o obj-$(CONFIG_ETRAX_ELPHEL353) += fpgajtag.o obj-$(CONFIG_ETRAX_ELPHEL353) += fpgaclocks.o obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_sdram.o #obj-$(CONFIG_ETRAX_ELPHEL353) += fpga_io.o #obj-$(CONFIG_ETRAX_ELPHEL353) += fpgactrl.o #obj-$(CONFIG_ETRAX_ELPHEL353) += cc3x3.o #obj-$(CONFIG_ETRAX_ELPHEL353) += cxi2c.o #obj-$(CONFIG_ETRAX_ELPHEL353) += cxsdram.o #obj-$(CONFIG_ETRAX_ELPHEL353) += cxdma.o #obj-$(CONFIG_ETRAX_ELPHEL353) += hist.o #obj-$(CONFIG_ETRAX_ELPHEL353) += ext353.o #obj-$(CONFIG_ETRAX_ELPHEL353) += circbuf.o #obj-$(CONFIG_ETRAX_ELPHEL353) += exif353.o #obj-$(CONFIG_ETRAX_ELPHEL353) += stream.o #obj-$(CONFIG_ETRAX_ELPHEL_MT9X001) += mt9x001.o #obj-$(CONFIG_ETRAX_ELPHEL347) += ???.o #obj-$(CONFIG_ETRAX_ELPHEL_KAI11002) += ???.o #obj-$(CONFIG_ETRAX_ELPHEL_KAI16000) += ???.o
Creation of a new device node
Edit the file elphel353-7.1.7.26/elphel353/packages/devices/elphel/Makefile and add the line
@$(MKNOD) -m 0666 $(DEV)/dma_test c 42 0
after the install target. Take into account that behind the character behind the @ is a TAB. It is not spaces.
Reflashing camera
Once both of these modifications are made, it is possible to reflash the camera following the usual procedure (Firmware_upgrade)
Driver compilation
The tarball attached to this page has a Makefile. To use it you have to edit it to correct the path to the kernel Makefile of the elphel tree (elphel353-7.1.7.26/elphel353/...). You also have to make sure of putting gcc-cris into the PATH (by sourcing the init_env located in the elphel tree).
Once this is done, you can compile the driver by simply issuing the command make
Testing DMA
Log into the camera, upload there both the driver (frame_reader.ko) and the FPGA image (fpga.bit). A good place to put these files is /var. Once there, issue the following commands
fpcf -X 0 100 cat fpga.bit > /dev/fpgaconfjtag insmod frame_reader.ko
after this, it is possible to test the DMA transfer by reading from the file /dev/dma_test. Like this
cat /dev/dma_test > out_file
The FPGA is programmed to write a sequence of numbers into the DMA buffer, and the driver is programmed to read the DMA buffer (once the FPGA has filled it) and pass it to user space. If the transfer was successful, the resulting file will have a size of 307200 bytes, or 76800 DWORDs. Each DWORD contains a number. The first DWORD has the number 0, the second has a 1, and so on. Inside the tarball attached there is a python program that reads the file and says if the data is consistent or not.
Notes/bugs/etc
- Every once in a while the reading of the /dev/dma_test fails. It is still not clear why this happens. The failure doesn't seem to be related to the amount of times the device is read. Tracking the driver execution with printk shows that the system hangs when writing to the FPGA (when passing the DMA address).
#define MEM_CSP4_START (0x20000000)
#define MEM_NON_CACHEABLE (0x80000000)
...
volatile unsigned long *fpga;
...
... (on module (driver) inititialization)
fpga = ioremap(MEM_CSP4_START|MEM_NON_CACHEABLE, 8192);
... (on device reading)
printk("Yadda Yadda\n");
fpga[0x84] = dmabuf_phys_addr;
The last thing heard from the kernel (on the serial port) is the "Yadda Yadda" message. No kernel dumps/oops/etc. We have also made tests with and without interruptions from the FPGA, and the problem arises on both cases.
- There seem to be issues with the frequency used. So far a frequency of 100Mhz is needed for the data transfer to work properly. Higher frequencies (e.g. 160Mhz) lead to problems when the FPGA writes the data to the DMA buffer, and sometimes to problems when passing the address of the DMA buffer to the FPGA (which makes the FPGA fill some random location of the memory with a nice sequence). It is important to note that since the SDRAM memory works with at most 130Mhz, the FPGA uses a fake clock of half the frequency, so it works with the SDRAM memory at, at most, 80Mhz.
