Porting Theora to 353 cameras

Elphel model 333 cameras were able to encode and stream Ogg Theora with 6.3.9 firmware. However, this model is no longer produced by Elphel. The goal of this project is to use the full potential of 353 cameras (larger FPGA, faster CPU) in order to port the Ogg Theora feature on the new hardware. First, the camera should be able to upload small Theora clips via FTP.

Different solutions are possible as previously mentioned in the Theora page :

• Instantiate both the Theora compressor and the MJPEG compressor in the FPGA since the new Spartan 3E seems to be large enough to handle the two features.
• Replace the existing MJPEG compressor with the Theora compressor and implement software Theora to JPEG conversion if JPEG is needed.
• Replace the existing MJPEG compressor in the 353 camera with the Theora compressor and keep two configuration files on the camera filesystem (one for Theora the other for MJPEG). FPGA code could be changed at run-time according to the needs.

Dealing with the last solution seems to be a good start for this project as the other features mentioned can be added later. In all cases, both FPGA code and software code will need to be adapted.

Hardware updates

Compressor

Replace the MJPEG compressor in the 353 with the 333 Theora compressor (333 Theora FPGA code). Maybe some adaptations need to be done within the Theora compressor itself. However, assuming that the compressor worked in 333 cameras a simple instantiation should suffice.

Memory controller

Replace the 4-channel memory controller with an 8-channel memory interface as in the 333 model in order to handle Theora specific data.

Channel	333 Theora	353
0	8/16 bit data from sensor to SDRAM	Data from the sensor (it can be combined with FPN correction data) to the SDRAM in either 8-bit mode or 16-bit
1	16 bit data from SDRAM to FPN correction module	Reads FPN data from SDRAM (8bit - subtract, 8 bit - scale) and feeds it to FPGA for FPN correction of the data on the way from sensor to SDRAM (through channel 0)
2	20x20 tiles of raw data from SDRAM to the color space converter and compressor	Provides data to JPEG encoder in 16x16x8bit MCUs. It can also provide data directly to CPU through DMA bypassing any compression
3	PIO read/write of SDRAM data	Provides bidirectional access from CPU through 8x32 bit window.
4	Writing current (restored) frame as YCbCr 4:2:0 data to SDRAM
5	Reading previous frame as YCbCr 4:2:0 data from SDRAM
6	Writing 12-bit pre-tokens to SDRAM (in macroblock-scan order, starting with AC[63]
7	reading 12-bit pre-tokens from SDRAM in indexed coded order

Sensor interface

sensorpix block must be adapted in order to provide a coherent interface between the memory controller and the sensor (channels 0 and 1)

System Interface

Adapt the system interface to assure communication between FPGA and its environment.

A feature of the system interface is to establish communication between the system CPU and the different blocks within the FPGA (compressor, memory controller etc). This is done by selecting a block with the address bus a[12:0] connected to the internal address bus ial[7:0]:

x353.v
always @ (posedge cwr) begin
   a0[7:0]<=ial[7:0];  // may want to change to just ia if ia will be late enough;
   wnr0                <= ~irnw;
   da_ctl0             <= (ial[7:0]==8'h00);  // 0x00 WE to control 32-bit register (1 loc)
   da_dmamode0         <= (ial[7:0]==8'h01);  // 0x01 select writing to dma_cntr/dma_raw (1 loc)
   da_sensormode0      <= (ial[7:0]==8'h02);  // 0x02 select writing to sensorpix (1 loc)
   ...
   da_compressor0      <= (ial[7:1]==7'h06);  // 0x0c..0x0d - will be farther decoded in the compressor module
   ...

Since memory controller and compressor are to be replaced these addresses have to be adapted.

353 system interface address table

Address	da signal	Comments
0x00	da_ctl0	WE to control 32-bit register (1 loc)
0x01	da_dmamode0	select writing to dma_cntr/dma_raw (1 loc)
0x02	da_sensormode0	select writing to sensorpix (1 loc)
0x03	da_virttrig0	write virtual trigger threshold
0x04	da_sensortrig0	select writing to sensorpix (1 loc)
0x05	da_sensortrig_lines0	write number of lines to be transferred in a frame (or after trigger)
0x07	da_sens_dc0	write to sensor DCDC converter frequency divider
0x08	da_dcm0	tune SDRAM clock phase
0x09	da_saturation0	write color saturation values
0x0a	da_framesync_dly0	write frame sync interrupt delay (in scan lines)
0x0c..0x0d	da_compressor0	will be farther decoded in the compressor module
0x0e	da_table_a0	write tables address
0x0f	twr0	write tables data
0x10		Status register
0x1b	dcmrst	async signal to restart DCMs (if irnw=0)
0x1c..0x1f	da_interrupts0	interrupt control
0x20..0x2f	da_dswe0	select reading/writing to mcontr (16 locations)
0x30	da_mem0	read/write to SDRAM buffer, autoincrement address
0x40..0x43	da_hist0	write/read histogram related data/registers
0x44..0x47	da_rtc0	real time clock
0x48	da_timestamp0	write timestamp mode
0x74	da_xjtag0	write to external (sensor board) JTAG
0x78..0x7b	da_extsync0	control of external sync module 0x78 - 0x7b

333 system interface address table

Address	da signal	Comments
0x00	da_ctl0	WE to control 32-bit register (1 loc)
0x01	da_dmamode0	select writing to dma_cntr/dma_raw (1 loc)
0x02	da_sensormode0	select writing to sensorpix (1 loc)
0x03	da_virttrig0	write virtual trigger threshold
0x04	da_sensortrig0	select writing to sensorpix (1 loc)
0x05	da_sensortrig_lines0	write number of lines to be transferred in a frame (or aftre trigger)
0x06	da_interrupts0	interrupt mask
0x07	da_sens_dc0	write to sensor DCDC converter frequency divider
0x08	da_dcm0	tune SDRAM clock phase
0x10	da_debug	write 0x10 any - load debug shift, read (through cs4) -shift
0x20..0x27	da_mcontr0	select reading/writing to mcontr (8 locations)
0x28..0x2f	da_compressor0	will be farther decoded in the compressor module
0x2c..0x2d	da_table0	write gamma tables address (same as in compressor)
0x30	da_mem0	read/write to SDRAM buffer, autoincrement address

Addresses 0x01,2,3,4,5,6,8 and 0x30 can be kept in the 353 system interface. The interrupt mask address 0x06 in the 333 sys interface is no longer used (353 uses interrupt vectors). The Theora compressor and the new memory controller addresses have to be mapped again in the new design (addresses 0x0c..0x0d and 0x20 to 0x2f in the 353 sys interface).

Interrupts

Adapt the Theora compressor interrupts to the 353 interrupt vector .

The 353 interrupt vector can handle up to 16 different irq. However, only irq 0 to 7 are used :

x353.v

assign irq_in= {8'b0,                    // extra
               compressor_done_compress, // 7 - will go high after all data is sent out (reset by compressor)
               compressor_done_input,    // 6 - will go high after EOT and persist until DCT is enabled (reset by compressor)
               dcc_rdy,                  // 5 - obsolete in 333 - channel3 has 128 more dc coefficients (or some if the compression is finished)
               compressor_eot,           // 4 - compressor read in the last MCU (predictable time to end of transfer)
               sr_sensortrig[2],         // 3 - (sync to pclk) - level, reset by writing to sensor triggering command register
               xfer_over_irq,            // 2 - (sync to pclk) - frame acquisition over
               trig_irq,                 // 1 - (sync to pclk) - external trigger
               vacts};                   // 0 - (sync to pclk) - frame sync

353 MJPEG compressor triggers irqs 4, 6 and 7. 333 Theora compressor is the source of two irqs :

compressor_all.v

done,           // single-cycle pulse when the compressed frame is sent out (IRQ source).
reset_compress, // reset compressor done interrupt (writing any compressor command)

Adaptation should consist of assigning Theora compressor irq sources to vectors 6 and 7 for example if the MJPEG compressor is to be replaced.

DMA FIFOs

353 design uses 2 DMA FIFOs between the compressor and the system interface instead of one for the 333 Theora compressor. The two DMA FIFOs have also slight differences between the two design. Maybe a single 333 FIFO should be kept for the final design (?)

FPGA -> CPU data out MUX

The purpose of this MUX is to route data out from the different FPGA blocks to the system interface and then to the data bus to the CPU. The selection is made with the internal address bus ia.

                                  _
                               --| \
                               --|  |
                               --|  |
                               --|  |--- iod[31:0] -> to system interface
                               --|  |
                               --|  |
                                 |_/
from system interface -> ia[7:0]__|

333 MUX

ia(5)	ia(4)	iod[31:0]	Comments
0	0	dma_d[31:0]	DMA data
0	1	rd_regs[31:0]	Data from internal registers
1	0	bdo[31:0]	32-bit data from SDRAM channel 3
1	1	{8'h0,xfer_bytes[23:0]}	Frame length (in bytes)

353 MUX

ia(6)	ia(5)	ia(4)	ia(2)	ia(0)	iod[31:0]	Comments
0	0	0	X	0	dma_d0[31:0]	DMA 0 data
0	0	0	X	1	dma_d1[31:0]	DMA 1 data
0	0	1	X	X	rd_regs[31:0]	Data from internal registers
0	1	0	X	X	dsdo[31:0]	Data out from descriptor memory
0	1	1	X	X	bdo[31:0]	32-bit data from SDRAM channel3
1	0	X	0	X	hist_do[31:0]	Histogram data out, actully only [17:0]
1	0	X	1	0	{12'h0,pio_usec[19:0]}	usec output from RTC
1	0	X	1	1	pio_sec[31:0]	sec output from RTC
1	1	0	X	X	32'h0	0x0000
1	1	1	X	X	{14'b0,xfpgatdo,xfpgastat,4'b0,io_pins[11:0]}	TDO read from an external FPGA / Multiplexed read data from xjtag / Status of 12 i/o pins

Internal registers MUX and status register

333 MUX

Address	ia(1)	ia(0)	rd_regs[31:0]	Comments
0x10	0	0	Status register	See below
0x11	0	1	{24'b0,irqr[7:0]}	Interrupts
0x12	1	0	{5'b0,trig_v[10:0],4'b0,trig_h[11:0]}	Trigger phase
0x13	1	1	MODELREV	Model revision

353 MUX

Address	ia(2)	ia(1)	ia(0)	rd_regs[31:0]	Comments
0x10	0	0	0	Status register	See below
0x11	0	0	1	irqr[31:0]	Interrupts
0x12	0	1	0	{5'b0,trig_v[10:0],4'b0,trig_h[11:0]}	Trigger phase
0x13	0	1	1	MODELREV	Model revision
0x14	1	X	0	{8'h0,imgptr[23:0]}	Image pointer in 32-byte chunks
0x15	1	X	1	hifreq[31:0]	Accumulated high-freq components

333 status register

Bit #	Signal	Comments
0	sr_sda0	I2C to sensor
1	sr_scl0	I2C clock0
2	sr_sda1	I2C
3	sr_scl1	I2C clock1
4..12	rq_busy[8:0]	Request or busy - per mcontr channel
13..15	sr_sensortrig[2:0]	x00 - idle, x01 - waiting vacts to start, x10 - waiting for trigger, x11 - frame over ,1xx - frame done (reset by writing command to sensor_trig
16	sr_memrdy	Ready to r/w next 8x32 page (channel 3) CPU<->SDRAM
17	sr_wrempty	CPU write buffer empty (valid only in write mode)
18	1'b0	0x0
19	dma_empty	dma output buffer empty and dma is enabled (pessimistic - with delay)
20	itrig	TRIG input
21..22	dcm_err[1:0]	00 - no data (SDRAM reads) since last change (wclk*wcmd) , 01 - clock is too late , 10 - clock is too early , 11 - OK (some measurements show too late, some - too early)
23	dcm_done
24	sr_pipe_rdy	Will go active with delay to overlap with DMA busy (external)
25	sr_pipe_nempty	Compressor pipe not empty (updated with done -> IRQ)
26	sr_pipe_full	Compressor pipe has 3 frame compress commands in the input queue/in progress
27	en_refresh	Status bit to find out if SDRAM is programmed
28	tok_frame_num_wr	Odd/even frame of tokens written
29	1'b0	0x0
30	debug_out	Debug shift register output
31	'b0	0x0

353 status register

Bit #	Signal	Comments
0	sr_sda0	I2C to sensor
1	sr_scl0	I2C clock
2	sr_ch0rdy	Ready to write next page (channel 0) SENSOR->SDRAM
3	sr_ch1rdy	Ready to read next page (channel 1) SDRAM->FPN(SENSOR)
4	sr_ch2rdy	Ready to read next page (channel 2) SDRAM->DMA
5	sr_memrdy	Ready to r/w next 8x32 page (channel 3) CPU<->SDRAM
6	sr_wrempty	CPU write buffer empty (valid only in write mode). May reprogram channel3.
7..10	sr_nextFrame[3:0]	(level) generated before actual block is filled - needed to be combined with the pulse from buffer control
11..13	sr_sensortrig[2:0]	x00 - idle, x01 - waiting vacts to start, x10 - waiting for trigger, x11 - frame over ,1xx - frame done (reset by writing command to sensor_trig
14	dcc_rdy	128 of DC components from JPEG encoder are available (same as interrupt)
15	compressor_done_input	will go high after EOT and persist until DCT is enabled
16	compressor_done_compress	will go high after all data is sent out
17	dma_empty	dma output buffer empty and dma is enabled (pessimistic - with delay)
19..18	dcm_err[1:0]	SDRAM DCM phase error
20	dcm_done	SDRAM DCM cmd (inc/dec) done
21	dcm_locked	SDRAM DCM overflow
22	dcm_status[0]	SDRAM DCM overflow
24..23	sens_ph_err[1:0]	Sensor DCM phase error
25	sens_dcm_done	Sensor DCM cmd (inc/dec/reset) done
26	sens_dcm_locked	Sensor DCM overflow
27	sens_dcm_status[0]	Sensor DCM overflow
28	clockios_locked	dcm locked
29..30	clockios_status[1:0]	dcm status (bit 1 - dcm clkin stopped)
31	1'b0	0x0

Control register

The dcr[31:0] bus is used to control the different blocks according to the control register values. The control register can be directly written by the CPU at the address 0x00.

333 control register

bit #	destination block	Comments
0..2	Not used
3	sensorpads	Inverts idclk phase
4	sensorpads	fillfactory sensor mode
5	sensorpads	delay hact pulse by 1
6	sensorpads	cb_zoran
7	sensorpads	skip first scan line from the sensor
8	sensortrig	external trigger polarity (0: 1->0, 1: 0->1)
9	sensorpads	iarst
10	sensorpads	iaro
11	iobuf	EXPS (output pin)
12	iobuf	EXPS (output pin)
13	sensorpads	imrst
14	iobuf	XRST (output pin)
15	i2cpads	scl0 out
16	i2cpads	scl0 enable
17	i2cpads	sda0 out
18	i2cpads	sda0 enable
19	i2cpads	scl1 out
20	i2cpads	scl1 enable
21	i2cpads	sda1 out
22	i2cpads	sda1 enable
23	Not used
24..27	clock_pclk	pclk source
28..30	Not used
31	clock_pclk	clk enable

353 control register

bit #	destination block	Comments
0	Not used
1..2	compressor, sensorpix, histogram
3	sensorpads	Inverts idclk phase
4	sensorpads	fillfactory sensor mode
5	sensorpads	delay hact pulse by 1
6	sensorpads	cb_zoran
7	sensorpads	skip first scan line from the sensor
8	sensortrig	external trigger polarity (0: 1->0, 1: 0->1)
9	sensorpads	iarst
10	sensorpads	iaro
11	sensorpads	0 - use 12-bit sensor data, 1 - use converter output pins
12	sensorpads	sensor trigger
13	sensorpads	imrst
14	iobuf	XRST (output pin)
15	i2cpads	scl0 out
16	i2cpads	scl0 enable
17	i2cpads	sda0 out
18	i2cpads	sda0 enable
19	sensorpads	dclkmode
20	sensorpads	use {vact,hact} as 2 LSB in 14-bit data
21..22	sensorpads	register hact, vact, bpf N/4 Tpclk later than data
23	Not used
24..25	BUFGMUX	pclk source
28..30	compressor
31	clock_pclk

Others

• Looks like Theora compressor requires 90° phase clock that was abandoned in the 353 design.

Using different phases reduces AC component of the current through the power/ground lines of the FPGA by spreading register transitions. Theora has larger core so it may help spreading phases.--Andrey.filippov 13:45, 25 February 2008 (CST)

• ...?

Software updates

• Adapt FPGA definitions (x353.h) and most likely some other FPGA drivers.
• ...?

These lists of update are obviously incomplete. They need to be more detailed in order to make an exhaustive list of the changes that have to be done.

353 Theora new design

Memory controller

Channel	Description
0	8/16 bit data from sensor to SDRAM
1	16 bit data from SDRAM to FPN correction module
2	20x20 tiles of raw data from SDRAM to the color space converter and compressor
3	PIO read/write of SDRAM data
4	Writing current (restored) frame as YCbCr 4:2:0 data to SDRAM
5	Reading previous frame as YCbCr 4:2:0 data from SDRAM
6	Writing 12-bit pre-tokens to SDRAM (in macroblock-scan order, starting with AC[63]
7	Reading 12-bit pre-tokens from SDRAM in indexed coded order

System interface

353 Theora system interface address table

Address	da signal	Comments
0x00	da_ctl0	WE to control 32-bit register (1 loc)
0x01	da_dmamode0	select writing to dma_cntr/dma_raw (1 loc)
0x02	da_sensormode0	select writing to sensorpix (1 loc)
0x03	da_virttrig0	write virtual trigger threshold
0x04	da_sensortrig0	select writing to sensorpix (1 loc)
0x05	da_sensortrig_lines0	write number of lines to be transferred in a frame (or after trigger)
0x07	da_sens_dc0	write to sensor DCDC converter frequency divider
0x08	da_dcm0	tune SDRAM clock phase
0x0a	da_framesync_dly0	write frame sync interrupt delay (in scan lines)
0x10		Status register
0x1b	dcmrst	async signal to restart DCMs (if irnw=0)
0x1c..0x1f	da_interrupts0	interrupt control
0x20..0x27	da_mcontr0	select reading/writing to mcontr (8 locations)
0x28..0x2f	da_compressor0	will be farther decoded in the compressor module
0x2c..0x2d	da_table0	write gamma tables address (same as in compressor)
0x30	da_mem0	read/write to SDRAM buffer, autoincrement address
0x40..0x43	da_hist0	write/read histogram related data/registers
0x44..0x47	da_rtc0	real time clock
0x48	da_timestamp0	write timestamp mode
0x74	da_xjtag0	write to external (sensor board) JTAG
0x78..0x7b	da_extsync0	control of external sync module 0x78 - 0x7b

Interrupts

irq_in	Signal	Comments
15..8	GND	Extra
7	compressor_done_compress	single-cycle pulse when the compressed frame is sent out
6	compressor_reset_compress	reset compressor done interrupt (writing any compressor command)
5	GND	dcc_rdy was tied to GND in the 333 design
4	GND	eot irq was tied to GND in the 333 design
3	sr_sensortrig[2]	(sync to pclk) - level, reset by writing to sensor triggering command register
2	xfer_over_irq	(sync to pclk) - frame acquisition over
1	trig_irq	(sync to pclk) - external trigger
0	vacts	(sync to pclk) - frame sync

DMA FIFOs

A single DMA FIFO from the 333 design will be instanciated.

See also

• Theora
• FPGA Development in Elphel cameras
• FPGA Theora Encoder for Videoconferencing
• 10353