[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: Kernel oops using DMA on MCM



Thank you for your quick response Mikeal.
 
> There are some minor differences in the synchronous serial 
> port between R1
> and R2. But I don't think that is related to your problem. 
> Are the OOPSes 
> identical each time or does tey look different?

The kernel backtraces are not identical at all times. To be more accurate,
the initial
Trace; c0014e8a <do_generic_file_read+3a8/3ae>
Trace; c0026312 <search_binary_handler+52/fe>
Trace; c0025836 <copy_strings+0/1ae>
Trace; c002650e <do_execve+150/1aa>
Trace; c00455f0 <sys_execve+2a/42>
Trace; c00460be <system_call+50/58>
part is identical in each trace I have done so far. Then, at some apparently
random level of function calls, we get a mmu_bus_fault(). The mmu_bus_fault
and some functions that are called from this can repeat a number of times as
well.

I tried using other applications than "cat". I had some test programs for
the driver along with the real user-level application. They all cause kernel
oops when they have the device driver open and the dma is working, similar
to cat. The actual point of crashing seems to be out of when they are
calling read(), and outside interrupt handlers and tasklets either.
Puzzling, really..

> Short version: If the input DMA buffers is in the cache you may 
>                corrupt data at other "random" addresses.
> 		   The bug is present in all ETRAX 100LX revisions.
> Long version at the end of the email if you really want to know.
> 
> To make sure that the DMA buffers two meassures must be taken:
> 1. Make sure that the buffers doesn't share cache line with any
> other data. This is achieved by aligning the buffer to cacheline
> boundary and the size a multiple of the cacheline size.
> 2. Make sure that the buffer is not present in cache when DMA runs.
> This is achieved by calling prepare_rx_descriptor when descriptors
> are returned to the DMA.
> 
> In the Ethernet driver this is rather complicated to increase
> performance. In most other drivers it is simple to add the
> workaround.
> 
> For the synchronous serial port I suggest:
> 
> 1. Align buffers by modifying struct sync_port:
> char in_buffer[IN_BUFFER_SIZE] __attribute__ ((aligned(32)));
> 2. In start_dma_in call prepare_rx_descriptor just before
>    *port->input_dma_first = virt_to_phys(&port->in_descr2); and
>    *port->input_dma_first = virt_to_phys(&port->in_descr1);
> e.g. prepare_rx_descriptor(&port->in_descr2);

I am now going to try your suggestions, thank you. I have used the original
sync_serial driver as a base, but eventually moved very far from it to suit
my requirements (which is continuous full-duplex data transfer regardless of
user-level processes reading/writing). I am pretty sure I can do the
equivalent in my driver, though. If you are interested I can send you its
source as well.

> Now the really long and boring bug description for the people
> with too much time:
> 
> The problem occurs when the CPU performs a cached memory write that 
> reaches over two cache lines, the first cache line is present in the
> cache and the second cache line gives a dirty miss. If the DMA at 
> the same time writes to the cache, the first dword flushed by the 
> CPU miss may get corrupted. The problem is only related to DMA data
> input buffers. It does not occur with DMA descriptors (not even when
> the DMA writes the status field), and it does not occur with output 
> DMA. All input DMA channels are affected.
> Possible workarounds:
> 1. Make sure that the DMA data input buffers are not present in the 
> cache. This can be acheived by accessing an address n * 8 k away from
> the buffer position before the buffer is entered into the DMA receive
> list. One address must be accessed for each cache line. 
> 2. Have the DMA input buffers in non-cached memory.
> 3. Avoid CPU writes that stretch over cache line borders where the
> first cache line is clean and the second cache line is dirty.
> Probably this solution is too difficult to be used in practice. 
*yawn* :)

Actually, option two may also be worthwhile to consider as well. If the CPU
was just not caching the area, the driver would not need to be made any more
complex than it currently is. Do you think this is something feasible to do?