[LinuxBIOS] Opteron caching of device memory

[Myles Watson]

> -----Original Message-----
> From: linuxbios-bounces at linuxbios.org [mailto:linuxbios-
> bounces at linuxbios.org] On Behalf Of Marc Jones
> Sent: Thursday, June 14, 2007 2:39 PM
> To: Roman Kononov
> Cc: myles at mouselemur.cs.byu.edu; linuxbios at linuxbios.org
> Subject: Re: [LinuxBIOS] Opteron caching of device memory
> 
> Roman Kononov wrote:
> > On 06/13/2007 09:39 AM, Myles Watson wrote:
> >
> >> I'm using LinuxBIOS on my Tyan s2892.  I have a device that maps a lot
> >> of the memory space, but I'm struggling trying to get the Opteron to
> >> read and write to my device in larger blocks.  I have set the variable
> >> MTRRs in the device driver to writeback (witnessed by /proc/mtrr), but
> I
> >> still get 64-bit accesses instead of 64-byte (cache line).
> >>
> >
> > Some time ago I officially asked an AMD's rep about this. Someone from
> AMD
> > unofficially said that MemIO can be only Write-Combined. And I gave up
> with
> > caching.
> >
> > Below is what I asked. Maybe you can spot what I missed. Please let me
> know
> > if you succeed.
> >
> > Roman
> > --------------------------------------------------------------------
> > I have a difficulty related to cacheability of Memory-Mapped I/O memory
> in
> > Opteron. Can you please assist me?
> >
> > I have an Opteron-based system, where the Opteron and an FPGA are
> connected
> > using a HyperTransport link. As a target of HT traffic, the FPGA
> responses
> > to certain range of physical addresses. I am trying to configure the CPU
> and
> > the North Bridge to treat the FPGA as Writeback memory. I cannot do
> this.
> >
> > I've been able to make the FPGA Uncacheable, Write-Combining, Write-
> Protect
> > and Writethrough. When I try to make it Writeback it behaves as if it
> were
> > Writethrough.
> >
> > Is it possible to make the FPGA Writeback? If yes, what should I change?
> >
> > I have observability of the HT traffic from the FPGA point of view, so I
> > know what is written and what is requested by the CPU.
> >
> > Here is how I program the Opteron:
> >
> > The processor is running in the Long Mode with paging enabled.
> >
> > A pair of NB PCI Function 1, Memory-Mapped I/O Base and Limit registers
> is
> > programmed with the FPGA physical address range, proper link and node
> IDs,
> > posted.
> >
> > A pair of the variable MTRR Base and Mask (MSR 0x200-0x20f) registers is
> > programmed with the FPGA physical address range, valid bit and the
> desired
> > caching method. Another pair of MTRR registers describes the DRAM, all
> > others are disabled.
> >
> > The PAT register (MSR 0x277) is 0x0606060606060606ull.
> >
> > The Page Table Entry for the virtual address mapped to the FPGA has PAT,
> PCD
> > and PWT bits cleared.
> >
> > Bit CD in the CR0 register is cleared.
> >
> >
> >
> I am not an expert but these are my thoughts....
> 
> Now that I think about it, it makes sense that you can't set writeback
> to noncoherent (nonsystem) memory space. What if another device wants to
> write to that memory. There is no way for the cache to snoop that it was
> written. You would need a coherent HT link to to your FPGA to get the
> cache snoop messaging.
> 
> Marc

I think that the FPGA should be able to set the coherent bit and cause
snooping, even if it is not on the coherent bus, because it targets coherent
memory.  

An example is DMA:
1. The processor writes data to DRAM (some of it may/will still be cached)
2. The processor writes to the DMA controller (say a SATA controller)
3. The DMA transfers the data to the disk drive

If there's no snooping, the DMA controller reads stale data from the DRAM.

Feel free to point out shortcomings in my example.  I need to learn more
about it.

Myles