[LinuxBIOS] Opteron caching of device memory

[Marc Jones]

Myles Watson wrote:
>   
>> -----Original Message-----
>> From: linuxbios-bounces at linuxbios.org [mailto:linuxbios-
>> bounces at linuxbios.org] On Behalf Of Marc Jones
>> Sent: Thursday, June 14, 2007 2:39 PM
>> To: Roman Kononov
>> Cc: myles at mouselemur.cs.byu.edu; linuxbios at linuxbios.org
>> Subject: Re: [LinuxBIOS] Opteron caching of device memory
>>
>> Roman Kononov wrote:
>>     
>>> On 06/13/2007 09:39 AM, Myles Watson wrote:
>>>
>>>       
>>>> I'm using LinuxBIOS on my Tyan s2892.  I have a device that maps a lot
>>>> of the memory space, but I'm struggling trying to get the Opteron to
>>>> read and write to my device in larger blocks.  I have set the variable
>>>> MTRRs in the device driver to writeback (witnessed by /proc/mtrr), but
>>>>         
>> I
>>     
>>>> still get 64-bit accesses instead of 64-byte (cache line).
>>>>
>>>>         
>>> Some time ago I officially asked an AMD's rep about this. Someone from
>>>       
>> AMD
>>     
>>> unofficially said that MemIO can be only Write-Combined. And I gave up
>>>       
>> with
>>     
>>> caching.
>>>
>>> Below is what I asked. Maybe you can spot what I missed. Please let me
>>>       
>> know
>>     
>>> if you succeed.
>>>
>>> Roman
>>> --------------------------------------------------------------------
>>> I have a difficulty related to cacheability of Memory-Mapped I/O memory
>>>       
>> in
>>     
>>> Opteron. Can you please assist me?
>>>
>>> I have an Opteron-based system, where the Opteron and an FPGA are
>>>       
>> connected
>>     
>>> using a HyperTransport link. As a target of HT traffic, the FPGA
>>>       
>> responses
>>     
>>> to certain range of physical addresses. I am trying to configure the CPU
>>>       
>> and
>>     
>>> the North Bridge to treat the FPGA as Writeback memory. I cannot do
>>>       
>> this.
>>     
>>> I've been able to make the FPGA Uncacheable, Write-Combining, Write-
>>>       
>> Protect
>>     
>>> and Writethrough. When I try to make it Writeback it behaves as if it
>>>       
>> were
>>     
>>> Writethrough.
>>>
>>> Is it possible to make the FPGA Writeback? If yes, what should I change?
>>>
>>> I have observability of the HT traffic from the FPGA point of view, so I
>>> know what is written and what is requested by the CPU.
>>>
>>> Here is how I program the Opteron:
>>>
>>> The processor is running in the Long Mode with paging enabled.
>>>
>>> A pair of NB PCI Function 1, Memory-Mapped I/O Base and Limit registers
>>>       
>> is
>>     
>>> programmed with the FPGA physical address range, proper link and node
>>>       
>> IDs,
>>     
>>> posted.
>>>
>>> A pair of the variable MTRR Base and Mask (MSR 0x200-0x20f) registers is
>>> programmed with the FPGA physical address range, valid bit and the
>>>       
>> desired
>>     
>>> caching method. Another pair of MTRR registers describes the DRAM, all
>>> others are disabled.
>>>
>>> The PAT register (MSR 0x277) is 0x0606060606060606ull.
>>>
>>> The Page Table Entry for the virtual address mapped to the FPGA has PAT,
>>>       
>> PCD
>>     
>>> and PWT bits cleared.
>>>
>>> Bit CD in the CR0 register is cleared.
>>>
>>>
>>>
>>>       
>> I am not an expert but these are my thoughts....
>>
>> Now that I think about it, it makes sense that you can't set writeback
>> to noncoherent (nonsystem) memory space. What if another device wants to
>> write to that memory. There is no way for the cache to snoop that it was
>> written. You would need a coherent HT link to to your FPGA to get the
>> cache snoop messaging.
>>
>> Marc
>>     
>
> I think that the FPGA should be able to set the coherent bit and cause
> snooping, even if it is not on the coherent bus, because it targets coherent
> memory.  
>
> An example is DMA:
> 1. The processor writes data to DRAM (some of it may/will still be cached)
> 2. The processor writes to the DMA controller (say a SATA controller)
> 3. The DMA transfers the data to the disk drive
>
> If there's no snooping, the DMA controller reads stale data from the DRAM.
>
> Feel free to point out shortcomings in my example.  I need to learn more
> about it.
>
> Myles
>
>
>   
I am sorry, I don't have the expertise to answer your question. I 
recommend that you try the Hyper Transport Consortium and if you still 
can't get an answer let me know. I will try to find the AMD Torrenza 
guys to get us an answer.

http://www.hypertransport.org/
http://enterprise.amd.com/us-en/AMD-Business/Technology-Home/Torrenza.aspx

Marc


-- 
Marc Jones
Senior Software Engineer
(970) 226-9684 Office
mailto:Marc.Jones at amd.com
http://www.amd.com/embeddedprocessors