[coreboot] PAT memory type issues and cbmem

[Ben Gardner]

On Fri, Oct 30, 2015 at 11:52 AM, Aaron Durbin <adurbin at google.com> wrote:
> On Fri, Oct 30, 2015 at 9:19 AM, Ben Gardner <gardner.ben at gmail.com> wrote:
>> cbmem attempts to map 1 MB of memory to read the CB tables.
>> This is failing on my board due to the PAT configuration under Linux.
>> If I boot Linux with "nopat", the issue goes away.
>>
>> I am using the upstream coreboot and SeaBIOS with a custom board based
>> on the bayleybay_fsp/fsp_baytrail and Linux 4.2.
>>
>> ~ # ./cbmem -V -l
>> Looking for coreboot table at 0 1048576 bytes.
>> Mapping 1MB of physical memory at 0x0 (requested 0x0).
>> Found!
>>   coreboot table entry 0x11
>>     Found forwarding entry.
>> Unmapping 1MB of virtual memory at 0x7fee18224000.
>> Looking for coreboot table at 7ad9f000 1048576 bytes.
>> Mapping 1MB of physical memory at 0x7ad9f000 (requested 0x7ad9f000).
>>   ... failed.  Mapping 1052671B of physical memory at 0x7ad9e000.
>> Failed to mmap /dev/mem: Resource temporarily unavailable
>> ~ # dmesg | tail -n 4
>> [  221.700621] x86/PAT: cbmem.orig:1058 conflicting memory types
>> 7ad9f000-7ae9f000 uncached-minus<->write-back 7adb8000-7adbc000
>> [  221.705419] x86/PAT: reserve_memtype failed [mem
>> 0x7ad9f000-0x7ae9efff], track write-back, req write-back
>> [  221.710435] x86/PAT: cbmem.orig:1058 conflicting memory types
>> 7ad9e000-7ae9f000 uncached-minus<->write-back 7adb8000-7adbc000
>> [  221.715501] x86/PAT: reserve_memtype failed [mem
>> 0x7ad9e000-0x7ae9efff], track write-back, req write-back
>> ~ # cat /sys/kernel/debug/x86/pat_memtype_list
>> PAT memtype list:
>> uncached-minus @ 0x7ada7000-0x7ada8000
>> write-back @ 0x7adb8000-0x7adbc000
>> write-back @ 0x7adbb000-0x7adbd000
>> write-back @ 0x7addd000-0x7adde000
>> uncached-minus @ 0x80600000-0x80601000
>> uncached-minus @ 0x80601000-0x80602000
>> uncached-minus @ 0x80602000-0x80603000
>> write-combining @ 0xc0000000-0xd0000000
>> ... snip ...
>>
>> From the Linux kernel log:
>>  MTRR variable ranges enabled:
>>    0 base 0FF800000 mask FFF800000 write-protect
>>    1 base 000000000 mask F80000000 write-back
>>    2 base 07B000000 mask FFF000000 uncachable
>>    3 base 07C000000 mask FFC000000 uncachable
>>
>> I'm not sure how to read MTRR ranges, but it looks like the cbtable
>> falls in the MTRR 1 with type write-back.
>>
>> From the SeaBIOS log:
>> e820 map has 15 items:
>>   0: 0000000000000000 - 000000000009fc00 = 1 RAM
>>   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
>>   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
>>   3: 0000000000100000 - 000000007ad8e000 = 1 RAM
>>   4: 000000007ad8e000 - 0000000080000000 = 2 RESERVED
>>   5: 00000000e0000000 - 00000000f0000000 = 2 RESERVED
>> ... snip ...
>>
>> The issue appears to be that the coreboot table memory is marked as
>> uncached-minus and then Linux (?) is marking the ACPI tables as
>> write-back.
>> The 1 MB size is overlapping the ACPI tables.
>> That makes mmap attempts of the whole coreboot table to fail due to
>> the mismatched memory types.
>>
>>
>> I have an older version of coreboot from Sage that does not have this issue.
>> The difference is that the first uncached-minus entry is not present.
>>
>> On the Sage-based (4.0) coreboot image:
>> ~ # cat /sys/kernel/debug/x86/pat_memtype_list
>> PAT memtype list:
>> write-back @ 0x7add1000-0x7add2000
>> write-back @ 0x7add4000-0x7add9000
>> write-back @ 0x7adda000-0x7addc000
>> uncached-minus @ 0x7ade2000-0x7ade3000
>> write-combining @ 0xc0000000-0xd0000000
>> ... snip ...
>>
>> So, I'm trying to figure out a few things:
>>
>> 1) Where does Linux get the PAT table from? The e820 map?
>
> It's generated a runtime based on the requests the kernel makes for
> specific mapping.
>>
>> 2) What piece of code sets the coreboot table as uncached-minus?
>
> The kernel does it. e820 is sort of a poor indicator for memory type.
> It merges the concept of address space usage and type but it doesn't
> do a very good job at it.
>
>>
>> 3) Why is that range set as uncached-minus?  Would write-back work?
>
> Please see this thread:
> http://www.coreboot.org/pipermail/coreboot/2015-September/080381.html
>
> The actual issue stems from
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/firmware/dmi_scan.c?id=d7f96f97c4031fa4ffdb7801f9aae23e96170a6f
> which maintains a persistent mapping of smbios tables. It uses
> dmi_remap() which is '#define dmi_remap ioremap' which is where the
> uncached-minus PAT entry comes from. It should be using the same
> mechanism as  the ACPI table mappings which uses ioremap_cache().
>
> Hope that helps.
>
> -Aaron

It does help.
I can confirm that changing that #define in arch/x86/include/asm/dmi.h
fixes the issue:

#define dmi_remap ioremap_cache

~ # cat /sys/kernel/debug/x86/pat_memtype_list
PAT memtype list:
write-back @ 0x7ada7000-0x7ada8000
write-back @ 0x7adb8000-0x7adbc000
write-back @ 0x7adbb000-0x7adbd000
write-back @ 0x7addc000-0x7addd000
write-combining @ 0xc0000000-0xd0000000
... snip ...

Thanks,
Ben