<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7653.38">
<TITLE>ACPI S3 with coreboot-v2 on DBM690T</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<BR>
<P><FONT SIZE=2 FACE="Arial">Hi, All,</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">After a dozens of day's struggling, I implement the ACPI S3 with coreboot-v2 on DBM690T. I write a note as below, please help to review it. Even though it is so primary and platform-specific, I appreciate any comments. Thank you in advance.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">---------------------------------------------------------------------------------------------------------------------------------</FONT></P>
<P><FONT SIZE=2 FACE="Arial">Platform: DBM690T populated 1G RAM, internal GFX</FONT>
<BR><FONT SIZE=2 FACE="Arial"> Fedora 6 shipping with 6.2.18 kernel</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">The basic principle of ACPI S3 with coreboot-v2 on DBM690T:</FONT>
<BR><FONT SIZE=2 FACE="Arial">Before Payload running, two major memory regions are used by CAR and coreboot:</FONT>
<BR><FONT SIZE=2 FACE="Arial">region 1: start address = 0x4000, length = 0x2ca0c(I think this only includes code and const), coreboot will run here after loading.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">region 2: start address = 0x1f8000, length = 0x8000, CAR uses this region as stack at the final stage.</FONT>
<BR><FONT SIZE=2 FACE="Arial">Furthermore, some tables occupy other small memory regions, such as IRQ table, MP table, ACPI table and coreboot table. When resuming, all these memory regions need to be protected from coreboot's overwriting. In the function of post_cache_as_ram, just before switching the stack from cache to RAM, the lowest 2M data are copied into the topmost 2M RAM. In the case of DBM690T, 1G RAM is populated while the last 128M RAM is reserved for the internal GFX as video memory. Linux of FC6 runs under the text mode on my DBM690T, so most video memory is not used. The 2M data can be kept intact in the topmost 2M RAM and no OS memory will be ruined by this moving. After all devices initialized, coreboot will check whether it is S3 resuming or not. If S3 resuming, the 2M data will be copied back into the lowest 2M RAM from the topmost 2M RAM and coreboot shifts the control to OS by jumping into the waking vector contained in the FACS table.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">The source code is a big mess and full of tricks, furthermore, it is so platform-specific by now: DBM690T populated 1G RAM, the internal GFX applied and FC 6 shipping with 6.2.18 kernel, it is so experimental that I can not check it in. The source code and comments are listed below. Hope it helpful for others.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">STEP 1. check the sleep resuming state and move the 2M data in the function of</FONT>
<BR><FONT SIZE=2 FACE="Arial">post_cache_as_ram from the lowest 2M RAM to the topmost 2M RAM:</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> /* BTDC: implement ACPI S3. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> sleep_type = inw(0x804); /* BTDC: hard-coded, modify it later. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> sleep_type = (sleep_type>>10) & 0x07; /* BTDC: get the sleep type. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> print_debug_pcar("BTDC: sleep_type = \n", sleep_type);</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> if(3 == sleep_type)</FONT>
<BR><FONT SIZE=2 FACE="Arial"> { /* BTDC: resuming from S3. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> msr_t msr_tmp;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> u32 * pt;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> u32 temp;</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> print_debug("BTDC: save the memory for OS resuming.\n");</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> /* BTDC: at this point, only the fixed MTRR 0x269 for stack in cache</FONT>
<BR><FONT SIZE=2 FACE="Arial"> * and the variable MTRR 0x202 for source code in FLASH are set, so if</FONT>
<BR><FONT SIZE=2 FACE="Arial"> * I want move the 2M data, I have to set some MTRRs myself.</FONT>
<BR><FONT SIZE=2 FACE="Arial"> */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> __asm__ volatile (</FONT>
<BR><FONT SIZE=2 FACE="Arial"> /* BTDC: disable the fixed MTRRs temporarily. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "movl $0xC0010010, %ecx\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "rdmsr\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "andl $(~(3<<18)), %eax\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "wrmsr\n\t"</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> /* BTDC: enable the default MTRR, so I can access the whole RAM. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "movl $0x2ff, %ecx\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "xorl %edx, %edx\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "movl $0x00000800, %eax\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "wrmsr\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> );</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> /* BTDC: save the memory for OS resuming. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> pt = 0;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> temp = *pt;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> print_debug_pcar("\nBTDC: the memory of 0 = ", temp);</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> pt = 2*1024*1024 - 4;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> temp = *pt;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> print_debug_pcar("\nBTDC: the memory of 2M = ", temp);</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> memcopy(0x3fe00000, 0, 2*1024*1024); /* BTDC: move the 2M data. */</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> pt = 0x3fe00000;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> temp = *pt;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> print_debug_pcar("\nBTDC: the memory of mirror 0 = ", temp);</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> pt = 2*1024*1024 - 4 + 0x3fe00000;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> temp = *pt;</FONT>
<BR><FONT SIZE=2 FACE="Arial"> print_debug_pcar("\nBTDC: the memory of mirror 2M = ", temp);</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> /* BTDC: restore the MTRR previously modified. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> __asm__ volatile (</FONT>
<BR><FONT SIZE=2 FACE="Arial"> /* BTDC: enable the fixed MTRRs again. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "movl $0xC0010010, %ecx\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "rdmsr\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "orl $(3<<18), %eax\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "wrmsr\n\t"</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> "movl $0x2ff, %ecx\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "xorl %edx, %edx\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "movl $0x00000c00, %eax\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> "wrmsr\n\t"</FONT>
<BR><FONT SIZE=2 FACE="Arial"> );</FONT>
<BR><FONT SIZE=2 FACE="Arial"> print_debug("BTDC: saving finished\n");</FONT>
<BR><FONT SIZE=2 FACE="Arial"> }</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial"> /* BTDC: just before the 2M data is cleared. */</FONT>
<BR><FONT SIZE=2 FACE="Arial"> set_init_ram_access(); /* So we can access RAM from [1M, CONFIG_LB_MEM_TOPK) */</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">STEP 2. if the system is resuming from S3, move the 2M data from the topmost 2M memory back into the lowest 2M memory in the function of hardwaremain and jump into the waking vector in the FASC table instead of loading payload.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">first, I set up a GDT table and two pseudo-descriptions for GDT and IDT respectively at 0x3fdfffe8, 0x3fdfffe0 and 0x3fdfffd8 for the 32bit-16bit opcode switching.</FONT></P>
<P> <FONT SIZE=2 FACE="Arial">memcpy((void *)(0x3fe00000-sizeof(real_mode_gdt_entries)), real_mode_gdt_entries, sizeof(real_mode_gdt_entries));</FONT></P>
<P> <FONT SIZE=2 FACE="Arial">memcpy((void *)0x3fdfffe0, real_mode_gdt, sizeof(real_mode_gdt));</FONT>
<BR> <FONT SIZE=2 FACE="Arial">memcpy((void *)0x3fdfffd8, real_mode_idt, sizeof(real_mode_idt));</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">the three tables are as below, thanks to Rudolf,</FONT>
<BR> <FONT SIZE=2 FACE="Arial">unsigned long long real_mode_gdt_entries [3] =</FONT>
<BR> <FONT SIZE=2 FACE="Arial">{</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x0000000000000000ULL, /* Null descriptor */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x008f9b000000ffffULL, /* 16-bit real-mode 64k code at 0x00000000 */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x008f93000000ffffULL /* 16-bit real-mode 64k data at 0x00000100 */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">};</FONT>
<BR> <FONT SIZE=2 FACE="Arial">struct Xgt_desc_struct {</FONT>
<BR> <FONT SIZE=2 FACE="Arial">unsigned short size;</FONT>
<BR> <FONT SIZE=2 FACE="Arial">unsigned long address __attribute__((packed));</FONT>
<BR> <FONT SIZE=2 FACE="Arial">unsigned short pad;</FONT>
<BR> <FONT SIZE=2 FACE="Arial">} __attribute__ ((packed));</FONT>
</P>
<P> <FONT SIZE=2 FACE="Arial">struct Xgt_desc_struct real_mode_gdt = { sizeof (real_mode_gdt_entries) - 1, (long)real_mode_gdt_entries };</FONT>
<BR> <FONT SIZE=2 FACE="Arial">struct Xgt_desc_struct real_mode_idt = { 0x3ff, 0 };</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">then, a small machine code are copied into the memory of 0x3fdfff00:</FONT>
<BR> <FONT SIZE=2 FACE="Arial">unsigned char bincode[] = {</FONT>
<BR> <FONT SIZE=2 FACE="Arial">/* BTDC: move 2M data back into the lowest 2M RAM.*/</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0xbe, 0x00, 0x00, 0xe0, 0x3f, /* mov esi, 3fe00000h */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0xbf, 0x00, 0x00, 0x00, 0x00, /* mov edi, 00000000h */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0xb9, 0x00, 0x00, 0x08, 0x00, /* mov ecx, 00080000h */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0xfc, /* cld */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0xf3, 0xa5, /* rep movsd es:[edi], ds:[esi] */</FONT>
</P>
<P> <FONT SIZE=2 FACE="Arial">/* BTDC: load new GDT and IDT for 32bit-16bit opcode switching. */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0xfa, /* cli */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x0f, 0x01, 0x1d, 0xd8, 0xff, 0xdf, 0x3f, /* lidt [3fdfffd8h] */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x0f, 0x01, 0x15, 0xe0, 0xff, 0xdf, 0x3f, /* lgdt [3fdfffe0h] */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0xb8, 0x10, 0x00, 0x00, 0x00, /* mov eax, 00000010h */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x8e, 0xd8, /* mov ds, ax */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x8e, 0xc0, /* mov es, ax */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x8e, 0xe0, /* mov fs, ax */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x8e, 0xe8, /* mov gs, ax */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x8e, 0xd0, /* mov ss, ax */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0xea, 0x37, 0xff, 0xdf, 0x3f, 0x08, 0x00, /* jmp $0x0008:$0x3fdfff37 */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x90, /* nop */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x90, /* nop */</FONT>
</P>
<P> <FONT SIZE=2 FACE="Arial">/* BTDC: enter the real mode. */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x0f, 0x20, 0xc0, /* movl %cr0, %eax */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x24, 0xfe, /* andb $0xfe, %al */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0x0f, 0x22, 0xc0, /* movl %eax, %cr0 */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">/* BTDC: jump into the waking vector in FASC. */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">/* BTDC: hard-coded and platform-specific: FC6 with 6.2.18. modify it later. */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">0xea, 0x00, 0x00, 0x00, 0x02 /* jmp 0x200:0x00 */</FONT>
<BR> <FONT SIZE=2 FACE="Arial">};</FONT>
<BR><FONT SIZE=2 FACE="Arial">In my case, the FACS table is located at 0xf0620, and the waking vector is</FONT>
<BR><FONT SIZE=2 FACE="Arial">0x2000, I hard-code just to make life easier. then, OS takes the control from</FONT>
<BR><FONT SIZE=2 FACE="Arial">coreboot. Linux resumes from S3.</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">Now, two questions arise to me:</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">1. Is it really necessary for CAR to move the stack from 0xc8000 in cache into 0x1f8000 in RAM at the final stage of CAR? Now that the stack works well in cache, why does CAR move the stack into RAM? For verifying RAM or other stuff?</FONT></P>
<P><FONT SIZE=2 FACE="Arial">2. When resuming from S3, I initialize RAM again instead of exiting self-Refresh. Lucky enough, RAM content is also kept intact in this way. I will try the exiting self-refresh later.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">My first attempt is to jump into the waking vector in the function of post_cache_as_ram, at this moment, RAM is accessible, I can get the waking vector. However, many devices are not initialized, the system is very unstable, I got different trace every time, the best was as below. So, after stuck a couple of days, I gave up and followed Rodulf's way as above.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">============================================================</FONT>
<BR><FONT SIZE=2 FACE="Arial"> usbdev5.1_ep81: PM: suspend 0->2, parent 5-0:1.0 already 1</FONT>
<BR><FONT SIZE=2 FACE="Arial"> usbdev4.1_ep81: PM: suspend 0->2, parent 4-0:1.0 already 1</FONT>
<BR><FONT SIZE=2 FACE="Arial"> usbdev3.1_ep81: PM: suspend 0->2, parent 3-0:1.0 already 1</FONT>
<BR><FONT SIZE=2 FACE="Arial"> usbdev2.1_ep81: PM: suspend 0->2, parent 2-0:1.0 already 1</FONT>
<BR><FONT SIZE=2 FACE="Arial">BUG: sleeping function called from invalid context at kernel/rwsem.c:20</FONT>
<BR><FONT SIZE=2 FACE="Arial">in_atomic():0, irqs_disabled():1</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">Call Trace:</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff8026929b>] show_trace+0x34/0x47</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff802692c0>] dump_stack+0x12/0x17</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff8029dc68>] down_read+0x15/0x23</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff80296254>] blocking_notifier_call_chain+0x13/0x36</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff803fde58>] cpufreq_resume+0x129/0x14c</FONT>
<BR><FONT SIZE=2 FACE="Arial">DWARF2 unwinder stuck at cpufreq_resume+0x129/0x14c</FONT>
<BR><FONT SIZE=2 FACE="Arial">Leftover inexact backtrace:</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff803a4350>] __sysdev_resume+0x2a/0x66</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff803a44fe>] sysdev_resume+0x1d/0x63</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff803a8efc>] device_power_up+0x9/0xf</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff802a5f42>] suspend_enter+0x3e/0x47</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff802a608e>] enter_state+0x143/0x19b</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff802a6155>] state_store+0x5e/0x79</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff802fb1ac>] sysfs_write_file+0xca/0xf9</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff802162b6>] vfs_write+0xce/0x174</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff80216b26>] sys_write+0x45/0x6e</FONT>
<BR><FONT SIZE=2 FACE="Arial"> [<ffffffff8025c00e>] system_call+0x7e/0x83</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">PCI: Enabling device 0000:00:13.0 (0000 -> 0002)</FONT>
<BR><FONT SIZE=2 FACE="Arial">PCI: Enabling device 0000:00:13.1 (0000 -> 0002)</FONT>
<BR><FONT SIZE=2 FACE="Arial">PCI: Enabling device 0000:00:13.2 (0000 -> 0002)</FONT>
<BR><FONT SIZE=2 FACE="Arial">PCI: Enabling device 0000:00:13.3 (0000 -> 0002)</FONT>
<BR><FONT SIZE=2 FACE="Arial">PCI: Enabling device 0000:00:13.4 (0000 -> 0002)</FONT>
<BR><FONT SIZE=2 FACE="Arial">PCI: Enabling device 0000:00:13.5 (0000 -> 0002)</FONT>
<BR><FONT SIZE=2 FACE="Arial">PCI: Enabling device 0000:00:14.2 (0000 -> 0002)</FONT>
<BR><FONT SIZE=2 FACE="Arial">hda_intel: azx_get_response timeout, switching to single_cmd mode...</FONT>
<BR><FONT SIZE=2 FACE="Arial">Restarting tasks... done</FONT>
<BR><FONT SIZE=2 FACE="Arial">Enabling non-boot CPUs ...</FONT>
<BR><FONT SIZE=2 FACE="Arial">================================================================</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">By the way, the sequence of Linux(6.2.18) resuming is as below, if you are curious:</FONT>
<BR><FONT SIZE=2 FACE="Arial">The waking vector points to the function of wakeup_code in the file of arch/x86_64/kernel/acpi/wakeup.s. </FONT>
<BR><FONT SIZE=2 FACE="Arial">Then, the function of restore_processor_state in the file of arch/x86_64/kernel/suspend.c is called.</FONT>
<BR><FONT SIZE=2 FACE="Arial">Then, the function of __restore_processor_state is called in the same file.</FONT>
<BR><FONT SIZE=2 FACE="Arial">Finally, I was lost here, I couldn't match the assembly language with the C code any more.</FONT>
</P>
<P><SPAN LANG="en-us"><FONT FACE="Arial">Best Regards</FONT></SPAN><SPAN LANG="zh-cn"></SPAN>
</P>
<P><SPAN LANG="zh-cn"><FONT FACE="宋体">丰立波</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Arial"> Feng Libo @ AMD Ext</FONT></SPAN><SPAN LANG="zh-cn"><FONT FACE="Arial">:</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Arial"> 20906</FONT></SPAN>
<BR><SPAN LANG="zh-cn"><FONT FACE="Arial">Mobile Phone</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Arial">: 13683249071</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT FACE="Arial">Office Phone: 0086-010-</FONT><FONT FACE="Arial">62801406</FONT></SPAN><SPAN LANG="zh-cn"></SPAN>
</P>
</BODY>
</HTML>