Difference between revisions of "Rating System"

From coreboot
Jump to: navigation, search
m (RatingSystem moved to Rating System: No CamelCase in wiki names.)
(Introduction)
 
(29 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== Rating System ==
+
At the coreboot summit 2008 in Denver we talked about a rating system for supported boards. The idea is to make it clear which boards are most highly recommended because the vendors cooperate. Thus the 'Vendor Cooperation Score' rating system was born.
  
At the coreboot summit in Denver we talked about a rating system for supported boards. The idea is to make it clear which boards are most highly recommended because the vendors cooperate.
+
= Introduction =
  
 
To get to such a rating for a particular board, we should establish a list of categories with an associated score.  Each fulfilled criteria should be easily verifiable as a yes or no answer.
 
To get to such a rating for a particular board, we should establish a list of categories with an associated score.  Each fulfilled criteria should be easily verifiable as a yes or no answer.
There should be no subjective elements to the rating system - only measurable criteria should be used to avoid bias or favoritism.
+
There should be no subjective elements to the rating system — only measurable criteria should be used to avoid bias or favoritism.
  
Adding up the scores for the major components on a board (cpu, chipsets, mainboard, others?) would give us a rating that results in a number of 'stars'.
+
Adding up the scores for the major components on a board (CPU, chipsets, mainboard, others?) would give us a rating that results in a number of 'stars'.
  
 
Some ideas for those categories:
 
Some ideas for those categories:
  
* availability of documentation (nothing/NDA restricted == 0, NDA but free to publish code == 3, online with click through == 7, public URL == 10)
+
* Availability of documentation (nothing/NDA restricted == 0, NDA but free to publish code == 3, online with click through == 7, public URL == 10)
** There should be multiple categories of documentation (register set, BIOS programming guide, errata, schematics or pinouts (for motherboards)
+
** There should be multiple categories of documentation (register set, BIOS programming guide, errata, schematics or pinouts (for motherboards))
* vendor participation in the coreboot project
+
** How to quantify?
+
* Availability of example and support code
+
** ACPI tables
+
 
* "Hackability"
 
* "Hackability"
 
** LPC header, JTAG header, BIOS socket, etc.
 
** LPC header, JTAG header, BIOS socket, etc.
 
** Should we dock a board because it requires soldering or a difficult process for flashing coreboot?
 
** Should we dock a board because it requires soldering or a difficult process for flashing coreboot?
 
* ...
 
* ...
 +
* Availability of example and support code
 +
** ACPI tables
 +
** flashrom support
 +
** driver code
  
 
As we list boards, we should also make it clear if a board is actually available for purchase. A board might get a high rating, but be unavailable for purchase, in which case it should be carefully marked as such. Board availability will change over a board's lifespan.
 
As we list boards, we should also make it clear if a board is actually available for purchase. A board might get a high rating, but be unavailable for purchase, in which case it should be carefully marked as such. Board availability will change over a board's lifespan.
Line 27: Line 27:
 
= Criteria =
 
= Criteria =
  
When all the criteria have been evaluated, each platform will end up with a total score.  To assign "stars" to the platform, we will divide the number of scored points by the number of maximum possible points and multiply that with the number of maximum stars
+
When all the criteria have been evaluated, each platform will end up with a total score.  To assign "hares" to the platform, we will divide the number of scored points by the number of maximum possible points and multiply that with the number of maximum hares (probably 5) and round down to the nearest half hare (bunny?).
(probably 5) and round down to the nearest half star.
+
  
 
== Documentation ==
 
== Documentation ==
Line 43: Line 42:
 
|  7 || Documentation is freely available and redistributable.  It can be downloaded on the web after agreeing to a click-through license.
 
|  7 || Documentation is freely available and redistributable.  It can be downloaded on the web after agreeing to a click-through license.
 
|-
 
|-
|  3 || Documentation is restricted and only available under Non Disclosure Agreement, however the NDA allows source code written with the documentation to be freely available under the GPL.
+
|  3 || Documentation is restricted and only available under Non Disclosure Agreement, however the NDA allows source code written with the documentation to be freely available under the GPL (or another Free Software license).
 
|-
 
|-
|  0 || Documentation is not available or the NDA does now allow release of code.
+
|  0 || Documentation is not available or the NDA does not allow release of code.
 
|}
 
|}
  
Line 54: Line 53:
 
Evaluate the criteria against the following three sets of documentation (30 total points possible):
 
Evaluate the criteria against the following three sets of documentation (30 total points possible):
  
* Datasheet / register set (Detailed information about programming registers and/or memory locations)
+
* Datasheet / register set (detailed information about programming registers and/or memory locations)
 
* BIOS programming guide
 
* BIOS programming guide
 
* Errata
 
* Errata
Line 62: Line 61:
 
Evaluate the criteria against the following three sets of documentation (30 total points possible):
 
Evaluate the criteria against the following three sets of documentation (30 total points possible):
  
* Datasheet / register set (Detailed information about programming registers and/or memory locations)
+
* Datasheet / register set (detailed information about programming registers and/or memory locations)
 
* BIOS programming guide
 
* BIOS programming guide
 
* Errata
 
* Errata
  
=== Other chipsets (SuperIO, etc) ===
+
=== Other chipsets (Super I/O, etc) ===
  
 
Evaluate the criteria against the following: (10 total points possible).  Add 10 points if not applicable to the platform.
 
Evaluate the criteria against the following: (10 total points possible).  Add 10 points if not applicable to the platform.
  
* Datasheet / register set (Detailed information about programming registers and/or memory locations)
+
* Datasheet / register set (detailed information about programming registers and/or memory locations)
  
 
=== Mainboard ===
 
=== Mainboard ===
Line 77: Line 76:
  
 
* Schematics / Pinout (Sufficient documentation to detail the platform specific GPIO & IRQ assignments)
 
* Schematics / Pinout (Sufficient documentation to detail the platform specific GPIO & IRQ assignments)
 +
 +
== Hackability ==
 +
 +
Use the following criteria to evaluate the hackability of a board. Maximum possible score: 24.
 +
 +
Rom chip:
 +
 +
{| border="1"
 +
|- bgcolor="#6699ff"
 +
!Points
 +
!Description
 +
|-
 +
| 10 || Socketed rom chip. 
 +
|-
 +
|  1 || Soldered rom chip with space on the board for a secondary rom chip (pads unpopulated).
 +
|-
 +
|  0 || Soldered rom chip, no space for secondary rom chip.
 +
|}
 +
 +
LPC header:
 +
 +
{| border="1"
 +
|- bgcolor="#6699ff"
 +
!Points
 +
!Description
 +
|-
 +
|  9 || LPC header, can be used to emulate rom chip. 
 +
|-
 +
|  0 || No LPC header.
 +
|}
 +
 +
JTAG header:
 +
 +
{| border="1"
 +
|- bgcolor="#6699ff"
 +
!Points
 +
!Description
 +
|-
 +
|  5 || JTAG header. 
 +
|-
 +
|  0 || No JTAG header.
 +
|}
 +
 +
== Example and support code ==
 +
 +
Is example and support code readily available? Maximum possible score: 10.
 +
 +
{| border="1"
 +
|- bgcolor="#6699ff"
 +
!Points
 +
!Description
 +
|-
 +
| 10 || Code is freely available under a free software license.  It can be downloaded through a direct URL with no click-through pages. 
 +
|-
 +
|  7 || Code is freely available under a free software license.  It can be downloaded on the web after agreeing to a click-through license.
 +
|-
 +
|  7 || Vendor provides code via e-mail, no NDA/license agreement required.
 +
|-
 +
|  3 || Code is restricted and only available under Non Disclosure Agreement, however the NDA allows source code written using the sample code to be freely available under a free software license.
 +
|-
 +
|  0 || Code is not available or the NDA does now allow release of code.
 +
|}
 +
 +
= Adding it all up =
 +
 +
The three categories above give us a scale with a maximum of 114 points. We devide that range up in 5 bands, meaning that boards can get a score from zero to five 'hares':
 +
 +
{| border="1"
 +
|- bgcolor="#6699ff"
 +
!Points
 +
!Score
 +
|-
 +
| 0 || [[Image:zero-hares.png]]
 +
|-
 +
| 1-22 || [[Image:one-hare.png]]
 +
|-
 +
| 23-45 || [[Image:two-hares.png]]
 +
|-
 +
| 46-68 || [[Image:three-hares.png]]
 +
|-
 +
| 69-91 || [[Image:four-hares.png]]
 +
|-
 +
| 92-114 || [[Image:five-hares.png]]
 +
|}
 +
 +
= Vendor Participation =
 +
 +
While we don't formally score vendor participation for a board, ongoing vendor participation has many benefits to all parties involved. Regular particpation in the mailing list and other outlets results in higher quality code, faster turn around time on bugs, and more satisfied end users.
  
 
= Example Board =
 
= Example Board =
 +
 
To demonstrate how this will work, we will apply the above criteria to the db800 platform from AMD, which is widely regarded as one of the better supported platforms in coreboot.
 
To demonstrate how this will work, we will apply the above criteria to the db800 platform from AMD, which is widely regarded as one of the better supported platforms in coreboot.
 
(Please fill in this section as new criteria are added).
 
(Please fill in this section as new criteria are added).
Line 87: Line 175:
  
 
== Documentation ==
 
== Documentation ==
=== CPU (Geode LX) ===
+
 
* Datasheet / register setFreely available [http://www.amd.com/files/connectivitysolutions/geode/geode_lx/33234F_LX_databook.pdf] - '''10 points'''
+
{| border="0" style="font-size: smaller"
* BIOS programming guide: NDA allowing GPLed code - '''3 points'''
+
|- bgcolor="#6699ff"
* Errata:  NDA allowing GPL - '''3 points'''
+
! align="left" | Item
== Chipset (CS5536) ===
+
! align="left" | Availability
* Datasheet / register setFreely available [http://www.amd.com/files/connectivitysolutions/geode/geode_lx/33238G_cs5536_db.pdf] - '''10 points'''
+
! align="left" | Points
* BIOS programming guide: NDA allowing GPLed code - '''3 points'''
+
 
* Errata:  Freely available [http://www.amd.com/files/connectivitysolutions/geode/geode_gx/34472D_CS5536_B1_specupdate.pdf] - '''10 points'''
+
|- bgcolor="#6699ff"
== SuperIO ==
+
| colspan="3" | '''CPU (Geode LX)'''
* Datasheet / register setFreely available [http://www.itox.com/pages/support/wdt/W83627HF.pdf] - '''10 points'''
+
 
==Mainboard==
+
|- bgcolor="#eeeeee"
* Schematics: NDA allowing GPLed code - '''3 points'''
+
| '''Datasheet / register set'''
 +
| Freely available [http://www.amd.com/files/connectivitysolutions/geode/geode_lx/33234F_LX_databook.pdf]
 +
| style="background:lime" | 10
 +
 
 +
|- bgcolor="#dddddd"
 +
| '''BIOS programming guide'''
 +
| NDA allowing GPL'd code
 +
| style="background:yellow" | 3
 +
 
 +
|- bgcolor="#eeeeee"
 +
| '''Errata'''
 +
| NDA allowing GPL'd code
 +
| style="background:yellow" | 3
 +
 
 +
|- bgcolor="#6699ff"
 +
| colspan="3" | '''Chipset (CS5536)'''
 +
 
 +
|- bgcolor="#eeeeee"
 +
| '''Datasheet / register set'''
 +
| Freely available [http://www.amd.com/files/connectivitysolutions/geode/geode_lx/33238G_cs5536_db.pdf]
 +
| style="background:lime" | 10
 +
 
 +
|- bgcolor="#dddddd"
 +
| '''BIOS programming guide'''
 +
| NDA allowing GPL'd code
 +
| style="background:yellow" | 3
 +
 
 +
|- bgcolor="#eeeeee"
 +
| '''Errata'''
 +
| Freely available [http://www.amd.com/files/connectivitysolutions/geode/geode_gx/34472D_CS5536_B1_specupdate.pdf]
 +
| style="background:lime" | 10
 +
 
 +
|- bgcolor="#6699ff"
 +
| colspan="3" | '''Super I/O'''
 +
 
 +
|- bgcolor="#eeeeee"
 +
| '''Datasheet / register set'''
 +
| Freely available [http://www.itox.com/pages/support/wdt/W83627HF.pdf]
 +
| style="background:lime" | 10
 +
 
 +
|- bgcolor="#6699ff"
 +
| colspan="3" | '''Mainboard'''
 +
 
 +
|- bgcolor="#eeeeee"
 +
| '''Schematics'''
 +
| NDA allowing GPL'd code
 +
| style="background:yellow" | 3
 +
 
 +
|}

Latest revision as of 16:12, 21 August 2008

At the coreboot summit 2008 in Denver we talked about a rating system for supported boards. The idea is to make it clear which boards are most highly recommended because the vendors cooperate. Thus the 'Vendor Cooperation Score' rating system was born.

Introduction

To get to such a rating for a particular board, we should establish a list of categories with an associated score. Each fulfilled criteria should be easily verifiable as a yes or no answer. There should be no subjective elements to the rating system — only measurable criteria should be used to avoid bias or favoritism.

Adding up the scores for the major components on a board (CPU, chipsets, mainboard, others?) would give us a rating that results in a number of 'stars'.

Some ideas for those categories:

  • Availability of documentation (nothing/NDA restricted == 0, NDA but free to publish code == 3, online with click through == 7, public URL == 10)
    • There should be multiple categories of documentation (register set, BIOS programming guide, errata, schematics or pinouts (for motherboards))
  • "Hackability"
    • LPC header, JTAG header, BIOS socket, etc.
    • Should we dock a board because it requires soldering or a difficult process for flashing coreboot?
  • ...
  • Availability of example and support code
    • ACPI tables
    • flashrom support
    • driver code

As we list boards, we should also make it clear if a board is actually available for purchase. A board might get a high rating, but be unavailable for purchase, in which case it should be carefully marked as such. Board availability will change over a board's lifespan.

Should we provide a separate rating for coreboot support (i.e. the stuff above) and how good our code actually is?

Criteria

When all the criteria have been evaluated, each platform will end up with a total score. To assign "hares" to the platform, we will divide the number of scored points by the number of maximum possible points and multiply that with the number of maximum hares (probably 5) and round down to the nearest half hare (bunny?).

Documentation

Use the following criteria to evaluate each set of documentation:

Points Description
10 Documentation is freely available and redistributable. It can be downloaded through a direct URL with no click-through pages.
7 Documentation is freely available and redistributable. It can be downloaded on the web after agreeing to a click-through license.
3 Documentation is restricted and only available under Non Disclosure Agreement, however the NDA allows source code written with the documentation to be freely available under the GPL (or another Free Software license).
0 Documentation is not available or the NDA does not allow release of code.

X86 based platforms can have up to 80 points for documentation.

CPU

Evaluate the criteria against the following three sets of documentation (30 total points possible):

  • Datasheet / register set (detailed information about programming registers and/or memory locations)
  • BIOS programming guide
  • Errata

Southbridge (chipset)

Evaluate the criteria against the following three sets of documentation (30 total points possible):

  • Datasheet / register set (detailed information about programming registers and/or memory locations)
  • BIOS programming guide
  • Errata

Other chipsets (Super I/O, etc)

Evaluate the criteria against the following: (10 total points possible). Add 10 points if not applicable to the platform.

  • Datasheet / register set (detailed information about programming registers and/or memory locations)

Mainboard

Evaluate the criteria against the following: (10 total points available)

  • Schematics / Pinout (Sufficient documentation to detail the platform specific GPIO & IRQ assignments)

Hackability

Use the following criteria to evaluate the hackability of a board. Maximum possible score: 24.

Rom chip:

Points Description
10 Socketed rom chip.
1 Soldered rom chip with space on the board for a secondary rom chip (pads unpopulated).
0 Soldered rom chip, no space for secondary rom chip.

LPC header:

Points Description
9 LPC header, can be used to emulate rom chip.
0 No LPC header.

JTAG header:

Points Description
5 JTAG header.
0 No JTAG header.

Example and support code

Is example and support code readily available? Maximum possible score: 10.

Points Description
10 Code is freely available under a free software license. It can be downloaded through a direct URL with no click-through pages.
7 Code is freely available under a free software license. It can be downloaded on the web after agreeing to a click-through license.
7 Vendor provides code via e-mail, no NDA/license agreement required.
3 Code is restricted and only available under Non Disclosure Agreement, however the NDA allows source code written using the sample code to be freely available under a free software license.
0 Code is not available or the NDA does now allow release of code.

Adding it all up

The three categories above give us a scale with a maximum of 114 points. We devide that range up in 5 bands, meaning that boards can get a score from zero to five 'hares':

Points Score
0 Zero-hares.png
1-22 One-hare.png
23-45 Two-hares.png
46-68 Three-hares.png
69-91 Four-hares.png
92-114 Five-hares.png

Vendor Participation

While we don't formally score vendor participation for a board, ongoing vendor participation has many benefits to all parties involved. Regular particpation in the mailing list and other outlets results in higher quality code, faster turn around time on bugs, and more satisfied end users.

Example Board

To demonstrate how this will work, we will apply the above criteria to the db800 platform from AMD, which is widely regarded as one of the better supported platforms in coreboot. (Please fill in this section as new criteria are added).

Total available points: 80 Total platform points: 52 Total "stars": 3

Documentation

Item Availability Points
CPU (Geode LX)
Datasheet / register set Freely available [1] 10
BIOS programming guide NDA allowing GPL'd code 3
Errata NDA allowing GPL'd code 3
Chipset (CS5536)
Datasheet / register set Freely available [2] 10
BIOS programming guide NDA allowing GPL'd code 3
Errata Freely available [3] 10
Super I/O
Datasheet / register set Freely available [4] 10
Mainboard
Schematics NDA allowing GPL'd code 3