Repository navigation
Svpbmt: Page-Based Memory Types - #663
Daniel Lustig (daniellustig) wants to merge 12 commits into
Conversation
| \end{commentary} | ||
|
|
||
| Memory accesses are ordered according to the effective memory type for the | ||
| address in question. A region of memory which is considered main memory by the |
There was a problem hiding this comment.
I can see how this paragraph (after the first sentence) may be a little confusing for some readers to parse correctly. Maybe this and the following paragraph can all be condensed to:
The ordering of memory accesses is based on the effective memory type and its 'main memory' versus 'I/O' attribute distinction. This includes for the purposes of FENCE, .aq, and .rl.
A non-normative comment can provide any desired clarification(s). Although it seems like the commentary in the two paragraphs all follows directly from the current arch specs and the above statement.
There was a problem hiding this comment.
P.S. These paragraphs (and some later text) tend to mix together two distinct forms of terminology: Architectural "main memory" and "I/O" (as PMA attributes), and "main memory" and "I/O" as colloquial terms. The former refer to how an access is handled or performed; the latter refer to the physical nature of what is at a physical address in a system. For example, one can map an address with a "I/O" memory type to what physically is DRAM "main memory", or an address can have a "main memory" memory type but maps to a RAM buffer within an I/O device.
There was a problem hiding this comment.
There should be no imprecise colloquial usage, we should use proper definitions for these things. I suggested "effective memory attributes" vs "physical memory attributes" above...if we go with that, this would become something like "If a memory access has an effective memory attribute of main memory, it obeys RVWMO (or RVTSO, if enabled). If it has an effective memory attribute of I/O, it obeys I/O channel 0 ordering rules."
| order after all prior operations from the same hart to main memory or to the | ||
| same ``I/O region'', and it will appear earlier in the global memory order than | ||
| all subsequent accesses from the same hart to main memory or to the same ``I/O | ||
| region''. Such operations are not guaranteed to remain ordered with respect to |
There was a problem hiding this comment.
For example, this is already true in the current PMA and memory ordering architecture.
There was a problem hiding this comment.
Channel 0 specifies point-to-point strong ordering with respect to a particular "I/O region". In this case that PBMTs turn I/O into main memory, we have to define what that "I/O region" means, because there is no underlying physical I/O region. I decided to declare that "I/O region" to include both main memory and other accesses in the same category, as a seeming natural choice. It's up for discussion, but that's what the text was getting at.
| or other ordering mechanisms if necessary. | ||
|
|
||
| \begin{commentary} | ||
| A device driver written to rely on I/O strong ordering rules will not |
There was a problem hiding this comment.
This paragraph seems semi-misleading. While appropriate wrt hyprvisors, an OS is not going to go off on its own and override PMAs, i.e. an OS will apply a pbmt override in response to a request from a device driver.
But I also now realize that this issue can't happen. If a device driver is relying on I/O strong ordering, then presumably either the PMAs specify "I/O" or the device requested the OS to apply an "I/O" pbmt. (Otherwise the device driver won't work properly even in a non-virtualized environment.) In which case, even if the hypervisor maps the address to colloquial "main memory", the address will still be treated as part of an architectural "I/O" region.
There was a problem hiding this comment.
I slightly misspoke. A hypervisor could turn PMA "I/O" into effectively "noncacheable main memory". But the motivating concern is when a hypervisor maps a guest memory-mapped I/O device into physical main memory for purposes of emulating that device. This would simply involve G-stage mapping the I/O device's VA pages to physical DRAM main memory. The G-stage PTE would not override the attributes. Or if it did, and specifically to "noncacheable main memory", that would be problematic and unnecessary (i.e. to quote Allen: "if it hurts, then don't do it").
There was a problem hiding this comment.
Isn't "...then don't do it" exactly what this non-normative text is warning about? Ultimately either we forbid the combination, or we define what its behavior is. For now it's not forbidden, so this comment was meant to make the warning explicit rather than leaving it quiet.
| actually needed. | ||
|
|
||
| In spite of the caveat above, it will often still be useful to map | ||
| device memory regions in I/O as main memory so that write combining |
There was a problem hiding this comment.
To some readers this can be confusing or ambiguous since it isn't clear whether "I/O" and "main memory" and "device memory regions" are referring to architectural I/O and main memory regions, or to colloquial physical main memory or memory-mapped I/O.
This is also misleading since it doesn't matter that an address is mapped to colloquial physical DRAM main memory. The effective memory type will determine how the access is handled/performed.
There was a problem hiding this comment.
How about "...it will still be useful to map physical I/O regions containing physical device memory into effective main memory regions..."
| RVWMO and RVTSO still require eventual visibility of writes from one harts to | ||
| other harts in the system. | ||
|
|
||
| When two-level paging is enabled within the H extension, the page-based memory |
There was a problem hiding this comment.
This text doesn't spell out how one combines the pbmt from VS-stage and G-stage PTEs when both are specifying overrides, i.e. which one wins out.
If one had names for the pbmt encodings, one could say something like:
The combined PTE memory type is the more conservative of the VS-stage and G-stage memory types, i.e. IO wins out over NC which wins out over PMA. The result is always one of IO, NC, or PMA.
There was a problem hiding this comment.
Conceptually I would view the PMA and G-stage PTE attributes being combined first. This intermediate set of attributes then represents what is seen by a VM as PMA attributes - which the guest OS can then choose to conditionally override.
This also provides an easy solution to specifying how VS-stage and G-stage pbmt's are combined. One applies the earlier description - for how a PTE modifies PMA attributes - first with the G-stage pbmt, and then take that and apply the VS-stage pbmt.
There was a problem hiding this comment.
I don't understand...isn't the note about first/second exactly what the text below says?
| \begin{center} | ||
| \begin{tabular}{|r|l|} | ||
| \hline | ||
| Value & Page-Based Memory Attributes \\ |
There was a problem hiding this comment.
To avoid any potential misunderstanding by a reader, it might be better to say "Page-Based Memory Attribute Combination". (Which is what a "page-based memory type" is.)
There was a problem hiding this comment.
I'm not sure this is a "combination" either...it's really the "effective" memory attribute. To your other comments, maybe we should use that term "effective memory attribute" throughout, so we can contrast it to "physical memory attribute" in a clearer way.
There was a problem hiding this comment.
Perhaps it is "effective" after the combination, but from a given PTE it is the "requested" memory attribute.
|
For non-leaf PTEs, what effect (if any) does the PBMT field have? Does it affect (only?) the next level of tablewalk? |
|
All the Svpbmt spec proposals including the final one (before Dan turned that into a Priv spec PR), have said: What this proposal is not trying to address:
If the current PR is felt to not be clear that Svpbmt only applies to leaf PTEs (and for all page sizes), then that should certainly be made clear. |
| \hline | ||
| 0 & None \\ | ||
| 1 & Non-cacheable, idempotent, weakly-ordered (RVWMO or RVTSO), main memory \\ | ||
| 2 & Non-cacheable, non-idempotent, strongly-ordered (channel 0), I/O \\ |
There was a problem hiding this comment.
Where is "channel 0" explained? The Unprivileged ISA also has a couple of mentions of channels, but no real explanation of what they mean.
There was a problem hiding this comment.
Nevermind...found this in the Privileged ISA section 3.6.4 Memory-Ordering PMAs. (I was expecting it in the RVWMO material in the Unprivileged ISA.)
| \end{commentary} | ||
|
|
||
| When two-level paging is enabled within the H extension, the page-based memory | ||
| types are applied in two stages. First, the G-stage PTE PBMT bits (if enabled) |
There was a problem hiding this comment.
What does "if enabled" mean here? Is there some global enable for Svpbmt? Or does this mean "if the PTE's PBMT bits are nonzero"?
There was a problem hiding this comment.
I meant "if hgatp.MODE != 0" and "vsatp.MODE != 0", because otherwise there are no PTE bits to consider at all, but I will reword to clarify.
I can't see in this PR where it says anything about that. |
|
Scott Johnson (@scottj97) I will push a clarification about non-leaf PTEs, thank you for pointing that out. |
| with adequate care. | ||
| \end{commentary} | ||
|
|
||
| When two-level paging is enabled within the H extension, the page-based memory |
There was a problem hiding this comment.
The H spec calls it "two-stage", not "two-level".
Thanks to @scottj97 for pointing out the unclear wording.
|
There are four possible combinations of |
With PMA=main memory and PBMT=1, accesses to the page continue to obey RVWMO (or RVTSO), just as they otherwise would. With PMA=I/O and PBMT=2, accesses to the page continue to obey I/O ordering rules. Those are the straightforward combinations, because we're not switching the memory type attribute. The other two cases are trickier because in those cases we are switching the memory type attribute. That means we have to define what it means to apply RVWMO to PMA=I/O regions, or to apply I/O ordering rules to PMA=main memory regions. That's what the text you referred to describes. |
| \begin{center} | ||
| \begin{tabular}{|r|l|} | ||
| \hline | ||
| Value & Page-Based Memory Attributes \\ |
There was a problem hiding this comment.
Perhaps it is "effective" after the combination, but from a given PTE it is the "requested" memory attribute.
| that page with PBMT=1 will obey RVWMO or RVTSO rather than I/O strong ordering | ||
| rules, and accesses to such pages are considered main memory rather than I/O | ||
| for the purposes of FENCE, {\em.aq}, and {\em.rl}. |
There was a problem hiding this comment.
It is unclear if these are
- examples
- the full subset of attributes that are affected
- point clarifications for specific attributes that are affected, with the rest being "obvious"
Same for the next paragraph.
| than a concurrent access to the same page performed by M-mode or when {\tt | ||
| satp}.MODE=Bare. If accesses are performed simultaneously to the same region |
There was a problem hiding this comment.
Is this list incomplete only mentioning satp?
| satp}.MODE=Bare. If accesses are performed simultaneously to the same region | ||
| of memory using different attributes, there may be a loss of coherence and/or | ||
| of normal RVWMO, RVTSO, or I/O ordering semantics. In such cases, | ||
| platform-specific mechanisms must be used to restore corrent coherence and |
There was a problem hiding this comment.
s/corrent/correct/
| satp}.MODE=Bare. If accesses are performed simultaneously to the same region | ||
| of memory using different attributes, there may be a loss of coherence and/or |
There was a problem hiding this comment.
Since "accesses" here can be speculative, this should be changed to describe the architectural scenario, which is something about having conflicting valid mappings exist across all harts logically concurrently. If SW robustly changes attributes with no overlap (e.g., some mutually globally invalid period), then is no platform-specific mechanism required?
I don't think that this is attempting to say that only explicit architectural accesses can cause this.
Thanks to @jscheid-ventana for the suggestions
Closes #3. Thanks to @jhauser-us
|
Merged via #763, closing |
Opening a draft PR to enable discussion on the Svpbmt extension being discussed by the RISC-V virtual memory TG.
This extension is created as a branch off of #628 for convenience, and to avoid conflating Svpbmt discussion with other virtual memory TG discussions, but Svnapot and Svpbmt will be reviewed for ratification purposes as independent entities.
PDF versions of the specs are available on the virtual memory TG GitHub page:
https://github.com/riscv/virtual-memory/tree/main/specs