Bug 407 - XICS interrupt controller is needed
Summary: XICS interrupt controller is needed
Status: RESOLVED FIXED
Alias: None
Product: Libre-SOC's first SoC
Classification: Unclassified
Component: Source Code (show other bugs)
Version: unspecified
Hardware: Other Linux
: High enhancement
Assignee: Luke Kenneth Casson Leighton
URL:
Depends on:
Blocks: 383
  Show dependency treegraph
 
Reported: 2020-06-30 20:09 BST by Luke Kenneth Casson Leighton
Modified: 2020-12-06 13:00 GMT (History)
2 users (show)

See Also:
NLnet milestone: NLNet.2019.10.Wishbone
total budget (EUR) for completion of task and all subtasks: 450
budget (EUR) for this task, excluding subtasks' budget: 450
parent task for budget allocation: 383
child tasks for budget allocation:
The table of payments (in EUR) for this task; TOML format:
"lkcl"={amount=450,submitted=2020-12-06}


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Luke Kenneth Casson Leighton 2020-06-30 20:09:51 BST
https://github.com/antonblanchard/microwatt/blob/master/xics.vhdl

an XICS controller is needed, to be reasonably compatible with other power implementations
Comment 1 Luke Kenneth Casson Leighton 2020-07-05 17:55:42 BST
external interrupt causes trap to 0x500

	irq_valid := '0';
	if ctrl.msr(MSR_EE) = '1' then
	    if ctrl.dec(63) = '1' then
		v.f.redirect_nia := std_logic_vector(to_unsigned(16#900#, 64));
		report "IRQ valid: DEC";
		irq_valid := '1';
	    elsif ext_irq_in = '1' then
		v.f.redirect_nia := std_logic_vector(to_unsigned(16#500#, 64));
		report "IRQ valid: External";
		irq_valid := '1';
	    end if;
	end if;
Comment 3 Luke Kenneth Casson Leighton 2020-07-26 21:28:07 BST
commit 0ee5e92264b539e4938427c296e8d1477ae704a7 (HEAD -> master, origin/master)
Author: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Date:   Sun Jul 26 21:27:53 2020 +0100

    start on conversion of xics.vhdl to nmigen
Comment 4 Luke Kenneth Casson Leighton 2020-07-27 22:44:22 BST
commit 46a368d998f225d2b07c0c9c2a2a4e14a48a39fc (HEAD -> master, origin/master)
Author: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Date:   Mon Jul 27 22:36:55 2020 +0100

    add 2nd part of XICS interrupt interface
Comment 5 Luke Kenneth Casson Leighton 2020-07-28 13:30:51 BST
Author: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Date:   Tue Jul 28 13:28:22 2020 +0100

    add preliminary investigative test of XICS ICS

documentation (a specification) on the XICS appears not to be locatable.
there exists *implementations* (qemu, linux kernel), just no documentation.
this preliminary unit test therefore explores how the (converted microwatt)
xics ICS works.
Comment 7 Cole Poirier 2020-07-28 17:17:47 BST
It looks like the documentation they have here, except for the part specific to qemu was taken from the XICS spec. So odd that we can't find the actual spec.

https://lists.sr.ht/~philmd/qemu/patches/5642#%3C20190517094435.32765-1-clg@kaod.org%3E+docs/specs/ppc-xive.rst

Well mostly at the bottom third of the page:

```
+++ b/docs/specs/ppc-xive.rst
@@ -0,0 +1,199 @@
+================================
+POWER9 XIVE interrupt controller
+================================
+
+The POWER9 processor comes with a new interrupt controller
+architecture, called XIVE as "eXternal Interrupt Virtualization
+Engine".
+
+Compared to the previous architecture, the main characteristics of
+XIVE are to support a larger number of interrupt sources and to
+deliver interrupts directly to virtual processors without hypervisor
+assistance. This removes the context switches required for the
+delivery process.
+
+
+XIVE architecture
+=================
+
+The XIVE IC is composed of three sub-engines, each taking care of a
+processing layer of external interrupts:
[more useful documentation of Power9 XIVE]
```
Comment 8 Luke Kenneth Casson Leighton 2020-07-28 17:52:38 BST
(In reply to Cole Poirier from comment #7)
> It looks like the documentation they have here, except for the part specific
> to qemu was taken from the XICS spec. So odd that we can't find the actual
> spec.
> 
> https://lists.sr.ht/~philmd/qemu/patches/5642#%3C20190517094435.32765-1-
> clg@kaod.org%3E+docs/specs/ppc-xive.rst

this again is the virtualisation of interrupts, not the actual interrupts.  cinsequently whilst the same registers appear (CPRR, PIPR) there's nothing about those.

my feeling is that the spec has never been released: the qemu XICS code for example is written *by* someone at IBM who had internal access to it.
Comment 9 Cole Poirier 2020-07-28 18:10:29 BST
(In reply to Luke Kenneth Casson Leighton from comment #8)
 
> this again is the virtualisation of interrupts, not the actual interrupts. 
> cinsequently whilst the same registers appear (CPRR, PIPR) there's nothing
> about those.
> 
> my feeling is that the spec has never been released: the qemu XICS code for
> example is written *by* someone at IBM who had internal access to it.

Oy gevalt! Such a headache, although I think Jacob may have found what we are looking for:... will edit with reference to mailing list archives once they are up-to-date enough to include jacob's mail.

```
from: Jacob Lifshay <programmerjake@gmail.com>
to: Libre-RISCV General Development <libre-riscv-dev@lists.libre-riscv.org>
date: Jul 28, 2020, 9:51 AM PDT
subject: Re: [libre-riscv-dev] how do we test external interrupts?

I found what looks like the reference source:
in
https://github.com/torvalds/linux/blob/ab02b61f24c76b1659086fcc8b00cbeeb6e95ac7/Documentation/virt/kvm/devices/xics.rst
> This device emulates the XICS (eXternal Interrupt Controller
Specification) defined in PAPR.

https://en.wikipedia.org/wiki/Power_Architecture_Platform_Reference

https://github.com/OpenPOWERFoundation/Linux-Architecture-Reference

Jacob
```

I think this is the key part:

https://github.com/OpenPOWERFoundation/Linux-Architecture-Reference/blob/master/LoPAR/ch_interrupt_controller.xml
PowerPC External Interrupt Architecture:
```
<chapter  xmlns="http://docbook.org/ns/docbook"
          xmlns:xl="http://www.w3.org/1999/xlink"
          xml:id="dbdoclet.50569331_37856"
          version="5.0"
          xml:lang="en">
  <title>Interrupt Controller</title>

  <para>This chapter specifies the requirements for the LoPAR interrupt
  controller. Platforms may chose to virtualize the interrupt controller or to
  provide the PowerPC External Interrupt option. </para>

  <section>
    <title>Interrupt Controller Virtualization</title>
    <para>Virtualization of the interrupt controller is done through the
    Interrupt Support hcalls. See <xref linkend="dbdoclet.50569344_26787"/>.</para>
  </section>

  <section xml:id="dbdoclet.50569331_29157">
    <title>PowerPC External Interrupt Option</title>
    <para>The PowerPC External Interrupt option is based upon a subset of the
    PowerPC External Interrupt Architecture. The PowerPC External Interrupt
    Architecture contains a register-level architectural definition of an interrupt
    control structure. This architecture defines means for assigning properties
    such as priority, destination, etc., to I/O and interprocessor interrupts, as
    well as an interface for presenting them to processors. It supports both
    specific and distributed methods for interrupt delivery. See also
    <!-- FIXME: xref linkend="error_section"/--><citetitle>A PowerPC External
    Interrupt</citetitle>.htm#38341.--&gt;</para>
    <para>In NUMA platform configurations, the interrupt controllers may be
    configured in disjoint domains. The firmware makes the server numbers visible
    to any single OS image appear to come from a single space without duplication.
    This may be done by appropriately initializing the interrupt presentation
    controllers or the firmware may translate the server numbers presented to it in
    RTAS calls before entering them into the interrupt controller registers. The OS
    is made aware that certain interrupts are only served by certain servers by the
    inclusion of the <emphasis role="bold"><literal>&#x201C;ibm,interrupt-domain&#x201D;</literal></emphasis>
    property in the interrupt controller nodes.</para>

    <section xml:id="sec_ext_int_opt_req">
      <title>PowerPC External Interrupt Option Requirements</title>
      <para>The following are the requirements for the PowerPC External
      Interrupt option. Additional requirements and information relative to the MSI
      option, when implemented with this option, are listed in <xref
      linkend="dbdoclet.50569331_33067"/>.</para>
```

I'll upload just this 'chapter' (only 750 lines of text) to the wiki?
Comment 11 Luke Kenneth Casson Leighton 2020-07-28 18:34:50 BST
(In reply to Cole Poirier from comment #9)
> (In reply to Luke Kenneth Casson Leighton from comment #8)
>  
> > this again is the virtualisation of interrupts, not the actual interrupts. 
> > cinsequently whilst the same registers appear (CPRR, PIPR) there's nothing
> > about those.
> > 
> > my feeling is that the spec has never been released: the qemu XICS code for
> > example is written *by* someone at IBM who had internal access to it.
> 
> Oy gevalt! Such a headache, although I think Jacob may have found what we
> are looking for:... will edit with reference to mailing list archives once
> they are up-to-date enough to include jacob's mail.
> 
> ```
> from: Jacob Lifshay <programmerjake@gmail.com>
> to: Libre-RISCV General Development <libre-riscv-dev@lists.libre-riscv.org>
> date: Jul 28, 2020, 9:51 AM PDT
> subject: Re: [libre-riscv-dev] how do we test external interrupts?
> 
> I found what looks like the reference source:
> in
> https://github.com/torvalds/linux/blob/
> ab02b61f24c76b1659086fcc8b00cbeeb6e95ac7/Documentation/virt/kvm/devices/xics.
> rst
> > This device emulates the XICS (eXternal Interrupt Controller
> Specification) defined in PAPR.
> 
> https://en.wikipedia.org/wiki/Power_Architecture_Platform_Reference
> 
> https://github.com/OpenPOWERFoundation/Linux-Architecture-Reference
> 
> Jacob
> ```
> 
> I think this is the key part:
> 
> https://github.com/OpenPOWERFoundation/Linux-Architecture-Reference/blob/
> master/LoPAR/ch_interrupt_controller.xml
> PowerPC External Interrupt Architecture:
> ```
> <chapter  xmlns="http://docbook.org/ns/docbook"
>           xmlns:xl="http://www.w3.org/1999/xlink"
>           xml:id="dbdoclet.50569331_37856"
>           version="5.0"
>           xml:lang="en">
>   <title>Interrupt Controller</title>
> 
>   <para>This chapter specifies the requirements for the LoPAR interrupt
>   controller. Platforms may chose to virtualize the interrupt controller or
> to
>   provide the PowerPC External Interrupt option. </para>

no it's not that one - XICS is like the PLIC for RISC-V.  it's the central (generic) "hub" for interrupts and defines the layout of in-memory registers.

for example for the source layer (microwatt xics.vhdl):

-- Register map
    --     0  : Config
    --     4  : Debug/diagnostics
    --   800  : XIVE0
    --   804  : XIVE1 ...
    --
    -- Config register format:
    --
    --  23..  0 : Interrupt base (hard wired to 16)
    --  27.. 24 : #prio bits (1..8)
    --
    -- XIVE register format:
    --
    --       31 : input bit (reflects interrupt input)
    --       30 : reserved
    --       29 : P (mirrors input for now)
    --       28 : Q (not implemented in this version)
    -- 30 ..    : reserved
    -- 19 ..  8 : target (not implemented in this version)
    --  7 ..  0 : prio/mask


and for the presentation layer there's something similar.

so the interrupt wires come in from a "source" (a single bit,
each given a number), and it is possible to set the individual
priority (by writing to the XIVEnnn register).

then the "presentation" layer receives a notification about one
of those, and generates - to the core - a *single* interrupt to
say "please come and look at the presentation layer registers
XIRR and MFRR to find out which one of the source interrupts
has occurred".

from the microwatt source code, _reading_ the XIRR register actually
clears the interrupt.

so that's how it fits together.
Comment 12 Cole Poirier 2020-07-28 19:03:37 BST
(In reply to Luke Kenneth Casson Leighton from comment #11)
> no it's not that one - XICS is like the PLIC for RISC-V.  it's the central
> (generic) "hub" for interrupts and defines the layout of in-memory registers.
> 
> for example for the source layer (microwatt xics.vhdl):
> 
> -- Register map
>     --     0  : Config
>     --     4  : Debug/diagnostics
>     --   800  : XIVE0
>     --   804  : XIVE1 ...
>     --
>     -- Config register format:
>     --
>     --  23..  0 : Interrupt base (hard wired to 16)
>     --  27.. 24 : #prio bits (1..8)
>     --
>     -- XIVE register format:
>     --
>     --       31 : input bit (reflects interrupt input)
>     --       30 : reserved
>     --       29 : P (mirrors input for now)
>     --       28 : Q (not implemented in this version)
>     -- 30 ..    : reserved
>     -- 19 ..  8 : target (not implemented in this version)
>     --  7 ..  0 : prio/mask
> 
> 
> and for the presentation layer there's something similar.
> 
> so the interrupt wires come in from a "source" (a single bit,
> each given a number), and it is possible to set the individual
> priority (by writing to the XIVEnnn register).
> 
> then the "presentation" layer receives a notification about one
> of those, and generates - to the core - a *single* interrupt to
> say "please come and look at the presentation layer registers
> XIRR and MFRR to find out which one of the source interrupts
> has occurred".
> 
> from the microwatt source code, _reading_ the XIRR register actually
> clears the interrupt.
> 
> so that's how it fits together.

Ok, so if I understand correctly xics is the internal interrupt infrastructure and the "LoPAR interrupt controller [is something] platforms may chose to virtualize or to provide the PowerPC External Interrupt option." (https://github.com/OpenPOWERFoundation/Linux-Architecture-Reference/blob/master/LoPAR/ch_interrupt_controller.xml) That deals with PCIE interrupts? Sorry I'm still a little confused...
Comment 13 Luke Kenneth Casson Leighton 2020-07-28 19:26:32 BST
(In reply to Cole Poirier from comment #12)
> master/LoPAR/ch_interrupt_controller.xml) That deals with PCIE interrupts?
> Sorry I'm still a little confused...

so the interrupt wires from PCIe would go *into* the source vector XICS_ICS.

e.g. bit 5 would be PCIe device 1's hardware interrupt wire.  or however it is done.

bit 6 of the XICS source would be connected e.g. to SPI0 and so on.
Comment 15 Luke Kenneth Casson Leighton 2020-07-29 12:54:34 BST
(In reply to Luke Kenneth Casson Leighton from comment #14)
> from mikey:
> "Yeah I was just look at LoPAPR and it's not there"
> https://openpowerfoundation.org/wp-content/uploads/2020/07/LoPAR-20200611.pdf

p437 the HCALLs are there.  just no hardware details
Comment 16 Luke Kenneth Casson Leighton 2020-07-31 15:15:00 BST
https://lists.sr.ht/~philmd/qemu/patches/5642#%3C20190517094435.32765-1-clg@kaod.org%3E+docs/specs/ppc-xive.rst

that doc is particularly informative.

i have the preliminary investigative unit tests working, enough to be able to progress to adding XICS as a peripheral.
Comment 17 Luke Kenneth Casson Leighton 2020-09-04 16:16:56 BST
commit ffe70742b1fb287ba1766df8a8e55ec2190db797 (HEAD -> master, origin/master)
Author: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Date:   Fri Sep 4 16:11:10 2020 +0100

    adding option to include XICS external interrupts.
    XICS ICP and ICS are included, the wishbone slave ports added to TestIssuer
    then if ext_irq is raised in core, execution jumps to 0x500 through a TRAP


commit 39f8d6aa73fd74610eeb354f76532fa2413ba2a4 (HEAD -> master)
Author: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Date:   Fri Sep 4 16:16:40 2020 +0100

    bring out XICS ICS interrupt levels so that they can be wired to peripherals
Comment 18 Luke Kenneth Casson Leighton 2020-09-05 15:10:06 BST
commit 2a187089be6e063f6bcaf8c27bfe82a5fe76570c
Author: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Date:   Sat Sep 5 14:38:08 2020 +0100

    add simple GPIO peripheral to verilog TestIssuer


commit 63ccd4c6351103b904a956b7963cf42f53e748ca (HEAD -> master, origin/master)
Author: Luke Kenneth Casson Leighton <lkcl@lkcl.net>
Date:   Sat Sep 5 15:03:21 2020 +0100

    add simple GPIO wishbone bus to litex sim.py
Comment 19 Luke Kenneth Casson Leighton 2020-09-06 09:37:07 BST
microwatt xics.bin unit test works after some modifications to use
a simple GPIO rather than a 16550 UART.