SoC framework, part 2: layer 2/3 protocols

Introduction

This is the second post in a series on the SoC framework I'm developing for my research. I'm going to get into more interesting topics (such as my build/test framework and FPGA cluster) shortly, but to understand how all of the parts communicate it's necessary to understand the basics of the SoC interconnect.

I'm omitting some of the details of link-layer flow control and congestion handling for now as it's not necessary to understand the higher-level concepts. If anyone really wants to know the dirty details, comment and I'll do a post on it at some point in the future.

As I mentioned briefly in part 1 of the series, my interconnect actually consists of two independent networks with the same topology. The RPC network is intended for control-plane transactions and supports function call/return semantics (request followed by response) as well as interrupts (one-way datagrams). The DMA network is meant for bulk data transfers between cores and memory devices.

Layer-2 header

The layer-2 header is the same for both networks:

31:24	23:16	15:8	7:0
Source address		Dest address

This is then followed by the layer-3 header for the protocol of interest. Which protocol is in use depends on the interface; the routers are optimized for one or the other. I may consider changing this in the future.

DMA network

Packet format

31:24	23:0
Layer-2 header
Opcode	Payload length in words (only rightmost 10 bits implemented)
Physical memory address
Zero or more application-layer data words

Protocol description

The DMA network is meant for bulk data transfers and is normally memory mapped when used by a CPU.

It supports read and write transactions of an integer number of 32-bit words, up to 512 data words plus three header words. This size was chosen so that a DMA transfer could transport an entire Ethernet frame or typical NAND page in one packet.

Byte write enables are not supported; it is expected that a CPU core requiring this functionality will use read-modify-write semantics inside the L1 cache and then move words (or cache lines containing several words) over the DMA network.

The physical DMA address space is 48 bits: each of the 2^16 possible cores in the SoC has 32 bits of address space. If one core requires more than 4GB of address space it may respond to several consecutive DMA addresses. CPU cores are expected to translate the 48-bit physical addresses into 32 or 64 bit virtual addresses as required by their microarchitecture.

Write transactions are unidirectional: a single packet with opcode set to "write request" is all that is required. The destination host may send an RPC interrupt back on success or failure of the write however this is not required by the layer 3 protocol. Specific application layer APIs may mandate write acknowledgements.

Read transactions are bidirectional: a "read request" packet with length set to the desired read size, and no data words, is sent. The response is a "read data" packet with the appropriate length and data fields. As with write transactions, failure interrupts are optional at layer 3 but typically required by application layer APIs.

RPC network

Packet format

31:24	23:21	20:0
Layer-2 header
Callnum	Type	Application-layer data
Application-layer data
Application-layer data

Protocol description

The RPC network is meant for small, low-latency control transfers and is normally register mapped when used by a CPU.

It supports fixed length packets of four word length so as to easily fit into standard register-based calling conventions.

The "callnum" field uniquely identifies the specific request / interrupt being performed. The meaning of this field is up to the application-layer protocol.

The "type" field can be one of the following:

• Function call request
  The source host is requesting the destination host to perform some action. A response is required.
• Function return (success)
  The source host has completed the requested action successfully. The application-layer protocol may specify a return value.
• Function return (fail)
  The source host attempted the requested operation but could not complete it. The application-layer protocol may specify an error code.
• Function return (retry)
  The source host is busy with a long-running operation and cannot complete the requested operation now, but might be able to in the future. The source host may re-send the request in the future or consider this to be a failure.
• InterruptSomething interesting happened at the source host, and the destination host has previously requested to been notified when this happened.
• Host prohibited
  Sent by a router to indicate that the destination host attempted to reach a host in violation of security policy. The source address of the packet is the prohibited address.
• Host unreachable
  Sent by a router to indicate that the destination host attempted to reach a nonexistent address. The source address of the packet is the invalid address.