Starting Small: Compute Tile and Embedded Software (Simulated)¶

It is a good starting point to simulate a single compute tile of a distributed memory system. Therefore a simple example is included and demonstrates the general simulation approach and gives an insight in the software building process.

Simulating only a single compute tile is essentially an OpenRISC core plus memory and the network adapter, where all I/O of the network adapter is not functional in this test case. It can therefore only be used to simulate local software.

You can find this example in $OPTIMSOC/examples/sim/compute_tile.

In addition to the simulated SoC hardware, you also need software that runs on the system. Our demonstration software is available in an extra repository:

git clone https://github.com/optimsoc/baremetal-apps
cd baremetal-apps

Build a simple “Hello World” example:

cd hello
make

You will then find the executable elf file as hello/hello.elf. Furthermore some other files are built. They are essentially transformed versions of the ELF file, i.e. the software binary.

• hello.dis is the disassembly of the file

• hello.bin is the elf representation of the binary file

• hello.vmem is a textual copy of the binary file

Now you have everything you need to run the hello world example on a simulated SoC hardware:

$OPTIMSOC/examples/sim/compute_tile/compute_tile_sim_singlecore --meminit=hello.vmem

And you’ll get roughly this output:

[                  22, 0] Software reset
[               63128, 0] Terminated at address 0x0000e958 (status:          0)
- ../src/optimsoc_trace_monitor_trace_monitor/verilog/trace_monitor.sv:89: Verilog $finish

Furthermore, you will find a file called stdout.000 which shows the actual output:

# OpTiMSoC trace_monitor stdout file
# [TIME, CORE] MESSAGE
[               39614, 0] Hello World! Core 0 of 1 in tile 0, my absolute core id is: 0
[               48764, 0] There are 1 compute tiles:
[               57162, 0]  rank 0 is tile 0

Congratulations, you’ve ran your first OpTiMSoC system!

static void sim_putc(unsigned char c) {
    asm("l.addi r3,%0,0" : : "r" (c) : "r3");
    asm("l.nop 4");
}

#define OPTIMSOC_TRACE(id,v)                \
   asm("l.addi r3,%0,0" : : "r" (v) : "r3"); \
   asm("l.nop %0": :"K" (id));

See the Waves¶

One major benefit of simulating a SoC is the possibility to inspect every signal inside the hardware design quite easily. When running a Verilator simulation, as we did in the previous step, you can simply add the --vcd command line option. It instructs Verilator to write all signals into a file. You can then start a waveform viewer, like GTKWave to display it.

$OPTIMSOC/examples/sim/compute_tile/compute_tile_sim_singlecore --meminit=hello.vmem --vcd

This command will run the hello world example like it did before, but this time Verilator additionally writes a sim.vcd waveform file. You can now view this file.

gtkwave -o sim.vcd

The screenshot is similar to what you should see when running GTKWave.

On the left side you find a hierarchy of all signals in the system. Add them to the wave view and explore all internals of a working SoC at your fingertips! Can you find the program counter? The instruction and data caches? The branch predictor?

Going Multicore: Simulate a Multicore Compute Tile¶

Next you might want to build an actual multicore system. In a first step, you can just start simulations of compute tiles with multiple cores.

Inside $OPTIMSOC/examples/sim/compute_tile you’ll find a dual-core version and a quad-core version of the system with just one compute tile that you just simulated in the previous step. You can run those examples like you did before.

The first thing you observe: the simulation runs become longer. After each run, inspect the stdout.* files.

Welcome to the multicore world!

Tiled Multicore SoC: Simulate a Small 2x2 Distributed Memory System¶

Next we want to run an actual NoC-based tiled multicore system-on-chip, with the examples you get system_2x2_cccc. The nomenclature in all pre-packed systems first denotes the dimensions and then the instantiated tiles, here cccc as four compute tiles. In our pre-built example, each compute tile has two CPU cores, meaning you have eight CPU cores in total.

Execute it again to get the hello world experience:

$OPTIMSOC/examples/sim/system_2x2_cccc/system_2x2_cccc_sim_dualcore --meminit=hello.vmem

In our simulation all cores in the four tiles run the same software. Before you shout “that’s boring”: you can still write different code depending on which tile and core the software is executed. A couple of functions are useful for that:

• optimsoc_get_numct(): The number of compute tiles in the system

• optimsoc_get_numtiles(): The number of tiles (of any type) in the system

• optimsoc_get_ctrank(): Get the rank of this compute tile in this system. Essentially this is just a number that uniquely identifies a compute tile.

There are more useful utility functions like those available, find them in the file $OPTIMSOC/soc/sw/include/baremetal/optimsoc-baremetal.h.

A simple application that uses those functions to do message passing between the different tiles is hello_mpsimple. This program uses the simple message passing facilities of the network adapter to send messages. All cores send a message to core 0. If all messages have been received, core 0 prints a message “Received all messages. Hello World!”.

# start from the the baremetal-apps source code directory
cd hello_mpsimple
make
$OPTIMSOC/examples/sim/system_2x2_cccc/system_2x2_cccc_sim_dualcore --meminit=hello_mpsimple.vmem

Have a look what the software does (you find the code in hello_mpsimple.c). Let’s first check the output of core 0.

$> cat stdout.000
# OpTiMSoC trace_monitor stdout file
# [TIME, CORE] MESSAGE
[               42844, 0] Wait for 3 messages
[               48734, 0] Received all messages. Hello World!

Finally, let’s have a quick glance at a more realistic application: heat_mpsimple. You can find it in the same place as the previous applications, hello and hello_mpsimple. The application calculates the heat distribution in a distributed manner. The cores coordinate their boundary regions by sending messages around.

Can you compile this application and run it? Don’t get nervous, the simulation can take a couple of minutes to finish. Have a look at the source code and try to understand what’s going on. Also have a look at the stdout log files. Core 0 will also print the complete heat distribution at the end.

Observing Software During Execution: The Debug System¶

Up to now, you have seen the output of the software that runs on your SoC. And you had a look deep into the inner works of the SoC by looking at the waveforms.

In a real-world system, you need something in between: a way to observe the software as it executes on a chip, but without observing or understanding all the signals inside the hardware. This is what the debug system provides: hardware inside the chip which allows you to observe what’s going on during software execution.

OpTiMSoC also comes with an extensive debug system. In this section, we’ll have a look at this system, how it works and how you can use it to debug your applications. But before diving into the details, we’ll have a short discussion of the basics which are necessary to understand the system.

Many developers know debugging from their daily work. Most of the time it involves running a program inside a debugger like GDB or Microsoft Visual Studio, setting a breakpoint at the right line of code, and stepping through the program from there on, running one instruction (or one line of code) at a time. This technique is what we call run-control debugging. While it works great for single-threaded programs, it cannot easily be applied to debugging parallel software running on possibly heterogeneous many-core SoC. Instead, the debug support in OpTiMSoC mainly relies on tracing. Tracing does not stop or otherwise influence the SoC itself; it only “records” what’s going on during software execution, and transmits this data to the developer.

The debug system consists of two main parts: the hardware part runs on the OpTiMSoC system itself and collects all data. The other part runs on a developer’s PC (often also called host PC) and controls the debugging process and displays the collected data.

After this introduction, let’s make use of the debug system to obtain various traces. Just like in the previous examples, our SoC hardware is still running in Verilator. This tutorial works best if you have multiple terminal windows open at the same time, as we’ll need to have multiple programs running at the same time.

So, open a new terminal (or a new tab inside your terminal), and start the simulation of the SoC hardware.

$OPTIMSOC/examples/sim/system_2x2_cccc/system_2x2_cccc_sim_dualcore_debug

The first and most common task using the debug system is to run a program (just like we did before with the --meminit parameter). Open a second terminal (leave the first one running!) and type

osd-target-run -e hello.elf -vvv

The osd-target-run command takes a couple seconds to finish, so don’t get nervious. If everything goes to plan osd-target-run just does its job: run the provided ELF file hello.elf on all CPUs in the system. To do that, it internally performs the following steps:

• Connect to the simulation over TCP (on port 23000 and 23001)

• Halt all CPUs

• Load all memories in the system (since this is a 2x2 system, there are four memories) with the ELF file

• Reset and start all CPUs

• Close the TCP connection

If you switch back to the first console where you started the simulation you should see something like this:

$> $OPTIMSOC/examples/sim/system_2x2_cccc/system_2x2_cccc_sim_dualcore_debug
Glip TCP DPI listening on port 23000 and 23001
[                  24, 0] Software reset
[                  24, 1] Software reset
[                  24, 2] Software reset
[                  24, 3] Software reset
Client connected
Disconnected
[             1035016, 0] Terminated at address 0x0000ee38 (status:          0)
[             1035016, 1] Terminated at address 0x0000ee38 (status:          0)
[             1035016, 2] Terminated at address 0x0000ee38 (status:          0)
[             1035016, 3] Terminated at address 0x0000ee38 (status:          0)
- ../src/optimsoc_trace_monitor_trace_monitor_0/verilog/trace_monitor.sv:94: Verilog $finish
- ../src/optimsoc_trace_monitor_trace_monitor_0/verilog/trace_monitor.sv:94: Verilog $finish
- ../src/optimsoc_trace_monitor_trace_monitor_0/verilog/trace_monitor.sv:94: Second verilog $finish, exiting

Just like in the previous examples you can see the output of the program runs as captured by the simulation software in the files stdout.NNN in the directory where you started the simulation.

Reading the stdout files works great as long as OpTiMSoC runs in simulation – but how can you access the program’s output when it runs on an FPGA? The answer is called “system trace”, and you’ll learn more about that in the next section.

osd-target-run -e hello.elf --systrace -vvv

osd-target-run -e hello.elf --coretrace -vvv

Our SoC on an FPGA¶

Welcome to the fun of real hardware! Before we can get started, you need to clarify some prerequisites.

optimsoc-pgm-fpga $OPTIMSOC/examples/fpga/nexys4ddr/compute_tile/compute_tile_nexys4ddr_singlecore.bit xc7a100t_0

ls /dev/ttyUSB*

osd-target-run -e hello.elf -b uart -o device=/dev/ttyUSB1,speed=12000000 --coretrace --systrace --verify -vvv
# let it run for a couple of seconds, then press CTRL-C to stop collecting traces

Make Message Passing More Simple¶

So far the example programs you have seen used the low level message passing buffers to exchange data between the tiles. You may remember that exchanging this data involved forming and parsing messages including the low level network-on-chip details.

To abstract from these low level details and to encapsulate certain extensions OpTiMSoC comes with the message passing library (libmp). It is a rather simple, straight-forward message passing API. Two different styles of communication are supported: message-oriented and connection-oriented. Message-oriented communication is preferred when you have spurious communication between many different communication partners. Connection-oriented communication is preferred when you have a fixed setup of channels between communication partners.

In this part of the tutorial you will learn the basic usage of the message passing library using message-oriented communication. In the baremetal-apps you can find the hello_mp example. Inspecting hello_mp.c you can see that it is much less code than the low level example from before.

Lets have a look at how it works. It starts with initializing the hardware and software:

optimsoc_init(0);
optimsoc_mp_initialize(0);

The parameters of those functions can be ignored for now. After calling those functions you can use the message passing library.

Communication in the message passing library takes place between so called endpoints. In the next step we create an endpoint in each tile:

optimsoc_mp_endpoint_handle ep;
optimsoc_mp_endpoint_create(&ep, 0, 0, OPTIMSOC_MP_EP_CONNECTIONLESS, 2, 0);

optimsoc_mp_endpoint_handle is the opaque type used to identify an endpoint in your code. You create and initialize the endpoint by calling optimsoc_mp_endpoint_create() that takes a reference to this handle as first parameter. The second and third parameter initialize the endpoint with a node and port. Each endpoint is globally addressable with its (tile, node, port) identifier. In our case the node 0 and port 0 endpoint is created in each tile.

The remaining parameters of optimsoc_mp_endpoint_create() configure the endpoint. By using OPTIMSOC_MP_EP_CONNECTIONLESS we create it to receive messages from arbitrary tiles. The last two parameters configure the number of messages it can hold and the maximum message size (0 says it is the default).

Now the code of the example diverts again, all but tile 0 execute:

optimsoc_mp_endpoint_handle ep_remote;
optimsoc_mp_endpoint_get(&ep_remote, 0, 0, 0);

optimsoc_mp_msg_send(ep, ep_remote, (uint8_t*) &rank, sizeof(rank));

So what they do is to define a second endpoint. But in this case it is not locally generated but points to a remote endpoint. It is the one we want to send a message too: tile 0, node 0, port 0. What happens under the hood it blocks until the remote endpoint is created and ready and than stores some information locally. In the final step the software sends a word to the remote endpoint using the local endpoint for sending.

In tile zero the software waits to receive all messages using:

optimsoc_mp_msg_recv(ep, (uint8_t*) &remote, 4, &received);

You can now run the example using:

# start from the the baremetal-apps source code directory
cd hello_mp
make
$OPTIMSOC/examples/sim/system_2x2_cccc/system_2x2_cccc_sim_dualcore --meminit=hello_mp.vmem

... (we've skipped some output here) ...
[               37812, 0] Event 0x0380: 0x00018c08
[               37844, 2] Event 0x0380: 0x00018c08
[               37872, 4] Event 0x0380: 0x00018c08
[               37900, 6] Event 0x0380: 0x00018c08
[               39984, 2] External interrupt exception
[               40012, 4] External interrupt exception
[               40040, 6] External interrupt exception
[               42048, 2] Return from exception
[               42076, 4] Return from exception
[               42104, 6] Return from exception
... (we've skipped some output here) ...
[              171970, 6] Event 0x0303: 0x00018d10
[              171982, 6] Event 0x0303: 0x00000000
[              172212, 6] Event 0x0304: 0x00018d10
[              172224, 6] Event 0x0304: 0x00000000
[              172240, 6] Event 0x0304: 0x00000004
[              172782, 0] External interrupt exception
[              173528, 6] Event 0x0305: 0x00018d10
[              174822, 0] Return from exception
[              174944, 6] Terminated at address 0x00011364 (status:          0)
[              185912, 0] Terminated at address 0x00011364 (status:          0)
- ../src/optimsoc_trace_monitor_trace_monitor_0/verilog/trace_monitor.sv:94: Verilog $finish

The simulation is currently very verbose, the events are emitted by the library to debug the message passing protocol. More important is the output of tile 0 in stdout.000:

# OpTiMSoC trace_monitor stdout file
# [TIME, CORE] MESSAGE
[               72050, 0] Received from 1
[               78792, 0] Received from 2
[              179834, 0] Received from 3

Run Linux on OpTiMSoC¶

Up to now all software running on OpTiMSoC was “baremetal” software, similar to software run on a microcontroller. For many purposes “baremetal” software is sufficient. However, if you want to write more advanced software an operating system (OS) can help: it provides task management (scheduling), separates resources between tasks, and provides standardized interfaces which are expected by many of today’s applications (e.g. pthreads). For many, the operating system of choice is Linux, and it’s natively supported by OpTiMSoC. This tutorial section explores how to build a Linux “image,”. i.e. a binary which contains both the Linux kernel (the actual operating system), together with a root filesystem containing all userspace components.

# get the OpTiMSoC buildroot configuration (a "br2-external tree")
git clone https://github.com/optimsoc/optimsoc-buildroot.git
cd optimsoc-buildroot
OPTIMSOC_BUILDROOT_DIR=$PWD
OPTIMSOC_BUILDROOT_VERSION=$(cat $OPTIMSOC_BUILDROOT_DIR/buildroot_version)
cd .. # back to your source directory

# get buildroot itself
git clone https://git.busybox.net/buildroot
cd buildroot
git checkout $OPTIMSOC_BUILDROOT_VERSION
make BR2_EXTERNAL=$OPTIMSOC_BUILDROOT_DIR optimsoc_computetile_singlecore_defconfig
make

This leaves a file output/images/vmlinux in the buildroot directory, which is in fact a regular ELF file for OpenRISC, which can be loaded on the system like a baremetal application. To see the output of the Linux during boot, and to have a console to interact with the Linux system we make use of the UART device emulation provided by Open SoC Debug, and built into the compute_tile designs.

To continue with this tutorial we use the compute_tile design with a with a single core and the debug system for the Nexys 4 DDR board. You can find this design in the folder $OPTIMSOC/examples/fpga/nexys4ddr/compute_tile/compute_tile_nexys4ddr_singlecore.

# Program the FPGA on the Nexys 4 DDR board
optimsoc-pgm-fpga $OPTIMSOC/examples/fpga/nexys4ddr/compute_tile/compute_tile_nexys4ddr_singlecore.bit xc7a100t_0

Now you can load the Linux image on the FPGA. See notes earlier in this tutorial for a discussion on the correct parameters for osd-target-run.

osd-target-run -e YOUR_BUILDROOT_DIR/output/images/vmlinux -b uart -o device=/dev/ttyUSB1,speed=12000000 --systrace -vvv

Watch the output of this command. If all goes well the output should contain a line similar to libosd: DEM-UART pseudo-terminal available at /dev/pts/19. Keep note of the device file (starting with /dev/pts/), you’ll need this path to connect to the Linux console on the OpTiMSoC system.

To connect, open a second terminal window on your machine, and use screen to connect to the remote console (use the appropriate device name as displayed by osd-target-run instead of /dev/pts/19):

screen /dev/pts/19

You should now see the output of Linux booting, and as soon as the boot process is done you can log into the system as root user (no password is required). You can now interact with the system as it would be a normal Linux system.

If you have some time to spare, how about playing a round of pacman?

# convince Linux that our console supports colors
stty cols 80 rows 80
export TERM=linux

# and run pacman
/usr/games/pacman4linux

This concludes our tutorial session, and hands over to you: modify the software as you wish, program it again, analyze the simulations and explore your first multicore SoC.