The Xilinx Microblaze soft processor, which is implemented on the FPGA’s logic fabric, is indeed a stable and fully capable processor, but its rather low clock frequency — 70-100 MHz on a Spartan-6 — makes it a problematic candidate for data capture and frame grabbing.

When running Linux on Microblaze, the current kernel allows for a data rate of approximately 1 MByte/sec due to internal overhead. It appears like there’s a lack of optimization of the parts in the kernel copying data.

So while Linux on Microblaze is a great solution for making the FPGA talk with storage and network in a high-level manner, it suffers from a very slow I/O, rendering it useless for data capture to a network shared disk, for example.

How it’s tackled

Technically speaking, the solution is to capture data directly into the processor’s DDR memory using DMA. Since the 32-bit bus’ frequency is the same as the processor’s, even the lower end of 70 MHz allows for a theoretic throughput of 280 MBytes/sec. In practice, the Xillybus IP core has the proven capability of capturing data arriving at a continuous rate of 250 MBytes/sec, on bursts of 8 MBytes each.

Keep it simple

Another bonus with using Xillybus, is that the data is fed into a standard asynchronous FIFO on the FPGA. There is no need to interface with Microblaze’s buses, just connect data and read enable to a FIFO. The IP core supplies additional signals for synchronizing events with the processor, but their use is optional.

On the Linux side, it all boils down to opening a device file, reading data normally, and closing the file. The FPGA can signal EOF (end-of-file), so making a high speed data capture can be done from the shell prompt with the “cat” or “dd” commands. There’s no need to write complicated software nor a driver. Just a single standard UNIX command, and the data is stored in a regular disk file.

One thing to take into account, is that even though an 8 MBytes chunk of data is captured into the processor’s RAM in a split second, the I/O operation of copying it into some other media will typically take around 8 seconds. The memory access is fast, but the processor isn’t all that so.

A few technicalities

A working Linux distribution for Microblaze is available for download at Xillybus’ site. While this distribution has the Xillybus IP core and kernel driver included, that version captures data at the processor’s slow rates. For an evaluation kit supporting fast data capture, please contact Xillybus directly.

Another thing to mention is the reason for the 8 MBytes limit: The DDR memories come in larger sizes, but DMA memory is inherently within Linux kernel space. Allocating large physically continuous segments of RAM is difficult, and doing too well on that can make the entire system unstable.

There is a well-known workaround for this, though: It’s possible to give the kernel a boot parameter limiting the RAM it’s allowed to access. Using this simple trick, it’s possible to use the untouched chunk as a huge buffer. This requires a simple modification on the Xillybus driver. So it’s not so difficult to allow a capture segment of any size, as long as there’s enough RAM for both the buffer and the kernel itself.

This distribution is a software kit, which allows you to run a functional Linux system on the SP605 or ML605 hardware evaluation kit (for Spartan-6 and Virtex-6, respectively). All components necessary to build it will be available for download at no cost (partly from Xilinx’ site). The distribution is not a demo kit; it’s a kickstart for a real-life application.

The steps to reaching a functional system include installing pieces of software, and perform specific and rather trivial operations as described in detail. No prior knowledge on either FPGA nor Linux is necessary. Your computer may run either Windows or Linux for running this through.

This distribution will allow you to

• Run a Linux system which is ready for working with, including a DHCP client, telnet, ftp, a small web server, mounting of NFS as well as Windows (SMB) shares. The CompactFlash card will also serve as the local disk which can be written to.
• Easily compile your own user-space applications in with “make” and gcc (a ready-to-go cross-compilation environment will be available for download as well). Compilation is possibly dynamic against libraries which are part of the distro.
• Easily develop your own Linux-reachable peripherals on the FPGA using the Xillybus IP core, and transport data between the FPGA and Linux user space applications with minimal effort.

Required hardware

• A Xilinx SP605 evaluation kit or ML605 ditto.
• A Sandisk CompactFlash card, with at least 512 MB. Let me emphasize this: Make it a Sandisk card, and not anything else. You may get away with using other cards for a one-off loading with data and multiple reads, but the Linux system is going to use this storage as its hard disk. Anything else than Sandisk, including the card which comes with the evaluation kit itself, is most likely to have reliability problems, which will give you a very wrong impression regarding the Linux system’s stability. Believe me, I’ve tried quite a few. Also, try to make the card not bigger than 8 GB, as the SysACE chip’s support of larger cards is questionable.
• A CompactFlash to USB adapter. There is no special requirements here, as long as your operating system recognizes it, and it’s reliable. Which holds true even for those cheap universal card readers available on Ebay.

As mentioned above, this is in the Xillybus mini-distro for Microblaze’s page.

GNUPREFIX=../gnutools/microblazeel-unknown-linux-gnu/bin/microblazeel-unknown-linux-gnu-

CC=$(GNUPREFIX)gcc
AR=$(GNUPREFIX)ar
AS=$(GNUPREFIX)as
CXX=$(GNUPREFIX)g++
LD=$(GNUPREFIX)ld
STRIP=$(GNUPREFIX)strip

CFLAGS=-Wall -I. -O3

APPLICATION=hello

OBJECTS=#somefile.o somefile2.o etc

all: $(APPLICATION)

$(APPLICATION): $(OBJECTS) $(APPLICATION).o
	$(CC) $(CFLAGS) $(OBJECTS) $(APPLICATION).o -o $(APPLICATION)

clean:
	rm -f *~ $(APPLICATION) *.o

# mount -o remount,rw /

# mount -o remount,rw /dev/xsa3

# mount -o remount,ro /

# mount -o remount,ro /dev/xsa3

# halt

#!/bin/sh  

ifconfig eth0 10.12.14.16 netmask 255.0.0.0

/ # chmod a+x /rw/system/userrc

/ # dhcp
udhcpc (v1.17.1) started
Sending discover...
PHY: c0729a0c:07 - Link is Up - 100/Full
Sending discover...
Sending select for 10.12.14.16...
Lease of 10.12.14.16 obtained, lease time 21600
deleting routers
route: SIOCDELRT: No such process
adding dns 10.0.0.1
adding dns 10.0.0.2
/ # eth0: no IPv6 routers present

/ # ifconfig eth0 10.12.14.16 netmask 255.0.0.0

/ # route add default gw 10.11.12.13

# mount -o user=theuser,password=thepass //10.11.12.13/mydisk /mnt/try

# mount 10.11.12.13:/thedir -o nolock,proto=tcp /mnt/try

/ # mount -o remount,rw /

Scope

Even though Xilinx supplies a cute wizard for creating peripherals in its EDK (version 13.2 in my case), it’s just enough to work as a demo. For a real-life case there’s no escape from getting down to the system’s guts. As it turns out, things are pretty well organized under EDK’s hood, which makes the attempt to cover it all up with a wizard even more questionable.

This post is a jot-down of the technicalities behind designing a minimal bare-boned peripheral and its Linux driver. With “bare-boned” I mean that it has the absolutely simplest bus interface (AXI4 Lite without even decoding the addresses), and that it’s in pure Verilog. No external IP is used.

This peripheral connects to the SP605′s four LEDs. Any write to its address region updates the LED’s state according to the written value’s four LSBs. Reading back from any of its covered addresses gives the four LSBs back. That’s all.

Sources of information

This post assumes that you’re familiar with running Linux on Microblaze. I have a pretty extensive tutorial on the subject for reference.

These are worth to look at:

• The official cores’ sources (in VHDL) can be found at ISE_DS\EDK\hw\XilinxProcessorIPLib\pcores (path from where ISE is installed). It’s interesting in particular to look at axi_lite_ipif_v1_00_a.
• The AMBA AXI protocol specification, downloaded free from ARM’s site.
• Platform Specification Format Reference Manual (UG642, psf_rm.pdf): Describes the file formats in detail. Necessary when editing the files.
• EDK Concepts, Tools, and Techniques (UG683, edk_ctt.pdf) : The chapter about Creating Your Own Intellectual Property is somewhat helpful to try out the Wizard.

Understanding the process

It looks like the missing link in Xilinx’ documentation is to explain how the whole machinery works with regard to adopting a custom made peripheral. I’ll try to fill in that gap now.

Generally speaking, the minimal core consists of the following files, which should be in a dedicated directory under the “pcores” directory, which is under the EDK project’s top directory:

• data/minimal_v2_1_0.mpd: This file is what EDK looks at when an IP is added to a project. It contains all the information used directly by the EDK. The peripheral’s GUI is set up according to this information, and it’s also used when the EDK generates wrappers and connections for it. Its format is well documented, but it looks like it’s easier to just copy snippets from existing core’s MPD files. It’s also possible to generate this file automatically with PsfUtility from the toplevel source file, but it’s not clear if it’s worth the effort to learn yet another tool.
• data/minimal_v2_1_0.pao: This file supplies EDK with a list of HDL files which need to be synthesized to create the peripheral. It also sets the order of synthesis.
• hdl/verilog/minimal.v: The Verilog file constituting the peripheral. Of course there may be several files, which need to be mentioned in the PAO file.
• Note that “black box” modules (presynthesized netlists) are listed in BBD files, which are not necessary in this example. When used, the MPD file is set to reflect this.

The file names above relate to a peripheral called “minimal”. They change according to the project’s setting and version numbers.

All in all, the flow is pretty simple: Only the MPD file is considered by EDK, and only at platform netlist generation are the HDL files synthesized according to the PAO file. The instantiation and connection depend on the settings within the EDK (actually, the MHS file).

It’s easiest to create just any peripheral with the wizard, see what they do, and then modify the files.

Going from Wizard’s output to minimal peripheral

This is a short outline of the stages. The result is given in the next section.

• Edit the data/*.pao file: Remove all files and insert the single Verilog file, changing the type to verilog.
• In the data/*.mpd file, change OPTION HDL = VHDL to VERILOG too. Also add, under ##ports, PORT minimal_leds = “”, DIR = O, VEC = [3:0] (so that the I/O port is known. Note the =”" part).
• Remove data/*.prj file  so no unnecessary files are included (even though this file seems to be ignored).
• Roughly translate the VHDL file to Verilog. Make it Verilog parameters instead of VHDL generics.
• Rename/remove the devl directory, since its information is not in sync with the new situation, and Custom IP Wizard can’t do any good at this point.
• And finally, in EDK, Project > Rescan User Repositories
• Remove the LED_4bits core from the project, choosing “Delete instance and any connections to internal nets”. This will keep the net names used for connecting to the LEDs, and make them available for connection to the new peripheral. Otherwise, the external net names need to be set, and the system.ucf given at the “project” tab updated to reflect the new nets.
• Add the minimal core to the project, and connect the just released LEDs_4Bits_TRI_O to its minimal_leds port.
• Create bitfile

The synthesis of the peripheral’s HDL takes place during the “create netlist” flow (which is, of course, part of generating bitfile). For example, the synthesis of an instance named minimal_0 will appear as follows in the console

INSTANCE:minimal_0 - C:\tryperipheral\system.mhs line 424 - Running XST
synthesis
PMSPEC -- Overriding Xilinx file
<C:/ise13_2/ISE_DS/EDK/spartan6/data/spartan6.acd> with local file
<C:/ise13_2/ISE_DS/ISE/spartan6/data/spartan6.acd>

And if there are errors in the HDL, they will show up at this point.

Sample files

These are the files used for the minimal peripheral. They are a sloppy adoption of the files generated by the Custom IP Wizard, so they’re very likely to contain unnecessary declarations.

First, the Verilog file:

module minimal #(
 parameter C_S_AXI_DATA_WIDTH = 32,
 parameter C_S_AXI_ADDR_WIDTH = 32,
 parameter C_S_AXI_MIN_SIZE = 'h000001FF,
 parameter C_USE_WSTRB = 0,
 parameter C_DPHASE_TIMEOUT = 8,
 parameter C_BASEADDR = 'hFFFFFFFF,
 parameter C_HIGHADDR = 'h00000000,
 parameter C_FAMILY = "spartan6",
 parameter C_NUM_REG = 1,
 parameter C_NUM_MEM = 1,
 parameter C_SLV_AWIDTH = 32,
 parameter C_SLV_DWIDTH = 32
 )
 (
 input S_AXI_ACLK,
 input S_AXI_ARESETN,
 input [(C_S_AXI_ADDR_WIDTH-1):0] S_AXI_AWADDR,
 input S_AXI_AWVALID,
 input  [(C_S_AXI_DATA_WIDTH-1):0] S_AXI_WDATA,
 input  [((C_S_AXI_DATA_WIDTH/8)-1):0] S_AXI_WSTRB,
 input S_AXI_WVALID,
 input S_AXI_BREADY,
 input  [(C_S_AXI_ADDR_WIDTH-1):0] S_AXI_ARADDR,
 input S_AXI_ARVALID,
 input S_AXI_RREADY,
 output S_AXI_ARREADY,
 output  [(C_S_AXI_DATA_WIDTH-1):0] S_AXI_RDATA,
 output  [1:0] S_AXI_RRESP,
 output S_AXI_RVALID,
 output S_AXI_WREADY,
 output [1:0] S_AXI_BRESP,
 output reg S_AXI_BVALID,
 output S_AXI_AWREADY,

 output reg [3:0] minimal_leds
 );

 assign  S_AXI_RDATA = minimal_leds;
 assign  S_AXI_RRESP = 0; // OKAY on AXI4
 assign  S_AXI_ARREADY = 1; // Always ready for read address
 assign  S_AXI_AWREADY = 1; // Always ready for write address
 assign  S_AXI_RVALID = 1; // Read data always valid (ILLEGAL)
 assign  S_AXI_WREADY = 1; // Always ready to write
 assign  S_AXI_BRESP = 0; // OKAY on AXI4

 // This will not work OK if several "bursts" are sent with no BVALIDs
 // inbetween. Not an expected scenario.

 always @(posedge S_AXI_ACLK)
   if (S_AXI_WVALID)
     begin
        S_AXI_BVALID <= 1;
        minimal_leds <= S_AXI_WDATA;
     end
   else if (S_AXI_BREADY && S_AXI_BVALID) // Active BRESP cycle
     S_AXI_BVALID <= 0;

endmodule

Most of the parameters at the top can be removed, I believe. It appears like they are necessary only when creating the MPD file with PsfUtility.

All ports, except minimal_leds are standard AXI4 lite ports. The implementation of the interface isn’t example for anything except a quick and dirty peripheral which responds to bus requests. The only thing it does actively is to update minimal_leds when necessary, and toggle the AXI_BVALID, so that only one burst response is sent for each write cycle (which is always one clock long in AXI4 lite). It’s OK not to decode the address, since it’s the interconnect’s job to make sure each peripheral gets only what it directed to it.

Holding S_AXI_RVALID high all the time violates the AXI4 spec, since it’s required to be asserted only after ARVALID and ARREADY. But the interconnect tolerated this anyhow.

Now to minimal_v2_1_0.mpd:

BEGIN minimal

## Peripheral Options
OPTION IPTYPE = PERIPHERAL
OPTION IMP_NETLIST = TRUE
OPTION HDL = VERILOG
OPTION IP_GROUP = MICROBLAZE:USER
OPTION DESC = MINIMAL
OPTION LONG_DESC = A minimal peripheral to start off with
OPTION ARCH_SUPPORT_MAP = (others=DEVELOPMENT)

## Bus Interfaces
BUS_INTERFACE BUS = S_AXI, BUS_STD = AXI, BUS_TYPE = SLAVE

## Generics for VHDL or Parameters for Verilog
PARAMETER C_S_AXI_DATA_WIDTH = 32, DT = INTEGER, BUS = S_AXI, ASSIGNMENT = CONSTANT
PARAMETER C_S_AXI_ADDR_WIDTH = 32, DT = INTEGER, BUS = S_AXI, ASSIGNMENT = CONSTANT
PARAMETER C_S_AXI_MIN_SIZE = 0x000001ff, DT = std_logic_vector, BUS = S_AXI
PARAMETER C_USE_WSTRB = 0, DT = INTEGER
PARAMETER C_DPHASE_TIMEOUT = 8, DT = INTEGER
PARAMETER C_BASEADDR = 0xffffffff, DT = std_logic_vector, MIN_SIZE = 0x0, PAIR = C_HIGHADDR, ADDRESS = BASE, BUS = S_AXI
PARAMETER C_HIGHADDR = 0x00000000, DT = std_logic_vector, PAIR = C_BASEADDR, ADDRESS = HIGH, BUS = S_AXI
PARAMETER C_FAMILY = virtex6, DT = STRING
PARAMETER C_NUM_REG = 1, DT = INTEGER
PARAMETER C_NUM_MEM = 1, DT = INTEGER
PARAMETER C_SLV_AWIDTH = 32, DT = INTEGER
PARAMETER C_SLV_DWIDTH = 32, DT = INTEGER
PARAMETER C_S_AXI_PROTOCOL = AXI4LITE, TYPE = NON_HDL, ASSIGNMENT = CONSTANT, DT = STRING, BUS = S_AXI

## Ports
PORT S_AXI_ACLK = "", DIR = I, SIGIS = CLK, BUS = S_AXI
PORT S_AXI_ARESETN = ARESETN, DIR = I, SIGIS = RST, BUS = S_AXI
PORT S_AXI_AWADDR = AWADDR, DIR = I, VEC = [(C_S_AXI_ADDR_WIDTH-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_AWVALID = AWVALID, DIR = I, BUS = S_AXI
PORT S_AXI_WDATA = WDATA, DIR = I, VEC = [(C_S_AXI_DATA_WIDTH-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_WSTRB = WSTRB, DIR = I, VEC = [((C_S_AXI_DATA_WIDTH/8)-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_WVALID = WVALID, DIR = I, BUS = S_AXI
PORT S_AXI_BREADY = BREADY, DIR = I, BUS = S_AXI
PORT S_AXI_ARADDR = ARADDR, DIR = I, VEC = [(C_S_AXI_ADDR_WIDTH-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_ARVALID = ARVALID, DIR = I, BUS = S_AXI
PORT S_AXI_RREADY = RREADY, DIR = I, BUS = S_AXI
PORT S_AXI_ARREADY = ARREADY, DIR = O, BUS = S_AXI
PORT S_AXI_RDATA = RDATA, DIR = O, VEC = [(C_S_AXI_DATA_WIDTH-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_RRESP = RRESP, DIR = O, VEC = [1:0], BUS = S_AXI
PORT S_AXI_RVALID = RVALID, DIR = O, BUS = S_AXI
PORT S_AXI_WREADY = WREADY, DIR = O, BUS = S_AXI
PORT S_AXI_BRESP = BRESP, DIR = O, VEC = [1:0], BUS = S_AXI
PORT S_AXI_BVALID = BVALID, DIR = O, BUS = S_AXI
PORT S_AXI_AWREADY = AWREADY, DIR = O, BUS = S_AXI
PORT minimal_leds = "", DIR = O, VEC = [3:0]

END

This file is exactly as generated by the Wizard, except for the HDL option in the beginning changed to VERILOG, and the added port minimal_leds at the end. Note its assignment to “”. This file is best created by looking at examples of existing cores.

Now to minimal_v2_1_0.pao:

lib minimal_v1_00_a minimal verilog

which was rewritten to reflect that the peripheral consists of one single Verilog file.

The device tree file

The device tree file needs to be generated as described in one of my posts. The relevant section is given here, since it relates to kernel code presented next:

minimal_0: minimal@7ae00000 {
 compatible = "xlnx,minimal-1.00.a";
 reg = < 0x7ae00000 0x10000 >;
 xlnx,dphase-timeout = <0x8>;
 xlnx,family = "spartan6";
 xlnx,num-mem = <0x1>;
 xlnx,num-reg = <0x1>;
 xlnx,s-axi-min-size = <0x1ff>;
 xlnx,slv-awidth = <0x20>;
 xlnx,slv-dwidth = <0x20>;
 xlnx,use-wstrb = <0x0>;
 }

It’s pretty evident that some of these parameters have no use.

The driver

First, it’s convenient to create a makefile for cross compilation. Even though the correct way is to set the environment variables in the shell, and run the module compilation in the same way the kernel itself is compiled, it’s much more convenient to go just “make” or “make clean” with this makefile. It’s not good for distribution, as the paths to both the kernel tree and cross compiler are hardcoded.

So here’s a dirty, but yet convenient makefile:

export CROSS_COMPILE=/path/to/microblazeel-unknown-linux-gnu/bin/microblazeel-unknown-linux-gnu-
export ARCH=microblaze

ifneq ($(KERNELRELEASE),)
obj-m    := minimal.o

else
KDIR := /path/to/linux-2.6.38.6

default:
 @echo $(TARGET) > module.target
 $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules

clean:
 @rm -f *.ko *.o modules.order Module.symvers *.mod.? .minimal.* *~
 @rm -rf .tmp_versions module.target

minimal.ko:
 $(MAKE)
endif

And now to the driver itself, minimal.c:

#include <linux/platform_device.h>
#include <linux/module.h>
#include <linux/of.h>
#include <linux/of_address.h>
#include <linux/of_device.h>
#include <linux/of_platform.h>
#include <asm/io.h>

/* Match table for of_platform binding */
static struct of_device_id minimal_of_match[] __devinitdata = {
 { .compatible = "xlnx,minimal-1.00.a", },
 {}
};

MODULE_ALIAS("minimal");

static void __iomem *regs;
static struct resource res;

static int __devinit
minimal_of_probe(struct platform_device *op, const struct of_device_id *match)
{

 const int *width;
 int ret;
 int val;

 ret = of_address_to_resource(op->dev.of_node, 0, &res);
 if (ret) {
 printk(KERN_WARNING "minimal: Failed to obtain device tree resource\n");
 return ret;
 }

 printk(KERN_WARNING "minimal: Physical address to resource is %x\n", (unsigned int) res.start);

 if (!request_mem_region(res.start, 32, "mimimal")) {
 printk(KERN_WARNING "minimal: Failed to request I/O memory\n");
 return -EBUSY;
 }

 regs = of_iomap(op->dev.of_node, 0); /* Verify it's non-null! */

 printk(KERN_WARNING "minimal: Access address to registers is %x\n", (unsigned int) regs);

 width = of_get_property(op->dev.of_node, "xlnx,slv-dwidth", NULL);

 printk(KERN_WARNING "minimal: Obtained width=%d\n", be32_to_cpu(*width));

 val = ioread32(regs);
 printk(KERN_WARNING "minimal: Read %d, writing %d\n", val, val+1);

 iowrite32(++val, regs);

 return 0; /* Success */
}

static int __devexit minimal_of_remove(struct platform_device *op)
{
 iounmap(regs);
 release_mem_region(res.start, 32);
 return 0; /* Success */
}

static struct of_platform_driver minimal_of_driver = {
 .probe = minimal_of_probe,
 .remove = __devexit_p(minimal_of_remove),
 .driver = {
 .name = "minimal",
 .owner = THIS_MODULE,
 .of_match_table = minimal_of_match,
 },
};

int __init minimal_init(void)
{
 int ret;
 ret = of_register_platform_driver(&minimal_of_driver);
 return ret;
}

void __exit minimal_exit(void)
{
 of_unregister_platform_driver(&minimal_of_driver);
}

module_init(minimal_init);
module_exit(minimal_exit);

MODULE_AUTHOR("Eli Billauer");
MODULE_DESCRIPTION("Microblaze minimal module");
MODULE_LICENSE("GPL")

It doesn’t do anything special, except for change the state of the LEDs every time it’s loaded. The drivers also reads one of the parameters from the device tree structure. Not fascinating, but keeps the code, well, minimal.

This code should be pretty straightforward to programmers who are familiar with PCI device drivers, with probing and removal working in more or less the same way. I’ve chosen a hardcoded segment of 32 bytes as the requested region. This depends on the peripheral, of course.

A test run

This is the transcript of the session on the UART console, as run on a freshly booted system. LEDs did indeed go on and off as reflected by the numbers.

/ # insmod minimal.ko
minimal: Physical address to resource is 7ae00000
minimal: Access address to registers is c87e0000
minimal: Obtained width=32
minimal: Read 0, writing 1
/ # lsmod
minimal 1978 0 - Live 0xc8056000
ipv6 305961 10 - Live 0xc8763000
/ # cat /proc/iomem
40600000-4060000f : uartlite
40a00000-40a0ffff : xilinx_spi
40e00000-40e0ffff : xilinx_emaclite
7ae00000-7ae0001f : mimimal
/ # rmmod minimal
rmmod: module 'minimal' not found
/ # cat /proc/iomem
40600000-4060000f : uartlite
40a00000-40a0ffff : xilinx_spi
40e00000-40e0ffff : xilinx_emaclite
/ # lsmod
ipv6 305961 10 - Live 0xc8763000
/ # insmod minimal.ko
minimal: Physical address to resource is 7ae00000
minimal: Access address to registers is c8820000
minimal: Obtained width=32
minimal: Read 1, writing 2

Note that rmmod produces an error message, which makes it look as if it failed to remove the module, when in fact all went well.

The physical address was indeed detected correctly (see device tree), and mapped to another kernel virtual address each time.

• Part I: Introduction and setting up the Microblaze processor
• Part II: Compiling the kernel
• Part III: Preparing for boot and booting
• Part IV: Compiling user space applications (this page)

Compiling user space applications

We shall now look at how to compile applications for execution under the Microblaze Linux machine. This is pretty straightforward for programs written for the specific environment. The problems may occur when compiling sources which were originally written for fullblown computers, as the build system may not have taken cross compilation into account. And as software projects tend to be hacked to death, with new features added all the time, the code may depend on libraries which are installed on every desktop, but not necessarily on an embedded system. These dependencies are at times a result of a completely offbeat feature, but it’s often simpler to compile the necessary library than to remove the feature. Since there is no single recipe for solving that kind of problems, we’ll stick to the basics of compiling for user space.

Background

Cross compilation of user space applications is actually more difficult than compiling the kernel, mainly because the kernel itself is, after all, a standalone application. There are a few things to take care of:

• Make sure libraries for dynamic linking are in place in the target runtime filesystem, as well as the dynamic linker itself.
• The compilation should be done against the header files corresponding to the libraries present in the target.
• The linked libraries used during compilation should correspond to those in the target.
• The C Runtime stubs (crt1.o, crti.o, crtbegin.o, crtend.o and crtn.o) should fit the Linux user space environment (different files with similar names are used for Microblaze standalones).

This may sound complicated, but most of the job has already been done. So it all boils down to a few simple things to bare in mind, as shown next.

Preparing the target’s file system

In part II it was shown how to download and extract the cross compiler for Microblaze. The same tarball also has the entire package for the target’s root under microblazeel-unknown-linux-gnu/microblazeel-unknown-linux-gnu/sys-root. This directory should be copied into the target’s root as is.

But the target’s root directory is already populated with files as necessary to boot Linux, and run command line utilities with busybox. Some of the files in sys-root, dynamic libraries in particular, already exist in target root, and they’re not identical. But since busybox is statically linked, overwriting the dynamic libraries seems harmless. Overwriting the previous files where applicable is therefore the way to resolve these conflicts, since dynamically linked applications will be compiled against the newer libraries.

All in all, the sys-root directory is ~164 MB, which isn’t too bad when stored on a flash memory. ~129 of these are /lib. The library files include libm, libpthread, libresolv and several other important libraries.

/usr/include takes up ~14 MB, which is probably not necessary on the target system.

Compilation

For cross compilation we use the same compiler used for the kernel. To compile an application:

/path/to/microblazeel-unknown-linux-gnu-gcc --sysroot=/path/to/nfsroot/ -o hello hello.c -lm

where /path/to/nfsroot is the directory which will be the root directory when the executable runs, or a copy of it. The truth is I’m not really sure –sysroot is really necessary, but given the pretty wild search the GNU tools do to find include files and libraries, it looks like a good measure to point directly at where these should be found.

Note that if we omit –sysroot, compiling for Microblaze Linux user space is done simply by using the cross compiler normally, and that works too. This happens because the compiler was configured to look for libraries and includes from its own sys-root. The C runtime stubs are always taken from the compiler’s own.

This is a good time to repeat something said in part II: There is no need to “install” the Microblaze compiler, and neither do the files need to be owned by root. A simple untar anywhere is fine. The compiler uses relative paths to find its resources.

Static linking

To be somewhat safer, static compilation may be preferred, in particular when the whole system’s functionality consists of a single application. The executable file is considerably larger, but doesn’t depend on libraries, so it works even without the sys-root copy mentioned above. Just add the –static flag to gcc. e.g.

/path/to/microblazeel-unknown-linux-gnu-gcc --static --sysroot=/path/to/nfsroot/ -o hello hello.c -lm

The -lm flag is here to demonstrate that libraries should be at the end of the command line. This has no significance when the executable is compiled to be dynamic, but for static linking, failing to put the -lm (and other loadables) at the end will cause misleading errors such as:

/tmp/cckqzZqo.o: In function `main':
: undefined reference to `sin'
collect2: ld returned 1 exit statu

So put the -lm at the end, OK?

Compiling from SDK

Note: I don’t present a working solution in this subject.

Since the BSP, which is generated by the SDK along with the compilation directories, are just include files and libraries, it was appealing to try compiling Linux user space applications from the SDK. The method suggested is to overwrite the BSP created by SDK with include files and libraries for the Linux environment.

The problem I didn’t bother to solve in the end, was that the C runtime libraries used by the linker remained those for a standalone application, so the executable couldn’t run on Linux. Most likely, this can be solved easily, but since I don’t like IDEs myself, I left this issue as is.

Another problem with this method is that the BSP is erased every time the project is rebuilt (but it survives recompilation, of course). So it’s best to keep a copy of the entire BSP directory structure.

In short, this is the procedure, minus the part that makes the linker work with the Linux-related CRTs.

• Create a BSP by creating a C project
• Copy all .a files from sys-root’s  /usr/lib and replace the lib directory in the BSP with a directory containing these (only)
• Replace the BSP’s include directory with sys-root’s /usr/include as is
• In SDK (Eclipse, actually), right-click the C project on the Project Explorer and pick Properties. Under C/C++ Build > Settings > Microblaze gcc linker > Linker script,  remove the linker script given (edit the text, and remove the part saying ../src/lscript.ld)

And again, this almost works. If someone takes this to the finish line, please let me know.

• Part I: Introduction and setting up the Microblaze processor
• Part II: Compiling the kernel
• Part III: Preparing for boot and booting (this page)
• Part IV: Compiling user space applications

Generating the ACE file

The ACE file is what the System ACE chip reads from, and programs the FPGA accordingly. It consists of a sequence of JTAG operations for each necessary task: Configure the FPGA itself, load the software into memory, set the software execution entry point, and kick the software off. All is done with JTAG commands, which the System ACE generates as it scans through its ACE file.

So let’s get down to business.

Create a directory to gather the relevant files, and copy the following into it:

• The Tcl script for generating ACE file: Found at ISE_DS/EDK/data/xmd/genace.tcl (relative to the path where Xilinx ISE is installed)
• The bitstream (system.bit) file created by the EDK (explained in part I). Found in the ‘hw’ subdirectory in the export bundle from EDK to SDK. Or just under ‘implementation’ in the processor’s working directory. It’s the same file.
• The kernel ELF file (simpleImage.xilinx, or the unstripped simpleImage.xilinx.unstrip) created by the kernel build system (explained in part II), found in arch/microblaze/boot/ in the kernel source tree.

Open a command shell (Project > Launch Xilinx Shell if you like), change to this directory and go:

xmd -tcl genace.tcl -hw system.bit -elf simpleImage.xilinx -ace linuxmb.ace -board sp605 -target mdm

which generates a lot of junk files (.svf most notably, which contain JTAG commands in a portable format), and eventually the linuxmb.ace is created (any file name is OK).

In the example above, I assumed that the target is the SP605 board. Looking at the genace.tcl script reveals easily which boards are supported. If it isn’t, it’s not such a big deal. The only reason the board matters is because the System ACE needs to know which device in the JTAG chain to talk with plus some programming parameters. The -board flags to this scrips allows setting the options in a “genace option file” (whatever that means). I would hack the script, though. It looks easier. See here for more information.

Writing to the Compact Flash

First and foremost: If you have a compact flash which boots anything to the FPGA, don’t format it unless you really have to. The System ACE chip (by Xilinx) which reads from the flash directly is a bit picky about the file system format. Preferably use the card which came with the development kit.

And this too: If you just bought a 2 GB flash or so in a general electronics store, odds are that you’ll need to format it.

I explain how to format the flash in another post of mine.

Assuming that the flash is formatted OK, copy the ACE file to the Compact Flash’ root directory. Make sure that

• there is no other *.ace file in the root directory
• there is no xilinx.sys in the root directory

It is perfectly OK to have unrelated directories on the flash, so if there are some files on the flash already, I’d suggest creating a directory with just any name (say, “prevroot”) and move everything in the root directory into that one. And then copy the desired ACE file (linuxmb.ace in the example above) into the root directory.

That’s it. The Linux kernel should now boot, but it will complain (the kernel will panic, actually) that it doesn’t have any root filesystem. So…

Setting up the root filesystem

Once the kernel is up, it needs something to mount as a root filesystem, in which it expects to find its init executable and quite a few other files. Xilinx supplies an image of this bundle which were downloaded along with the cross compilers (see part II), in the same directory.

You may recall that I chose to mount root over the network, using NFS. So to create a useful root directory to work with, just change directory to whatever is going to be root (in my case, the one exposed via NFS) and go

zcat /path/to/microblaze_v1.0_le/initramfs_minimal_le.cpio.gz | cpio -i -d -H newc --no-absolute-filenames

This bundle includes a practical set of executables (well, it’s actually a lot of symbolic links to busybox) including vi, watch, dd, grep, gzip, tar, rpm, nc and even httpd (a web server…!). There’s also a rootfs.cpio.gz in the kernel sources when downloaded from Xilinx’ git (linux-2.6-xlnx.git in part II) which I haven’t tried out. But it’s opened in the same way.

You may, of course, compile your own programs, which is discussed in part IV.

There’s no “shutdown” executable, though. There’s “halt” instead.

A test run

Well, plug in the Compact Flash card, turn the power on, and hope to see a green LED blinking, which turns to steady green after a few seconds. When the LED is steady, expect some output on the UART. A typical log for SP605 is given at the end of this post.

At times, the SP605 board’s green LED went on, but nothing runs until SYS_ACE_RESET is pressed (the middle button out of three close to the Compact Flash jack). Looks like a powerup issue.

Is it fast? Is it fast?

This is maybe not such a fair comparison, and still the facts speak for themselves:

On Microblaze @ 75 MHz clock (37 BogoMIPS):

# dd if=/dev/zero of=/dev/null bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (10.0GB) copied, 1058.304486 seconds, 9.7MB/s
# dd if=/dev/zero of=/dev/null bs=512 count=100k
102400+0 records in
102400+0 records out
52428800 bytes (50.0MB) copied, 9.531130 seconds, 5.2MB/s

The same thing on my own computer  @ 1.2 GHz (5600 BogoMIPS):

$ dd if=/dev/zero of=/dev/null bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 0.941238 s, 11.4 GB/s
$ dd if=/dev/zero of=/dev/null bs=512 count=100k
102400+0 records in
102400+0 records out
52428800 bytes (52 MB) copied, 0.0443318 s, 1.2 GB/s

According to the BogoMIPSes, Microblaze should have been 150 times slower, not 1000 times slower!

A typical boot log

early_printk_console is enabled at 0x40600000
Ramdisk addr 0x00000003, Compiled-in FDT at 0xc03c2348
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.38.6 (eli@localhost.localdomain) (gcc version 4.1.2) #19 Fri Aug 5 16:40:02 IDT 2011
setup_cpuinfo: initialising
setup_cpuinfo: Using full CPU PVR support
cache: wt_msr_noirq
setup_memory: max_mapnr: 0x8000
setup_memory: min_low_pfn: 0xc0000
setup_memory: max_low_pfn: 0xc8000
On node 0 totalpages: 32768
free_area_init_node: node 0, pgdat c04f515c, node_mem_map c05ca000
 Normal zone: 256 pages used for memmap
 Normal zone: 0 pages reserved
 Normal zone: 32512 pages, LIFO batch:7
pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
pcpu-alloc: [0] 0
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
Kernel command line: console=ttyUL0 ip=::::::dhcp rootfstype=nfs root=/dev/nfs rw nfsroot=10.11.12.13:/shared/nfsroot,tcp
PID hash table entries: 512 (order: -1, 2048 bytes)
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
allocated 655360 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Memory: 123204k/131072k available
SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:32
xlnx,xps-intc-1.00.a #0 at 0xc8000000, num_irq=8, edge=0x60
xlnx,xps-timer-1.00.a #0 at 0xc8004000, irq=7
Heartbeat GPIO at 0xc8008000
microblaze_timer_set_mode: shutdown
microblaze_timer_set_mode: periodic
Console: colour dummy device 80x25
Calibrating delay loop... 37.17 BogoMIPS (lpj=185856)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
Initializing cgroup subsys ns
ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
devtmpfs: initialized
NET: Registered protocol family 16
PCI: Probing PCI hardware
bio: create slab <bio-0> at 0
XGpio: /axi@0/gpio@40040000: registered
XGpio: /axi@0/gpio@40020000: registered
XGpio: /axi@0/gpio@40000000: registered
vgaarb: loaded
Switching to clocksource microblaze_clocksource
microblaze_timer_set_mode: oneshot
Switched to NOHz mode on CPU #0
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
PCI: CLS 0 bytes, default 32
Skipping unavailable RESET gpio -2 (reset)
GPIO pin is already allocated
audit: initializing netlink socket (disabled)
type=2000 audit(0.429:1): initialized
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
fuse init (API version 7.16)
msgmni has been set to 240
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
40600000.serial: ttyUL0 at MMIO 0x40600000 (irq = 6) is a uartlite
console [ttyUL0] enabled
brd: module loaded
loop: module loaded
of:xsysace 41800000.sysace: Xilinx SystemACE revision 1.0.12
of:xsysace 41800000.sysace: capacity: 3980592 sectors
 xsa: xsa1
Xilinx SystemACE device driver, major=254
Generic platform RAM MTD, (c) 2004 Simtec Electronics
xilinx_spi 40a00000.spi: at 0x40A00000 mapped to 0xc8080000, irq=0
of:xilinx_emaclite 40e00000.ethernet: Device Tree Probing
Xilinx Emaclite MDIO: probed
of:xilinx_emaclite 40e00000.ethernet: MAC address is now 00:0a:35:49:b2:00
of:xilinx_emaclite 40e00000.ethernet: Xilinx EmacLite at 0x40E00000 mapped to 0xC80A0000, irq=5
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.19.1-ioctl (2011-01-07) initialised: dm-devel@redhat.com
nf_conntrack version 0.5.0 (1925 buckets, 7700 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
Registering the dns_resolver key type
registered taskstats version 1
Sending DHCP requests .
PHY: c0020918:07 - Link is Up - 100/Full
., OK
IP-Config: Got DHCP answer from 10.11.12.13, my address is 10.11.12.155
IP-Config: Complete:
 device=eth0, addr=10.11.12.155, mask=255.255.255.0, gw=10.11.12.13,
 host=10.11.12.155, domain=, nis-domain=(none),
 bootserver=10.11.12.13, rootserver=10.11.12.13VFS: Mounted root (nfs filesystem) on device 0:13.
devtmpfs: mounted
Freeing unused kernel memory: 147k freed
Starting rcS...
++ Mounting filesystem
++ Starting telnet daemon
rcS Complete
/bin/sh: can't access tty; job control turned off
/ # NET: Registered protocol family 10
eth0: no IPv6 routers present

• Part I: Introduction and setting up the Microblaze processor
• Part II: Compiling the kernel (this page)
• Part III: Preparing for boot and booting
• Part IV: Compiling user space applications

Kernel compilation in general

Compiling a Linux kernel traditionally consists of the following steps (some of which are elaborated further below):

• Obtaining a kernel source tree.
• Configure the kernel. Which all in all means to set up a file named “.config” in the kernel source’s root directory.
• Compile actual kernel, ending up with an executable image.
• Compile the post-boot loadable kernel modules.
• Put everything in its place, set up the bootloader
• Pray and boot

When compiling for Microblaze, the process is somewhat different:

• Cross compilation: The compiled binaries run on a processor different from the one doing the compilation.
• Kernel modules are most likely not used at all. They are a bit pointless when the hardware is known in advance, and also add some complexity in setting up the entire system for boot. Besides, modprobe on a Microblaze can take forever.
• The hardware configuration is custom made, and the kernel needs to be informed about it (through the Device Tree Structure)

Downloading kernel sources

Note that all kernels compile for all target architectures. If you download a kernel from Xilinx’ repository, it may have the parts relevant to Xilinx slightly more updated. The emphasis is on “may”.
The “vanilla” kernel (maintained by Linus Torvalds) can be downloaded from the main kernel archive or one of its mirrors. Several other flavors float around, including Xilinx own git

git clone git://git.xilinx.com/linux-2.6-xlnx.git

or Petalogix’ git (after all, they do a lot of maintenance on the Xilinx devices):

git clone git://developer.petalogix.com/linux-2.6-microblaze.git

The question is always which kernel is best. The answer is that it’s a bit of a gamble. It’s usually almost exactly the same piece of software, with git version having the latest changes. That means the latest bug fixes, new drivers, but also the latest, undocumented and undiscovered bugs. Vanilla kernels tend to be more conservative, but the only rule is that there are no rules. So in short, toss a coin and pick one.

Personally, I compiled the kernel which happened to be on my hard disk for other purposes.

Cross compilers

The good news is that there’s no need to compile the GNU tools. As a matter of fact, this part turned out to be surprisingly painless. The cross compiler and binutils binaries + initramfs images can be downloaded with

$ git clone git://git.xilinx.com/xldk/microblaze_v1.0_le.git
$ git clone git://git.xilinx.com/xldk/microblaze_v1.0.git

Choose one, depending on whether you prefer little endian or big endian for your processor. I picked little endian, but there’s one initramfs in the big endian bundle which isn’t there for the little endian set (which only has the “minimal” image).

One of the files fetched by git is microblazeel-unknown-linux-gnu.tar.gz (gzipped tarball) for the little endian version and mb_gnu_tools_bin.tar.bz (bzipped tarball) for big endian. I’ll leave the latter, because I didn’t use it.

There’s no need to install anything, and no need to be root (actually, doing this as root is pretty unwise). Just untar the tarball of your choice in any directory. Tar generates several subdirectories, but we’re after the cross compilers. Or more precisely, to make the kernel build system use them. This boils down to this:

export CROSS_COMPILE=/home/myhomedir/untarred-to/microblazeel-unknown-linux-gnu/bin/microblazeel-unknown-linux-gnu-

First of all, note the dash at the end of the statement. The whole string is a prefix for all compilation commands made by the kernel build system. It is often recommended to set the path to where the compilers are, and then set CROSS_COMPILE to a shorter prefix. I don’t see the point in polluting the overall path. The build environment has no problem with the statement above.

It has also crossed my mind to use the mb-gcc and friends, which are part of the SDK. But that may require another paid-for license, in case different people do the FPGA and software (which usually is the case).

And to wrap this up: If I’ll ever need to build a cross compiler from scratch, I would start with looking at Buildroot (and another page about it) or following this guide (I haven’t tried either, though).

Kernel configuration

Setting this up correctly is a tedious process, and even the most seasoned kernel hackers may not get it right on the first go. If it’s your first time, prepare to spend quite a few hours on this. The less experienced you are with Linux in general, the more time will you need to spend to make an educated guess about your need for each feature offered.

You can try to use my configuration file, or at least start off with it. It was made for against a 2.6.38 kernel, and booted well as shown in part III. Copy the file as .config on the kernel source’s root, and start with oldconfig.

The commands involved are basically (all “make” commands issues at the kernel source’s top directory):

• Clean up everything, including the .config file if present. This is not necessary if you just uncompressed your kernel. It’s actually rarely necessary at all: “make ARCH=microblaze mrproper”. This will delete .config! (I know I just said it).
• Adopt an existing .config file: “make ARCH=microblaze oldconfig”. This is useful in particular when switching to another kernel version or flavor. Only questions about new features are asked. If you downloaded my configuration file, I would suggest not to turn on options that are offered while running oldconfig, unless they are clearly Xilinx related.
• Configure the kernel: “make ARCH=microblaze xconfig”, “make ARCH=microblaze gconfig” or “make ARCH=microblaze menuconfig” (pick one). These applications present the kernel options in a fairly user-friendly manner, and eventually save the result to .config. I recommend xconfig, because it’s graphic and has a search feature, which turns out very useful.

When targeting an embedded platform, the strategy is to enable whatever is necessary in the kernel itself, and not count on kernel modules. A second issue is to eliminate anything unnecessary from the kernel. This is not just a matter of the kernel image’s size and speed, but enabling components which have nothing to do there can cause the kernel compilation to fail, and even worse, the kernel to crash at boot. Each architecture maintains a set of #include headers, and some kernel components may assume certain things that these architecture-dependent parts haven’t caught up with. So the rule that is whatever hasn’t been tested, won’t necessarily work. Enabling an esoteric keyboard driver on a Microblaze processor may very well fail the boot, simply because nobody cares.

In particular, you most likely want to follow these:

• Under Platform Options, set CONFIG_KERNEL_BASE_ADDR to where your DDR RAM begins (0xC0000000 on my processor), the targeted FPGA family as well as the other parameters (USE_*). The USE_* parameters’ correct values can be found in the .dts file. Just copy the values of the processor elements with the same names.
• Also set
  CONFIG_SERIAL_UARTLITE=y
  CONFIG_SERIAL_UARTLITE_CONSOLE=y
  CONFIG_SERIAL_CORE=y
  CONFIG_SERIAL_CORE_CONSOLE=y
• Since we’re not going to use any boot loader, the kernel command line needs to be compiled into the kernel itself: Enable CMDLINE_BOOL (default bootloader kernel argument) and set it to something useful. As for the console, set it to console=ttyUL0, or nothing goes to console after the two first lines sent to console from early_printk_console (CONFIG_CMDLINE_FORCE may be necessary as well. It doesn’t hurt in the absence of a boot loader anyhow)
• Enable CONFIG_MSDOS_FS and CONFIG_VFAT_FS in kernel (not module), so that the SystemACE can be read.
• Enable CONFIG_XILINX_SYSACE
• Enable CONFIG_XILINX_EMACLITE and CONFIG_FB_XILINX
• Disable the FTRACE config option (under kernel hacking, compilation fails) instead of using patch.

And for your own sake, make a copy of the .config file every now and then as you work on it. It’s very easy to delete it by mistake or to mess it up in general.

Setting the Linux boot parameters correctly is very important, because if they’re wrong, kernel recompilation is they only way to fix it in the absence of a boot loader. I’ve chosen to mount the root directory from the network, but note that /dev/sxa is the Compact flash itself (with /dev/sxa1 is the first partition, for example). So it’s fairly simple to add a partition to the flash device, and put a regular root filesystem there. Maybe I’ll do that myself and update this post.

Anyhow, my choice for the Linux boot parameters was

console=ttyUL0 ip=::::::dhcp rootfstype=nfs root=/dev/nfs rw nfsroot=10.11.12.13:/shared/nfsroot,tcp

where “/shared/nfsroot” is the shared NFS directory on the server with IP 10.11.12.13. This command is suitable for getting the root from the network, which is very convenient for development. This setting requires a DHCP server on the LAN. In case you don’t want to configure a DHCP server, use the ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:::off format instead. Documentation/filesystems/nfs/nfsroot.txt in the kernel sources has more about booting from NFS. I’ve also written a post about booting a diskless PC from network, but it’s a bit of an overkill.

In case you’re interested in how the whole configuration thing comes together, let’s take CONFIG_EARLY_PRINTK for example. In arch/microblaze/kernel/Makefile, one of the lines says:

obj-$(CONFIG_EARLY_PRINTK)    += early_printk.o

On the other hand, in the config file it can say

CONFIG_EARLY_PRINTK=y

So when the Makefile is executed, the target early_prink.o is added to either obj-y, obj-m or obj-n. obj-y is the list of objects to be inserted into the kernel, obj-m is the list of modules, and obj- is the junk list. The configuration rules are given in the Kbuild files, next to the Makefiles.

A small Makefile fix

As of 2.6.38, there is a small error in the arch/microblaze/boot/Makefile, which makes the build system always attempt making an U-Boot image, which is not necessary in our case. This may result in an error message (saying “mkimage” wasn’t found), when everything is actually OK. So in the part saying

$(obj)/simpleImage.%: vmlinux FORCE
 $(call if_changed,cp,.unstrip)
 $(call if_changed,objcopy)
 $(call if_changed,uimage)
 $(call if_changed,strip)
 @echo 'Kernel: $@ is ready' ' (#'`cat .version`')'

remove or comment out the line saying “$(call if_changed,uimage)”.

Compiling the kernel

Before starting: You didn’t forget to set CROSS_COMPILE and copy the updated xilinx.dts file to its place… right?

I prefer cleaning up before compiling:

make ARCH=microblaze clean
rm arch/microblaze/boot/simpleImage.*

This is a good time to ask why the image file isn’t cleaned by “make clean”. To be fixed, I suppose.

And then, the compilation is just

make -j 8 ARCH=microblaze simpleImage.xilinx

Note that the “.xilinx” suffix corresponds to the xilinx.dts file in the arch/microblaze/boot/dts/ directory. If another .dts file should be made effective, change the suffix.

The “-j 8″ means that 8 compilation processes run in parallel, which is suitable for a quad processor with hyperthreading. Skip this option or use another number, depending on your computer, your spare time and your need to see the logic of the events.

The basic UNIX rule is that everything went fine unless an error message appeared. A more explicit confirmation is that it said

OBJCOPY arch/microblaze/boot/simpleImage.xilinx

somewhere close to the end, and that the arch/microblaze/boot/simpleImage.xilinx is indeed there, and has a date stamp that makes sense.

If and when you get errors, well, there’s no simple  recipe to solve that. The easiest way is to eliminate the need to compile that certain file by changing the kernel configuration, if the functionality is indeed unnecessary. Otherwise your best friends are Google and your brain, not necessarily in that order.

As for the Device Tree, it was compiled into a .dtb file (the Device Tree binary blob), which can be found in the same directory as the just generated kernel image. The Device Tree Compiler (dtc) comes with the kernel sources, and can be found in scripts/dtc.

And just to wrap this up: If you insist on seeing all the commands issued instead of the otherwise laconic output, there the KBUILD_VERBOSE flag. For example,

make ARCH=microblaze KBUILD_VERBOSE=1 clean

With a compiled kernel image at hand (which already has the Device Tree built-in), all that’s left is to set up the Compact Flash and boot. Go to part III of this HOWTO.

A few other make statements

For completeness:

• Clean up any compiled binaries: Recommended after a change in .config: “make ARCH=microblaze clean”
• Generate loadable modules: “make ARCH=microblaze modules”. Not necessary if everything needed is compiled into the kernel.
• And then gather the modules in a neat directory (making sure you don’t have a /lib/modules directory with the same version number): “make ARCH=microblaze modules_install”. This will write to /lib/modules on the local machine, so if you happen to compile exactly the same kernel version for your own PC and the embedded target, the kernel modules the PC relies on will be overwritten.

• Part I: Introduction and setting up the Microblaze processor (this page)
• Part II: Compiling the kernel
• Part III: Preparing for boot and booting
• Part IV: Compiling user space applications

Introduction

This HOWTO goes through the procedures for getting a simple Linux system running on a Xilinx Microblaze processor. The examples are given for an SP605 evaluation board, but almost everything here applies for other FPGAs and boards as well. The Xilinx software version used here is 13.2.

There are quite a few variants on how to get the bitstream and Linux kernel into their right places in the FPGA. The approach taken here is to boot up from the Compact Flash alone by writing a file to it. No bootloader is used in this howto; the SystemACE chip is responsible for loading both the FPGA bitstream and Linux kernel image, and it will do so reading one single (.ace) file. The main advantage of this approach is that there’s no need to set up a boot loader, which is yet another piece of software that can go wrong. The main disadvantage is that a bootloader allows some tweaking of the kernel configuration at boot time, which has to be done by recompiling the kernel otherwise.

The root filesystem is mounted from network (NFS) in this HOWTO.

I’m assuming the following prerequisites:

• You have the Xilinx tools set up properly, and have managed to compile and run a simple standalone “Hello, World” application with the EDK/SDK (having loaded the code to the FPGA in any way, we’ll get to that)
• You’ve actually seen the RS-232 console data on a terminal, and feel confident about it (otherwise you may work hard to figure out why everything is stuck, when it’s actually your terminal window’s problem).
• You’re running on one of the evaluation boards, or know how to set up the processor to work with your own (and have that tested already)
• Your board has a systemACE flash chip (recent evaluation boards usually do)
• You have access to a machine running Linux on a computer. Compiling the kernel will require this. The Xilinx tools can be run on whatever’s your cup of tea.
• You have the ability to read and write files to a Compact Flash. This is most easily done with a simple adapter to a PC computer, which should be available in camera or computer accessories shops. Chances are you have one without necessarily being aware of it.

An outline of the steps

So this is what we’ll do:

• Set up a Microblaze processor in the Xilinx EDK so it can run Linux.
• Generate the processor, so an FPGA bitstream is at hand.
• Export the processor to the Xilinx SDK and compile a dummy C application, so that  necessary metadata files are generated
• Generate a Device Tree file (.dts) based upon files created by EDK/SDK, and copy it into the Linux kernel sources, so Linux is in sync with the EDK regarding what it’s running on.
• Configure the kernel and compile it.
• Create a .ace file from the FPGA bitstream and kernel image just compiled.
• Set up the Compact Flash card.
• Boot and hope for good

And of course, certain software tools will need to be downloaded for this. We’ll come to this.

Setting up the processor

If you’re really lazy about this, you can use the minimal processor I’ve generated for the SP605 board. Unzip, double-click system.xmp, and skip to after the bullets below. It will work on that board only, of course.

Otherwise: Start Platform Studio (EDK) and create a new platform, based upon the Wizard’s defaults.

Following a Microblaze on Linux guide, in particular the part regarding minimal hardware requirements, there a need to make sure that the hardware has an MMU with two regions, a timer, an interrupt controller and a UART with an interrupt line. In the platform studio it goes like this:

Starting off with the Wizard’s defaults,

• Double click “microblaze_0″ on the Ports tab, and set the Linux with MMU preset on the Configuration wizard showing up. This will take care of most settings.
• Still in the ports view, add an AXI Interrupt Controller (under Clock, Reset and Interrupt in the IP Catalog). Accept default settings. Make a new connection for its irq output, and connect it to the microblaze_0′s interrupt input pin.
• Pick the RS232_Uart_1 and make a new connection for the interrupt line. Connect that signal to the interrupt controller.
• Add an AXI Timer/Counter, and accept defaults. Make a new connection for the interrupt, and connect it to the interrupt controller.
• Connect the interrupts of the Ethernet lite, SPI Flash, IIC SFP, IIC EEPROM, IIC_DVI, and SysACE cores to the interrupt controller as well.

Then generate bitstream, export to SDK, and run the SDK, adopting this hardware platform. The goal of this is to generate a .mss file, which will be needed later. For this to happen, make a new C project (“Hello World” will do just fine) and compile it.

There is no need to “update the bitstream” like with standalone applications: The Linux kernel can take care of itself, without having its entry address hardwired in the FPGA’s block RAM. We’ll use the system.bit, and not the download.bit (even though the latter works too).

Creating a Device Tree file

The purpose of this stage is to generate a .dts file, which is the format expected by the kernel build environment. It informs the kernel about the structure of the processor and its peripherals. The device tree structure is discusses further here.

If you chose to download and use my processor with no changes whatsoever, you can also get my DTS file. Just copy it to arch/microblaze/boot/dts/ in the to-be compiled kernel source tree.

To make your own .dts file, first create a special directory, and make it the working directory of your shell.

The device tree file is generated automatically with the libgen utility with the help of a Tcl script. As of ISE 13.2, this script needs to be loaded separately with git:

bash> git clone git://git.xilinx.com/device-tree.git

This generates a device-tree directory. Another web page explains how to make SDK recognize the script, but I prefer command line for things like this. Another post of mine explains the device tree further.

Copy the system.xml file from the directory to which you exported to SDK (in the “hw” subdirectory), into the current one. Then copy system.mss from the project’s BSP directory. It will have a name like hello_world_bsp_0.

Edit the copy you made of system.mss, so that the BEGIN OS to END part reads

BEGIN OS
 PARAMETER OS_NAME = device-tree
 PARAMETER OS_VER = 0.00.x
 PARAMETER PROC_INSTANCE = microblaze_0
END

and not “standalone” for OS.

And then run libgen as follows (make sure it’s in the PATH. The easiest way is to launch a “Xilinx shell” from the EDK’s project menu):

libgen -hw system.xml -lp device-tree -pe microblaze_0 -log libgen.log system.mss

Which generates a xilinx.dts in microblaze_0/libsrc/device-tree_v0_00_x. Copy this file to arch/microblaze/boot/dts/ in the to-be compiled kernel source tree. If you can’t find the file there, and libgen didn’t complain about some error, you may have forgotten to edit system.mss as mentioned just above.

Now let’s go on to compiling the kernel, in part II.

Spoiler

It’s very likely that you don’t need to read this. If all you want is to get a Linux kernel to detect a Microblaze processor on an Xilinx FPGA, the relevant information is in another post of mine. This post goes into the details which are necessary to understand, if you want to write a kernel driver for a device tree mapped peripheral.

Why a device tree is necessary

The main issue with running Linux on an FPGA is that the Linux kernel needs to know what peripherals it has and where it can find them. On PC computers this problem was solved many years ago with the PCI bus: The BIOS detects the peripherals, allocates their addresses and interrupts and tells the operating system what it has and where it can be found. In the embedded world, this information was hardcoded into pieces of the kernel sources, which were written specifically for every board. With many boards out there, the kernel source grew way too fast. This far-from-optimal solution is not feasible with a soft processor, whose peripherals are configured per case. Hacking the kernel sources to match the FPGA is a recipe for bugs, crashes and being stuck with a certain kernel forever.

The elegant solution for this is the Flattened Device Tree. The idea is to create some binary data structure, which is either linked into the kernel image or given to it during boot. This binary blob contains the information about the processor itself and its peripherals, including the addresses, interrupts and several application-specific parameters. So the drivers for these peripherals are written very similar to PCI drivers: They declare what peripherals they support, and obtain their resources from a standard kernel API.

The code for Flattened Device Tree and Open Firmware resides in drivers/of in the kernel tree. The relevant include file is include/linux/of.h.

Generation

Note that at least for Xilinx FPGAs, there is no need to generate the device tree manually. Rather, get a copy of the device tree generator with

bash> git clone git://git.xilinx.com/device-tree.git

which basically consists of a TCL script run by libgen and a configuration file. The device tree generator’s page explains how to make SDK recognize the script, but there’s no reason to play around with SDK for that.

Instead, go

libgen -hw /path/to/system.xml -lp /path/to/device-tree -pe microblaze_0 -log libgen.log system.mss

Which generates a system.dts in microblaze_0/libsrc/device-tree_v0_00_x

The system.mss file is generated as a byproduct when compiling just any a project within SDK, and is found under the directory with the _bsp_n suffix. I still need to find out how to create the file from the command line.

It needs to be modified, so that the BEGIN OS to END part reads

BEGIN OS
 PARAMETER OS_NAME = device-tree
 PARAMETER OS_VER = 0.00.x
 PARAMETER PROC_INSTANCE = microblaze_0
END

and not “standalone” for OS.

To get the system.xml file (which was necessary to create the system.mss), go Project > Export Hardware to SDK in the EDK platform studio. Or

make -f system.make exporttosdk

from the project’s home directory.

The correct setup of the device tree entry can be found in the Documentation/devicetree/bindings directory of the kernel sources. The xilinx.txt file describes the bindings for Xilinx peripherals, and explains how information in the system.mhs file is translated into a xilinx.dts.

As part of a full kernel compilation, the .dts is compiled into a .dtb file (the Device Tree binary blob), which can be found in the same directory as the generated kernel image. The Device Tree Compiler (dtc) comes with the kernel sources, and can be found in scripts/dtc.

A sample entry

The following example is given there for a Uartlite (which we’ll follow on below):

opb_uartlite_0: serial@ec100000 {
 device_type = "serial";
 compatible = "xlnx,opb-uartlite-1.00.b";
 reg = <ec100000 10000>;
 interrupt-parent = <&opb_intc_0>;
 interrupts = <1 0>; // got this from the opb_intc parameters
 current-speed = <d#115200>;     // standard serial device prop
 clock-frequency = <d#50000000>; // standard serial device prop
 xlnx,data-bits = <8>;
 xlnx,odd-parity = <0>;
 xlnx,use-parity = <0>;
 };

It’s recommended to have a look at arch/microblaze/platform/generic/system.dts in the kernel sources for a fullblown file. Or one you’ve generated yourself, for that matter.

Declarations in a kernel module driver

Device tree mapped instances are treated by the kernel very much like PCI devices, only the source of information is the DTB (Device Tree Binary) rather than from the BIOS.

The parallel to PCI’s Vendor/Product IDs is an entry looking like this (taken from uartlite.c):

static struct of_device_id ulite_of_match[] __devinitdata = {
 { .compatible = "xlnx,opb-uartlite-1.00.b", },
 { .compatible = "xlnx,xps-uartlite-1.00.a", },
 {}
};

MODULE_DEVICE_TABLE(of, ulite_of_match)

Which is then bound to a driver with

static struct of_platform_driver ulite_of_driver = {
 .probe = ulite_of_probe,
 .remove = __devexit_p(ulite_of_remove),
 .driver = {
 .name = "uartlite",
 .owner = THIS_MODULE,
 .of_match_table = ulite_of_match,
 },
}

and then, finally, exposed to the kernel with

static inline int __init ulite_of_register(void)
{
 pr_debug("uartlite: calling of_register_platform_driver()\n");
 return of_register_platform_driver(&ulite_of_driver);
}

somewhere at the end of the driver’s code. This format is very similar to the declaration of PCI devices, so if this is unclear, I’d suggest learning how to do it the PCI way, which is by far more documented.

And by the way, when the kernel is configured to support it, the device tree can be viewed in human-readable format in /proc/device-tree.

The of_device_id structure

The structure is defined in include/linux/mod_devicetable.h as

struct of_device_id
{
 char    name[32];
 char    type[32];
 char    compatible[128];
#ifdef __KERNEL__
 void    *data;
#else
 kernel_ulong_t data;
#endif
};

Surprisingly enough, the lengths of the entries are fixed and limited.The three strings, name, type and compatible are compared as strings (with strcmp(), see of/base.c) with the device tree’s node’s data. Everything declared (that is, non-NULL) in the structure must be equal with the node’s info for a match. In other words, NULLs are wildcards.

In the declaration example above, only the “compatible” part was declared, so any device matching the string exactly triggers off a probe on the driver.

Introduction

The Xilinx Platform Studio (EDK) has this “update bitstream” function, which I wasn’t so clear about, despite its documentation page. Its icon says “BRAM INIT” which turns out to be more accurate than expected. So what happens during this process? When is it necessary?

If you’re into running a Linux kernel, you’re most likely wasting your time reading this, because the Linux kernel is kicked off directly from the external RAM, and hence this mangling isn’t necessary. To set up a Linux bitstream, see another post of mine.

Having that said, let’s look at the problem this functions solves: A Microblaze processor starts executing at address 0 unless told otherwise. Its interrupt vectors are at near-zero addresses as well. These addresses are mapped to an FPGA block RAM.

What this block RAM should contain is a jump to the application’s entry point. On a SP605 board, this is most likely the beginning of the DDR memory, Oxc0000000. So when the processor kicks off, this block RAM’s address zero should contain:

00000000 <_start>:
 0:    b000c000     imm    -16384
 4:    b8080000     brai    0

Which is Microblazish for “Jump to Oxc0000000″ (note the lower 16 bits of both commands).

When a system is booted, there are two phases: First, the FPGA is loaded with its bitstream, and then the external memory, containing the bulk of execution code. And then the processor is unleashed.

So the block memory’s correct content needs to be included in the bitstream itself. But when the processor is implemented from its logic elements, it isn’t clear what should be written there. It’s only when the software is linked, that the addresses of the different segments are known.

But software compilation and linking requires the knowledge of the processor’s memory map, which is generated while the processor is implemented. So there’s a chicken-and-egg situation here.

The egg was first

The solution is that block RAM’s content is fixed after the software is compiled and linked. The reset and interrupt vectors are included in the ELF file generated by the software linker, and are mapped to the block RAM’s addresses. The “update bitstream” process reads the ELF file, finds the relevant region, and updates the bitstream file, producing the download.bit file. That’s why choosing the ELF file is necessary for this process.

Necessity

The original problem was that the execution starts from address zero. But if the ELF file points at the real starting point, and this is properly communicated to the processor at startup, there’s no need to set up the block RAM at all. Well, assuming that the executable takes care of interrupts and exception vectors soon enough. This is the case with Linux kernel images, for example, for which there is no need to update the bitstream.

Some gory details

The “update bitstream” process launches a command like

bitinit -p xc6slx45tfgg484-3 system.mhs -pe microblaze_0 sdk/peripheral_tests_0/Debug/peripheral_tests_0.elf \
 -bt implementation/system.bit -o implementation/download.bit

which takes place in two phases. In the first phase, the system.mhs file is read and parsed, so that the memory map is known and the block RAM is identified. This program then runs something like

data2mem -bm "implementation/system_bd" -p xc6slx45tfgg484-3 -bt "implementation/system.bit" -bd "sdk/peripheral_tests_0/Debug/peripheral_tests_0.elf" tag microblaze_0 -o b implementation/download.bit

Which is the action itself. Data2mem is a utility for mangling bitstreams so that their block RAMs contain desired data. The -bm flag tells data2mem to get the block RAM map from implementation/system_bd.bmm, which can be

// BMM LOC annotation file.
//
// Release 13.2 - Data2MEM O.61xd, build 2.2 May 20, 2011
// Copyright (c) 1995-2011 Xilinx, Inc.  All rights reserved.

///////////////////////////////////////////////////////////////////////////////
//
// Processor 'microblaze_0', ID 100, memory map.
//
///////////////////////////////////////////////////////////////////////////////

ADDRESS_MAP microblaze_0 MICROBLAZE-LE 100

 ///////////////////////////////////////////////////////////////////////////////
 //
 // Processor 'microblaze_0' address space 'microblaze_0_bram_block_combined' 0x00000000:0x00001FFF (8 KBytes).
 //
 ///////////////////////////////////////////////////////////////////////////////

 ADDRESS_SPACE microblaze_0_bram_block_combined RAMB16 [0x00000000:0x00001FFF]
 BUS_BLOCK
 microblaze_0_bram_block/microblaze_0_bram_block/ramb16bwer_0 [31:24] INPUT = microblaze_0_bram_block_combined_0.mem PLACED = X3Y30;
 microblaze_0_bram_block/microblaze_0_bram_block/ramb16bwer_1 [23:16] INPUT = microblaze_0_bram_block_combined_1.mem PLACED = X2Y30;
 microblaze_0_bram_block/microblaze_0_bram_block/ramb16bwer_2 [15:8] INPUT = microblaze_0_bram_block_combined_2.mem PLACED = X2Y32;
 microblaze_0_bram_block/microblaze_0_bram_block/ramb16bwer_3 [7:0] INPUT = microblaze_0_bram_block_combined_3.mem PLACED = X2Y36;
 END_BUS_BLOCK;
 END_ADDRESS_SPACE;

END_ADDRESS_MAP;

So this file defines the addresses covered as well as the physical positions of these block RAMs in the logic fabric.

The -bd flag points at the ELF file to get the data from, with the “tag microblaze_0″ part saying that only the memories tagged microblaze_0 in the .bmm file should be handled, and the rest ignored.

ELF headers

Looking into the ELF file, we have something like this:

> mb-objdump --headers sdk/peripheral_tests_1/Debug/peripheral_tests_1.elf

sdk/peripheral_tests_1/Debug/peripheral_tests_1.elf:     file format elf32-microblazele

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
 0 .vectors.reset 00000008  00000000  00000000  000000b4  2**2
 CONTENTS, ALLOC, LOAD, READONLY, CODE
 1 .vectors.sw_exception 00000008  00000008  00000008  000000bc  2**2
 CONTENTS, ALLOC, LOAD, READONLY, CODE
 2 .vectors.interrupt 00000008  00000010  00000010  000000c4  2**2
 CONTENTS, ALLOC, LOAD, READONLY, CODE
 3 .vectors.hw_exception 00000008  00000020  00000020  000000cc  2**2
 CONTENTS, ALLOC, LOAD, READONLY, CODE
 4 .text         0000653c  c0000000  c0000000  000000d4  2**2
 CONTENTS, ALLOC, LOAD, CODE
 5 .init         0000003c  c000653c  c000653c  00006610  2**2
 CONTENTS, ALLOC, LOAD, READONLY, CODE
 6 .fini         0000001c  c0006578  c0006578  0000664c  2**2
 CONTENTS, ALLOC, LOAD, READONLY, CODE
 7 .ctors        00000008  c0006594  c0006594  00006668  2**2
 CONTENTS, ALLOC, LOAD, DATA
 8 .dtors        00000008  c000659c  c000659c  00006670  2**2
 CONTENTS, ALLOC, LOAD, DATA
 9 .rodata       00000986  c00065a4  c00065a4  00006678  2**2
 CONTENTS, ALLOC, LOAD, READONLY, DATA
 10 .sdata2       00000006  c0006f2a  c0006f2a  00006ffe  2**0
 ALLOC
 11 .sbss2        00000000  c0006f30  c0006f30  000071d8  2**0
 CONTENTS
 12 .data         000001d0  c0006f30  c0006f30  00007000  2**2
 CONTENTS, ALLOC, LOAD, DATA
 13 .eh_frame     00000004  c0007100  c0007100  000071d0  2**2
 CONTENTS, ALLOC, LOAD, DATA
 14 .jcr          00000004  c0007104  c0007104  000071d4  2**2
 CONTENTS, ALLOC, LOAD, DATA
 15 .sdata        00000000  c0007108  c0007108  000071d8  2**0
 CONTENTS
 16 .sbss         00000000  c0007108  c0007108  000071d8  2**0
 CONTENTS
 17 .tdata        00000000  c0007108  c0007108  000071d8  2**0
 CONTENTS
 18 .tbss         00000000  c0007108  c0007108  000071d8  2**0

 19 .bss          00000d78  c0007108  c0007108  000071d8  2**2
 ALLOC
 20 .heap         00000400  c0007e80  c0007e80  000071d8  2**0
 ALLOC
 21 .stack        00000400  c0008280  c0008280  000071d8  2**0
 ALLOC
 22 .debug_line   0000779f  00000000  00000000  000071d8  2**0
 CONTENTS, READONLY, DEBUGGING
 23 .debug_info   00008b11  00000000  00000000  0000e977  2**0
 CONTENTS, READONLY, DEBUGGING
 24 .debug_abbrev 000028e7  00000000  00000000  00017488  2**0
 CONTENTS, READONLY, DEBUGGING
 25 .debug_aranges 000006c0  00000000  00000000  00019d70  2**3
 CONTENTS, READONLY, DEBUGGING
 26 .debug_macinfo 0007f541  00000000  00000000  0001a430  2**0
 CONTENTS, READONLY, DEBUGGING
 27 .debug_frame  00000f10  00000000  00000000  00099974  2**2
 CONTENTS, READONLY, DEBUGGING
 28 .debug_loc    00003f80  00000000  00000000  0009a884  2**0
 CONTENTS, READONLY, DEBUGGING
 29 .debug_pubnames 00000fbe  00000000  00000000  0009e804  2**0
 CONTENTS, READONLY, DEBUGGING
 30 .debug_str    000018d5  00000000  00000000  0009f7c2  2**0
 CONTENTS, READONLY, DEBUGGING
 31 .debug_ranges 00000078  00000000  00000000  000a1097  2**0
 CONTENTS, READONLY, DEBUGGING

Even though this is a lot of mumbo-jumbo, there are three main parts. The reset and interrupt vectors, around address zero, the main parts of the ELF (.text, .data and such) at Oxc0000000 and on, and the debug parts which have no memory allocation at all.

The reset branch to application

This is interesting to compare with the Microblaze’s memory map. It can be deduced from the .mhs file, but hey, the log file (with .log suffix) has this segment:

Address Map for Processor microblaze_0
 (0000000000-0x00001fff) microblaze_0_d_bram_ctrl    microblaze_0_dlmb
 (0000000000-0x00001fff) microblaze_0_i_bram_ctrl    microblaze_0_ilmb
 (0x40000000-0x4000ffff) Push_Buttons_4Bits    axi4lite_0
 (0x40020000-0x4002ffff) LEDs_4Bits    axi4lite_0
 (0x40040000-0x4004ffff) DIP_Switches_4Bits    axi4lite_0
 (0x40600000-0x4060ffff) RS232_Uart_1    axi4lite_0
 (0x40800000-0x4080ffff) IIC_SFP    axi4lite_0
 (0x40820000-0x4082ffff) IIC_EEPROM    axi4lite_0
 (0x40840000-0x4084ffff) IIC_DVI    axi4lite_0
 (0x40a00000-0x40a0ffff) SPI_FLASH    axi4lite_0
 (0x40e00000-0x40e0ffff) Ethernet_Lite    axi4lite_0
 (0x41800000-0x4180ffff) SysACE_CompactFlash    axi4lite_0
 (0x74800000-0x7480ffff) debug_module    axi4lite_0
 (0xc0000000-0xc7ffffff) MCB_DDR3    axi4_0

So obviously all the main ELF parts go directly to the DDR memory (that isn’t much of a surprise), and the reset/interrupt go to the internal block ram.

A quick disassembly reveals the gory details:

> mb-objdump --disassemble sdk/peripheral_tests_1/Debug/peripheral_tests_1.elf
sdk/peripheral_tests_1/Debug/peripheral_tests_1.elf:     file format elf32-microblazele

Disassembly of section .vectors.reset:

00000000 <_start>:
 0:    b000c000     imm    -16384
 4:    b8080000     brai    0
Disassembly of section .vectors.sw_exception:

00000008 <_vector_sw_exception>:
 8:    b000c000     imm    -16384
 c:    b8081858     brai    6232
Disassembly of section .vectors.interrupt:

00000010 <_vector_interrupt>:
 10:    b000c000     imm    -16384
 14:    b80818a4     brai    6308
Disassembly of section .vectors.hw_exception:

00000020 <_vector_hw_exception>:
 20:    b000c000     imm    -16384
 24:    b8081870     brai    6256
Disassembly of section .text:

c0000000 <_start1>:
c0000000:    b000c000     imm    -16384
c0000004:    31a07108     addik    r13, r0, 28936
c0000008:    b000c000     imm    -16384
c000000c:    30406f30     addik    r2, r0, 28464
(... and it goes on and on ...)

So let’s look at the reset vector at address zero. The first IMM opcode loads C000 as the upper 16 bits for the command following, which is a branch immediate command. Together, they make a jump to Oxc000000. Likewise, the software exception jumps to Oxc0001858 and so on.

Since only the block RAM part can be included in the download.bit bitfile, only these jump vectors depend on the ELF file during the “Update bitfile” process. That’s why one gets away with not running this process, even when the ELF has been modified with a plain recompilation.

And now to the bootloop ELF

So what is the bootloop code doing? The headers are no more impressive than

> mb-objdump --headers bootloops/microblaze_0.elf

bootloops/microblaze_0.elf:     file format elf32-microblazele

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
 0 .boot         00000004  00000000  00000000  00000074  2**0
 CONTENTS, ALLOC, LOAD, READONLY, CODE
 1 .text         00000000  00000000  00000000  00000074  2**0
 CONTENTS, ALLOC, LOAD, READONLY, CODE
 2 .data         00000000  00000000  00000000  00000074  2**0
 CONTENTS, ALLOC, LOAD, DATA
 3 .bss          00000000  00000000  00000000  00000078  2**0
 ALLOC

Note the Size column: All entries are empty, except for the .boot section, which is four bytes small (one single instruction). That doesn’t leave room for sophisticated software, and the disassembly is indeed

> mb-objdump --disassemble bootloops/microblaze_0.elf

bootloops/microblaze_0.elf:     file format elf32-microblazele

Disassembly of section .boot:

00000000 <_boot>:
 0:    b8000000     bri    0        // 0

Which is simply an endless loop. So they called it bootloop for a reason.