STARSHIPRAIDER: Input buffer rev 0.1 design and characterization

Working as an embedded systems pentester is a lot of fun, but it comes with some annoying problems. There's so many tools that I can never seem to find the right one. Need to talk to a 3.3V UART? I almost invariably have an FTDI cable configured for 5 or 1.8V on my desk instead. Need to dump a 1.8V flash chip? Most of our flash dumpers won't run below 3.3. Need to sniff a high-speed bus? Most of the Saleae Logic analyzers floating around the lab are too slow to keep up with fast signals, and the nice oscilloscopes don't have a lot of channels. And everyone's favorite jack-of-all-trades tool, the Bus Pirate, is infamous for being slow.

As someone with no shortage of virtual razors, I decided that this yak needed to be shaved! The result was an ongoing project I call STARSHIPRAIDER. There will be more posts on the project in the coming months so stay tuned!

The first step was to decide on a series of requirements for the project:

• 32 bidirectional I/O ports split into four 8-pin banks.
  This is enough to sniff any commonly encountered embedded bus other than DRAM. Multiple banks are needed to support multiple voltage levels in the same target.
• Full support for 1.2 to 5V logic levels.This is supposed to be a "Swiss Army knife" embedded systems debug/testing tool. This voltage range encompasses pretty much any signalling voltage commonly encountered in embedded devices.
• Tolerance to +/- 12V DC levels.Test equipment needs to handle some level of abuse. When you're reverse engineering a board it's easy to hook up ground to the wrong signal, probe a power rail, or even do both at once. The device doesn't have to function in this state (shutting down for protection is OK) but needs to not suffer permanent damage. It's also OK if the protection doesn't handle AC sources - the odds of accidentally connecting a piece of digital test equipment to a big RF power amplifier are low enough that I'm not worried.
• 500 Mbps input/output rate for each pin.This was a somewhat arbitrary choice, but preliminary math indicated it was feasible. I wanted something significantly faster than existing tools in the class.
• Ethernet-based interface to host PC.I've become a huge fan of Ethernet and IPv6 as communications interface for my projects. It doesn't require any royalties or license fees, scales from 10 Mbps to >10 Gbps and supports bridging between different link speeds, supports multi-master topologies, and can be bridged over a WAN or VPN. USB and PCIe, the two main alternatives, can do few if any of these.
• Large data buffer.Most USB logic analyzers have very high peak capture rates, but the back-haul interface to the host PC can't keep up with extended captures at high speed. Commodity DRAM is so cheap that there's no reason to not stick a whole SODIMM of DDR3 in the instrument to provide an extremely deep capture buffer.
• Multiple virtual instruments connected to a crossbar.Any nontrivial embedded device contains multiple buses of interest to a reverse engineer. STARSHIPRAIDER needs to be able to connect to several at once (on arbitrary pins), bridge them out to separate TCP ports, and allow multiple testers to send test vectors to them independently.

The brain of the system will be fairly straightforward high-speed digital. It will be a 6-8 layer PCB with an Artix-7 FPGA in FGG484 package, a SODIMM socket for 4GB of DDR3 800, a KSZ9031 Gigabit Ethernet PHY, a TLK10232 10gbit Ethernet PHY, and a SFP+ cage, plus some sort of connector (most likely a Samtec Q-strip) for talking to the I/O subsystem on a separate board.

The challenging part of the design, from an architectural perspective, seemed to be the I/O buffer and input protection circuit, so I decided to prototype it first.

STARSHIPRAIDER v0.1 I/O buffer design

A block diagram of the initial buffer design is shown above. The output buffer will be discussed in a separate post once I've had a chance to test it; today we'll be focusing on the input stage (the top half of the diagram).

During normal operation, the protection relay is closed. The series resistor has insignificant resistance compared to the input impedance of the comparator (an ADCMP607), so it can be largely ignored. The comparator checks the input signal against a threshold (chosen appropriately for the I/O standard in use) and sends a differential signal to the host board for processing. But what if something goes wrong?

If the user accidentally connects the probe to a signal outside the acceptable voltage range, a Schottky diode connected to the +5V or ground rail will conduct and shunt the excess voltage safely into the power rails. The series resistor limits fault current to a safe level (below the diode's peak power rating). After a short time (about 150 µs with my current relay driver), the protection relay opens and breaks the circuit.

The relay is controlled by a Silego GreenPAK4 mixed-signal FPGA, running a small design written in Verilog and compiled with my open-source toolchain. The code for the GreenPAK design is on Github.

All well and good in theory... but does it work? I built a characterization board containing a single I/O buffer and loaded with test points and probe connectors. You can grab the KiCAD files for this on Github as well. Here's a quick pic after assembly:

STARSHIPRAIDER I/O characterization board

Initial test results were not encouraging. Positive overvoltage spikes were clamped to +8V and negative spikes were clamped to -1V - well outside the -0.5 to +6V absolute max range of my comparator.

Positive transient response

Negative transient response

After a bit of review of the schematics, I found two errors. The "5V" ESD diode I was using to protect the high side had a poorly controlled Zener voltage and could clamp as high as 8V or 9V. The Schottky on the low side was able to survive my fault current but the forward voltage increased massively beyond the nominal value.

I reworked the board to replace the series resistor with a larger one (39 ohms) to reduce the maximum fault current, replaced the low-side Schottky with one that could handle more current, and replaced the Zener with an identical Schottky clamping to the +5V rail.

Testing this version gave much better results. There was still a small amount of ringing (less than five nanoseconds) a few hundred mV past the limit, but the comparator's ESD diodes should be able to safely dissipate this brief pulse.

Positive transient response, after rework

Negative transient response, after rework

Now it was time to test the actual signal path. My first iteration of the test involved cobbling together a signal path from an FPGA board through the test platform and to the oscilloscope without any termination. The source of the signal was a BNC-to-minigrabber flying lead test clip! Needless to say, results were less than stellar.

PRBS31 eye at 80 Mbps through protection circuit with flying leads and no terminator

After ordering some proper RF test supplies (like an inline 50 ohm BNC terminator), I got much better signal quality. The eye was very sharp and clear at 100 Mbps. It was visibly rounded at 200 Mbps, but rendering a squarewave at that rate requires bandwith much higher than the 100 MHz of my oscilloscope so results were inconclusive.

PRBS31 eye at 100 Mbps through protection circuit with proper cabling

PRBS31 eye at 200 Mbps, limited by oscilloscope bandwidth

I then hooked the protection circuit up to the comparator to test the entire inbound signal chain. While the eye looked pretty good at 100 Mbps (plotting one leg of the differential since my scope was out of channels), at 200 Mbps horrible jitter appeared.

PRBS31 eye at 100 Mbps through full input buffer

PRBS31 eye at 200 Mbps through full input buffer

After quite a bit of scratching my head and fumbling with datasheets, I realized my oscilloscope was the problem by plotting the clock reference I was triggering on. The jitter was visible in this clock as well, suggesting that it was inherent in the oscilloscope's trigger circuit. This isn't too surprising considering I'm really pushing the limits of this scope - I need a better one to do this kind of testing properly.

PRBS31 eye at 200 Mbps plus 200 MHz sync clock

 At this point I've done about all of the input stage testing I can do with this oscilloscope. I'm going to try and rig up a BER tester on the FPGA so I can do PRBS loopback through the protection stage and comparator at higher speeds, then repeat for the output buffer and the protection run in the opposite direction.

I still have more work to do on the protection circuit as well... while it's fine at 100 Mbps, the 2x 10pF Schottky diode parasitic capacitance is seriously degrading my rise times (I calculated an RC filter -3dB point of around 200 MHz, so higher harmonics are being chopped off). I have some ideas on how I can cut this down much less but that will require a board respin and another blog post!

Open Verilog flow for Silego GreenPak4 programmable logic devices

I've written a couple of posts in the past few months but they were all for the blog at work so I figured I'm long overdue for one on Silicon Exposed.

So what's a GreenPak?

Silego Technology is a fabless semiconductor company located in the SF Bay area, which makes (among other things) a line of programmable logic devices known as GreenPak. Their 5th generation parts were just announced, but I started this project before that happened so I'm still targeting the 4th generation.

GreenPak devices are kind of like itty bitty PSoCs - they have a mixed signal fabric with an ADC, DACs, comparators, voltage references, plus a digital LUT/FF fabric and some typical digital MCU peripherals like counters and oscillators (but no CPU).

It's actually an interesting architecture - FPGAs (including some devices marketed as CPLDs) are a 2D array of LUTs connected via wires to adjacent cells, and true (product term) CPLDs are a star topology of AND-OR arrays connected by a crossbar. GreenPak, on the other hand, is a star topology of LUTs, flipflops, and analog/digital hard IP connected to a crossbar.

Without further ado, here's a block diagram showing all the cool stuff you get in the SLG46620V:

SLG46620V block diagram (from device datasheet)

They're also tiny (the SLG46620V is a 20-pin 0.4mm pitch STQFN measuring 2x3 mm, and the lower gate count SLG46140V is a mere 1.6x2 mm) and probably the cheapest programmable logic device on the market - $0.50 in low volume and less than $0.40 in larger quantities.

The Vdd range of GreenPak4 is huge, more like what you'd expect from an MCU than an FPGA! It can run on anything from 1.8 to 5V, although performance is only specified at 1.8, 3.3, and 5V nominal voltages. There's also a dual-rail version that trades one of the GPIO pins for a second power supply pin, allowing you to interface to logic at two different voltage levels.

To support low-cost/space-constrained applications, they even have the configuration memory on die. It's one-time programmable and needs external Vpp to program (presumably Silego didn't want to waste die area on charge pumps that would only be used once) but has a SRAM programming mode for prototyping.

The best part is that the development software (GreenPak Designer) is free of charge and provided for all major operating systems including Linux! Unfortunately, the only supported design entry method is schematic entry and there's no way to write your design in a HDL.

While schematics may be fine for quick tinkering on really simple designs, they quickly get unwieldy. The nightmare of a circuit shown below is just a bunch of counters hooked up to LEDs that blink at various rates.

Schematic from hell!

As if this wasn't enough of a problem, the largest GreenPak4 device (the SLG46620V) is split into two halves with limited routing between them, and the GUI doesn't help the user manage this complexity at all - you have to draw your schematic in two halves and add "cross connections" between them.

The icing on the cake is that schematics are a pain to diff and collaborate on. Although GreenPak schematics are XML based, which is a touch better than binary, who wants to read a giant XML diff and try to figure out what's going on in the circuit?

This isn't going to be a post on the quirks of Silego's software, though - that would be boring. As it turns out, there's one more exciting feature of these chips that I didn't mention earlier: the configuration bitstream is 100% documented in the device datasheet! This is unheard of in the programmable logic world. As Nick of Arachnid Labs says, the chip is "just dying for someone to write a VHDL or Verilog compiler for it". As you can probably guess by from the title of this post, I've been busy doing exactly that.

Great! How does it work?

Rather than wasting time writing a synthesizer, I decided to write a GreenPak technology library for Clifford Wolf's excellent open source synthesis tool, Yosys, and then make a place-and-route tool to turn that into a final netlist. The post-PAR netlist can then be loaded into GreenPak Designer in order to program the device.

The first step of the process is to run the "synth_greenpak4" Yosys flow on the Verilog source. This runs a generic RTL synthesis pass, then some coarse-grained extraction passes to infer shift register and counter cells from behavioral logic, and finally maps the remaining logic to LUT/FF cells and outputs a JSON-formatted netlist.

Once the design has been synthesized, my tool (named, surprisingly, gp4par) is then launched on the netlist. It begins by parsing the JSON and constructing a directed graph of cell objects in memory. A second graph, containing all of the primitives in the device and the legal connections between them, is then created based on the device specified on the command line. (As of now only the SLG46620V is supported; the SLG46621V can be added fairly easily but the SLG46140V has a slightly different microarchitecture which will require a bit more work to support.)

After the graphs are generated, each node in the netlist graph is assigned a numeric label identifying the type of cell and each node in the device graph is assigned a list of legal labels: for example, an I/O buffer site is legal for an input buffer, output buffer, or bidirectional buffer.

Example labeling for a subset of the netlist and device graphs

The labeled nodes now need to be placed. The initial placement uses a simple greedy algorithm to create a valid (although not necessarily optimal or even routable) placement:

1. Loop over the cells in the netlist. If any cell has a LOC constraint, which locks the cell to a specific physical site, attempt to assign the node to the specified site. If the specified node is the wrong type, doesn't exist, or is already used by another constrained node, the constraint is invalid so fail with an error.
2. Loop over all of the unconstrained cells in the netlist and assign them to the first unused site with the right label. If none are available, the design is too big for the device so fail with an error.

Once the design is placed, the placement optimizer then loops over the design and attempts to improve it. A simulated annealing algorithm is used, where changes to the design are accepted unconditionally if they make the placement better, and with a random, gradually decreasing probability if they make it worse. The optimizer terminates when the design receives a perfect score (indicating an optimal placement) or if it stops making progress for several iterations. Each iteration does the following:

1. Compute a score for the current design based on the number of unroutable nets, the amount of routing congestion (number of nets crossing between halves of the device), and static timing analysis (not yet implemented, always zero).
2. Make a list of nodes that contributed to this score in some way (having some attached nets unroutable, crossing to the other half of the device, or failing timing).
3. Remove nodes from the list that are LOC'd to a specific location since we're not allowed to move them.
4. Remove nodes from the list that have only one legal placement in the device (for example, oscillator hard IP) since there's nowhere else for them to go.
5. Pick a node from the remainder of the list at random. Call this our pivot.
6. Find a list of candidate placements for the pivot:
1. Consider all routable placements in the other half of the device.
2. If none were found, consider all routable placements anywhere in the device.
3. If none were found, consider all placements anywhere in the device even if they're not routable.
7. Pick one of the candidates at random and move the pivot to that location. If another cell in the netlist is already there, put it in the vacant site left by the pivot.
8. Re-compute the score for the design. If it's better, accept this change and start the next iteration.
9. If the score is worse, accept it with a random probability which decreases as the iteration number goes up. If the change is not accepted, restore the previous placement.

After optimization, the design is checked for routability. If any edges in the netlist graph don't correspond to edges in the device graph, the user probably asked for something impossible (for example, trying to hook a flipflop's output to a comparator's reference voltage input) so fail with an error.

The design is then routed. This is quite simple due to the crossbar structure of the device. For each edge in the netlist:

1. If dedicated (non-fabric) routing is used for this path, configure the destination's input mux appropriately and stop.
2. If the source and destination are in the same half of the device, configure the destination's input mux appropriately and stop.
3. A cross-connection must be used. Check if we already used one to bring the source signal to the other half of the device. If found, configure the destination to route from that cross-connection and stop.
4. Check if we have any cross-connections left going in this direction. If they're all used, the design is unroutable due to congestion so fail with an error.
5. Pick the next unused cross-connection and configure it to route from the source. Configure the destination to route from the cross-connection and stop.

Once routing is finished, run a series of post-PAR design rule checks. These currently include the following:

• If any node has no loads, generate a warning
• If an I/O buffer is connected to analog hard IP, fail with an error if it's not configured in analog mode.
• Some signals (such as comparator inputs and oscillator power-down controls) are generated by a shared mux and fed to many loads. If different loads require conflicting settings for the shared mux, fail with an error.

If DRC passes with no errors, configure all of the individual cells in the netlist based on the HDL parameters. Fail with an error if an invalid configuration was requested.

Finally, generate the bitstream from all of the per-cell configuration and write it to a file.

Great, let's get started!

If you don't already have one, you'll need to buy a GreenPak4 development kit. The kit includes samples of the SLG46620V (among other devices) and a programmer/emulation board. While you're waiting for it to arrive, install GreenPak Designer.

Download and install Yosys. Although Clifford is pretty good at merging my pull requests, only my fork on Github is guaranteed to have the most up-to-date support for GreenPak devices so don't be surprised if you can't use a bleeding-edge feature with mainline Yosys.

Download and install gp4par. You can get it from the Github repository.

Write your HDL, compile with Yosys, P&R with gp4par, and import the bitstream into GreenPak Designer to program the target device. The most current gp4par manual is included in LaTeX source form in the source tree and is automatically built as part of the compile process. If you're just browsing, there's a relatively recent PDF version on my web server.

If you'd like to see the Verilog that produced the nightmare of a schematic I showed above, here it is.

Be advised that this project is still very much a work in progress and there are still a number of SLG46620V features I don't support (see the manual for exact details).

I love it / it segfaulted / there's a problem in the manual!

Hop in our IRC channel (##openfpga on Freenode) and let me know. Feedback is great, pull requests are even better,

You're competing with Silego's IDE. Have they found out and sued you yet?

Nope. They're fully aware of what I'm doing and are rolling out the red carpet for me. They love the idea of a HDL flow as an alternative to schematic entry and are pretty amazed at how fast it's coming together.

After I reported a few bugs in their datasheets they decided to skip the middleman and give me direct access to the engineer who writes their documentation so that I can get faster responses. The last time I found a problem (two different parts of the datasheet contradicted each other) an updated datasheet was in my inbox and on their website by the next day. I only wish Xilinx gave me that kind of treatment!

They've even offered me free hardware to help me add support for their latest product family, although I plan to get GreenPak4 support to a more stable state before taking them up on the offer.

So what's next?

Better testing, for starters. I have to verify functionality by hand with a DMM and oscilloscope, which is time consuming.

My contact at Silego says they're going to be giving me documentation on the SRAM emulation interface soon, so I'm going to make a hardware-in-loop test platform that connects to my desktop and the Silego ZIF socket, and lets me load new bitstreams via a scriptable interface. It'll have FPGA-based digital I/O as well as an ADC and DAC on every device pin, plus an adjustable voltage regulator for power, so I can feed in arbitrary mixed-signal test waveforms and write PC-based unit tests to verify correct behavior.

Other than that, I want to finish support for the SLG46620V in the next month or two. The SLG46621V will be an easy addition since only one pin and the relevant configuration bits have changed from the 46620 (I suspect they're the same die, just bonded out differently).

Once that's done I'll have to do some more extensive work to add the SLG46140V since the architecture is a bit different (a lot of the combinatorial logic is merged into multi-function blocks). Luckily, the 46140 has a lot in common architecturally with the GreenPak5 family, so once that's done GreenPak5 will probably be a lot easier to add support for.

My thanks go out to Clifford Wolf, whitequark, the IRC users in ##openfpga, and everyone at Silego I've worked with to help make this possible. I hope that one day this project will become mature enough that Silego will ship it as an officially supported extension to GreenPak Designer, making history by becoming the first modern programmable logic vendor to ship a fully open source synthesis and P&R suite.

TDR updates

A few months ago, I wrote about a project I had been thinking of for a while but not had time to work on: a time-domain reflectometer (TDR) for testing twisted pair Ethernet cables.

TDR background

The basic theory of operation is simple: Send a pulse down a transmission line and measure the reflected voltage over time to get a plot of impedance discontinuities over time. Unfortunately, doing this with sufficient temporal resolution (sub-nanosecond) requires extremely high analog sampling rates, and GHz A/D converters are (to say the least) not cheap: the least expensive 1 GSA/s ADC on Digi-Key is the HMCAD1520, which sells for $120 each at the time of this writing. Higher sampling rates cost even more, the 1.5 GSa/s ADC081500CIYB is listed at $347.

One possible architecture would consist of a pre-amplifier for each channel, a 4:1 RF mux, and a single high-speed ADC sampled by an FPGA. This would work, but seemed quite expensive and I wanted to explore lower-cost options.

ADC architecture

After thinking about the problem for a while, I realized that the single most expensive component in a classical TDR was probably the ADC - but there was no easy way to make it cheaper. What if I could eliminate the ADC entirely?

I drew inspiration from the successive-approximation-register (SAR) ADC architecture, which essentially converts a DAC into an ADC by binary searching. The basic operating algorithm is as follows:

• For each point T in time
• vstart = 0
• vend = Vref
• Set DAC to (Vstart + Vend) / 2
• Compare Vin against Vdac
• If Vin > Vdac
• Set current bit of sample to 1
• Set Vstart to Vdac, update ADC, repeat
• else
• Set current bit of sample to 0
• Set Vend to Vdac, update ADC, repeat

The problem with SAR for high speeds is that N-bit analog resolution at M samples per second requires a DAC that can run at O(M log N) samples per second - hardly an improvement!

In order to work around this problem, I began to think about ways to represent the data generated by a SAR ADC. I ended up modeling a simplified SAR ADC which performed a linear, rather than binary, search. We can represent the intermediate data as a matrix of 2^N rows by M columns, one for each of M data points.

The sampling algorithm for this simplified ADC works as follows:

• For each point T in time 
• Set DAC to 0
• Compare Vin against Vdac
• If Vin > Vdac
• Set column[Vdac] to 1
• Increment Vdac
• Repeat comparison
• Otherwise stop and capture the next sample

Once we have this matrix, we can simply sum the number of 1s in each column to calculate the corresponding sample value.

While this approach will clearly work, it is exponentially slower than the conventional SAR ADC since it requires 2^N samples instead, instead of N, for N-bit precision. So why is it useful?

Now consider what happens if we acquire the data from a transposed version of the same matrix:

• For each Vdac from 0 to Vref
• For each point T in time
• Compare Vin against Vdac
• If Vin > Vdac
• Set row Vin of column T to 1
• Increment Vref
• Go back in time and loop over the signal again

This version clearly captures the same data, since matrix[T][V] is set to true iff sample T is less than V. We simply switch the inner and outer loops.

It also has a very interesting property for cost optimization: Since it only updates the DAC after sampling the entire signal, we can now use a much slower (and cheaper) DAC than with a conventional SAR. In addition, the comparator can now update at the sampling frequency instead of 2^N times the sampling frequency.

There's just one problem: It requires time travel! Why are we wasting our time analyzing a circuit that can't actually be built?

Well, as it turns out we can solve this problem too - with "parallel universes". Since the impedance of the cable is (hopefully) fairly constant over time, if we send multiple pulses they should return identical reflection waveforms. We can thus send out a pulse, test one candidate Vdac value against this waveform, then increment Vdac and repeat.

The end result is that with a cheap SPI DAC, a high-speed comparator, and an FPGA with a high-speed digital input we can digitize a repetitive signal to arbitrary analog precision, with sampling rate limited only by comparator bandwidth and FPGA input buffer performance!

Pulse generation

The first step in any TDR, of course, is to generate the pulse.

I spent a while looking over FPGAs and ended up deciding on the Xilinx Kintex-7 series, specifically the XC7K70T. The -1 speed grade can do 1.25 Gbps LVDS in the high-performance I/O banks (matching the -2 and -3 speed Artix-7 devices) and the higher speed grades can go up to 1.6 Gbps.

The pulse is generated by using the OSERDES of the FPGA to produce a single-cycle LVDS 1 followed by a long series of 0s. The resulting LVDS pulse is fed into a Micrel SY58603U LVDS-to-CML buffer. This slightly increases the amplitude of the output pulse and sharpens the rise time up to 80ps.

The resulting pulse is then sent through the RJ45 connector onto the cable being tested.

Output buffer

Input preamplifier

The reflected signal coming off the differential pair is AC coupled with a pair of capacitors to prevent bus fights between the (unequal) common-mode voltages of the output buffer and the preamplifier. It is then fed into a LMH6881 programmable-gain preamplifier.

This is by far the most pricey analog component I've used in a design: nearly $10 for a single amplifier. But it's a very nice amplifier (made on a SiGe BiCMOS process)- very high linearity, 2.4 GHz bandwidth, and gain from 6 to 26 dB programmable over SPI in 0.25 dB steps.

Input preamplifier

The optional external terminator (R25) is intended to damp out any reflections coming off of the preamplifier if they present a problem; during the initial assembly I plan to leave it unpopulated. Since this is my first high-speed mixed signal design I'm trying to make it easy to tweak if I screwed up somehow :)

The output of the preamplifier is a differential signal with 2.5V common mode offset.

Differential to single-ended conversion

The next step is to convert the differential voltage into a single-ended voltage that we can feed into the comparator. I use an AD8045 unity gain voltage feedback amplifier for this, configured to compute CH1_VDIFF = (CH1_BUF_P - CH1_BUF_N)  + 2.5V.

Digitization

The single-ended voltage is compared against the DAC output (AFE_VREF) using one half of an LMH7322 dual comparator.

The output supply of the comparator is driven by a 2.5V supply to produce LVDS-compatible differential output voltage levels.

PCB layout

The board was laid out in KiCAD using the new push-and-shove router. All of the differential pairs were manually length-matched to 0.1mm or better.

The upper left corner of the board contains four copies of the AFE. The AD8045s are on the underside of the board because the pinout made routing easier this way. Hopefully the impedance discontinuities from the vias won't matter at these signal speeds...

AFE layout, front side

AFE layout, back side

The rest of the board isn't nearly as complex: the lower left has a second RJ45 connector and a RGMII PHY for interfacing to the host PC, the power supply is at the upper right, and the FPGA is bottom center.

The power supply is divided into two regions, digital and analog. The digital supply is on the far right side of the board, safely isolated from the AFE. It uses an LTC3374 to generate an 1.0V 4A rail for the FPGA core, a 1.2V 2A rail for the FPGA transceivers and Ethernet PHY, a 1.8V 1A rail for digital I/O, and a 2.5V rail for the CML buffers and Ethernet analog logic.

The analog supply was fairly close to the AFE so I put a guard ring around it just to be safe. It consists of a LTC3122 boost converter to push the 5V nominal input voltage up to 6V, followed by a 5V LDO to give a nice clean analog rail. I ran the output of the LDO through a pi filter just to be extra safe.

The TDR subsystem didn't use any of the four 6.6 Gbps GTX serial transceivers on the FPGA because they are designed to recover their clock from the incoming signal and don't seem to support use of an external reference clock. It seemed a shame to waste them, though, so I broke them (as well as 20 0.95 Gbps LVDS channels) out to a Samtec Q-strip header for use as high-speed GPIO.

Without further ado, here's the full layout. I could have made the board quite a bit smaller in the vertical axis but I needed to keep a constant 100mm high so it would fit in the card guides on my Eurocard rack.

The board is at fabs for quotes now and I'll make another post once the boards come back.

Layer 1

Layer 2 (ground)

Layer 3 (power)

Layer 4, flipped to make text readable

Managed DC PDU

As I mentioned in my last post, powering all of the prototyping boards on my desk presents some unique challenges. With only one exception (the Xilinx AC701 board), each of the 22 boards requires 5VDC at somewhere between 0.1 and 2 amps. Some are strictly USB powered, some have a 5.5/2.1mm barrel jack, and some can be powered by either USB or a barrel jack.

Powered USB hubs would reduce the number of power sources required, so I did just that. Lots of cables would get in the way so I designed a custom "backplane" USB hub with male mini-B ports which could plug directly into small prototyping boards. (As a side note, the connectors for this board were nearly impossible to find. There are very few uses for a male mini-B connector that mounts to a PCB rather than being attached to a cable so nobody makes them!)

USB backplane hub

These reduced the problem, but did not come close to eliminating it. I still had to power three backplane hubs, six standalone FPGA boards, and four standalone MCU/SoC dev boards. All needed 5V except for the AC701 (which runs on 12V) but I wanted additional 12V capability for the future if I expanded into higher-power design.

The obvious first idea was an ATX supply. My calculations of peak power for the apparatus (including room for growth) were fairly high, though, and most ATX supplies put the bulk of their output on the 12V rail and have fairly limited (well under 100W) 5V capacity.

The next thing I considered was an off-the-shelf 5V supply. This looked like a nice idea, but (as with an ATX supply) the high output current capability would represent a fire hazard if something shorted. I would obviously need overcurrent protection.

Thinking a bit more, I realized that fusing was probably not the best option. Fuses need to be replaced once blown and in a lab environment overcurrent events happen fairly often. Classical current limiting techniques would be problematic as well since many of my boards have switching power supplies. Since a switcher is a nonlinear load, reducing the input voltage doesn't actually reduce the current. Instead, load current actually increases to maintain the output voltage, which can lead to runaway failure conditions. The safer way to handle overcurrent on a switcher is to shut it down entirely.

I also wanted the ability to power cycle boards on command to reset a stuck board or test power-up behavior. While jiggling cables may work in a hands-on lab environment, it isn't a viable option in the remote-controlled "embedded cloud" platform I'm trying to build.

This would obviously require some intelligence on the part of the power management system. The natural solution was a managed power distribution unit (PDU) of the sort commonly used in datacenters for feeding power to racks of servers. Managed PDUs often include current metering as well, which could be very useful to me when trying to minimize power consumption in a design.

There's just one problem: As far as I can tell, nobody makes managed PDUs for 5V loads. The only ones I saw were for 12/24/48V supplies and massively overpriced: this 8-channel 12V unit costs a whopping $1,757.

What to do? Build one myself, of course!

The first step was to come up with the requirements:

• Remote control via SNMP
• Ten DC outputs fed by external supply
• 4A max load for any single channel, 20A max for entire board
• Independent overcurrent shutdown for each channel with adjustable threshold
• Inrush timers for overcurrent shutdown to prevent false positives during powerup
• Remote switching
• Current metering
• Thermal shutdown
• Under/overvoltage shutdown
• Input reverse voltage protection
• Able to operate at 5V or 12V (jumper selected)

Now that I had a good idea of what I was building, it was time to start the actual design. I decided to use an FPGA instead of a MCU since the parallel nature made it easy to meet the hard-realtime demands of the overcurrent protection system. I also wanted an opportunity to field-test my softcore gigabit-Ethernet MAC, one of my CPU designs, and several other components of my thesis architecture under real-world load conditions.

PDU block diagram

The output stage is key to the entire circuit so it was very important that it be designed correctly. I put quite a bit of effort into component selection here... perhaps a bit too much, as I missed a few bugs elsewhere on the board! More on that later.

Output stage

Working from the output terminal (right side, VOUT_1) we first encounter a a 5 mΩ 4-terminal shunt resistor which feeds the overcurrent shutdown circuit and current metering. This is followed by a an LC filter to smooth the output power and reduce coupling of noise between downstream devices.

The fuse is provided purely as a second line of defense in the event that the soft overcurrent protection fails. As a firmware/HDL developer I know all too well what bugs are capable of, so I like to include passive safeguards whenever reasonably possible. Assuming that my code works correctly, this fuse should never blow even if the output of the PDU was connected to a dead short. (This of course requires that my protection mechanism trip faster than the fuse. Given the 1ms response time of typical fuses to small overcurrents, this isn't a very difficult task.)

Power switching is done by a high-side P-channel MOSFET connected to VOUT (the main high-current power rail). The logic-level input from the control subsystem is shifted up to VOUT level by an N-channel MOSFET. A pullup and pulldown resistor ensure that the output is kept safely in the "off" state when the system is booting.

Current monitoring

The monitoring stage is even simpler: the shunt voltage is amplified by a TI INA199A2 instrumentation amplifier, then fed to an ADC (not shown in this schematic) for metering. A comparator checks the amplified voltage against a reference voltage set by a DAC (also not shown) and if the threshold is exceeded the overcurrent alarm output is asserted.

A module in the FPGA controls the output enables based on the overcurrent flags and internal state. When an output is first turned on the overcurrent flag is ignored for a programmable delay (usually a few ms) in order to avoid false triggering from inrush spikes. After this period, if the overcurrent flag is ever asserted the channel is turned off and placed in the "error-disable" state. In order to clear an error condition the channel must be manually cycled, much like a conventional circuit breaker.

Here's a view of the finished first-run prototype. As you can see the first layout revision had a few bugs ;) The dead-bugged oscillator turned out to not be necessary but it would have been more work to remove it so I'm keeping it until I do a respin with all of these fixes incorporated.

PDU board on my desk

The SNMP interface and IP protocol stack runs on a custom softcore CPU of my own design. The CPU is named GRAFTON, in keeping with my tradition of naming my processors after nearby towns. It is fairly similar to MIPS-1 at the ISA level and can be targeted by mips-linux-gnu gcc with carefully chosen flags, but does not implement unaligned load/store, interrupts, or the normal coprocessors. Coprocessor 0 exists but is used to interface with the RPC network.

GRAFTON's programming model is largely event-driven, in a model that will be somewhat familiar to anyone who has done raw Windows API programming. The CPU sleeps until an RPC interrupt packet shows up, then it is processed and it goes back to sleep. Unlike classical interrupt handling, user code running on GRAFTON cannot be pre-empted by an interrupt; it just sits in the queue until retrieved.

int main()
{
	//Do one-time setup
	Initialize();

	//Main message loop
	RPCMessage_t rmsg;
	while(1)
	{
		GetRPCInterrupt(&rmsg);
		ProcessInterrupt(&rmsg);
	}
	
	return 0;
}

RPCFunctionCall(), a simple C wrapper around the low-level SendRPCMessage and RecvRPCMessage() functions, abstracts the RPC network with a blocking C function call semantics. Any messages other than return values of the pending call are queued for future processing.

In the example below, I'm initializing the SPI modules for the A/D converters with a clock divisor computed on the fly from the system clock rate.

void ADCInitialize()
{
	//SPI clock = 250 KHz
	RPCMessage_t rmsg;
	RPCFunctionCall(g_sysinfoAddr, SYSINFO_GET_CYCFREQ, 0, 250 * 1000, 0, &rmsg);
	int spiclk = rmsg.data[1];
	for(unsigned int i=0; i<3; i++)
		RPCFunctionCall(g_periphAddr, PERIPH_SPI_SET_CLKDIV, spiclk, i, 0, &rmsg);
}

The firmware is about 4300 lines of C in total, including comments but not the 1165 lines of C and assembly in my C runtime library shared by all GRAFTON designs. It implements IPv4, UDP, DHCP, ARP, ICMP echo, and SNMPv2c. SNMPv3 security and IPv6 are planned but are on hold until I move firmware out of block RAM and into flash so I have some space to work in. Other than that, it's essentially feature-complete and I've been using the PDU in my lab for a while while working on my flash controller and some support stuff.

The PC-side UI, intended to control several PDUs, is written in C++ using gtkmm and communicates with the board over SNMP. One tab (not shown) contains summary information with one graph trace per PDU.

PDU control panel

With a few minutes of PHP scripting I was also able to get my Munin installation to connect to the PDU and collect long-term logs even when I don't have the panel up.

Munin logs of PDU

The board runs quite cool, the spikes of heat caused by my furnace kicking in are quite visible and dwarf thermal variations caused by changes in load.

It needs a little bit more work to be fully production-ready but is already saving me time around the lab.

My desk with the PDU installed

Here's a look at my desk after deploying the PDU. The power cable mess is almost completely gone :) I do need to tidy up the Ethernet cables at some point, though...

Desktop raised floor

It's been a while since I've posted about a project I've done rather than a tool or some of my reversing work. This one is purely mechanical too!

First, a little background. I have a lot of FPGA/CPLD/MCU dev boards on my desk. By "a lot" I don't mean two or three... more like 20. Powering this much hardware presents some interesting problems. I don't have that many USB ports (and many of them need more power than USB can provide). Wallwarts are another obvious solution, but I don't have enough outlets or wallwarts to power 20 boards either!

I made three bar-shaped USB hubs with male mini-B ports, to plug into small development boards backplane-style. This helped a bit, but as my collection of boards grew the situation got worse.

By last May, my desk looked something like this:

My desk full of cables

Despite extensive efforts to manage the cable disaster with split tubing, there was still a giant octopus. Worse yet, my power strips were full and half of my boards didn't even have power.

The first step was to replace the loose boards with a datacenter-style "raised floor". I bought a 2x3 foot sheet of clear blue acrylic from McMaster-Carr, carefully floorplanned where all of the boards would go, and then drilled holes for each board's mounting standoffs.

Drilling holes

This operation had to be done out on the kitchen table because my office was too small to work comfortably in.

Mounting USB hubs

I mounted all of the USB hubs to the underside of the board in order to save space on top for dev boards and things I was likely to need to probe. While this seemed a good idea at first, reaching underneath them to run cables was a little tricky. After finishing the build I replaced the legs with ones several inches longer to provide the necessary hand clearance.

Before running cables, I attached all of the boards and brought it back to my desk to test the fit.

The apparatus on my desk

The "hostnames" on labels below each board are used as node names for my batch scheduler and unit test framework (more on that in a future post). In addition, those boards with Ethernet interfaces are assigned a constant IP address by my DHCP server, recorded in DNS with that hostname so I can write test cases using hostnames instead of raw IP addresses.

In an effort to reduce cable mess, I made custom cut-to-size USB cables out of cat5 cable and soldered on USB plugs. This was a very slow and laborious process because the connectors tended to melt very easily no matter what temperature I ran the iron at. BGA is no problem for me but these connectors gave me a hard time; I had yields somewhere around 60-70% even after rework. The rest of the time the connectors were melted beyond repair.

Despite the pain, I think the results were worth it. I was a little worried about signal quality as USB is supposed to be 90 ohm Zdiff and cat5e is 100, but I've noticed no problems. I did try to find 90 ohm cables but had trouble locating any.

Custom USB cables

After running all of the cables I could, a few of the boards were still unpowered and there were wallwarts everywhere, but the data wiring was a bit neater. Definitely a step in the right direction, but more work was needed.

After initial deployment

After taking that picture, I replaced most of the red electrical tape with zip ties and stick-on mount points. This made the setup a lot neater but I don't have any photos of that handy.

In order to tidy it up properly, I needed to tackle the power problem. My solution to that is a bit of a long story so I'll save that for next post :)

SoC framework, part 5: JtagDebugController and nocswitch

All of the JTAG utilities I've been mentioning are quite handy if you need to load a bitstream onto a board from one of several workstations. But JTAG is capable of much more, including powerful on-chip debug features.

One of the often-overlooked hard IP blocks in Xilinx FPGAs is BSCAN. This primitive (usually described in the FPGA's configuration user guide) connects a JTAG data register for certain special instructions to FPGA fabric.

Xilinx 6 and 7 series FPGAs each contain four BSCANs, one connected to each of the four JTAG instructions USER1...USER4. These are very rarely used by user designs, but Xilinx utilities like ChipScope and the in-system SPI programming cores use them to communicate with the FPGA without needing additional connections.

The primitive is named BSCAN_SPARTAN6 in Spartan-6 and BSCANE2 in 7 series. As far as I can tell, both are functionally equivalent.

BSCAN_SPARTAN6 #(
 .JTAG_CHAIN(1)
)
user1_bscan (
 .SEL(instruction_active),
 .TCK(tck),
 .CAPTURE(state_capture_dr),
 .RESET(state_reset),
 .RUNTEST(state_runtest),
 .SHIFT(state_shift_dr),
 .UPDATE(state_update_dr),
 .DRCK(tck_gated),
 .TMS(tms),
 .TDI(tdi),
 .TDO(tdo)
);

The JTAG_CHAIN parameter specifies which of the four user instructions to use. I'll summarize the interesting ports below including some notes:

• SEL goes high whenever USERx is loaded into the instruction register, regardless of the test state machine's current state.
• CAPTURE, RESET, RUNTEST, SHIFT, UPDATE are one-hot flags that go high when the corresponding DR state is active. When the state machine is in the IR shift path, all flags are held low.
• TMS is of little practical use since the state machine is already implemented for you.
• TCK provides direct access to the JTAG clock. (Be sure to create a timing constraint for any signals clocked by this net.) In my experience the Xilinx tools often do not recognize this signal as a clock and use high-skew local routing; manual insertion of a BUFG/BUFH is advised for optimal results.
• TDI and TDO are connected to the corresponding JTAG pins when in the SHIFT-DR state. You can connect any fabric logic you want to them.

Given this core plus libjtaghal on the PC side, we have a solid framework for building an on-chip debug system! The first step is to decide what sort of data to move over the link. Since my framework is NoC based, raw NoC frames seemed the natural choice. This would create a sort of layer-3 VPN encapsulating RPC/DMA transactions within JTAG scan operations.

After some experimenting with protocols I came up with one that seemed to work reasonably well. USER1 is the status/control register, USER2 is the RPC data register, and USER3 is the DMA data register. USER4 is left free for future expansion.

The FPGA side of the link is a module called JtagDebugController. It exposes RPC and DMA ports to the NoC; my current convention calls for addresses in subnet c000/2 to be routed to the debug bridge.

I'm deliberately not describing the actual on-wire protocol in depth because it's still in flux; when I get closer to a stable release I'll document it somewhere.

The PC side of the link is a C++ application using libjtaghal called "nocswitch". Example usage:

$./x86_64-linux-gnu/nocswitch --server localhost --port 50100 --lport 50101
Emulated NoC switch [SVN rev 1253:1254M] by Andrew D. Zonenberg.

License: 3-clause ("new" or "modified") BSD.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Connected to JTAG daemon at localhost:50100
Querying adapter...
    Remote JTAG adapter is a Dev board JTAG (232H) (serial number "FTWOON60", userid "FTWOON60", frequency 10.00 MHz)
Initializing chain...
Scan chain contains 1 devices
Device  0 is a Xilinx XC6SLX25 stepping 2
    Virtual TAP status register is  1000adba
    Valid NoC endpoint detected

This spawns a nocswitch listening on localhost:50101 connecting to a jtagd at localhost:50100.

Once nocswitch is running, it polls the status register on USER1 constantly waiting for the "new RPC message" or "new DMA message" bit to be set. (This causes a lot of traffic on the nocswitch-jtagd link and uses a decent amount of CPU on the host; my custom 8-port ICE will include FPGA based polling and an onboard nocswitch along with the jtagd's to avoid this problem.)

Client applications can then connect to nocswitch via a TCP-based protocol. The nocswitch assigns an address in c000/2 to each client in a manner somewhat reminiscent of DHCP; client applications (on the same machine or elsewhere on the LAN) can then send and receive NoC packets directly to the device under test. Multiple clients are fully supported; the nocswitch performs layer-2 switching between clients and the DUT as needed.

Nocswitch is able to switch frames from one client to another as well as just to the DUT; this permits a client to send messages to a NoC address without caring about whether it's a core in the SoC, a PC-side unit test, or even an RTL simulation (my mechanism for doing the latter will be described in a future post).

From a test case author's perspective, the NocSwitchInterface class implements the RPCAndDMAInterface class and supports the usual complement of operations.

printf("Connecting to nocswitch server...\n");
NOCSwitchInterface iface;
iface.Connect(server, port);

uint16_t eaddr = nameserver.ForwardLookup("eth0");
printf("eth0 is at %04x\n", eaddr);

printf("Resetting interface...\n");
iface.RPCFunctionCall(eaddr, ETH_RESET, 0, 0, 0, rxm);

Finally, here's a sneak peek at what's coming in future posts:

• Hardware cosimulation, including a workaround for ISim's lack of Verilog PLI support
• Splash, my build system inspired by Google Blaze
• RED TIN, my internal logic analyzer (ChipScope/SignalTap replacement with lots of features useful in my work, like state machine decoding, RLE, and time-scale compression)
• A look at both the hardware and software sides of the infrastructure for my dev board farm (batch scheduling, distributed build, automated testing, managed power distribution, and more). Hooking a single board up to a single JTAG dongle works fine if you only have one device but becomes a lot more of a pain to maintain when you have over twenty dev boards with more on the way!

SoC framework, part 4: jtagd

As I mentioned in my previous post, libjtaghal supports a socket-based protocol for communicating with JTAG adapters. This allows some very powerful capabilities, for example sharing a single dev board among multiple developers.

The core of this is a TCP server written in C++ known as jtagd. So far I have tested it on Debian 7 on both x86_64-linux-gnu and arm-linux-gnueabihf architectures (laptop computer and Beaglebone Black).

The main jtagd executable connects to a JtagInterface object and bridges it out to a TCP socket using my custom protocol. The protocol is not 100% finalized at this point, several features (like a magic-number banner to verify the client is actually talking to a valid jtagd and not a mistyped port number) will be added before a public release.

Starting a jtagd is quite simple: run "jtagd --list" to see what interfaces are available, then connect to one of them.

azonenberg@mars$ jtagd --list
JTAG server daemon [SVN rev 1230M] by Andrew D. Zonenberg.

License: 3-clause ("new" or "modified") BSD.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Digilent API version: 2.9.3
    Enumerating interfaces... 2 found
    Interface 0: JtagSmt1
        Serial number:  SN:210203825011
        User ID:        JtagSmt1
        Default clock:  10.00 MHz
    Interface 1: JtagHs1
        Serial number:  SN:210205812611
        User ID:        JtagHs1
        Default clock:  10.00 MHz

FTDI API version: libftd2xx 1.1.4
    Enumerating interfaces... 16 found
    Interface 0: Digilent Adept USB Device A
        Serial number:  210203825011A
        User ID:        210203825011A
        Default clock:  10.00 MHz
   [[ Output trimmed for brevity ]]
    Interface 10: Dev Board JTAG
        Serial number:  FTWB6M0W
        User ID:        FTWB6M0W
        Default clock:  10.00 MHz

azonenberg@mars$ jtagd --api ftdi --serial 210203825011A --port 50200
JTAG server daemon [SVN rev 1230M] by Andrew D. Zonenberg.

License: 3-clause ("new" or "modified") BSD.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Connected to interface "Digilent Adept USB Device A (2232H)" (serial number "210203825011A")

Once the jtagd is running, you can connect to it using command-line tools such as jtagclient, or directly from C code using libjtaghal. The example here connects to a Digilent Atlys and verifies the device ID of the XC6SLX45 FPGA.

NetworkedJtagInterface iface;
iface.Connect(server, port);

//note use of RAII-style mutexing
//since jtagd is multi-client capable
JtagLock lock(m_iface);
m_iface->InitializeChain();

int ndev = m_iface->GetDeviceCount();
if(ndev == 0)
{
 throw JtagExceptionWrapper(
  "No devices found - invalid scan chain?",
  "",
  JtagException::EXCEPTION_TYPE_BOARD_FAULT);
}

//Verify that the board is an Atlys
//Should have a single XC6SLX45
XilinxSpartan6Device* pfpga = dynamic_cast<XilinxSpartan6Device*>(m_iface->GetDevice(0));
if(pfpga == NULL)
{
 throw JtagExceptionWrapper(
  "Device does not appear to be a Spartan-6",
  "",
  JtagException::EXCEPTION_TYPE_BOARD_FAULT);
}
if(pfpga->GetArraySize() != XilinxSpartan6Device::SPARTAN6_LX45)
{
 throw JtagExceptionWrapper(
  "Device is not an XC6SLX45",
  "",
  JtagException::EXCEPTION_TYPE_BOARD_FAULT);
}

The library internally uses low-level chain operations in order to talk to the device. The code below retrieves the "Device DNA" die serial number from a Spartan-6.

void XilinxSpartan6Device::GetSerialNumber(unsigned char* data)
{
 JtagLock lock(m_iface);
 
 Erase();
 
 //Enter ISC mode (wipes configuration)
 ResetToIdle();
 SetIR(INST_ISC_ENABLE);
 
 //Read the DNA value
 SetIR(INST_ISC_DNA);
 unsigned char zeros[8] = {0x00};
 ScanDR(zeros, data, 57);
 
 //Done
 SetIR(INST_ISC_DISABLE);
}

Stay tuned for my next post on nocswitch and the NoC-to-JTAG debug bridge :)

SoC framework, part 3: libjtaghal

Almost all of my embedded development and debugging makes heavy use of JTAG, both for loading new bitstreams/firmware images and for interacting with on-chip debug systems.

When I first got into FPGA development I used the Xilinx Platform Cable USB II, which sells for $258.75 on Digikey as of this writing. It integrated nicely with the Xilinx IDE but I quickly grew frustrated. I wanted to use the BSCAN_SPARTAN6 primitive in the FPGA to move debug data on and off the FPGA using JTAG, but Xilinx does not provide any sort of API for scripting the platform cable. Although iMPACT allows manual bit twiddling in the chain as well as executing pre-made SVF files, there is no way to do interactive testing with it

My first step in deciding how to proceed was to see what made their adapter tick. I would have opened up the adapter to see what was inside, but Bunnie saved me the trouble by posting pictures a while ago as the Name That Ware for March 2011.

Xilinx Platform Cable USB II (image courtesy of Bunnie)

The vast majority of the footprints on the board aren't even populated... one can only guess what additional functionality may have been planned at one point. There's an XC3S200A FPGA, a Cypress USB MCU, USB descriptor EEPROM, flash for the FPGA, and then a bunch of passives for power regulation and level shifting. Overall, the design is quite simple and certainly not worth $250.

After browsing for something cheaper and based on a well-known chipset, I found the Digilent HS1, a $54.99 FT2232-based adapter which is supported by Digilent's documented JTAG API and integrated nicely with the Xilinx IDE. In addition, since it's a standard FTDI chipset it would be possible to interact with it at a lower level using libftd2xx. (I also built a custom FT232H-based programmer that I have half a dozen of around my lab, but I wanted a known-good design to verify my software on first.)

The HS1 worked quite well using the Xilinx tools, but I still needed JTAG code to talk to it and interact with the FPGAs for scripted tests. I looked at a couple of popular options and rejected each of them:

• OpenOCD (GNU GPL, incompatible with the BSD license used by my work)
• xc3sprog (GPL, standalone tool with no API, includes programming algorithms that can run directly off a .bit file)
• urjtag (GPL, has a socket-based JTAG server under development but not released yet)

It looked like I was going to have to write my own software, so I sat down and did just that. The result was a C++ library I call libjtaghal (JTAG hardware abstraction layer). It will be released publicly under the 3-clause BSD license once I've cleaned it up a bit; in the meantime if anyone wants a raw code drop with no documentation and a not-quite-finished build system leave a comment and I'll post something.

The basic structure of libjtaghal is built around two core object types: interfaces and devices. A JtagInterface represents a connection to a single JTAG adapter. As of now I support:

• FT*232H MPSSE (assumes ADBUS7 is the output enable, an option to configure this is planned for the future)
• Digilent API (for HS1 and integrated programmers on the Atlys etc)
• Generic socket-based protocol for talking to remote libjtaghal servers

My custom 8-port JTAG system (more to follow in a future post) will use my socket-based protocol and show up as 8 separate interfaces which can each be controlled independently (potentially from 8 separate client PCs).

 A JtagDevice represents a single chip in a scan chain. Support for multi-device scan chains needs a bit more work; this is one of the reasons I haven't released it yet.

A given JtagDevice may implement one or more additional interfaces. Some of these are:

• CPLD (generic complex programmable logic device)
• FPGA (generic FPGA device)
• ProgrammableDevice (any device which accepts firmware of some sort, including CPLDs, FPGAs, MCUs, and JTAG-capable ROMs)
• RPCNetworkInterface (a device which supports sending RPC messages over JTAG)
• DMANetworkInterface (a device which supports sending DMA messages over JTAG)
• RPCAndDMANetworkInterface (implements RPCNetworkInterface, DMANetworkInterface, and some logic to connect the two protocols)

This design allows several very handy design abstractions. For example, the below code is the sum total of the "program" mode for my "jtagclient" command-line application. It takes a JtagInterface object "iface" and programs the device at chain index "devnum" with the firmware image "bitfile". Note the complete lack of any device- or interface-specific code. The same function can configure a CoolRunner-II via one of my custom FTDI programmers or a Spartan-6 using the integrated Digilent programmer on a dev board without changing anything.

JtagDevice* device = iface.GetDevice(devnum);
if(device == NULL)
{
 throw JtagExceptionWrapper(
  "Device is null, cannot continue",
  "",
  JtagException::EXCEPTION_TYPE_BOARD_FAULT);
}

//Make sure it's a programmable device
ProgrammableDevice* pdev = dynamic_cast(device);
if(pdev == NULL)
{
 throw JtagExceptionWrapper(
  "Device is not a programmable device, cannot continue",
  "",
  JtagException::EXCEPTION_TYPE_BOARD_FAULT);
}

//Load the firmware image and program the device
printf("Loading firmware image...\n");
FirmwareImage* img = pdev->LoadFirmwareImage(bitfile);
printf("Programming device...\n");
pdev->Program(img);
printf("Configuration successful\n");
delete img;

This is part of the test case for my gigabit Ethernet MAC, allocating a page of memory on the device under test by talking to the RAM controller at NoC address "raddr" via the RPCNetworkInterface "iface". (Details on how this is implemented will be coming in a few posts.)

printf("Allocating memory...\n");
iface.RPCFunctionCall(raddr, RAM_ALLOCATE, 0, 0, 0, rxm);
uint32_t txptr = rxm.data[1];
printf("    Transmit buffer is at 0x%08x\n", txptr);

There's a lot more to the system than this but I'll save the rest for my next post :)