Sunday, December 2, 2012

CMake, CTest, and CDash for Xilinx FPGAs

Despite most of my posts lately having been on hardware topics, my PhD work (as well as my undergraduate degree) is in computer science. As a result, one of my many interests is applying software development methodologies to hardware, soft hardware, and firmware.

This goes well beyond the obvious stuff like using version control for layout files or firmware source code. I'm talking about stuff like continuous integration and test-driven development. While I do commit frequently and have some unit tests (not as many as I'd like) there is currently no formal methodology for nightly builds, running all of the tests each commit (right now they need to be run by hand in the simulator), or automatic regression testing.

I've also been getting increasingly fed up with Xilinx's IDE lately (and IDEs in general, but that's another story...) - the editor doesn't support regex search and replace, all of the toolbars and wizards make it way too complex to change one compiler flag, and generally it seems to make me less productive. Almost all of my "pure" software projects use CMake with makefile outputs; I develop in a standalone editor and then just "make" from a shell to compile the code.

This post documents my work to date on a CMake-based workflow for Xilinx devices. My hope is that I can eventually have everything completely integrated so that my firmware, FPGA bitstreams, and RTL simulation test cases can all be built with a single "make" command.

The first part of my script is still very hackish - there are way too many hard-coded paths and it assumes the 14.3 toolchain version on 64-bit Linux, but it works for now and, more importantly, provides a wrapper that all of my other CMake code can use without changing even if I improve the autodetection.


# Find /opt/Xilinx or similar
find_file(XILINX_PARENT NAMES Xilinx PATHS /opt)
if(XILINX_PARENT STREQUAL "XILINX_PARENT-NOTFOUND")
 message(FATAL_ERROR "No Xilinx toolchain installation found")
endif()

# Find /opt/Xilinx/VERSION
# TODO: Figure out a better way of doing this
find_file(XILINX NAMES 14.3 PATHS ${XILINX_PARENT})
if(XILINX STREQUAL "XILINX-NOTFOUND")
 message(FATAL_ERROR "No ISE 14.3 installation found")
endif()
message(STATUS "Found Xilinx toolchain... ${XILINX}")

# Set current OS architecture (TODO: autodetect)
set(XILINX_ARCH lin64)

# Find fuse
find_file(FUSE NAMES fuse PATHS "${XILINX}/ISE_DS/ISE/bin/${XILINX_ARCH}/")
if(FUSE STREQUAL "FUSE-NOTFOUND")
 message(FATAL_ERROR "No Xilinx fuse installation found")
endif()
message(STATUS "Found Xilinx fuse... ${FUSE}")

The next step was a little helper for argument parsing. In most languages supported by CMake there is no concept of a "top level" source file - the compiler finds your main() function but the build system doesn't need to know which file it's in.

macro(xilinx_parse_args _top_level _sources)
 set(${_top_level} FALSE)
 set(${_sources})
 set(_found_sources FALSE)
 set(_found_top_level FALSE)
 foreach(arg ${ARGN})
  if(${arg} STREQUAL "TOP_LEVEL")
   set(_found_top_level TRUE)
  elseif(${arg} STREQUAL "SOURCES")
   set(_found_sources TRUE)
  elseif(${_found_sources})
   list(APPEND ${_sources} ${arg})
  elseif(${_found_top_level})
   if(${_top_level})
    message(FATAL_ERROR "Multiple top-level files specified in xilinx_parse_args")
   else()
    set(${_top_level} ${arg})    
   endif()
  else()
   message(FATAL_ERROR "Unrecognized command ${arg} in xilinx_parse_args")
  endif()
 endforeach()
endmacro()

Once this was working it was time to actually create a simulation executable. This is a bit more involved than one might think for a couple of reasons:
  • The Xilinx tools ship their own custom C/C++ runtime libraries which are generally not the same version as that used by the host system. If you source /opt/Xilinx/[VERSION]/ISE_DS/settings[32|64].sh then all of the tools work (and are added to your $PATH) but any application depending on the host's glibc version won't start!
  • As a result, CMake and CTest require the host's glibc (so you can't run the sim executable or it'll segfault) and the sim executable requires the Xilinx glibc. This means that CTest cannot run the sim executable directly.
  • ISim does not seem to provide any way of setting the exit code for a simulation. CTest expects a test case to return 0 on success and nonzero on failure.
I started out by creating a tcl script (pretty much an exact copy of the one generated by the GUI toolchain except for the exit call) to run the simulation. Right now the 1000ns run time is hard coded so your simulation must finish sooner than that or not all the test cases will run. I'm going to make this parameterizable in the future.

onerror {resume}
wave add /
run 1000 ns;
exit;

This script is then launched by an automatically generated bash script which runs the simulation and then looks at the log file. If your simulation ever issues a $DISPLAY with the text "FAIL" in it, the test case is considered a failure; otherwise it's marked a success. The intention is to have a bunch of test cases in the testbench printing out something like "SPI flash read test: [PASS|FAIL]"

#!/bin/bash
cd /home/azonenberg/native/programming/verilogpractice/UnitTest02/build/tests/testNandGate
source /opt/Xilinx/14.3/ISE_DS/settings64.sh
./testNandGate -tclbatch /home/azonenberg/native/programming/verilogpractice/UnitTest02/build/tests/testNandGate/testNandGate.tcl -intstyle silent -vcdfile testNandGate.vcd -vcdunit ps || exit 1
cat isim.log | grep -q FAIL
if [ "$?" != "1" ]; then
    exit 1;
fi

Gluing all of the necessary parts and code generation together, we get this: 
function(add_isim_executable OUTPUT_FILE )
  
 # Parse args
 xilinx_parse_args(TOP_LEVEL SOURCES ${ARGN})
 
 # Get base name without extension of the top-level module
 get_filename_component(TOPLEVEL_BASENAME ${TOP_LEVEL} NAME_WE )
 
 # Write the .prj file
 set(PRJ_FILE "${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE}.prj")
 file(WRITE ${PRJ_FILE} "verilog work \"${TOP_LEVEL}\"\n")
 foreach(f ${SOURCES})
  file(APPEND ${PRJ_FILE} "verilog work \"${f}\"\n")
 endforeach()
 file(APPEND ${PRJ_FILE} "verilog work \"${XILINX}/ISE_DS/ISE/verilog/src/glbl.v\"\n")
 
 # Main compile rule
 # TODO: tweak this
 add_custom_target(
  ${OUTPUT_FILE} ALL
  COMMAND ${FUSE} ${FUSE_FLAGS} -o ${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE} -prj ${PRJ_FILE}
    work.${TOPLEVEL_BASENAME} work.glbl > ${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE}_build.log
    2> ${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE}_err.log
  DEPENDS ${SOURCES} ${TOP_LEVEL}
  COMMENT "Building ISim executable ${OUTPUT_FILE}..."
 )
 
 # Write the tcl script
 set(TCL_FILE "${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE}.tcl")
 file(WRITE ${TCL_FILE} "onerror {resume}\n")
 file(APPEND ${TCL_FILE} "wave add /\n")
 file(APPEND ${TCL_FILE} "run 1000 ns;\n")
 file(APPEND ${TCL_FILE} "exit;\n")
 
 # Write the run-test wrapper script
 set(TEST_WRAPPER "${CMAKE_CURRENT_BINARY_DIR}/run${OUTPUT_FILE}.sh")
 file(WRITE ${TEST_WRAPPER} "#!/bin/bash\n")
 file(APPEND ${TEST_WRAPPER} "cd ${CMAKE_CURRENT_BINARY_DIR}\n")
 file(APPEND ${TEST_WRAPPER} "source ${XILINX}/ISE_DS/settings64.sh\n")
 file(APPEND ${TEST_WRAPPER} "./${OUTPUT_FILE} -tclbatch ${TCL_FILE} -intstyle silent -vcdfile ${OUTPUT_FILE}.vcd -vcdunit ps || exit 1\n")
 file(APPEND ${TEST_WRAPPER} "cat isim.log | grep -q FAIL\n")
 file(APPEND ${TEST_WRAPPER} "if [ \"$?\" != \"1\" ]; then\n")
 file(APPEND ${TEST_WRAPPER} "    exit 1;\n")
 file(APPEND ${TEST_WRAPPER} "fi\n")
 add_custom_command(TARGET ${OUTPUT_FILE} POST_BUILD COMMAND chmod +x ${TEST_WRAPPER})
 
endfunction()

There are several issues with the system right now; the most notable is that the fuse command is run every build even if the source files haven't changed. I'm going to fix this in a future release; this is just a WIP.

The final piece of the puzzle is some glue to create the executable and a CTest test case that calls the bash script:

function(add_isim_test TEST_NAME)

 # Parse args
 xilinx_parse_args(TOP_LEVEL SOURCES ${ARGN})

 # Add the sim executable
 add_isim_executable(test${TEST_NAME}
  TOP_LEVEL
   ${TOP_LEVEL}
  SOURCES 
   ${SOURCES}
  )

 add_test(${TEST_NAME}
  "${CMAKE_CURRENT_BINARY_DIR}/runtest${TEST_NAME}.sh")
 set_property(TEST ${TEST_NAME} APPEND PROPERTY DEPENDS test${TEST_NAME})


endfunction()

This is the only external interface to the whole module for now; everything else is just internal helper routines. The intended use is as follows: 
cmake_minimum_required(VERSION 2.8)
include(FindXilinx.cmake)
enable_testing()
include (CTest)
add_isim_test(NandGate
 TOP_LEVEL
  ${CMAKE_CURRENT_SOURCE_DIR}/testNandGate.v
 SOURCES 
  ${CMAKE_SOURCE_DIR}/hdl/NandGate.v
 )
The full code for FindXilinx.cmake as it stands is here for convenience: 
########################################################################################################################
# @file FindXilinx.cmake
# @author Andrew D. Zonenberg
# @brief Xilinx ISE toolchain CMake module
########################################################################################################################

########################################################################################################################
# Autodetect Xilinx paths (very hacky for now)

# Find /opt/Xilinx or similar
find_file(XILINX_PARENT NAMES Xilinx PATHS /opt)
if(XILINX_PARENT STREQUAL "XILINX_PARENT-NOTFOUND")
 message(FATAL_ERROR "No Xilinx toolchain installation found")
endif()

# Find /opt/Xilinx/VERSION
# TODO: Figure out a better way of doing this
find_file(XILINX NAMES 14.3 PATHS ${XILINX_PARENT})
if(XILINX STREQUAL "XILINX-NOTFOUND")
 message(FATAL_ERROR "No ISE 14.3 installation found")
endif()
message(STATUS "Found Xilinx toolchain... ${XILINX}")

# Set current OS architecture (TODO: autodetect)
set(XILINX_ARCH lin64)

# Find fuse
find_file(FUSE NAMES fuse PATHS "${XILINX}/ISE_DS/ISE/bin/${XILINX_ARCH}/")
if(FUSE STREQUAL "FUSE-NOTFOUND")
 message(FATAL_ERROR "No Xilinx fuse installation found")
endif()
message(STATUS "Found Xilinx fuse... ${FUSE}")

########################################################################################################################
# Argument parsing helper

macro(xilinx_parse_args _top_level _sources)
 set(${_top_level} FALSE)
 set(${_sources})
 set(_found_sources FALSE)
 set(_found_top_level FALSE)
 foreach(arg ${ARGN})
  if(${arg} STREQUAL "TOP_LEVEL")
   set(_found_top_level TRUE)
  elseif(${arg} STREQUAL "SOURCES")
   set(_found_sources TRUE)
  elseif(${_found_sources})
   list(APPEND ${_sources} ${arg})
  elseif(${_found_top_level})
   if(${_top_level})
    message(FATAL_ERROR "Multiple top-level files specified in xilinx_parse_args")
   else()
    set(${_top_level} ${arg})    
   endif()
  else()
   message(FATAL_ERROR "Unrecognized command ${arg} in xilinx_parse_args")
  endif()
 endforeach()
endmacro()

########################################################################################################################
# ISim executable generation

function(add_isim_executable OUTPUT_FILE )
  
 # Parse args
 xilinx_parse_args(TOP_LEVEL SOURCES ${ARGN})
 
 # Get base name without extension of the top-level module
 get_filename_component(TOPLEVEL_BASENAME ${TOP_LEVEL} NAME_WE )
 
 # Write the .prj file
 set(PRJ_FILE "${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE}.prj")
 file(WRITE ${PRJ_FILE} "verilog work \"${TOP_LEVEL}\"\n")
 foreach(f ${SOURCES})
  file(APPEND ${PRJ_FILE} "verilog work \"${f}\"\n")
 endforeach()
 file(APPEND ${PRJ_FILE} "verilog work \"${XILINX}/ISE_DS/ISE/verilog/src/glbl.v\"\n")
 
 # Main compile rule
 # TODO: tweak this
 add_custom_target(
  ${OUTPUT_FILE} ALL
  COMMAND ${FUSE} ${FUSE_FLAGS} -o ${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE} -prj ${PRJ_FILE}
    work.${TOPLEVEL_BASENAME} work.glbl > ${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE}_build.log
    2> ${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE}_err.log
  DEPENDS ${SOURCES} ${TOP_LEVEL}
  COMMENT "Building ISim executable ${OUTPUT_FILE}..."
 )
 
 # Write the tcl script
 set(TCL_FILE "${CMAKE_CURRENT_BINARY_DIR}/${OUTPUT_FILE}.tcl")
 file(WRITE ${TCL_FILE} "onerror {resume}\n")
 file(APPEND ${TCL_FILE} "wave add /\n")
 file(APPEND ${TCL_FILE} "run 1000 ns;\n")
 file(APPEND ${TCL_FILE} "exit;\n")
 
 # Write the run-test wrapper script
 set(TEST_WRAPPER "${CMAKE_CURRENT_BINARY_DIR}/run${OUTPUT_FILE}.sh")
 file(WRITE ${TEST_WRAPPER} "#!/bin/bash\n")
 file(APPEND ${TEST_WRAPPER} "cd ${CMAKE_CURRENT_BINARY_DIR}\n")
 file(APPEND ${TEST_WRAPPER} "source ${XILINX}/ISE_DS/settings64.sh\n")
 file(APPEND ${TEST_WRAPPER} "./${OUTPUT_FILE} -tclbatch ${TCL_FILE} -intstyle silent -vcdfile ${OUTPUT_FILE}.vcd -vcdunit ps || exit 1\n")
 file(APPEND ${TEST_WRAPPER} "cat isim.log | grep -q FAIL\n")
 file(APPEND ${TEST_WRAPPER} "if [ \"$?\" != \"1\" ]; then\n")
 file(APPEND ${TEST_WRAPPER} "    exit 1;\n")
 file(APPEND ${TEST_WRAPPER} "fi\n")
 add_custom_command(TARGET ${OUTPUT_FILE} POST_BUILD COMMAND chmod +x ${TEST_WRAPPER})
 
endfunction()

########################################################################################################################
# Test generation
#
# Usage:
# add_isim_test(NandGate
# TOP_LEVEL
#  ${CMAKE_CURRENT_SOURCE_DIR}/testNandGate.v
# SOURCES 
#  ${CMAKE_SOURCE_DIR}/hdl/NandGate.v
# )

function(add_isim_test TEST_NAME)

 # Parse args
 xilinx_parse_args(TOP_LEVEL SOURCES ${ARGN})

 # Add the sim executable
 add_isim_executable(test${TEST_NAME}
  TOP_LEVEL
   ${TOP_LEVEL}
  SOURCES 
   ${SOURCES}
  )

 add_test(${TEST_NAME}
  "${CMAKE_CURRENT_BINARY_DIR}/runtest${TEST_NAME}.sh")
 set_property(TEST ${TEST_NAME} APPEND PROPERTY DEPENDS test${TEST_NAME})


endfunction()
Once I get it to a more stable state I'll probably set up a Google Code page but for now this is good enough. The code can be used under the same 3-clause BSD license as almost all of my other open source code.
Posted by at
Labels: ,

1 comment:

  1. tymon_xJanuary 21, 2018 at 1:33 AM

    For using CMake with FPGA you can try with this: https://github.com/tymonx/logic . It supports Quartus Lite/Standard/Pro Prime, Xilinx Vivado, Verilator and ModelSim. In future will be more like Vivado Simulator.

    ReplyDelete

Subscribe to: Post Comments (Atom)