Setting uart_name to 'bridge' results in linker errors

[cklarhorst]

I would like to use the uart bridge (for openocd debugging and for litescope).

So I changed my target file (atlys/base) and added:

[...]
def __init__(self, platform, **kwargs):
   kwargs['uart_name']='bridge'
[...]

But I'm unable to run make gateware for the following config:

export PLATFORM=atlys
export TARGET=base
export VARIANT=lite+debug

it results in linker errors:

riscv64-unknown-elf-ld: ../libbase/libbase-nofloat.a(uart.o): in function `uart_isr':
[...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:32: undefined reference to `uart_ev_pending_read'
riscv64-unknown-elf-ld: [...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:35: undefined reference to `uart_rxempty_read'
riscv64-unknown-elf-ld: [...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:46: undefined reference to `uart_ev_pending_write'
riscv64-unknown-elf-ld: [...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:47: undefined reference to `uart_txfull_read'
riscv64-unknown-elf-ld: [...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:38: undefined reference to `uart_rxtx_read'
riscv64-unknown-elf-ld: [...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:41: undefined reference to `uart_ev_pending_write'
riscv64-unknown-elf-ld: [...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:48: undefined reference to `uart_rxtx_write'
riscv64-unknown-elf-ld: ../libbase/libbase-nofloat.a(uart.o): in function `uart_write':
[...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:88: undefined reference to `uart_txfull_read'
riscv64-unknown-elf-ld: [...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:92: undefined reference to `uart_rxtx_write'
riscv64-unknown-elf-ld: ../libbase/libbase-nofloat.a(uart.o): in function `uart_init':
[...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:105: undefined reference to `uart_ev_pending_read'
riscv64-unknown-elf-ld: [...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:105: undefined reference to `uart_ev_pending_write'
riscv64-unknown-elf-ld: [...]litex-buildenv/third_party/litex/litex/soc/software/libbase/uart.c:106: undefined reference to `uart_ev_enable_write'
[...]litex-buildenv/third_party/litex/litex/soc/software/bios/Makefile:64: recipe for target 'bios.elf' failed
make[1]: *** [bios.elf] Error 1
make[1]: Leaving directory '[...]litex-buildenv/build/atlys_base_vexriscv/software/bios'
Traceback (most recent call last):
  File "./make.py", line 185, in <module>
    main()
  File "./make.py", line 167, in main
    vns = builder.build(**dict(args.build_option))
  File "[...]litex-buildenv/third_party/litex/litex/soc/integration/builder.py", line 198, in build
    self._generate_rom_software(not self.soc.integrated_rom_initialized)
  File "[...]litex-buildenv/third_party/litex/litex/soc/integration/builder.py", line 178, in _generate_rom_software
    subprocess.check_call(["make", "-C", dst_dir, "-f", makefile])
  File "[...]litex-buildenv/build/conda/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-C', '[...]litex-buildenv/build/atlys_base_vexriscv/software/bios', '-f', '[...]litex-buildenv/third_party/litex/litex/soc/software/bios/Makefile']' returned non-zero exit status 2.

So how do I deactivate that it builds the uart code?

I'm only able to run make gateware when I use the stub uart (to keep the sw happy) and an additional uart bridge:

       kwargs['uart_name']='stub'       
       sys_clk_freq = 75*1000000
       # SoCSDRAM ---------------------------------------------------------------------------------
       SoCSDRAM.__init__(self, platform, clk_freq=sys_clk_freq, **kwargs)

       self.submodules.uartbone  = uart.UARTWishboneBridge(
               pads     = self.platform.request("serial"),
               clk_freq = self.sys_clk_freq,
               baudrate = 115200)
       self.bus.add_master(name="uartbone", master=self.uartbone.wishbone)

Or is that maybe the only intended way? But even then I'm not able to use openocd over litex_server. I get 0xff for all reads inside the debug region (0xf00f0000). I also tried to read/write inside the sram region. That was working fine.

[mithro]

@mateusz-holenko @ewenmcneill -- Any idea?

[mateusz-holenko]

IIRC we stumbled upon similar problems when working on FOMU support. The code assumed there will always be a UART, which was not the case for that board.
The suggested solution was to use the stub UART to make LiteX happy.

Don't know about the wishbone bridge though, as I have never used it.

[ewenmcneill]

The linker errors definitely look like (upstream) litex is assuming there is always a console UART, but that the default console UART gets disabled if the UART is repurposed to be a serial wishbone bridge. I suspect at this point the work arounds are either (a) use another serial port for the console from the one used for the UART wishbone bridge, or (b) use the stub UART to "absorb" that serial console. (I don't think there's another way to turn off the serial console in litex.)

I have a vague memory that someone (xobs?) was working on (maybe just considering?) an option to tunnel the console UART output via the wishbone bridge. But I can't find any details on that now (having spent a while looking).

https://github.com/litex-hub/wishbone-utils might also be of some use for using the Wishbone UART bridge for debugging. Eg, there's a feature to bridge gdb via the wishbone UART bridge (eg, as used in the Fomu tutorial -- https://workshop.fomu.im/en/latest/riscv.html). I'm not sure about the openocd over litex option; that might be something to ask about in the upstream litex project.

Ewen

GitHub

litex-hub/wishbone-utils

Utilities for working with a Wishbone bus in an embedded device - litex-hub/wishbone-utils

Fomu as a CPU — FPGA Tomu (Fomu) Workshop 0.1-194-g810431e documentation

[cklarhorst]

Thank you all very much for your inputs.
@ewenmcneill I think I found the uart tunnel option in the wishbone-tool wiki: uart_name="crossover"
I will try it out.

[ewenmcneill]

I think I found the uart tunnel option in the wishbone-tool wiki: uart_name="crossover"

Good find! Yes, https://github.com/litex-hub/wishbone-utils/tree/master/wishbone-tool#crossover-uart is what I was thinking of. I obviously didn't read far enough down the README earlier (and was expecting to find something in the Litex / Wishbone Utils documentation, rather than the README).

Ewen

GitHub

litex-hub/wishbone-utils

Utilities for working with a Wishbone bus in an embedded device - litex-hub/wishbone-utils

[cklarhorst]

Nice, GDB and UART are working well with the wishbone-tool over the uart_bridge and the uart_name="crossover" setting.

So, what's the difference between the litex_server command and the wishbone tool? Why are there two tools?

Would the project benefit from a PR for a new debug target e.g targets/atlys/debug.py with uart bridge, crossover and litescope enabled?

[ewenmcneill]

So, what's the difference between the litex_server command and the wishbone tool? Why are there two tools?

litex_server is the original, and I think written in C/C++. From memory it only supports a few wishbone bridge types (Ethernet, UART, maybe one more?).

wishbone-tool is basically a rewrite in Rust, done by Sean Cross (xobs) in the context of the Fomu project development, which added a bunch of features Sean wanted including Wishbone bridging over SPI (for the Fomu bring up), GDB interfaces, serial port crossover, etc. My guess is that now that wishbone-tool has been adopted into Litex Hub it's likely to be the preferred tool, as I believe it's got a bunch of extra functionality, but that's a decision for upstream Litex.

Glad to hear that you got it all working!

Ewen

[ewenmcneill]

Would the project benefit from a PR for a new debug target e.g targets/atlys/debug.py with uart bridge, crossover and litescope enabled?

That's probably a question for @mithro.

I suspect it's useful to have an example of how to do that in the repository, but I'm not sure if a specific target is the right place for it (since anyone using it for anything else would probably have to cut'n'paste it to their own target definition anyway).

Possibly the example should just be in the Wiki -- https://github.com/timvideos/litex-buildenv/wiki -- or other documentation? In a small snippet of changes that are easier to find?

Ewen

GitHub

timvideos/litex-buildenv

An environment for building LiteX based FPGA designs. Makes it easy to get everything you need! - timvideos/litex-buildenv

[mithro]

I think an example "debug.py" target with documentation on how to talk to it and use it would be super awesome!

Even better if it could have CI run this an test.

[cklarhorst]

Initial rough Roadmap:

• Add debug target (with uart bridge, crossover and litescope) to the atlys board
• Write a litescope fetch script (I would also put this into the target directory if nobody disagrees)
• Write a target specific make rule to start the litescope script and add text to the help command
• Write a target specific make rule to start gdb and add text to the help command
• Write a target specific make rule to use uart over litex_server and add text to the help command
• CI

[mithro]

Sounds good to me!

[tcal-x]

[This is actually a general question, but since there are Wishbone experts here...]

Is Wishbone bridge capable of actually multiplexing the connections (e.g. both UART and CPU debug), or can the host only connect to one at a time?

[cklarhorst]

Multiplexing is possible e.g for the wishbone-tool you can use -s gdb together with -s terminal or for litex_server you can connect the litex_crossover_uart and fetch litescope data.
I didn't put that information into the debug.py to keep it as simple as possible but I can add a sentence about it if you like.

[tcal-x]

Ok, that's cool! Are both of the tools capable of (de)multiplexing, litex-server and wishbone-tool?
Yes, a note that the options can be combined would be useful I think.
Maybe even in the diagram show the demuxing on the HOST side to , "terminal", and "gdb"....

[cklarhorst]

Maybe like this?:
(I also found out that you can connect the wishbone-tool to the litex_server)

# Connection Overview:
#                            
#      FPGA:                        
#                                                   |--> LiteScope
#   --SERIAL--> UARTWishboneBridge <--WISHBONE-BUS--+--> Crossover UART
#   |                                               |--> CPU Debug Interface
#   |  
#   |  Host:          
#   |                                  |-> LiteScope
#   |      |--->  litex_server  <--TCP-+-> litex_crossover_uart -> terminal
#   |      |                           |-> client that writes/reads any address
#   |      |                           |-> wishbone-tool
#   ^-----or
#          |                           |-> terminal
#          |-->   wishbone-tool <------+-> gdb
#                                      |-> client that writes/reads any address
#
# Note: The CPU Debug Interface is only available 
#       if your CPU_VARIANT includes "debug".
#
# Note: You can run multiple litex_server clients or 
#       wishbone-tool clients at the same time.
#
# There are currently two ways to connect to your UARTWishboneBridge:
#   1. Litex Server:
#       Usage: litex_server --uart --uart-port /dev/ttyXXX
#       - Features:
#           - LiteScope: (todo)
#           - Crossover UART:
#               - cd into build/[target]/test/
#               - start litex_crossover_uart
#               - connect to /dev/pts/XXX (e.g minicom -D /dev/pts/XXX)
#           - CPU Debug Interface: (not supported)
#           - Wishbone-tool: wishbone-tool --ethernet-host 127.0.0.1 --ethernet-tcp
#   2. Wishbone Tool (https://github.com/litex-hub/wishbone-utils)
#       - Features:
#           - LiteScope: (not supported)
#           - Crossover UART: 
#               wishbone-tool -s terminal --csr-csv build/[target]/test/csr.csv
#           - CPU Debug Interface: 
#               - wishbone-tool -s gdb --csr-csv build/[target]/test/csr.csv
#               - start gdb
#                   - issue: target remote :3333

[mithro]

I started trying to make a diagram of of your ascii diagram at https://docs.google.com/drawings/d/13bRFtmgb7Og6hmWEYqxPrfbshxPwuIRDJ0NzPKeBoZs/edit

Google Docs

LiteX Wishbone Bridge Infrastructure

Peripheral Bridge WB2CSR Wishbone CSR FPGA Bridge Host Computer LiteScope litex_crossover_uart client that writes/reads any address Terminal gtkwave Python? litex_server Etherbone wishbone-tool Terminal client that writes/reads any address GDB LiteScope Crossover UART CPU Debug Interface

[cklarhorst]

I like your diagram very much. I admit that a graphical image is better suited than my ascii version. But the graphical version can't be added to the source file, so I would like to ask for guidance on what information I should put into the debug.py.
One idea that should be possible is:

• Incorporate the current header of the debug.py into a wiki page with a non ascii diagram
• Make a new header with only a link to the wiki page

Should we go this route?

[mithro]

Why not both!? :-P

[ewenmcneill]

I like your diagram very much. I admit that a graphical image is better suited than my ascii version. But the graphical version can't be added to the source file, so I would like to ask for guidance on what information I should put into the debug.py.

I'm with Tim -- doing both would be a great idea. I very much like "ASCII Art" high level context drawings in documentation to have the context by the code, which makes it easier to understand the code. But it's also helpful to have more detailed diagrams/documentation in a Wiki page or similar.

Personally I'd prefer that the comments in the code not just be a link to a Wiki page -- those links are helpful for "see also", but if you have to read external documentation to understand the code, it's just a rabbit trail to follow every time someone tries to understand it.

It probably does help to keep the "ASCII Art" diagram/comments high level so they're unlikely to get out of date as other code (eg, in Litex) changes. I think the level of detail in your example above (#481 (comment)) is ideal.

# - issue: target remote :3333

I'd be inclined to word that "in gdb run: target remote ..." to make it more obvious. (My brain wants to read "issue" and a number as a reference to a bug tracker somewhere :-) )

It looks like it might be possible to do:

gdb -ex 'target remote :3333'

on the command line to auto connect (eg, https://stackoverflow.com/a/2717992), rather than it being two steps; but I've not tried that precise version myself (https://sourceware.org/gdb/onlinedocs/gdb/File-Options.html implies "-ex" is "execute a single command, but I'm unsure if it then goes interactive after that). If it works, that might make it easier to explain/document.

Ewen

[cklarhorst]

Ok I updated the PR.
@ewenmcneill thx for the gdb -ex idea. I tested it and it goes interactive.

[ewenmcneill]

FTR, because it took me a while to figure out (and isn't linked here), the PR mentioned in this change seems to be #492 (which is currently still open, apparently due to CI failing on it, and it seems unclear if that's a simulation or real problem.).

There's also issue #497 to add an equivalent for the Digilent Arty. And I think if #492 is merged, this issue could probably be closed just leaving #497 as future work.

Ewen