DDR fails RAM test on Pano Logic Rev C (earlier HW revs work fine).

[skiphansen]

The LiteX BIOS reports lots and lots of memory test failures on the Rev C, but none on the Rev B (a log is attached). One clue is that the errors are always in the lower 16 bits of the word.

AFAIK the only difference between the rev A/B and the rev C is the FPGA and SPI flash. The markings on the DDR are the same, i.e. D9LPX.

I've visually compared the Rev B and Rev C and neither have the DDR termination resistors installed. The PCB layouts appear to be identical as best as I can tell by eye.

There are two projects that have used DDR successfully on the Rev C Pano, both are using MIG generated controllers:
https://github.com/tomverbeure/panologic-g2 (ddr branch)
https://github.com/crwulff/panologic-g2 (memory_controller branch)

I've run the crwulff version on my Rev C board and the RAM test passes.

I've compared the .ucf files and found differences in the .ucf file. The MIG generated code specifies either UNTUNED_SPLIT_50 or UNTUNED_SPLIT_25 on the data lines, but the LiteX code doesn't.

I tried changing the LiteX code to use UNTUNED_SPLIT_25 and UNTUNED_SPLIT_50 but instead of a one and two bit errors the SDRAM now always reads as 0xffffffff.

I'm mystified what the problem could be and DDR controllers are way over my head. I need help with a fix or a suggestion on how to debug the problem.

Thanks!
ddr_errors.log

[ewenmcneill]

From what I've seen in the #litex IRC channel (on FreeNode) such errors are typically DDR RAM timing access issues, which end up depending on the timing of other things on the system/build (modern DDR3/4 RAM needs dynamic "training" on startup to lock onto the best timing phase/offset to read/write reliably, as it varies depending on, eg, how other things were placed in the FPGA). Also beware that there were some RAM timing fixes added to upstream litex in the last month or so, and I'm not sure if the litex dependency in litex-buildenv has been updated since then.

You might want to try asking Florent (of Enjoy Digital / upstream litex) about the DDR timing tweaks for litex -- https://github.com/enjoy-digital/litex (main project) or https://github.com/enjoy-digital/litedram (DDR RAM interface). Both to find out if it's likely to been fixed in upstream litex already, and also if not on how to figure out how to tweak the litedram timings effectively. Florent seems to understand the DDR RAM timings pretty well, and since you seem to have two "works" / "doesn't work" combinations (rev B / rev C HW; other RAM controller / litex RAM controller) hopefully he can help you narrow down the relevant changes.

Ewen

GitHub

enjoy-digital/litex

Build your hardware, easily! Contribute to enjoy-digital/litex development by creating an account on GitHub.

GitHub

enjoy-digital/litedram

Small footprint and configurable DRAM core. Contribute to enjoy-digital/litedram development by creating an account on GitHub.

[skiphansen]

Thanks Ewen. I built the Pano target on bare Litex, but it doesn't support the DDR. I'll try to merge it from litex-buildenv or another example. I've also ordered a Linsn RV901T which is another Spartan 6 based board which has DDR support for comparison.

[povik]

To report here:

I built the litex-buildenv's Pano target with litex 01a2fc11 and litedram 2c608619, and the boot-up memtest on RevC passes. (Building with the mismatched litex version required some fixes related to builder.bios_options). I did not try building with other versions of litex/litedram so I do not know if the issue reproduces on the particular unit I have.

[skiphansen]

Thanks @povik!, I'll give those versions a try as soon as I get a chance. I only have one Rev C and I don't know if anyone else had even tried it.

[skiphansen]

No joy here using an image povik provided. I guess the timing is just very marginal.

litex> sdram_test
Memtest at 0x0x40000000 (2MiB)...
  Write: 0x40000000-0x40200000 2MiB
   Read: 0x40000000-0x40200000 2MiB
  bus errors:  0/256
  addr errors: 1/8192
  data errors: 171004/524288
Memtest KO

litex>