Modelling of block sizes in AVRDUDE

[stefanrueger]

The blocksize for reading or writing chunks of flash or EEPROM memory not only varies wildly between parts (that much is known) but can also vary, for a fixed part, with the interface it is programmed with, and worse, with the programmer that is used. Pertinent question at the end of the (long) post.

I took a look at the atdf files with a view to extract the parameters for parts used in different interfaces: parameters.txt

Q: How many parameter settings?

$ wc -l < parameters.txt 
22376

Q: Show me a few examples

$ grep 328P.ISP parameters.txt  | head
ATmega328P	ISP_INTERFACE	IspEnterProgMode_timeout	200
ATmega328P	ISP_INTERFACE	IspEnterProgMode_stabDelay	100
ATmega328P	ISP_INTERFACE	IspEnterProgMode_cmdexeDelay	25
ATmega328P	ISP_INTERFACE	IspEnterProgMode_synchLoops	32
ATmega328P	ISP_INTERFACE	IspEnterProgMode_byteDelay	0
ATmega328P	ISP_INTERFACE	IspEnterProgMode_pollIndex	3
ATmega328P	ISP_INTERFACE	IspEnterProgMode_pollValue	0x53
ATmega328P	ISP_INTERFACE	IspLeaveProgMode_preDelay	1
ATmega328P	ISP_INTERFACE	IspLeaveProgMode_postDelay	1
ATmega328P	ISP_INTERFACE	IspChipErase_eraseDelay	45

Q: Which interfaces are we talking about?

$ cut -f2 parameters.txt | sort | uniq -c | sort -n
    166 UPDI_INTERFACE
    177 JTAG_INTERFACE
    405 PDI_INTERFACE
    437 HVSP_INTERFACE_STK600
    483 HVSP_INTERFACE
   1248 PP_INTERFACE_AVRDRAGON
   3672 PP_INTERFACE_STK600
   3696 PP_INTERFACE
   6006 ISP_INTERFACE_STK600
   6086 ISP_INTERFACE

Q: Which interfaces define blocksizes for access?

$ grep -i blocksize parameters.txt | cut -f2 | sort | uniq -c | sort -n 
     76 HVSP_INTERFACE_STK600
     84 HVSP_INTERFACE
    208 PP_INTERFACE_AVRDRAGON
    612 PP_INTERFACE_STK600
    616 PP_INTERFACE
    728 ISP_INTERFACE_STK600
    736 ISP_INTERFACE

Q: ... and which blocksize parameters are we talking about?

$ grep -i blocksize parameters.txt | cut -f3 | sort | uniq -c | sort -n
     40 HvspProgramEeprom_blockSize
     40 HvspProgramFlash_blockSize
     40 HvspReadEeprom_blockSize
     40 HvspReadFlash_blockSize
    359 PpProgramEeprom_blockSize
    359 PpProgramFlash_blockSize
    359 PpReadEeprom_blockSize
    359 PpReadFlash_blockSize
    366 IspProgramEeprom_blockSize
    366 IspProgramFlash_blockSize
    366 IspReadEeprom_blockSize
    366 IspReadFlash_blockSize

Q: Blimey, there are 7 interfaces with 12 different blocksize parameters?
A: Actually, there are 28 different combinations

$ grep -i blocksize parameters.txt | cut -f2-3 | sort -u | wc -l
28

Q: Surely, once you've decided on the part, which memory and whether you want to read or write, then all the blocksizes are the same?
A: No such luck...

$ for rw in Read Program; do
     for mem in Flash Eeprom; do
       echo $rw$mem $(grep Size parameters.txt | grep $rw$mem | cut -f1,4 | \
         sort -u | cut -f1 | sort | uniq -c | awk '{ if($1 > 1) print; }' | wc -l)
     done
  done

ReadFlash 2
ReadEeprom 2
ProgramFlash 145
ProgramEeprom 172

This means there are 145 parts that have at least 2 different blocksizes to write flash to, which depend on the protocol; and there are 172 different parts that have at least 2 different blocksizes for writing EEPROM.

Q: Surely, at least, once one fixes PP, ISP or HVSP in addition then the blocksize is unique, ie, it does not depend on the device one programs with?
A: Not really, no.

$ for rw in Read Program; do
    for mem in Flash Eeprom; do
      for ifc in HVSP ISP PP; do
        echo $rw$mem $ifc $(grep Size parameters.txt | grep $rw$mem | grep $ifc.INTERF | \
           cut -f1,4 | sort -u | cut -f1 | sort | uniq -c | awk '{ if($1 > 1) print; }' | wc -l)
      done
    done
  done | grep -v " 0$"

ReadFlash ISP 2
ReadEeprom ISP 2
ProgramFlash HVSP 3
ProgramFlash ISP 13
ProgramFlash PP 1
ProgramEeprom HVSP 1
ProgramEeprom ISP 9
ProgramEeprom PP 1

There are 13 parts that have different blocksizes to write flash, depending whether it's STK600 or not, ...

Q: Surely, AVRDUDE models that zoo, not?

A: Well, there are four parameters in avrdude.conf, flash readsize (ReadFlash in atdf), eeprom readsize (ReadEeprom), flash blocksize (ProgamFlash) and eeprom blocksize (ProgramEeprom). There is no distinction as to the different programming interfaces, and even less as to the different programmers using the different interfaces. And there are not many places where readsize and blocksize are actually used.

$ grep "-e-> *"blocksize -r src
src/stk500v2.c:        if (mem->blocksize != 0)
src/stk500v2.c:            write_size = mem->blocksize;
src/avrpart.c:              m->mode, m->delay, m->blocksize, m->pollindex,
src/config_gram.y:      current_mem->blocksize = $3->value.number;

$ grep "-e-> *"readsize -r src
src/stk500v2.c:  page_size = m->readsize;
src/stk500v2.c:  page_size = m->readsize;
src/serialupdi.c:  if (n_bytes > m->readsize) {
src/serialupdi.c:                          remaining_bytes > m->readsize ? m->readsize : remaining_bytes);
src/serialupdi.c:        read_offset+=m->readsize;
src/serialupdi.c:        remaining_bytes-=m->readsize;
src/jtagmkII.c:  page_size = m->readsize;
src/jtag3.c:	if (m->readsize != 0 && m->readsize < m->page_size)
src/jtag3.c:	  PDATA(pgm)->flash_pagesize = m->readsize;
src/jtag3.c:        if (m->readsize != 0 && m->readsize < m->page_size)
src/jtag3.c:          PDATA(pgm)->flash_pagesize = m->readsize;
src/jtag3.c:	if (m->readsize != 0 && m->readsize < m->page_size)
src/jtag3.c:	  PDATA(pgm)->flash_pagesize = m->readsize;
src/jtag3.c:  page_size = m->readsize;
src/config_gram.y:      current_mem->readsize = $3->value.number;
src/jtagmkI.c:  page_size = m->readsize;

--

Now what?

[mcuee]

The blocksize for reading or writing chunks of flash or EEPROM memory not only varies wildly between parts (that much is known) but can also vary, for a fixed part, with the interface it is programmed with, and worse, with the programmer that is used.

Wow, this is really complicated. I guess we have to live some levels of complexity as some programmers may have some limitations to make it different. That being said, if we can combine certain things and reduce the complexities, why not? And if there are bugs in avrdude in this aspect, then we have to fix it.

[stefanrueger]

Wow, this is really complicated.

True. I looked into all this to figure out what a good cache size for Issue #1020 is (I still plan to solve this generically) only to discover the intricate complexities of page sizes and blocksizes. But I am glad I went down this road. I am sure we'll be able to use the 22k settings in parameters.txt file for some automated updating/rewriting of avrdude.conf in other ways at some point in the future. Pushing atdf file contents to avrdude.conf would improve its data quality.

In the first instance I might use max(256, mem->page_size) as the main port of call for a flash cache and 256 for an EEPROM cache and then use the paged read/write routines to fill and write modified segments of the cache.

There is still the question as to how well paged read/write routines should understand the 28 potentially different block sizes in addition to the flash/EEPROM page sizes that memories already have.

[stefanrueger]

The different readsize numbers that .atdf files stipulate don't seem to matter in practice. At least it looks like programming with various STK500, STK600 and Dragon programmers work. Hence, closing the discussion.