maxlen in data iterators

[Avmb]

In domain_iterator.py and domain_interpolation_data_iterator.py there is this fragment of code:

            if len(ss) > self.maxlen and len(tt) > self.maxlen:
                continue

which skips a sentence pair if both the source sentence and the target sentence exceed the maximum allowed length. Is this behavior correct? Shouldn't we skip the pair if any of the sentences exceeds the maximum length?

[rsennrich]

Hi Antonio,

yes, this should be changed to 'or'. The reason why the current code doesn't crash is that there's another length check in prepare_data().

[rsennrich]

This was fixed in data_iterator.py in commit 52ea880, but still remains TODO for domain_interpolation_data_iterator.py

[Sofiabella]

52ea880