K9-Mail Compatibility: list special-use type error

[iot-resister]

To get K-9 Mail working, I changed the follwing

 s := make([]string, len(fields))
 for i, v := range fields {
    s[i] = fmt.Sprint(v)
  }
 
   if mailbox, err := imap.ParseString(s[0]); err != nil {
...

to line: https://github.com/emersion/go-imap/blob/master/commands/list.go#L41

Please excuse my go. I've only been writing it for one day.

references: https://stackoverflow.com/questions/44027826/convert-interface-to-string-in-golang

[emersion]

This sounds like a bug with K9 mail. The RFC specifies:

list            = "LIST" SP mailbox SP list-mailbox
list-mailbox    = 1*list-char / string
list-char       = ATOM-CHAR / list-wildcards / resp-specials

[iot-resister]

This my first time every hearing about ABNF, but shouldn't mailbox be a string? And isn't that SPECIAL-USE ? Mbsync sends []string, but k9 logs out an []interface.... So they are sending the wrong type?

mailbox         = "INBOX" / astring
                    ; INBOX is case-insensitive.  All case variants of
                    ; INBOX (e.g., "iNbOx") MUST be interpreted as INBOX
                    ; not as an astring.  An astring which consists of
                    ; the case-insensitive sequence "I" "N" "B" "O" "X"
                    ; is considered to be INBOX and not an astring.
                    ;  Refer to section 5.1 for further
                    ; semantic details of mailbox names.

I created an issue for k9 mail.

thunderbird/thunderbird-android#4729

Feel free to ignore my questions if they are grossly misinformed.

[cketti]

K-9 Mail 5.600 will only send the extended LIST command when the SPECIAL-USE extension is advertised. My reading of the spec was (and really still is), that the SPECIAL-USE capability may only be advertised if the server also supports LIST-EXTENDED. K-9 Mail versions 5.7xx+ will check for both capabilities before sending the extended LIST command.

See also emersion/hydroxide#41

[emersion]

Ah, right. Sorry, I didn't remember about this.

[exander77]

This should not be closed as it is not fixed. It relates to this: emersion/go-imap-specialuse#1
And by extension, this is not fixed as well: #21

As per https://tools.ietf.org/html/rfc6154#page-7

When SPECIAL-USE capability is advertised, LIST (SPECIAL-USE) "" "*" must be handled. This is not an optional feature and it means, that it go-imap-specialuse breaks RFC6154. I would extend: https://github.com/emersion/go-imap-specialuse
But, I don't see the ability to extend command and partially handle it in extension, it seems to be all or nothing:

// Command gets a command handler factory for the provided command name.
func (s *Server) Command(name string) HandlerFactory {
  // Extensions can override builtin commands
  for _, ext := range s.extensions {
    if h := ext.Command(name); h != nil {
      return h
    }
  }

  return s.commands[name]
}

SPECIAL-USE is tightly intervened with the LIST and maybe it should be merged into go-imap instead of being an extension.

I did a hackish solution here: exander77@f85eae3

[emersion]

When SPECIAL-USE capability is advertised, LIST (SPECIAL-USE) "" "*" must be handled.

Your link doesn't justify this.

[exander77]

@emersion Why do you think that?

[cketti]

@emersion: I tried to illustrate in this comment that announcing the SPECIAL-USE capability doesn't make sense if all you support is the regular LIST command with special-use attributes. The spec specifically states that this feature is not associated with a capability string:

RFC 6154, section 2:

There is no capability string related to the support of special-use attributes on the non-extended LIST command.

[exander77]

For the extended list command [RFC5258], this extension adds a new
capability string, a new selection option, and a new return option,
all called "SPECIAL-USE". Supporting implementations MUST include
the "SPECIAL-USE" capability string in response to an IMAP CAPABILITY
command.

If the client specifies the "SPECIAL-USE"
return option, the LIST command MUST return the new special-use
attributes on those mailboxes that have them set.

@cketti You covered it well here: emersion/hydroxide#41

I do think it is way worse than it doesn't make sense to advertise it, it breaks RFC6154. The capability SPECIAL-USE is there only and only if the extended LIST command supports SPECIAL-USE. Having special flags for special use folders is no capability at all. Flags are already supported.

Uf, this is going on from 2018? The RFC is extremely clear.

[exander77]

@emersion So... what are we going to do about it? I have a kind of dirty patch (mentioned above) as I don't see an easy way how to implemented using extensions. And I think We can all agree that this is wrong and breaking RFC6154.

Received some comments from Barry Leiba:

I believe that Section 2 makes it clear that it works both ways: the
first paragraph says how special-use behave with non-extended LIST,
and the second paragraph says what it does with extended LIST.

Ah, wait, I see. The example in Section 5.2 says that it's an example of use of extended LIST, but the server doesn't advertise extended LIST... that's the problem. Yes, that's an error in Section 5.2, and this report should be marked "Verified".

(Sorry for the noise: one further comment.) When it's marked "Verified", a comment should be added that it is OK to support special-use without extended list, as specified in the first paragraph of Section 2, but that the example in 5.2 says it's an example of use with extended list, so the example is wrong.

The RFC guy actually confirms it. He is saying that it is ok to support special-use without an extended list and We all agree with that. Special-use folders are just flags. What is wrong is announcing SPECIAL-USE capability which is exclusive to extended LIST supporting SPECIAL-USE selection and return option.

There are 4 different meanings of special-use in RFC6154:

1. there are SPECIAL-USE Metadata for Special-Use Attributes, this relates to METADATA extension in RFC5464, see 4 and 8.6,
2. there is SPECIAL-USE IMAP Capability, see 8.3,
3. there is SPECIAL-USE Selection Option, see 8.4,
4. there is SPECIAL-USE Return Option, see 8.5.

Case 1. is just an extension of RFC5464 and is not announced (these are just flags):

There is no capability string related to the support of special-use attributes on the non-extended LIST command.

Cases 2. and 3. and 4. are bound together:

For the extended list command [RFC5258], this extension adds a new capability string, a new selection option, and a new return option, all called "SPECIAL-USE".

[exander77]

I have been reading more in: emersion/hydroxide#41

And there are misconceptions there:

So the "bug" in the second example is the CAPABILITY list is missing LIST-EXTENDED.
Yes, this is what I reported.

No, there is no bug in the example. RFC5258 clearly states:

Note that extensions that don't require support for an extended LIST command, but use extended LIST responses (see below), don't need to advertise the "LIST-EXTENDED" capability string.

This is exactly the case of SPECIAL-USE as it works without LIST-EXTENDED, but is using extended LIST responses. Also, I would note here, that LIST-EXTENDED does not need to be advertised at all based on RFC5258 (it only extends LIST command).

And:

Seems like K-9 is assuming the server supports RFC 5258 without checking it.

No, SPECIAL-USE implies extended LIST responses as RFC5258 notes. You are confusing extended LIST responses with extended LIST command and its options, but so does 5.2. which text is quite imprecise:

5.2.  Example of an Extended IMAP LIST Command

   This example shows an IMAP LIST response from a server that supports
   this extension.  The client uses the extended IMAP LIST command.

What they mean here is:

5.2.  Example of an Extended IMAP LIST Command 
   or IMAP LIST Command Extended by This Extension Which Is Using Extended IMAP LIST Command Response

   This example shows an IMAP LIST response from a server that supports
   this extension.  The client uses the extended (either RFC5258 or by this extension) IMAP LIST command.

It confuses the reader with Extended LIST Command as defined in RFC5258 and LIST command extended by this extension which is using Extended LIST Command responses.

K-9 does not assume EXTENDED-LIST support but extended LIST responses from a LIST command with SPECIAL-USE selection and return options.

There are commands and there are responses, there are selection options, there are return options and there are capabilities.

[cketti]

K-9 does not assume EXTENDED-LIST support but extended LIST responses from a LIST command with SPECIAL-USE selection and return options.

That's a stretch. K-9 Mail 5.600 does assume the server supports the extended LIST command when the SPECIAL-USE capability is present. It then uses the command LIST (SPECIAL-USE) "" "*". In 5.7xx beta versions the code was changed to check for both the SPECIAL-USE and LIST-EXTENDED capabilities. Due to other changes the issued command itself changed to LIST "" "*" RETURN (SPECIAL-USE).

RFCs are often not great at mentioning error cases. They state what you should do and hope you make the correct assumptions about what not to do. Technically, the behavior of go-imap falls into undefined behavior territory.

RFC 6154, section 2:

An IMAP server that supports this extension MAY include any or all of
the following attributes in responses to the non-extended IMAP LIST
command. The new attributes are included along with existing
attributes, such as "\Marked" and "\Noselect". A given mailbox may
have none, one, or more than one of these attributes. In some cases,
a special use is advice to a client about what to put in that
mailbox. In other cases, it's advice to a client about what to
expect to find there. There is no capability string related to the
support of special-use attributes on the non-extended LIST command.

For the extended list command [RFC5258], this extension adds a new
capability string, a new selection option, and a new return option,
all called "SPECIAL-USE". Supporting implementations MUST include
the "SPECIAL-USE" capability string in response to an IMAP CAPABILITY
command. If the client specifies the "SPECIAL-USE" selection option,
the LIST command MUST return only those mailboxes that have a
special-use attribute set. If the client specifies the "SPECIAL-USE"
return option, the LIST command MUST return the new special-use
attributes on those mailboxes that have them set. The "SPECIAL-USE"
return option is implied by the "SPECIAL-USE" selection option. The
extended LIST command MAY return SPECIAL-USE attributes even if the
client does not specify the return option.

Paragraph 1 states that there's no capability associated with including the attributes in the regular list response. It does not say you MUST NOT use the SPECIAL-USE capability if you only support this mechanism, but not the extended LIST command feature.

Paragraph 2 deals with an extension to the extended LIST command. It doesn't apply to go-imap because go-imap doesn't support the extended LIST command.

Unfortunately, the spec doesn't explicitly spell out what servers have to do when the SPECIAL-USE capability is announced. But I hope there's no doubt that announcing the SPECIAL-USE capability without supporting the extended LIST command (and announcing the LIST-EXTENDED capability) is not a useful thing to do and is just asking for this kind of compatibility issues.

[exander77]

Paragraph 1 states that there's no capability associated with including the attributes in the regular list response. It does not say you MUST NOT use the SPECIAL-USE capability if you only support this mechanism, but not the extended LIST command feature.

You cannot advertise capabilities without actually supporting them. Isn't this obvious? Why are We even talking about this? This is registered IANA IMAP 4 Capability and you can use it only as defined. I can look it up, but it is definitely part of IMAP specs. From the top of my had, you cannot even strap a custom capability there without prefixing it with X-.

For the extended list command [RFC5258], this extension adds a new capability string, a new selection option, and a new return option, all called "SPECIAL-USE".

If what you are suggesting was the case then you can just advertise all registered IANA IMAP 4 Capabilities and issue errors if needed. We may do it in the next release. :)

Capability registry: https://www.iana.org/assignments/imap-capabilities/imap-capabilities.xhtml

[exander77]

An IMAP server that supports this extension MAY include any or all of
the following attributes in responses to the non-extended IMAP LIST
command.

Btw, this talks about supporting the extension, not advertising the capability. There is one thing to support RFC 6154, which you do, and the other thing which is announcing SPECIAL-USE capability.

[exander77]

K-9 does not assume EXTENDED-LIST support but extended LIST responses from a LIST command with SPECIAL-USE selection and return options.

That's a stretch. K-9 Mail 5.600 does assume the server supports the extended LIST command when the SPECIAL-USE capability is present. It then uses the command LIST (SPECIAL-USE) "" "*". In 5.7xx beta versions the code was changed to check for both the SPECIAL-USE and LIST-EXTENDED capabilities. Due to other changes the issued command itself changed to LIST "" "*" RETURN (SPECIAL-USE).

I do not think it is stretch SPECIAL-USE does only require Extended LIST Responses. So this holds. This is more about general handling, the K-9 is just one case. There may be other clients.

I think they could do it better as well. Like fallback...

But this:

LIST (SPECIAL-USE) "" "*"

is not Extended LIST Command (RFC6154). It is standard LIST Command extended by SPECIAL-USE selection option. I can completely forget about RFC6154 and this would be valid based on RFC3501 and RFC6154. I only need RFC6154 for response:

Note that extensions that don't require support for an extended LIST command, but use extended LIST responses (see below), don't need to advertise the "LIST-EXTENDED" capability string.

[cketti]

The standard LIST command (RFC 3501, section 6.3.8, RFC 3501, page 85, list rule) doesn't support options.

RFC 6154 does not create a special version of the LIST command. It (optionally) extends the extended LIST command as defined in RFC 5258. If a server doesn't support the extended LIST command the sections in RFC 6154 about extending the extended LIST command don't apply.

[exander77]

The standard LIST command (RFC 3501, section 6.3.8, RFC 3501, page 85, list rule) doesn't support options.

RFC 6154 does not create a special version of the LIST command. It (optionally) extends the extended LIST command as defined in RFC 5258. If a server doesn't support the extended LIST command the sections in RFC 6154 about extending the extended LIST commands don't apply.

There is some truth to that, but that is not really a problem. The problem is the advertisement for the SPECIAL-USE capability. We don't need to even discuss this if the capability is not wrongly advertised.

My interpretation is that you can support this command without supporting RFC6154 Extended LIST Command, by allowing LIST command to have a single option (SPECIAL-USE). The 5.2. does not refer to RFC6154. So the "extended" cannot be assumed to be Extended LIST Command. You can see that 5.1. refers to RFC3348. So extended in 5.2 refers to extended by this extension.

You are assuming that Extended LIST Command refers to Extended LIST Command defined in RFC6154. But it is not cited.

[exander77]

I think the main problem and confusion here is that it was intended for servers who both support or don't support LIST-EXTENDED capability, but the clients would obviously use Extended LIST Command.

[cketti]

You are assuming that Extended LIST Command refers to Extended LIST Command defined in RFC6154 [I assume you meant 5258]. But it is not cited.

Well, section 2 contains this:
"For the extended list command [RFC5258], this extension adds a new capability string, a new selection option, and a new return option, all called "SPECIAL-USE"

I think the main problem and confusion here is that it was intended for servers who both support or don't support LIST-EXTENDED capability, but the clients would obviously use Extended LIST Command.

If you read it that way, there's indeed some confusion. Section 2, paragraph 1 deals with the response to the regular LIST command. It may include special-use attributes. See example 5.1.

Side note: Please try to be more considerate when posting comments. The fact that you post multiple subsequent comments and edit each of them multiple times is a good indication that your thought process isn't finished when you hit "Comment".

[exander77]

Well, section 2 contains this:
"For the extended list command [RFC5258], this extension adds a new capability string, a new selection option, and a new return option, all called "SPECIAL-USE"

Yes, that is true, it adds those options for Extended LIST Command.

Side note: Please try to be more considerate when posting comments. The fact that you post multiple subsequent comments and edit each of them multiple times is a good indication that your thought process isn't finished when you hit "Comment".

Sorry about this, it is not about the thought process, but that I reread comments several times to remove anything imprecise and make it as clear as is possible. I am never able to do it without at least ten edits.

[emersion]

I'm not convinced by @exander77, however @cketti's comment makes me think I'm wrong. go-imap uses the cap to advertise that it'll return SPECIAL-USE attributes in the regular LIST command, and the RFC citation says it's wrong.

[exander77]

@emersion I agree with @cketti as well. The discussion about extended LIST command is pointless if SPECIAL-USE is not advertised at all. So I would recommend doing what @cketti suggests and just don't advertise it. Or be nice as I suggest and support list by extending LIST by SPECIAL-USE. Both of these will make clients happy.

[foxcpp]

Related: #342

[foxcpp]

@emersion, if we can get this PR merged I might look into implementing extended list support this weekeed.

[exander77]

@foxcpp That would be really nice,.

[foxcpp]

Uhhh, why is this closed again? We have now SPECIAL-USE in go-imap directly but no EXTENDED-LIST yet so... K-9 will still cause errors?

[emersion]

SPECIAL-USE in go-imap doesn't advertise the cap.

[exander77]

@foxcpp SPECIAL-USE capability announcement is disabled. K-9 works without issues.

[foxcpp]

Okay, I see. Going to take a look at what's needed for EXTENDED-LIST...

[emersion]

Note to self: CHILDREN works differently and the cap is advertised.