Re: authenticated smtp problems

[Kyle Wheeler <kyle-mutt@xxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wednesday, May 20 at 03:06 PM, quoth dv1445@xxxxxxxxx:
> One of my email providers just migrated to a new email system 
> (Zimbra).  With just a tiny adjustment in my muttrc (change from 
> imaps://old.domain.com to imaps://new.domain.com) I seem to be 
> able to read mail just fine.

Good! As would be expected.

> They require authenticating smtp with ssl,

There's multiple ways of doing that. Are they any more specific?

Before we get into this, let me explain the basic options here. There 
are three different protocol-level variations on basic SMTP service: 
SMTP, SMTPS, and Submission. The first, plain old SMTP on port 25, is 
unencrypted. Some (many, these days) servers support the STARTTLS 
command, which allows the connection to be encrypted AFTER it has been 
established. The second, SMTPS, is encrypted from the very first 
packet. As SMTP is supposed to be on port 25, SMTPS is supposed to be 
on port 465. There is no unencrypted version of SMTPS; it's *always* 
encrypted. The third, Submission, is virtually identical to SMTP, 
except that you must authenticate yourself (in a manner identical to 
SMTP-AUTH) before you can submit email. It is ALSO unencrypted, and 
ALSO usually supports the STARTTLS command to begin encrypting the 
connection after it has been established. Because it is so similar, 
this is almost always handled by telling the sending application to 
use SMTP on an alternate port (namely, the Submission port, 587).

Generally speaking, SMTP clients detect the availability of the 
STARTTLS command and opportunistically use it; that applies to both 
SMTP and Submission, since they're so similar.

> so I have smtp_url=smtps://user@xxxxxxxxxxxxxxx  I keep getting 
> "connection refused".

Right, because that tells mutt to use SMTPS, which uses port 465 (and 
it seems to be a rare email provider that supports SMTPS). It would 
appear that your service provider does not provide the SMTPS service.

> Connection failed. errno: 61...  Could not connect to 
> new.domain.com (Connection refused). 
> Connected to new.domain.com:465 on fd=-1 mutt_socket_close:

Translation: mutt attempted to contact them on port 465 and could not 
even establish a connection. Further translation: they do not support 
SMTPS.

> If I try to put port 587 into smtp_url, I get the following:

Your alternative is to use the Submission protocol? ... let me guess, 
you told mutt to use SMTPS on the Submission port, rather than telling 
mutt to use SMTP on the Submission port.

> SSL failed: error:140770FC:SSL 
> routines:SSL23_GET_SERVER_HELLO:unknown protocol 
> Connected to new.domain.com:587 on fd=-1 mutt_socket_close:
> Attempt to close closed connection.

Looks like I guessed right. Your mistake was to use this:

     set smtp_url=smtps://new.domain.com:587

That tells mutt to use SMTPS on port 587. Since that means mutt 
expects the very first packet to be encrypted, and the very first 
packet WASN'T encrypted, the "connection" failed. If you had done 
this:

     set smtp_url=smtp://new.domain:587
  
...it probably would have worked. Your mistake is assuming that 
"smtps://" means encryption and that "smtp://" means no encryption. In 
reality, "smtps://" means the SMTPS protocol and "smtp://" means the 
SMTP (or Submission) protocol.

Does that make sense?

> I don't understand what's going on.  Apple Mail can send mail 
> with the new system just fine, using authenticating SSL smtp 
> server,

That's... descending into gibberish. But chances are Apple Mail knows 
that port 587 ought to be treated the same as port 25.

> as can Thunderbird (in Mail.app, the box for Use SSL is checked, and 
> next to "Authentication" it says "Password".

Since most people don't care about the difference between SMTPS and 
SMTP+STARTTLS, Apple uses the "Use SSL" option to mean both, with the 
specifics depending on the port specified. It's a user-friendliness 
thing, rather than a "here's how experts think about it" thing.

Thunderbird is usually a bit more pedantic about this sort of thing, 
but I wouldn't be surprised if it works for a similar reason.

Now, their "user-friendliness" means that they would be a bit harder 
to use if you wanted to do something like use the SMTPS protocol on an 
alternate port, or use SMTP on the SMTPS port, but since nobody wants 
to do that, they get away with hiding the details. Mutt, on the other 
hand, is very literal.

> I even tried reverting to msmtp, but couldn't get that to work at 
> all. Msmtp would tell me that the server doesn't understand TLS,

Interesting. Am I correct in assuming that msmtp is connecting to port 
25?

> but if I have tls off and auth on in the msmtprc, mutt complains 
> that the server doesn't understand authentication!

Well, that's easy to explain: most intelligent servers don't advertise 
authentication unless you're in an encrypted connection. Thus, without 
TLS, the server will deny that it knows anything about authentication.

> Oddly I can send mail from mutt *unauthenticated*, using 
> smtp://new.domain.com

That's not necessarily odd, depending on how you tested it and what 
your situation. For example, 100% of mail servers allow you to send 
mail to LOCAL USERS without authentication from just about anywhere. 
Thus, testing it by sending mail to your own email address without 
authentication will succeed. The reason is because it's equivalent to  
joe-shmoe (e.g. ME) sending mail to your address---I don't have to 
authenticate with your server, obviously, right?

Also, in lots of places (like universities or companies) where there 
is an internal network with a fixed range of IP addresses, they allow 
anyone within their network to use their mail server without 
authentication.

> Mutt is getting hung up on securely using smtp.  I would like to do 
> it securely, not only because my provider says we have to, but 
> because I'll bet a lot of money the server won't let me do it sans 
> authentication when I'm on the road (which I haven't been able to 
> test yet).

One would hope!

> I cannot tell if this is a bug in the new system, in mutt, or with 
> my setup.

It's most likely a problem with your setup. To recap, try this:

     set smtp_url=smtp://new.domain.com:587

I'm assuming that you're handling the smtp username and password 
elsewhere (e.g. via smtp_user); otherwise, you'll need to set that up 
as well.

~Kyle
- -- 
Mathematicians stand on each other's shoulders while computer 
scientists stand on each other's toes.
                                                       -- R. W. Hamming
-----BEGIN PGP SIGNATURE-----
Comment: Thank you for using encryption!

iQIbBAEBCAAGBQJKFGNgAAoJECuveozR/AWeeagP90qBWUZy4pRPlcbK/EBego9s
ZzKsH7q0u+IlHnAxzFMD+xPSmoQHUOVLV6CvL6t4Ax19YTipLaHZliU8xsxm19fy
7einPyD41XyeKDoI40ELBL/1IPhM3DuW2j2UPbrfK/K8BNtcwyY/sfex+bmqN94G
L37d4g3vr1WxzehaKxMlZVAPCcvy4xqoBAgtspKa+J7rvdZhbuJzktSrzreIa9Ey
1uUTZ5cvKFQj4xiyc/8APRcV62dMzWYgD2+2W8IrD/vcS9jLhal1+rFKPzjrlywS
NSkIvgD6//ZFb1NHOa5yyMp5VncBxHMKuL3swa6ZAu03WT7M8E7KJSjp9TXlWR6s
7yjWqx+34/GDjDBuMAoqOTv17lBiGfkwr3EQs40qwTRZUjfpVd7HskRwBNYhgFsF
sWmHdS1tLc5EOF9bnSdg+rclATEn+Pu1GJKtASl3MIvQamXiy/wql2OqcPjaEV52
Z+6UxQtl6N9nLza3PnDVM6bjejMhUCQSAKFKxI6OsmiaVXlgh+gBKQcTmBIdTa3z
4TfteqISs9HBYEh3kb54AeA4Icf+FYkxgWnmbS+5PVcing83QQy+CBpm4IHcc5Jz
GliKroKgaM5f1MWnH7evTwYDmKyjZcPbnIKwpCUqlLZyc1TweK7vOn4+8dYfZXsV
/fmIScj5FW+galc1fWM=
=tySY
-----END PGP SIGNATURE-----