Fallback from the Submission port to the SMTP port
Kazu Yamamoto
IIJ Research Laboratory
Written: November 9, 2005
Modified: December 28, 2005
Abstract
To solve the problem of spam messages, a mail-reader should use the Submission port (587) instead of the SMTP port (25), which is used currently, when sending a message. The author would propose to change a mail-reader so that it first tries the Submission port then makes a connection to the SMTP port in the case where the Submission port is not available. It is shown that this change does not give a significant impact to its users.
Background
Spammers use "bot"s (or zombies) or walk around ISPes whose restrictions are loose, and make an SMTP connection to a target mail server from their PCs then send messages. They repeat this procedure to send massive spam messages.
To prevent this, some ISPes blocks the outbound SMTP port at their boundary, and the number of such ISPes is increasing. This workaround is called OP25B (Outbound Port 25 Blocking).
ISPes with OP25B, of course, configure their routers so that their sending server can make SMTP connections to other ISP's receiving server. In this environment, if users of such an ISP use the sending server of the ISP, no problem happens when sending messages.
However, some users have legitimate accounts of other ISPes/ASPes, and use their sending servers when sending messages. This communication is blocked as spammers communication is so. To rescue such users, it is proposed to separate SMTP to "submission" and "transfer". "Submission" means message deliver from a mail-reader to a sending server while "transfer" refers to message deliver from a sending server to a receiving server.
For both submission and transfer, SMTP is used. For submission, port 587 is used as the Submission port to get over OP25B. For transfer, port 25 is used as it used to be.
If we change port number only (from 25 to 587), spammers would also use port 587. So, it is obvious that ISP should require user authentication (SMTP AUTH) for submission. If domain authentication is deployed for transfer, we can provide an ideal mail environment where mail addresses cannot be faked (see the figure blow).
The number of ISPes which provide the Sumission port is increasing. In contrast, there is no mail-reader which uses the Submission port by default at this moment.
Problem
Most mail-readers can change the port to be used for sending at this moment. However, with the current situation where it is users that configure the port, the deployment speed of the Submission port would be slow. To accelerate the deployment, it is desired that mail-readers use the Submission port by default.
If a mail-reader is released with the Submission port enabled, the mail-reader fails to send a message in the environment where its sending server does not provide the Submission port.
Fallback and its Technical Consideration
To solve the problem above, the author would propose a fallback mechanism. That is, a mail-reader first tries the Submission port. If the port is not available, the mail-reader makes a connection to the SMTP port.
Let's observe how this mechanism works with cases.
1) The Submission port is available
No problem. This mechanism has a big advantage in the OP25B environment.
2) TCP RST returns
The mail-reader is kept waiting for the time of RTT, and can fallback. Users could not recognize. Most ISPs could be categorized to this case.
3) ICMP hard error returns
The mail-reader is kept waiting for the time of RTT, and can fallback. Users could not recognize. This case would happen if a firewall filters the Submission port.
Here are summary of ICMP hard errors:
- ICMPv4 Destination Unreachable
- code 2 = protocol unreachable
- code 3 = port unreachable
- code 4 = fragmentation needed and DF set
- code 6 = destination network unknown
- code 7 = destination host unknown
- code 8 = source host isolated
- code 9 = communication with destination network administratively prohibited
- code 10 = communication with destination host administratively prohibited
- code 11 = network unreachable for type of service
- code 12 = host unreachable for type of service
- ICMPv6 Destination Unreachable
- code 1 = communication with destination administratively prohibited
- code 4 = port unreachable
4) ICMP soft error returns
ICMP soft error is produced by routing problems. During the problems, communication is impossible. Even mail-reader first tries the Submission port, there is no difference. That is, communication is impossible. We can consider that this is a rare case.
Here are summary of ICMP soft errors:
- ICMPv4 Destination Unreachable
- code 0 = net unreachable
- code 1 = host unreachable
- code 5 = source route failed
- ICMPv6 Destination Unreachable
- code 0 = no route to destination
- code 3 = address unreachable
I'm afraid that this is not directly relating to the proposed method, though, let me explain how TCP reacts when ICMP soft error returns with two sub-cases:
4.1) Old TCP Implementations
They treat soft errors literally as soft errors before a TCP connection is established. If TCP stacks has a retransmission limitation, this limitation is reached. In the case of BSD, it takes about 10 seconds. If TCP stacks does not has a retransmission limitation, we need to wait for TCP timeout.
4.2) New TCP Implementations
They treat soft errors as hard errors before a TCP connection is established. The mail-reader is kept waiting for the time of RTT, and can fallback. Users could not recognize. Linux adopts this mechanism.
5) TCP timeouts
Mail-readers is kept waiting for the TCP timeout, and fallbacks to the SMTP port. This case would happen if a firewall silently discards packets destined to the Submission port.
Here are summary of values of TCP timeout for each OS.
| OS | Value of TCP timeout |
| BSD | 75 seconds |
| Linux | 189 seconds |
| Solaris | 225 seconds |
| Windows XP | 21 seconds |
Even in this case, if mail-readers can send messages in background (with non-blocking connect()), users feel that it takes a little bit more time to deliver messages. If OSes have a long timeout value like Solaris, it would be a good idea that mail-readers can set a shorter alarm timer.
Also, mail-readers can negatively cache the fact that fallback happened, and they might use the SMTP port only from the next time. Or perhaps, mail-readers might show dialog and ask a user what he wants.
Note that this is not relating to "greet pause" which is getting popular recently. "Greet pause" is done in the SMTP level while this sub-section talks about the TCP level. (To implement "greet pause", a server waits for some seconds after accept() and write() a greeting. Thus, from the client point of view, connect() is not blocked and read() is blocked.)
Implementation and field test
The author has already implemented the fallback mechanism in background on Mew and confirmed that the proposed mechanism works exactly as consideration. Also to figure out percentage of each case, the author has carried out a field test.
The number of report is 39. The number of servers which indicates "localhost" or a server in a home is 7. This is not a typical usage. Subtracting this number from 39 results in 32. This includes reports whose both user and server are the same but clients locate different places. Investigating these report carefully, it appears such cases can be considered identical. Removing the redundancy from the server/user pair point of view, the total number becomes 26.
This is not statistically-meaningful however we can see a kind of tendency.
The following table describes breakdown of servers:
| ISP | 16 |
| ASP | 1 |
| Company | 6 |
| University | 2 |
| Unknown | 1 |
The following table shows breakdown of the cases:
| Case 1 (Submission) | 12 |
| Case 2 (TCP RST) | 7 |
| Case 3 (Hard error) | 0 |
| Case 4 (Soft error) | 0 |
| Case 5 (TCP timeout) | 4 |
| SSH(22) | 2 |
| SSL(465) | 1 |
Things learned:
- Case 5 exists and we cannot ignore. From the results above, its percentage is about 15%. So, mail-readers should set a short timer to get around a long TCP timeout of OSes.
- The Submission port has been deployed more than we expected. From the results above, its percentage is about 46%. Unfortunately, since many servers provide the Submission port without user authentication, spammer would probably abuse it.
- Mail-Readers should accomplish user authentication according to the AUTH capability which servers return. Mail-Readers should be implemented so as to handle the case where user authentication is required for the Submission port and it is not required for the SMTP port after fallback.
- Since there are servers which return the AUTH capability even though they does not support user authentication, mail-reader should provide a feture to disable user authentication.
- In addition to fallback in the port level, fallback from IPv6 to IPv4 is necessary. In other words, mail-readers should support fallback: Submission/IPv6, Submission/IPv4, SMTP/IPv6, and SMTP/IPv4 in order.
Conclusion
With the fallback mechanism, users are not effected significantly even if the Submission port is used by default.
For mail-readers which can send messages in background, there is no significant impact to users.
For mail-readers which can send messages only in foreground, it takes a time to fallback from the Submission port to the SMTP port in particular cases. However, the time can be shorten by the mail-readers. Also, the mail-readers can use the SMTP port only from the next time.
Note
Fallback is a common idea in the dual stack environment of IPv4 and IPv6. You can find thorough consideration on fallbak in Cho's article.
Appendix A: Implementation in Mew
Emacs 21 provides blocking connect() only. Fallback from IPv6 to IPv4 is done in the C level if blocking connect() is used. Emacs 22 implements non-blocking connect(), too. Unfortunately, due to implementation limitation, non-blocking connect() does not support fallback from IPv6 to IPv4.
Thus the author adopted a method that specifies an address family in the Emacs Lisp level when non-blocking connect() is used so as to ensure what kind of address family is used in the C level.
At this moment, the algorithm implemented in Mew is as follows:
- Non-blocking connect() to the Submission port with IPv4 specified
- Timer is set to 10 seconds
- Blocking connect () to the SMTP port if the above case fails
- Fallback from IPv6 to IPv4 in the C level
If we prepend "non-blocking connect() to the Submission port with IPv6 specified" to the first, the algorithm becomes symmetric. Since there are many situations where time out becomes longer, the author does not adopt this at this moment.
The algorithm above is implemented so as to co-exist TLS and/or an SSH tunnel. Also, if SSL is used, blocking connect() is used to the 465 port and fallback to the SMTP port is not carried out. (The port of SMTP over SSL is NOT officially assigned. The 465 port is carelessly used and has become de-facto standard. Note that the 465 port is officially assigned to a protocol provided Cisco Systems.)
Appendix B
Windows has a mechanism that a security software investigates all communications of any applications. It would be practical that a security software relays an SMTP connection from a mail-reader to a Submission connection. With this method, a security software must implement fallback.