Distributed Message Transfer Protocol (DMTP)

THIS IS A PRE-DRAFT, PLEASE DO NOT REDISTRIBUTE

Authors: Amir Malik «[email protected]», Drew Wilson «[email protected]»

Created: 2012-08-01

Preface

TODO: motivation

TODO: http://blog.superfeedr.com/federated-follow-button/

Introduction

The objective of Disitributed Message Transfer Protocol (DMTP) is to transfer messages reliably and efficiently. The DMTP design is heavily inspired by Simple Mail Transfer Protocol (SMTP) and Internet Mail 2000 (IM2000).

Over the past decade, email use has skyrocketed with an estimated 1.5 billion email accounts worldwide. Unsolicited bulk email (also known as spam) has also kept up with growth; many attempts have been made to eradicate it, initially with filtering capabilities in Mail User Agents (MUA), and more recently with more advanced statistical filtering systems implemented at the Mail Delivery Agent (MDA) layer.

In the last decade, we have also seen immense growth in social networking services such as Facebook, Twitter, Google+, Instagram, to name a few. None of these services have taken full advantage of SMTP for cross-service federation or communication due to lack of widely deployed standards for transmission non-email data, overly chatty nature of SMTP resulting in poor performance over high-latency links, and no standards for social graph manipulation.

DMTP addresses these points and provides a specification that immediately complements SMTP implementations, but allows for complete migration away from SMTP at some point in the future. Although DMTP is not backwards-compatible at the protocol level with SMTP, many of the same concepts are borrowed, the most viisble of which is the email address.

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

The DMTP Model

The DMTP design is based on a publish/subscribe (PubSub) model of communication with an inversion of message storage responsibility. The four basic units of the DMTP model are: users, messages, channels, and notifications.

TODO: Message Storage (Archive?) Agent (MSA) TODO: Message Transfer Agent (MTA, formerly Mail Transfer Agent) TODO: Message URI TODO: Distributed Message Storage Network (DMSN)

A user is identified by a DMTP-addres of the format “user@domain”. It is RECOMMENDED that sites providing both SMTP and DMTP service to use the same addresses for both services to avoid confusion.

A message MUST contain the sender’s DMTP address, one or more recipient addresses, date, an OPTIONAL signature, and at least one of the following fields: subject, message, attachment. A field MAY appear more than once to allow for multiple content types to be sent. For example, two “message” fields, one for each content type, “text/plain”, and “text/html”, may be present. The DMTP-sender SHOULD ensure that rendered versions of each instance of a field convey the same meaning to the recipient.

A channel is where messages arrive on the DMTP-receiver’s side. When sending a message, the DMTP-sender MUST specify a channel (see section ###Well Known Channel Identifiers###) for each recipient. Channels are loosely analogous to ports in Transmission Control Protocol (TCP), except that there are no limits on the number of channels. Implementations MUST support at least one channel per user.

A notification is an asynchronous event that is sent by the DMTP-sender to a recipient, informing them of a pending message for them. Because DMTP is based on the pickup model rather than delivery model, a notification is the only way in which a DMTP-receiver can know of a new message. A notification consists of one or more URIs pointing to the message. See the section ###DMTP Message URIs###.

Thus to send a message, the DMTP-sender MUST store the original version of the message on their DMTP Message Storage Agent (MSA), which may provide multiple fetch URIs for message retrieval. The MSA SHOULD store messages in the DMSN to avoid the thundering herd effect choking the DMTP-sender’s server when a message is retrieved by many clients at the same time.

The DMTP Procedure

In order to send a message, the DMTP-sender needs to determine the address and port of the MUST extract the hostname portion of the recipient’s DMTP address and perform a DNS lookup for the hostname’s _dmtp._tcp SRV record.

For example, to find the DMTP server for the DMTP address [email protected] the DMTP-sender needs to query _dmtp._tcp.example.com.. If mulitple SRV records are returned, DMTP-senders SHOULD use the priority and weight fields in each SRV record to distibute load across multiple servers, and to retry connections to another server when encountering an unresponsive one.

Upon receiving a new connection, a DMTP server MUST send a greeting banner consisting of a dictionary identifying any protocol limits that need to be observed. The DMTP-sender SHOULD store the contents of the dictionary in some form for the duration of the connection.

At this stage, the client MAY attempt to send a STARTTLS command to intiate the Transport Layer Security (TLS) handshake. The server MAY refuse to allow it, but it is RECOMMENDED that TLS be enabled on all DMTP servers to avoid Man-In-The-Middle (MITM) attacks. After a successful TLS handshake, the server MUST retransmit a greeting banner including the dictionary. The client MUST discard the previously received dictionary. This gives DMTP-senders the ability to use client-side certificates and allow the server to customize protocol limits on a per-certificate basis.

Once the greeting has been received, the DMTP-sender MAY issue one or more of the following commands:

• POST to send a notification to one or more users
• CHANNELS to retrieve a list of channels for one or more users
• NOOP to send a keep-alive to avoid connection timeouts
• QUIT to close the connection

The server MAY impose limits on the number of commands that can be sent at once (pipelining) without waiting for acknowledgments. The client MUST NOT attempt to send more commands than specified the server’s protocol limit dictionary. However, it is RECOMMENDED that DMTP-senders always pipeline as many commands as possible to reduce the effects of connection latency.

DMTP servers MAY process received commands in a pipeline in any order, but the responses to each command in the pipeline MUST be in the order that they were sent. Furthermore, DMTP servers MUST guarantee that any commands and data are safely stored for processing before sending acknowledgments to the client. When pipelined commands are received, the server MAY delay storing data until the pipeline is either full or a certain time interval has elapsed since the last command was received. In any case, servers SHOULD minimize delays to avoid blocking the client’s command queue.

The DMTP-sender MUST be prepared to handle a number of responses:

• I indicating general connection status information, such as throttling
• X indicating no more commands should be sent after the current pipeline (if any) has been acknowledged
• K indicating success of the last unacknowledged command
• E indicating a fatal error in the last last unacknowledged command, which the client SHOULD NOT retry again
• Z indicating a temporary failure of the last unacknowledged command, which the client SHOULD retry later

DMTP clients and servers SHOULD reuse connections to avoid connection overhead. Clients SHOULD use the NOOP command to send keep-alive commands, but up to the maximum number specified in the protocol limits dictionary. If there is no activity for a time interval, either side MAY close the connection.

The DMTP Specifications

DMTP Encoding

TODO: use BitTorrent encoding from http://bittorrent.org/beps/bep_0003.html

Greeting Banner

Commands

STARTTLS

POST

CHANNELS

NOOP

QUIT

Replies

I - Informational

X - No More Commands

K - OK

E - Fatal Error

Z - Temporary Error

Appendix

Protocol Limits Dictionary

Well-Known Channel Identifiers

inbox

This is default channel.

smtp

This is an OPTIONAL channel to be used by SMTP-to-DMTP gateways and for any SMTP-originated message being delivered to a DMTP address.

References

IM2000, “Internet Mail 2000”, Dan J. Bernstein, «http://cr.yp.to/im2000.html>>.

SMTP, “Simple Mail Transfer Protocol”, J. Postel, «http://tools.ietf.org/html/rfc821>>.