<b>1. What is RetchMail?</b>
Eric S. Raymond's Fetchmail is a full-featured, robust, well-documented
remote-mail retrieval and forwarding utility intended to be used over on-demand
TCP/IP links (such as SLIP or PPP connections). It supports every remote-mail
protocol now in use on the Internet: POP2, POP3, RPOP, APOP, KPOP, all flavors
of IMAP, ETRN, and ODMR. It can even support IPv6 and IPSEC.
Oh, you were asking about RETCHMAIL. Sorry.
RetchMail is the world's most stupidly fast POP3 retriever.
It is almost completely unlike fetchmail: RetchMail lacks features, isn't
particularly robust (although it won't delete a message until sendmail says it
was delivered okay), has nearly no documentation, and is actually fast.
Thanks to PatrickPatterson's WvSSLStream, RetchMail also supports POP3-SSL,
which is much more secure than the RPOP, APOP, KPOP and other password
obfuscation routines currently en vogue to hide your password. SSL has the nice
feature that not only is your password secure going across the wire, but so is
the rest of your mail.
<b>2. Where can I get it?</b>
You can download the latest and greatest version of RetchMail at
DownloadReleases. For the latest bleeding edge development (and complete
development history), see [wlach]'s [http://github.com/wlach/retchmail
repository] on github.
<b>3. What exactly do you mean by "stupidly" fast?</b>
Well, retchmail was written because I was annoyed with waiting for my e-mail to
download. I have a pretty active email box (several, actually), and I often have
to retrieve my mail through a rather high-latency Internet connection from
whatever odd places I might be in at the time. Traditional POP retriever
programs were slowly driving me crazier.
Here are some reasons that RetchMail seems to be and actually is faster than any
other mail retriever I know:
- RetchMail downloads from multiple mailboxes simultaneously; if you have
several email accounts, like I do, you don't have to wait for one download to
finish before starting the next one. If you want, you can deliver mail from each
mailbox into a different local mailbox. If one of your mail servers is slower
than the others, it can download in the background while your faster servers go
faster.
- RetchMail can be waiting for one message delivery to finish while it starts
the next one. Since mail delivery programs tend to be ultra-paranoid and fsync()
and run the disks a lot, you'd be surprised at how much faster this can make
things, without making them less safe.
- RetchMail downloads smaller messages before larger ones, without disturbing
the message order too much. This means you can start reading your small messages
before the giant attachment has finished downloading. However, retchmail only
reorders messages when it's actually useful, so your mailbox stays in mainly
chronological order.
Special bonus feature: absolutely nowhere in RetchMail's code is there an
arbitrary 3-second sleep(). Wow! What other mail retriever can say that? (Hint:
not fetchmail.)
<b>4. That sounds fast, but not "stupidly" fast. Are you forgetting to tell me
something?</b>
You caught me. The main reason RetchMail is faster - often more than 5x faster
than fetchmail - is that it pipelines message retrievals. That means it asks for
several messages from the server at once, rather than requesting a message,
waiting for it to download, deleting it, waiting for the delete request to be
acknowledged, and requesting the next one. For a mailbox with lots of short
messages (the common case) the request-wait-request-wait style of transaction is
_hideously_ slow, because no messages are being sent (the link is idle) in
between the previous message being sent and the next request being received.
If you do the math, you'll find that for the common case of a relatively slow or
overloaded Internet connection with relatively small messages, _most_ of the
time it takes to download your mailbox is actually spent just waiting while no
data is transferred.
Instead, RetchMail requests multiple messages at once, so it always has several
messages "in flight" at once. For example, it requests messages 1 through 5, and
the server starts sending those in order. When retchmail has finished receiving
message #1, it then requests message #6, and so on. The mail server won't have
finished sending message #5 by that time, so the download part of the connection
is _never_ idle after the transfer starts.
All that makes RetchMail go REALLY fast. Almost everyone will find RetchMail to
be at least 3x faster than most POP retrievers, and probably more than 5x
faster.
<b>5. Is RetchMail standards-compliant?</b>
Quick answer: HA HA HA HA HA HA no.
Longer answer:
I've read RFC:1939. As far as I can tell, nothing in there says we can't do what
we're doing. That said, I know perfectly well that what we're doing is
definitely NOT what the authors intended. So we're in compliance with the
letter, but not the spirit, of the standard.
Just to be extra clear about this: yes, it is morally wrong for us to have
written RetchMail, and it is morally wrong for you to use it. But try it, it's
really fast!
<b>6. Will the lack of standards-compliance cause problems?</b>
Maybe, if your POP server is badly written. It wouldn't surprise me terribly
much if this were the case.
The most likely type of bug will be "failure to split packets" or "failure to
merge packets." Most network programming libraries (not WvStreams) make it kind
of hard to do simple, obvious things like reading a single line of text, and
most people writing network code tend to screw it up. Your POP server is likely
to screw up in one or both of two ways:
- Failure to split packets: if extra commands arrive while data is being
transferred, or two commands arrive at the same time, stupid POP servers (since
they only ever expect one command to be sent before they send a response code)
might ignore or give an error to the unexpected commands.
- Failure to merge packets: retchmail sends several commands at once, so the
data it produces might take more than a single network packet to transmit. If
this happens, the packets will probably be split in the middle of a line, so
that (say) "RETR 5" is at the end of packet #1, and "6\r\n" is at the beginning
of packet #2. Stupid POP servers might not be able to merge those two requests.
The problem you're more likely to experience is the first one, since RetchMail
mostly doesn't produce huge packets. However, your POP server is probably more
likely to actually _contain_ the second problem, since it's harder to test
against.
<b>7. Do you support IMAP or any non-POP3 protocols?</b>
No, although it wouldn't be very hard to add. Thanks to the handy-dandy (if I do
say so myself) WvStreams library, retchmail is only 800 lines, so it's pretty
easy to extend and understand. (Fetchmail is more than 20000 lines - 25 times
bigger - but they didn't have the benefit of WvStreams or C++.)
In fact, IMAP properly supports pipelined requests, so it would even be
standards-compliant to do what retchmail does if we used IMAP.
If you want this feature, write it and submit a patch.
<b>8. Why do you call the local MTA instead of implementing the entire SMTP
protocol inside RetchMail? The fetchmail design notes said...</b>
Because that's what the local MTA is for. If you don't want a "full" MTA, you
should still install a basic one so other packages will work. "sSMTP" is a nice,
simple, basic /usr/sbin/sendmail implementation that does exactly what
fetchmail's does, but other programs can use it too.
<b>9. Okay, then, what do you think about "syntactic noise" in config files?</b>
!@#*@!#(*#%^!%$@#';,!@#
<b>10. Are you going to write a long-winded essay about the relationship between
this program and sociology?</b>
I actually rather liked "The Cathedral and the Bazaar," and it doesn't have
serious latency problems, so I don't think there's any need to duplicate that
effort.
In fact, I'll just link to it here:
http://www.catb.org/~esr/writings/cathedral-bazaar/
Eventually I _will_ write a long-winded essay about fixing latency problems.
<b>11. Are you at least tracking the number of users of retchmail?</b>
As far as I know, retchmail has exactly three users.
<b>12. And you'll put that in your essay?</b>
No.
<b>13. Who's responsible for this madness?</b>
First, Eric S. Raymond wrote fetchmail, which was really the inspiration for
retchmail. And I mean that in the nicest possible way.
Avery Pennarun ([apenwarr]) wrote the first version of retchmail and started
this FAQ. When I say "I", I mean me, Avery.
Patrick Patterson ([ppatters]) added the SSL support and keeps track of public
releases, since Avery is too busy/lazy.
IMPORTANT NOTE: - RetchMail requires WvStreams - please make sure that you have
this library installed prior to installing Retchmail.
<hr><b>Anonymous@80.58.1.235 (2003/08/31)</b>:
We want IMAP!
We want IMAP!
We want IMAP!
:)
<hr><b>Anonymous@24.223.189.214 (2003/11/22)</b>: does this work with exchange
and catchall pop3 accounts?
- <b>[jim] (2003/11/23)</b>: RetchMail does pop3, so if exchange does pop3
RetchMail should work. As for catchall accounts, nope it doesn't work. We'd
take a patch to make it work if you have one.
<hr><b>Anonymous@212.182.115.22 (2004/11/17)</b>:
Fails when password contains "#"
<hr><b>Anonymous@138.130.22.114 (2005/05/06)</b>:
Help! I've read everything I could find on configuration - which is not much.
How do I configure retchmail to collect mail through popfile and hand it on to
procmail? (I think I have the popfile side sorted).
<hr><b>Anonymous@219.95.95.110 (2005/08/05)</b>:
I'm having the same problem as Anonymous@138.130.22.114 above.
<p>How do I use procmail with retchmail? This is what i have in retchmail.conf:
<pre>
[POP Servers]
user@mailserver = password
[POP Targets]
user@mailserver = procmail
</pre>
<p>
This is the output I got when I run retchmail:
<pre>
A* Retrieve mail from user@mailserver into procmail.
A Recv(HELLO): +OK ready
A* 2 messages in 6433 bytes.
A [1] 2.1k <address@somewhere> test again
A [2] 4.1k
A* Error delivering message 1 to sendmail.
A <[snipped]@yahoo.co> GeForce 4 MX 400
A* Error delivering message 2 to sendmail.
A* Done.
</pre>
<hr><b>Anonymous@69.194.228.44 (2005/08/24)</b>:
Anon 219.95.95.110: actually I don't use retchmail yet,
I am just interested in it - but the answer to your "problem" is simply to
configure sendmail or whatever
compatible program is on your machine, to deliver through
procmail. They are ready for it, have been doing it since
procmail was born, so why duplicate that functionality
in retchmail?
<hr><b>Anonymous@203.114.131.188 (2006/03/22)</b>:
In response to question 5 (standards compliance),
<a href="http://www.faqs.org/rfcs/rfc2449.html">RFC 2449</a> [POP3-EXT] defines
pipelining as an optional capability. Therefore, if RetchMail were to issue the
CAPA command and recieve a PIPELINING response, it would be in compliance with
the "spirit" of the standard and be sure that pipelining will work with that
particular server.
Of course, given Retchmail's goal, it will probably choose to go ahead with
pipelining regardless of the server's response ;-)
- <b>[dcoombs] (2006/03/23)</b>: In four years of using it, we haven't met a
single POP3 server that RetchMail couldn't handle. Or vice versa. I guess that
means we're doing something right! Or everything not wrong enough!
- <b>Anonymous@203.114.131.240 (aka 203.114.131.188) (2006/03/23)</b>: Yeah, I
kind of figured that. I was half-joking that if Retchmail were to issue one
additional command at startup, all the standards-compliance naysayers could shut
up. ;-)
I am considering using Retchmail myself, as an alternative to Fetchmail (I'm a
little put off by its history of security-critical bugs, e.g. buffer overflows),
to set up a Debian mail server on my home LAN. I notice that Retchmail 1.0 has
been released, but Retchmail 0.9 is still in Debian Stable ("sarge"). Would
there be any advantage to downloading source for Retchmail 1.0 (and WvStreams
4.2.2) from Debian Testing and compiling it to run on Sarge?
- <b>[dcoombs] (2006/03/24)</b>: There were no super-duper critical fixes in
1.0, although there were a couple crashes and suchlike. You might as well start
with 0.9 since it's easy to get with apt, and then try 1.0 if you run into
trouble. The important thing is that they're both really fast!
<hr><b>Anonymous@69.194.204.172 (2006/08/16)</b>:
how does the SSL support work? Through STARTTLS (good) or through "alternate
port", i.e. assuming port 110 is plain POP3 and port 995 is POP3s? fetchmail
seems to do the latter and that's the current reason why I'm trying to replace
it, not that I don't have other reasons as well.
- <b>[pmccurdy] (2006/08/16)</b>: It appears to use the "alternate port" method.
If you connect to port 995, it uses SSL, otherwise it doesn't. It doesn't look
like it would be very hard to add STARTTLS support though, if you wanted to
write a patch.