In my many columns about what it will take to stop spam, including the most recent one about why greed is interfering with true progress, my primary message has always been that we're not going to get very far until the various providers of e-mail solutions and services agree to new interoperable e-mail standards.

Interoperable standards ensure that all systems capable of sending or receiving e-mail are in agreement over the different pieces of information required to complete an e-mail transaction, the way in which each of those "fields" is formatted (e.g., dates can be formatted in many different ways), the order of those fields, and how those fields are packaged for transmission across a network. Such standards are referred to as protocols. Although every e-mail sent over the Internet involves many such interoperable protocols, the one that deceptive senders of unwanted e-mail have had the easiest time manipulating is called the Simple Mail Transfer Protocol (SMTP).

Most manipulations disguise an e-mail sender's identity and location by falsifying the data found in the fields - collectively referred to as the e-mail header -- reserved for identifying the precise sources of such transmissions. In other words, the Internet has a real identity crisis on its hands.

All members of the e-mail ecosystem with whom I've spoken agree that the first step towards objectively determining the legitimacy (as defined by each recipient) of an e-mail is guaranteeing the authenticity of its origin. In other words, is the e-mail from whom it says it's from? The sooner we can verify the sender's identity, the sooner our systems can be reliably programmed to accept or refuse mail on the basis of whom it came from while simultaneously guaranteeing that the e-mail we send won't itself be denied safe passage on the basis of subjective criteria. The number one reason that legitimate e-mails don't reach their intended destination is because they're falsely classified as spam by an anti-spam technology that, in the absence of verifiable credentials, must resort to more subjective criteria.

To get from where we are today to the point where recipients can verify an e-mail's credentials, the sending systems must know something about the technical expectations of the recipient. This implicit understanding of tamper-proof credential interoperation between two systems (something that the current protocols are incapable of) means that we need one of two things: agreement on new standards of interoperation between e-mail systems (especially dissimilar ones like Microsoft Exchange and Lotus Notes) or significant fortification of the existing standards. For all systems to successfully send or receive such tamper-proof credentials, the developers of those systems must be in multilateral agreement over the technical specifics.

For almost 18 months now, my biggest gripe has been that no such agreement exists nor has enough progress been made towards reaching such an agreement. In my last column, I cited greed's triumph over civic duty as the reason for costly debacles such as Sobig and MyDoom. Today, the problem of unwanted e-mail is not only threatening to kill one of the Internet's top two killer applications (the other is the Web), it's threatening to overwhelm the Internet's infrastructure and wipe out our systems with e-mail-borne viruses.

It's for this reason that I have long warned against the dangers of using proprietary, unilaterally developed anti-spam systems in lieu of demanding that the purveyors of these systems come to a technical agreement. Such systems do not work on the basis of interoperation or agreement, but rather, on the selfish principle that if you can stop most spam from getting into your inbox or servers, the problem will go away. Since you have no control over the anti-spam techniques used by the recipients of your e-mail, these proprietary systems can do nothing to guarantee the delivery of the e-mail that you're sending. Ultimately, that lack of dependability is what will render the system useless.

Since advocating the use of such proprietary systems equates to supporting the destruction of the Internet's e-mail system (in my opinion), I've routinely turned down most invitations to meet with anti-spam vendors if the meeting was not about what those vendors were willing to do to bring about industry-wide agreement.

After a year of turning down briefings, IronPort's announcement of SMTPi caught my eye. I wondered about the acronym, which seemed to suggest a modified version of the old and beleaguered protocol. I also wondered, if SMTPi was a modified, secure version of the SMTP protocol, would IronPort be making it available to the rest of the world on a royalty-free, no-strings-attached basis? No single vendor should be in the position of placing an unacceptable burden on vendors that rally to support such an important and socially significant set of protocols.

In addition to the many proprietary offerings in the market, some vendors are beginning to test the interoperable standards waters with their own patented technology. Microsoft, for example, recently introduced an interoperable specification called Caller ID for E-Mail that, according to a Microsoft document, is all about being able to verify a sender's identity. But even though the license to implement the specification is apparently available on a royalty-free, no-burdensome-strings-attached basis, giving a single vendor - and particularly one with Microsoft's history of monopolizing -- exclusive control over such an important specification is asking for trouble.

Microsoft isn't the only host of an e-mail service with sufficient presence to push its proprietary technology on the market. Yahoo is working on a technology called Domain Keys that - through the use of public and private keys - allows for the application of digital signatures to domain-specific information in such a way that a recipient (or even an ISP through which an e-mail travels) could reject e-mail on the basis of a public/private key mismatch. Based on the way keys are stored in the Domain Name System servers, Domain Keys' basic premise is similar to a promising but nascent Internet Engineering Task Force (IETF) specification known as DNSSec that's been stuck in pre-ratification mud for over a decade. Because of DNSSec's open nature, I have been a proponent of extracting DNSSec from the mud and getting it onto the Internet where it might not only put an end to spam, but a host of other identity-related problems.

Knowing that one goal of any of these approaches is to allow for the free flow of legitimate bulk e-mail, Yahoo is already testing the technology with bulk-e-mail solution provider SendMail. But little else is known about Domain Keys and its licensing terms. Indicative of the problems that arise when vendors unilaterally act against spam with their proprietary technologies (instead of letting an open standards body serve as chaperone), the Philadelphia-based ePrivacyGroup dropped a strong but cryptic hint that Domain Keys may infringe on its patents.

While Microsoft and Yahoo are advocating their proprietary technologies, a third, caller-ID-like specification known as Sender Policy Framework is proving why open may be the way to go. SPF is already at the draft specification stage with the IETF. Though that's not a guarantee of becoming a standard (just look at DNSSec, SPF has a much better head start towards doing so than Microsoft's Caller-ID or Yahoo's Domain Keys. Whereas you wouldn't need many fingers to count the total number of domains supporting the proprietary technologies from Microsoft and Yahoo, SPF, as of the publication of this story, was being supported by 8,311 domains and growing daily.

Among SPF's biggest supporters are America Online, Amazon.com, Google, O'Reilly, Symantec and the World Wide Web Consortium. All this support comes in spite of the fact that the specification needs workarounds to overcome a glaring weakness or two (such as the way it breaks the current method for a type of forwarding that preserves the original sender's e-mail address. As SPF inventor Meng Weng Wong says on his site, however, "SPF tries to break as few eggs as it can by providing a range of conformance levels. Many people believe that SPF is a strong solution because it helps many while hurting few." In other words, a lot of us already do without automatic forwarding and maybe those who do rely on it can figure out a way to live without it, or find a workaround.

Another element of SPF is that once you have established with relative certainty that the sender of an e-mail is indeed who he or she says she is, you may still want to know more about that sender before letting the e-mail through. For example, if you've already established a trusted relationship with the sender, then all you need every time that sender sends you e-mail is some guarantee that it was indeed he or she that sent it (or that it came from their domain). But in cases where you don't know the sender, you may say to yourself "OK, I'm relatively certain this mail is from who it says it's from, but is that someone I want to hear from in the first place?" According to Wong, "SPF needs to work in hand with reputation schemes."

VeriSign chief scientist Philip Hallam-Baker apparently agrees. Hallam-Baker believes that unwanted e-mail exists because the SMTP lacks accountability, and that introducing accountability requires a three-layer system that involves authentication, accreditation, and consequences. Whereas technologies like Domain Keys, Caller-ID, and SPF take care of the authentication part, Hallam-Baker says "accreditation is where I need to know something about the person. Are they a company or are they a person and what is their reputation?" So, given that every company and individual has a different definition of spam, accreditation implies the extent to which an authenticated sender passes each recipient's reputation test.

Similar to the way that blacklists have been used in the past as public reputation management systems, Hallam-Baker told me that recipients will likely turn to third-party reputation lists to manage the accreditation step of their anti-spam strategy. But there's a difference between reputation lists of the future, and the blacklists that ISPs and others have traditionally relied on to identify sources of spam. Because data about suspected spammers is contributed anonymously, and the blacklists themselves are often run in anonymous fashion, there has been no accountability for the quality and accuracy of the data found in them. As a result, a lot of legitimate mail was being denied safe passage and the senders of that mail had no one to rectify the blockage.

Future reputation lists, according to Hallam-Baker, will have accountability baked in. They'll be run by reputable services (perhaps VeriSign will be one of them) that can be contacted in the event of a problem and, although Internet users will still be able to contribute their opinion to these lists, the better lists (and ones that recipients should gravitate to) will not allow the anonymous submission of spam reports. Internet users who like to report spammers - often because it's an easier way to get off distribution lists than going through a lengthy unsubscribe process -- will become accountable for their reports because reputation list managers will require the same sort of authentication and accreditation that e-mail recipients require of e-mail senders. In this system of authentication and accreditation, your reputation as a sender or as someone who files spam reports will begin to suffer consequences for chronically engaging in questionable practices (true spamming, falsely accusing spammers, etc.).

As it turns out, SMTPi is not a new version of SMTP. The "i" in SMTPi stands for identity, and by providing access to reputation-oriented data, according to Tom Gillis, chief marketing officer for Internet e-mail gateway appliance vendor IronPort Systems, SMTPi addresses the accreditation part of the three-layered approach described by Hallam-Baker. With appliances now installed on the networks of companies like AOL, Verizon Wireless, and Bell Canada that have a lot of in- and outbound e-mail, Gillis claims that IronPort's hardware touches about 20 percent of all e-mail traffic. This puts IronPort in the unique position of being able to profile some of the Internet's most prolific senders of e-mail. To wit, IronPort has created a database called SenderBase, which the company claims to be the first e-mail reputation service.

Whereas inclusion on a blacklist implies that a certain IP address is either the source of spam or, due to lax administrative policies, may be indirectly aiding and abetting such a source, Gillis claims SenderBase, which is free to use by all (and has already been adopted for use by anti-spam solution provider SpamAssassin), makes no editorial judgment about the entries it contains. "That's up to the recipients," says Gillis. "We just provide the raw data and the recipients can decide for themselves whether a source of e-mail is reputable or not." While a visit to the home page at www.senderbase.org gives an interesting snapshot of the sort of data it keeps and how it's organized, Gillis says that the database tracks approximately 40 different quantitative and qualitative parameters that recipients and ISPs can use to profile senders. In addition to volume of e-mail by IP address and domain, SenderBase tracks whether the e-mail passed through an open proxy or open relay (a common practice of spammers), what the country of origin is, whether the source's DNS servers resolve properly, and whether it will accept mail in return (using IronPort's spider technology). "Spammers can change their identity, but they can't mask a lot of this other data," says Gillis.

By itself, Gillis admits that SenderBase has one of the same weaknesses that blacklists do: Instead of having being able to assign all of this data to the identity of the sender, it can only be assigned to the IP address of the sending system. As I learned with e-mail from my personal domain that was falsely classified as spam, even if an IP address is a source of spam, that doesn't necessarily mean that all senders located behind it are spammers. But when layered on top of a system that requires authentication, the entire ballgame changes. "As one of the new technologies that can overlay authentication, SMTPi is a long-term initiative" said Gillis. "As technologies like SPF and Microsoft's CallerID allow us to get more granular than IP address, the situation will improve."

In addition to having no authentication layer on which to currently rely, perhaps the one thing missing from SMTPi is a way for authenticated users to contribute their editorial judgments, especially if they're willing to be held accountable for what they say. But since blacklists are the closest thing to reputation management that we've had so far and they've done more harm than good, any new reputation system - especially one with accountability - like SMTPi could be a huge towards a more robust accreditation layer. However, since it's not an IETF-approved protocol, technology, or enhancement to the SMTP standard, its misleading name should be changed.

You can write to me at david.berlind@cnet.com. If you're looking for my commentaries on other IT topics, check the archives.