How to Send Zillions of s a Day BY MIKE HILLYER

Transcription

1 How to Send Zillions of s a Day BY MIKE HILLYER

2 How Much Does Your Company Send? A hundred a day? A thousand a day? More? Believe it or not, many organizations send several million s a day, and enterprises, large online retailers, social networks and gaming sites often send as much as a billion a month. In the case of social networks, a few send as many as a billion s a day. No matter how much companies send, they all send for the same reason: They have a community of customers or users, and these companies need to keep their communities engaged and in-the-know. Why? Because a disengaged community is no community at all, and a disengaged customer base is not going to be your customer base for long. Is it easy to send high volumes of ? Yes and no. It s easy to do, but the way you do it is what matters. Large volumes of mail can be expensive to deploy. Worse, large volumes of mail may be perceived as spam by ISPs. If done incorrectly, high-volume sending can drain your resources and your revenue, and provide no value at all to your customers. How can you keep your customers happy and engaged if your message never makes it to the intended inbox? Here are four steps you need to take to do it right: MARKETING REPORTS IS A GREAT SOURCE FOR REPUTATION BUILDING. YOU LL FIND A WEALTH OF INFORMATION, ORIGINAL ARTICLES, REVIEWS AND LINKS TO HELP YOU START OR IMPROVE YOUR OPT-IN MARKETING. -marketing-reports.com 1 Optimize Your Sending Platform. As a sender, you have the option of using an service provider (ESP) or using your own dedicated sending service like SparkPost Elite, and you have to choose the best model for your situation. ESPs provide not only infrastructure but also expertise, to handle the myriad details of sending for their clients. This enables companies to focus on their core business rather than the many technical challenges of high-volume . When we talk with companies that rely on an ESP to send all their , we hear some common issues. Because ESPs charge by the , many companies face continuing high costs for their marketing programs. They d like to grow those programs, but they face budget and resource constraints. If their programs grow, so do their ESP costs. They also find that they can t get the kind of actionable data they need to optimize their campaigns because their ESP collects and compiles data in their own format according to their own schedule. So when the business does get the data, it s frequently days late and they re not able to enter it into their business systems very easily. We re also seeing dissatisfaction with low inbox delivery rates. Because ESPs typically report on s delivered to ISPs, and not ultimate inbox rates, the delta between what companies are told is being delivered and what is actually being delivered can be as high as 20%. The common alternatives to working with an ESP are to either outsource to a cloud service that specializes in delivery without the creative and strategy services provided by ESPs, or use in-house infrastructure. Going with a cloud provider is a smart choice for companies who don t want to deal with managing hardware and network infrastructure in-house. Especially if you re working with an ESP and not using much if any of their creative and strategic services, switching 2

3 A high reputation is critical in getting the majority of your mail safely in the hands of your intended recipients. to a dedicated cloud offering like SparkPost Elite can save you a lot of money. The better ones use tiered pricing, so you re buffered against steep and surprising price increases if you re business really takes off. The quality cloud services are built on high-grade commercial platforms, rather than commodity open source servers used by many smaller ESPs, meaning you can rely on consistently excellent performance and uptime. The high-end offerings on the market also offer flexible API integration, meaning you can easily link your content-generation systems and get fast access to your data. The overall uptick in performance using dedicated cloud offerings over ESPs can be substantial, and you still get the advantage of not having to run your own servers and systems. In the end, it s not a matter of insourcing versus outsourcing but of right sourcing, or doing what makes the most sense for you and your customers. You may very well find that a mixed environment works best, in which both you and an ESP share the sending responsibility, or you use a dedicated cloud instance to handle your transactional messages (purchase receipts, password resets) but stick with your ESP for your marketing messages. In-house or in the cloud, a simple rule of thumb when considering hosting options is: which configuration will drive the highest engagement rates and bottom-line value for my organization? Keep that question in mind, and you re likely to make the right call. 2 Secure Your Sending Reputation. The higher your sending volume, the more you need to improve your sending reputation among the ISPs that handle most of the incoming traffic on the Internet. This holds true for major ISPs such as MSN, AOL, Yahoo, AT&T, Gmail, EarthLink, Cox, Bell South etc., as well as smaller local ISPs. A high reputation is critical in getting the majority of your mail safely in the hands of your intended recipients. To build and maintain a good reputation you need to follow best practices for list acquisition, list hygiene, segmentation and relevancy. This guide focuses on the technical requirements for sending high volumes, but keep in mind that doing so without a solid reputation on your domain and sending IPs is very difficult. From a technical perspective there s a number of bases we need to cover regarding authentication, whitelisting, bounce processing and complaint handling. Authentication. As a reputable sender you will want to associate your IP addresses with your domain using the authentication standards available to you. These include SPF, SenderID, DomainKeys (DK) and DomainKeys Identified Mail (DKIM). There are indications that SPF (and SenderID by association) is ineffective, but given the low effort required to implement it, we recommend doing so. While SPF and SenderID are purely DNS-based, DK and DKIM require an implementation either during message creation or during relay by the MTA and as a result will impact the maximum throughput of your infrastructure. DK is quickly being superseded by DKIM, but with most solutions supporting them it is simple enough to use both when sending to an ISP that supports one or the other.implementation details will vary based on your sending solution. While some recommend selectively signing DK and DKIM only for messages sent to ISPs that are known to check authentication (in order to lower the impact signing has on throughput on a solution that takes a significant performance hit from signing), it is recommended that you sign all messages; you never know who is checking for authentication without announcing it. 3

4 CHECK YOUR REPUTATION. RETURN PATH, A LEADING CERTIFICATION AND SCORING COMPANY, OFFERS SENDERS, RECEIVERS OR CONSUMERS FREE ACCESS TO SENDER SCORE, AN ONLINE REPUTATION CHECKING TOOL. senderscore.org Whitelisting. One benefit of getting on the various whitelists provided by ISPs and reputation providers is that in some cases you can send higher volumes on whitelisted IP addresses than would otherwise be possible. Keep in mind that in most situations whitelisting comes after sending has already begun in order to allow the provider of the whitelist to examine your sending patterns as part of the whitelisting process, so put your best foot forward (and follow it up with consistent behavior). Bounce Processing. One quick way to lose reputation is to repeatedly send mail to invalid addresses. ISPs will track how many non-existent addresses you send to and throttle you accordingly. Even more seriously, ISPs will occasionally take inactive addresses and re-activate them as spam traps; any mail sent to the address will immediately get classified as a spam source and will be filtered accordingly. To prevent this from happening it is necessary to capture and act on the responses sent by the ISPs and unsubscribe those addresses identified as non-existent or inactive, while retaining those with responses that identify users on vacation, with full mailboxes, and other non-fatal conditions. Cloud sending solutions will usually perform this step automatically with varying levels of effectiveness, while other platforms will require a third-party solution. Complaint Handling. In an effort to help senders improve their practices, a number of ISPs have implemented feedback loop (FBL) programs using the abuse reporting format (ARF), the industry standard format for spam reporting. When a user on a supported ISP clicks the This Is Spam button, an automated message is sent to an address defined in advance (when signing up with the ISP for the feedback loop program). By processing these messages and unsubscribing the relevant users, you prevent further reputation damage that may result when sending them future messages. You can also use this feedback to refine your campaign objectives and approach, to provide your customers with the content and offers they are most likely to need or want. The ARF format used by the ISPs makes it relatively straightforward to process feedback loop messages and use them to unsubscribe the users who have complained about your messages. Some sending solutions can handle FBL messages natively while others require separate tools that can open ARF-formatted messages. 3 Scale Your Sending Infrastructure. Several key architectural components come into play to make it possible to send high volumes of , including network connectivity, server hardware and software. If you re working with an ESP or dedicated cloud service, then hardware, connectivity and network equipment are all included in the service bundle. You ll definitely want to make sure your service level agreement (SLA) provides for more-than-adequate equipment along with redundant backup. But if you re bringing your sending infrastructure in-house, you ll need to have the staff with the know-how to manage all of the following. Network Connectivity. Most professional sending operations are based in rented datacenters, simplifying the provisioning of network connectivity. For example, if we need to send a billion s per month, we re talking 2.5 million messages per hour, and we ll assume a sample message size of 50 kilobytes (51,200 bytes), meaning that we need to send at a rate of 2,500,000 * 51,200 = 128,000,000,000 bytes per hour or megabits per second. With this throughput, you will certainly need gigabit-speed networking in the datacenter. Do the math on your own expected throughput, to determine your datacenter needs. But keep in mind that has a significant impact on your bottom line so you won t be able to tolerate extended outages. You need to not only make sure that 4

5 the datacenter you choose has redundant power and backbone connections, you also need to consider using redundant datacenters. Server Hardware. Moving high volumes of does not require the purchase of custom server hardware but it does require making a proper investment in hardware. Generally speaking you will be using an infrastructure similar to the one shown in Figure 1. The Message Injector queries the database and uses the results to assemble one or more messages, which it relays to the Outbound Mail Server. The Outbound Mail Server queues the message, performs any necessary manipulations on the message and then sends it via the Internet to the destination server. In the event of a delivery failure message or a feedback loop message, the incoming message arrives via the Internet to the Inbound Mail Server. The Inbound Mail Server performs antivirus/anti-spam scanning and then, in the case of a legitimate message, processes the message and updates the subscriber information in the database. Note that not all server solutions can perform this processing in-stream. When using such solutions, an intermediate server will be needed to accept the clean message from the Inbound Mail Server and process it using custom code. In a production deployment there can be several variations on this example, typically with multiple servers used on the outbound and inbound roles, with multiple message injectors pushing to the outbound machines and often specialty servers on the inbound side dedicated to processing incoming feedback loop and bounce messages. To send in very high volumes (say, more than a billion a month), quality working mail servers should feature the following: 2x multi-core, 64-bit processors 16-32GB of RAM 8x15K RPM hard disks Battery-backed RAID-10 controller The specific details of your hardware selection will depend on your actual and expected sending volume and the ability of your specific software to leverage the resources provided. A large number of fast disks in a RAID-10 array is recommended for the message spool as standards-compliant mail servers must write messages to disk before accepting them for delivery, placing significant demands on storage resources. Software. For many years, the first choice for sending software was Figure 1: Server Hardware INTERNET Inbound Mail Server Outbound Mail Server Database Server Message Injector open source message transfer agents (MTAs), the most widely used of which were Postfix and Sendmail. The problem with open source MTAs is that their performance tops out at around 100,000 messages per hour. To support a onemillion-per-hour rate, for example, you d need 10 individual servers, each running an instance of an open source MTA. Popular commercial sending solutions such as PowerMTA from Port25, a Message Systems company, show real-world performance in the range of 2.5 million messages per hour per server far higher than open source. The highest-performing sending solution on the market today, the Momentum platform from Message Systems, can deliver at far higher rates than that. In a production environment using multi-core server hardware and ultrafast solid-state disk technology, Momentum has reached in excess of 15 million 5

6 A NUMBER OF TOOLS FOR CHECKING THE REPUTATION OF YOUR IP ADDRESSES CAN BE FOUND AT WORD TO THE WISE, A SOFTWARE PROVIDER THAT HELPS ISPS AND ESPS TO MANAGE SPAM COMPLAINTS AND DELIVERABILITY ISSUES. wordtothewise.com messages per hour. Not all high-volume senders will require that kind of top-end performance, but this is the state of the art today, and something to consider when reviewing your options. In addition, your sending software should support IP domain segregation. ISPs pay close attention to the source IP domains of flowing into their servers. If any domain sends a high rate of mail that looks like spam, or gets flagged as spam by recipients, or contains bad addresses, the ISP will filter traffic from that domain accordingly, or might block all traffic from that domain outright. If all of your mail comes from a single IP, you run the risk of all of your mail getting blocked. For example, let s say we have one million messages to send today, and they break down into three groups. The first third is going to addresses that we re very confident are legitimate because those addresses have been used in the past to send invoices, order confirmations and other transaction-based s. The second third of the list was generated with confirmed- or double-opt-in tactics, so we re confident that these won t be blocked, but not with the same level of certainty as group one. The final third of the address list was built using unconfirmed- or single-opt- in tactics; we need to separate out these addresses because they re likely to generate a higher number of spam complaints than group one or group two. Given our situation, we would want sending software that could accommodate at least three IP domains, but keep in mind that companies that send very high volumes of mail often need many more. For instance, large service providers that manage campaigns on behalf of numerous brands would, ideally, want the ability to create unlimited IP addresses. Some sending software goes beyond basic message queuing and sending, providing the additional functionality required for a high-volume sender. Here are some other critical features to look for: High Availability. With high-volume sending, you absolutely need a solution that provides high availability out of the box. If a server goes down, you can t afford to be frantically activating a warm spare, just to find out that it too has some issue. You need an active-active solution that reacts automatically to server failures and keeps the mail flowing. Centralized Management. You need a solution that can be easily managed on your terms, whether you prefer editing configuration files or using a web interface. In addition, you need something that grows with you, providing centralized management of an entire cluster of servers. Advanced Reporting. One key to successful sending at high volumes is keeping tabs on how your server is performing and how your mailings are doing. You need to know what is passing through your server, how quickly messages are moving, whether queues are backed up, how the various ISPs are treating your traffic, all with the ability to drill down on specific source IPs and specific destination ISPs. You need to be able to see all of this in real time and across your entire infrastructure. When choosing sending software, be careful not to choose a free or low-cost solution only to spend countless hours building workarounds, or writing scripts to automate administration or provide redundancy. This work requires time, and time spent creating tools that are already provided in an alternative solution is time (and money) wasted. You are always better off using your time to create your special sauce, that which makes your business unique and gives you a competitive advantage. 6

7 4 Strengthen Your Monitoring Capabilities. When sending high volumes of mail, you just can t afford to have a fire and forget mentality. You need to keep a constant eye on your environment, monitoring multiple key factors to ensure that you can continue to successfully send. Reputation Monitoring. Remember earlier when we said you needed to improve your sending reputation? You also need to maintain your good reputation. To do that you should take advantage of reputation monitoring tools provided by companies like Return Path (returnpath.net), and you should also keep a close eye on the reports produced by your sending software. You need to watch metrics such as bounce rates, FBL hit rates, blacklist hits, transient failures and response rates. Response Monitoring. Keep in mind that you get what you monitor for; if you focus too much on one metric it may improve without helping the big picture. In addition to making sure all the underlying pieces are in place, don t forget to keep an eye on things where the rubber meets the road. You may be sending at phenomenal rates with great metrics but failing to generate customer actions that lead to revenue. So don t forget to monitor your response rates. Start Sending! While by no means an exhaustive list, this should give you a good idea of what s required in a high-volume sending environment, while keeping your customers happy and engaged. If your company is growing, it pays to see the road ahead, and high-volume sending might be in your future sooner than you think. ABOUT SPARKPOST SparkPost is the cloud solution from the world s number one infrastructure provider, whose customers including Facebook, LinkedIn, Twitter, Groupon, Salesforce, Marketo, Pinterest, Zillow and Comcast send over 3 trillion messages a year, over 25% of the world s legitimate . Our service outperforms every other cloud or on-premises alternative, and these companies choose us to provide the deliverability, speed and insight they need to drive customer engagement for their business. Get Started Today! Call or hello@sparkpost.com. Follow us on or visit SparkPost.com. SparkPost.com 301 Howard St., Suite 1330 San Francisco, CA tel toll free usa Guide_Zillions_1115 7