Mailingee

email automation & sequencing system

26 Jul 2022

I have tried a few email automation & sequencing platforms over the last 10 years, starting with Toutapp (acquired since by Marketo) and up to Outreach in more recent days.

Though the products are good generally, I usually have precise ideas about what I want to achieve or the level of automation I expect.

Every time, I got frustrated - by product limitations, deliverability issues... and pricing.

Last year, as part of my "Sales-as-a-Service" engagements, I faced the need again to use an email sequencing solution.

Knowing now how to write automation scripts in Python, I decided to write my own email automation solution.

The idea was to have something that would fit my needs in terms of flexibility, including:

  • same basic capabilities as off-the-shelf tools: email sequencing, etc..
  • easy re-use & automations where possible: across both campaigns AND client engagements
  • multi-lingual: send emails in English, French & German (my main working languages) depending on the recipient's
  • time personalisation: greetings based on time of day
  • snippets insertion: customise emails very flexibly with any contact-level information
  • integration: working as part of my stack and other automations, eg Twittee (lead generation using the Twitter API) !projects/twittee

Overall structure

3 components to the system:

  • data management: spreadsheet-like "source of truth" of recipients and sequences, using Grist !apps/grist
  • email content: text files
  • sending emails: python script

DATA MANAGEMENT

I turned to my online spreadsheet solution of choice (Grist: !apps/grist) to manage the list of contacts and sequences.

Example

Here is an overview of a campaign in Grist:

campaign-overview

This particular campaign was based on reaching out to webinar organisers, based on webinar data I gathered from Twitter (see !projects/twittee)

The spreadsheet format enables to maintain quick and full visibility of the entire campaign, easy to filter as needed.

It includes the following fields:

  • email
  • domain: enables to filter by company (TODO: add logic to space out emails from same company)
  • li: Linkedin search link, with pattern https://www.linkedin.com/search/results/all/?keywords={first_name}%20{company_name}
  • first
  • title
  • track: different messaging by track, eg for webinar campaign each recipient assigned to track based on current solution used. Track name inserted in emails as personlisation.
  • search: search link when company URL not readily available, using pattern https://duckduckgo.com/?q=!ducky+{domain}
  • company
  • url_reg: webinar registration link
  • webinar_title: to be inserted in emails for personalisation.
  • tz: timezone from the webinar (assuming same as organiser)
  • status: SEND (ready to send), DISCARD, DNE (Do Not Email), Engaged, Hold, No, Queue, Ready
  • notes
  • tweet: original tweet where lead/webinar was found
  • answer: date answer received (average response time can be calculated)
  • src: source of the email address
  • webinar_date
  • v: version of the email, eg A, B, etc..
  • s_test_date: test field
  • s_1_date: send date of email #1
  • s_2_date: send date of email #2
  • s_3_date: send date of email #3, etc..
  • lang: EN, FR, DE
  • created: record creation date
  • updated: record last update
  • dup: duplicate flag
  • radar: add to Radar
  • send_from: logic to send from different email accounts

EMAIL CONTENT

Each email is a text file in a folder for the campaign.

The filename of the text file follows this pattern, enabling support for tracks, language, versions and sequences:

TRACK_LANGUAGE_VERSION_SEQUENCE_SUBJECT LINE.txt

eg. GoToWebinar_E_A_1_GoToWebinar contract.txt is the email content for the GoToWebinar track (ie recipients currently using GoToWebinar), in English (E), version A (enabling A/B testing, up to Z), 1st email in the sequence, where the subject line of the email will be GoToWebinar contract.

It's therefore very easy to quickly create a new email version, sequence or language, just by creating a new text file with the right name.

For a single email campaign:

eoy-emails

example of email cadence

Email #1

Filename: GoToWebinar_E_A_1_GoToWebinar contract.txt

= Track: GoToWebinar
= Language: English
= Version: A
= Sequence: 1
= Subject: GoToWebinar contract

Body (content of text file):

Hi{first},

I have seen with your webinar {webinar_title} that you are using GoToWebinar.

Who is responsible for the webinar contract at {company_name}?
And when does it expire/is set to renew?

We can help you improve participants' experience, increase engagement and results.

Webinar.net offers a platform built from the ground up for webinars.
It is like organising a seminar in an auditorium and not in a meeting room.

If you are looking to upgrade/improve your webinar program this year, let me know who would be the right person at {company_name}, and when we should connect?

Thanks,

Nic


{signature}

Email #2

Filename: GoToWebinar_E_A_2_Re GoToWebinar contract.txt

= Track: GoToWebinar
= Language: English
= Version: A
= Sequence: 2
= Subject: Re GoToWebinar contract

Body:

Hi{first},

I hate to pester, especially the wrong person at the wrong time :)

Our clients all were looking to upgrade their webinar program, be it the experience for participants/presenters/producers and/or the results (engagement, number and quality of leads, etc..).

If this might be of interest when renewal is in question for you, can you confirm who the person to talk to about webinars would be, and when to connect in 2022 (ie when your contract with GoToWebinar expires)?

Let me know,

Thanks,

Nic


{signature}

Email #3

Filename: GoToWebinar_E_A_3_Re GoToWebinar contract.txt

= Track: GoToWebinar
= Language: English
= Version: A
= Sequence: 3
= Subject: Re GoToWebinar contract

Body:

Hi{first},

Upgrading to a webinar platform designed from the ground up for that purpose, brings benefits for all involved - participants, presenters and producers.

From not losing participants because a plug-in needs to be installed, to a stress-free presenter experience, a participant experience leading to a more engaged audience, better qualified leads, and more.

Obviously if you are happy with what GoToWebinar has to offer and don't see improvement potential for better business results, just let me know.

Else, when should we connect this year?

Thanks,

Nic

{signature}

etc...

Cadences can go up to as many emails as desired and A/B testing is possible.
I have seen good success with the above campaign, as it was personalised with the solution they used, their own webinar title, and targeting the webinar organiser(s) directly.

Typical other cadences or tests might include dropping other clients in same industry, success metrics, etc...

See !b2b-sales/outbound for more.

SENDING EMAILS

Sending methods

Script allows for 3 ways to send emails:

send manually

MAN: email gets generated ready to send. Provides ability to personalise even more before hitting send.

Here is an example, with the added feature of launching both the recipient's Linkedin profile (this campaign had that info) and the company's website alongside generating the ready-to-send email. This makes hyper-personalisation of each email very efficient.

send automatically using keyboard automation

AUTO_KB: email gets generated on desktop and sent, using keyboard shortcuts automation.

This is my preferred method.
Why?
Emails are sent 100% as if sent manually, from email client/desktop, thus circumventing any spam filtering based on automation identification.
Plus this allows for inclusion of email signature without having to fiddle with inline image insertion via SMTP which can be a bit fiddly.

send automatically via SMTP

AUTO_SMTP: fully "behind-the-scenes"

The most "elegant" way to run it (vs the "hacky" approach above), used by all the emailing platforms.
Though generating emails that "look" as if they were sent manually from a metadata perspective (headers, etc..) is painful.
Sending is easy - circumventing complex spam filtering looking at email metadata is trickier.

test

+ TEST which prints what would be sent, without sending the email / or send to Mailtrap:

Script

06 Sep 2022

This is my current script - this one without opening Linkedin & Company Website (TODO: add example).

Comment inline.

# my default Python boilerplate
from datetime import datetime
print(f"Starting at {datetime.now().strftime('%H:%M:%S')}")
import time
start_time = time.time()
import os
# appending paths to my modules
import sys
sys.path.append("/path/to/project/with/my/modules/in/it")

from dotenv import load_dotenv # to store environment variables
load_dotenv() # initiate load of .env
# pretty printer, always handy to have
import pprint
pp = pprint.PrettyPrinter(indent=4)

# I always have a counter in my boilerplate/footer, to add easily when coding for verification purposes
count = 0

# my collection of utilities. See !python/my-utils
import my_utils

# my Grist interface modules. Anonymised here with "XX" - one module per Grist workspace (ie per client in my case). See !apps/grist
import grist_XX
import grist_XX
import grist_XX

# script specific imports
import webbrowser
import time
import random
from datetime import date
from urllib.parse import quote
import datetime
import re
from pynput.keyboard import Key, Controller


####################

import re
import webbrowser
from urllib.parse import quote
import random
from pynput.keyboard import Key, Controller
import smtplib
from email.mime.text import MIMEText

# for automated emails sent via Gmail/SMTP
import yagmail

per_run = 3
stop_mailtrap = 0

send_type = "AUTO_KB"  # TEST or MAN or AUTO_KB or AUTO_SMTP
# email_version = 'A'

wait_from = 1
wait_to = 3

####################
# Global Variables

date_time_today = datetime.datetime.now()
date_today = time.mktime(date_time_today.timetuple())


campaign = os.path.basename(__file__)[:-3]

# Client identifcation:
# I assign a 2-letter code to each of my client, as a unique identifier
# XXript name would start with that 2 letter code, enabling automations below
client = campaign[:2]

# Campaign identification
# same as above, each campaign gets a 5 character code assigned, comprising the first 2 being the client ID, with 3 digits added
# the full 5 characters represent the campaign ID
# the XXript name contains the campaign name after the ID
campaign_code = campaign[:5]


# initiate yagmail
if client == 'XX':
    yag = yagmail.SMTP(XX_EMAIL, XX_PASSWORD)
if client == 'yy':
    yag = yagmail.SMTP(YY_EMAIL, YY_PASSWORD)

# current_time = datetime.datetime.now()

# Bcc Hubspot based on client
if client == 'XX':
    bcc = 'xxxxxxx@bcc.hubspot.com'
elif client == 'YY':
    bcc = 'yyyyyyy@bcc.hubspot.com'
elif client == 'ZZ':
    bcc = 'zzzzzzzz@bcc.hubspot.com'

# Sender based on campaign name
if client == 'XX':
    sender = 'nicolas@xxxx.com'
elif client == 'YY':
    sender = 'nicolas@yyyy.com'
elif client == 'ZZ':
    sender = 'nicolas@zzzz.com'

# Language assignment based on code
french = ['FR', 'F']
german = ['DE', 'D', 'AT']

# Fetching campaign data from Grist (list of namedtuples)
data_grist_campaign = grist_XX.XX003.fetch_table('Master')

# Count left to send based on empty "send date" field
count_rest_to_send = 0
for record in data_grist_campaign:
    if record.send_date == None:
        count_rest_to_send += 1

# Counters
count_sent = 0
count_replied = 0
count_recipients = 0

recipients = {}
list_sent = []

# Regex to validate email format if needed
validEmail = "[^@]+@[^@]+\.[^@]+"


####################
# Fetch Messages

message_path = f'/path/to/folder/{campaign_code}/emails'

directory = os.fsencode(message_path)
dict_emails = {}

# Dict of email messages
for file in os.listdir(directory):
    filename = os.fsdecode(file)
    if filename.endswith(".txt"):
        print(filename)
        filename_regex = re.match(r"(.*)_(.*)_(.*)_(.*)_(.*).txt", filename)

        email_lan = filename_regex[1] # Language
        email_ver = filename_regex[2] # Version
        email_seq = filename_regex[3] # Sequence
        email_tra = filename_regex[4] # Track
        email_sub = filename_regex[5] # Subject
        email_path = f"{message_path}/{filename}"

        dict_emails[email_path] = {'email_ver': email_ver, 'email_lan': email_lan, 'email_sub': str(email_sub), 'email_seq': email_seq, 'email_tra': email_tra}


####################
# Functions


def select_email(email_seq, email_lan, email_tra): 
    path_to_return = ''
    email_version = ''
    for path, attribute in dict_emails.items():
        if attribute['email_seq'] == email_seq and attribute['email_lan'] == email_lan and attribute['email_tra'] == email_tra:
            if path_to_return == '':
                path_to_return = path
                email_version = attribute['email_ver']
            else:
                if path > path_to_return:
                    path_to_return = path
                    email_version = attribute['email_ver']
    if path_to_return == '': # Backup to generic email templates if none specific found
        email_tra = 'XXX'
        for path, attribute in dict_emails.items():
            if attribute['email_seq'] == email_seq and attribute['email_lan'] == email_lan and attribute['email_tra'] == email_tra:
                if path_to_return == '':
                    path_to_return = path
                    email_version = attribute['email_ver']
                else:
                    if path > path_to_return:
                        path_to_return = path
                        email_version = attribute['email_ver']
    # return path_to_return
    dict_to_return = {'email_version': email_version, 'email_path': path_to_return}
    return dict_to_return


def mailto(email, subject, body, bcc):
    return webbrowser.open(f'mailto:{email}?subject={quote(subject)}&body={quote(body)}&bcc={quote(bcc)}')


def send_email(email, subject, body, send_type, id):
    global bcc
    global count_sent
    global sender
    global client
    global campaign
    global count_test_send # to manage Mailtrap limit
    global stop_mailtrap
    # global send_type

    if send_type == "MAN":
        mailto(email.strip(), subject, body, bcc)
        print_output = f"Email generated for {email}"
        mark_as_sent(id, email_seq, email_version)
        time.sleep(1)
        count_sent += 1

    elif send_type == "TEST":

        msg = MIMEText(body)
        msg['Subject'] = subject
        msg['From'] = sender
        msg['To'] = email

        # if count_test_send < stop_mailtrap: # To minimise Mailtrap credit
        #     with smtplib.SMTP("smtp.mailtrap:2525") as server:

        #         server.ehlo()
        #         server.starttls()
        #         server.login("xxxxxxxxxxxx", "xxxxxxxxxxxxx")
        #         # message = 'Subject: {}\n\n{}'.format(subject, body)
        #         # server.sendmail(sender, email, message)
        #         server.sendmail(sender, email, msg.as_string())
        #         server.quit()

        #     mark_as_sent(id, email_seq, email_version)
        count_sent += 1
        # list_sent.append(email)
        print_output = f"func send_email: email sent to Mailtrap for {email}"


    elif send_type == "AUTO_KB":
        mailto(email.strip(), subject, body, bcc)
        random.randint(wait_from, wait_to)
        print(f"waiting {wait} seconds...")
        for x in range(wait):
            print(f"{x + 1}..")
            time.sleep(1)
        # Send automatically by clicking Cmd + Enter
        keyb = Controller()
        with keyb.pressed(Key.cmd):
            keyb.press(Key.enter)
        mark_as_sent(id)
        count_sent += 1
        print_output = f"Email sent to {email} via keyboard automation"


    elif send_type == "AUTO_SMTP":
        content = [body,
            # '',
            # yagmail.inline("minions.gif"),
            # '\n\n\n\n\n\nNic',
            ]

        # yag.send(email, subject, content)
        yag.send(to=email, bcc=bcc, subject=subject, contents=content)
        mark_as_sent(id)
        count_sent += 1
        # Print for verification
        # print()
        # print(subject)
        # print('----')
        # print(body[:160], "...")

        print_output = f"Email sent to {email} via SMTP automation"
        wait = random.randint(wait_from, wait_to)
        print(f"waiting {wait} seconds...")
        time.sleep(wait)

    return print(print_output)


def mark_as_sent(id):
    global date_today
    global campaign
    global client

    # Comment out to test marking as Sent in proper column
    if send_type == "TEST":
        email_seq = 'test'

    if client == 'XX':
    grist_XX.XX003.update_records('Master', [
                                        {   'id': id,
                                            f's_{email_seq}_date': date_today,
                                            f's_{email_seq}_v': email_version,
                                        }
                                            ])

    if client == 'YY':
        grist_YY.YY001.update_records('Master', [
                                            {   'id': id,
                                                f's_{email_seq}_date': date_today,
                                                f's_{email_seq}_v': email_version,
                                            }
                                                ])

    print(f"{id} marked as Sent")

####################
# Loop

list_dne = [x.email for x in grist_XX.Contacts.fetch_table('Master') if x.dne in [True, 'True']] 

# blacklist = my_utils.get_blacklist_domains() + my_utils.get_blacklist_freemail_domains()

count_row = 0
count_sequence_finished = 0
count_left_to_send = 0
count_test_send = 0

list_missing_data_to_send = []
list_missing_email_template = []

for contact in data_grist_campaign:
    # time.sleep(1)
    count_row += 1
    id = int(contact.id)
    first = contact.first
    email = contact.email

    # if email not in list_dne and my_utils.domain_from_email(email) not in blacklist:
    if email not in list_dne:

        ### Send only to marked as Send and without a Do Not Email flag
        # if contact.send_date in [None, '', 0] and contact.priority in [None, '']:
        if contact.send_date in [None, '', 0]:

            count_recipients += 1
            if count_recipients < per_run + 1:  # Limit per run
                # print(f"\n{count_recipients}/{count_rest_to_send}") # Need to find logic for count_rest_to_send
                print(f"\n\n{datetime.datetime.now().strftime('%H:%M:%S')} #{count_row} / Recipient {count_recipients} / Send {send_type}")
                print(email)
                print(f"First: {first}")

                message_path = '/path/to/folder/mailingee/XX003/emails/E_A_1_0_EMAIL SUBJECT LINE.txt'

                # Subject
                fileName = os.path.basename(os.path.splitext(message_path)[0])
                subject = re.match(r"(.*)_(.*)_(.*)_(.*)", fileName)
                subject = subject[4]

                # Body
                with open(message_path) as file:
                    body = file.read()

                    # Insert First Name
                    if first not in [None, '', '--']:
                        body = body.replace(
                            "{first}", f" {first}")
                    else:
                        body = body.replace("{first}", "")

                    # SEND EMAIL
                    if send_type == 'TEST':
                        send_email(email.strip(), subject, body, "TEST", id)
                        list_sent.append([email, first])
                    elif send_type == 'MAN':
                        send_email(email.strip(), subject, body, "MAN", id)
                        list_sent.append([email, first])
                    elif send_type == 'AUTO_KB':
                        send_email(email.strip(), subject, body, "AUTO_KB", id)
                        list_sent.append([email, first])
                    elif send_type == 'AUTO_SMTP':
                        send_email(email.strip(), subject, body, "AUTO_SMTP", id)
                        list_sent.append([email, first])

                    else:
                        count_recipients -= 1
                        print(f"NO EMAIL SENT FOR {email}")
                        continue
            else:
                count_left_to_send += 1

# my default Python footer

########################################################################################################

if __name__ == '__main__':
    print('\n\n')
    print('-------------------------------')
    print(f"count = {count}")
    print(f"count_row = {count_row}")
    print(f"count_recipients = {count_recipients}")
    print(f"count_left_to_send = {count_left_to_send} / For next run")
    print()
    print(f"count_sent = {count_sent}")
    print()
    print()
    print('-------------------------------')
    run_time = round((time.time() - start_time)/60, 1)
    print(f'{os.path.basename(__file__)} finished in {run_time} minutes.')
    print(datetime.datetime.now().strftime('%H:%M:%S'))
    print()

19 Apr 2023

Improved the system a lot over the last few weeks. Amongst other changes:

  • different intro/outro depending on day of the week
  • different call-to-action write up if email sent to a team, or an individual (“who best to connect with” vs “let’s connect?“)
  • different tagline depending if email is .edu /.ac(“.. platform for Universities”) or .org (“..platform for Associations”) or neither (just “…platform”).
  • different phone number based on country TLD (my German number only if .de, .at, UK only if .uk, both otherwise)

TODO write-up.

about tracking

this script/solution does not cater for tracking emails - though it could, leveraging services like this one:

Though I had deliverability issues in the past with off-the-shelves solutions (eg Toutapp), where I think the tracking pixel was a culprit (or perhaps in combination with other factors).

Furthermore, I experienced also "false positives" - engaging prospects thinking they had seen/read my email, when they clearly had not.

Last but not least, I'm a data privacy geek and avoid tracking myself as much as possible. So feels weird to employ those tactics on others.

Long story short, I'm underwhelmed by "tracking emails" and do not think at this point that my script/system should support it.

links

social