Scheduled maintenance

Our hosting provider is performing some hardware upgrades on the servers we run on, so there will be an outage tonight of several hours. Since this is the middle of the night in Europe, this should not affect many of our customers or their subscribers.

It's privacy time!

As you've no doubt gathered, at Smartmessages we are very keen on privacy and preserving the rights of subscribers. Today adds an important privacy upgrade: tracking consent.

When someone consents for us to send them email for marketing purposes, we are required to be able to demonstrate that they did so with full transparency (GDPR's accountability requirement) and via an explicit, informed opt-in mechanism (a requirement of PECR / EPD rather than GDPR). So when we gather someone'e email address for the purpose of sending them messages, that is all that we are asking them for; at no point are they asking to be tracked, and historically, nor have we asked them if it's ok if we do. This is a clear contravention of the purpose limitation. That changes today. All of our subscribe forms now include a tracking consent checkbox (with polite copy!), and if a subscriber does not check it, they are not tracked. As simple as that!

We have long allowed account holders to do without user tracking (indeed, it is turned off altogether by default), and we have also always honoured "do not track" requests from browsers. Today's addition is to request opt-in consent for tracking of opens and clicks as part of the subscribe process. The reason for adding this is quite straightforward: the law requires it. GDPR's principles include this requirement for purpose limitation:

"Personal data shall be collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes"

This means that we can't take someone's data for one thing, and then use it for something else, in this case, we can't take someone's email address so that we can then send them mail, and then use it for tracking.

Then the principle of data minimisation applies:

"Personal data shall be adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed"

Taking a minimal approach (as this says), the only thing we require in order to send them email is their email address. We can ask for further data or processing permissions (such as tracking), but it's not specifically needed, so we require consent and be disabled by default.

You can read more about how exactly our tracking of opens and clicks works, and how they interact with the account-wide tracking setting and the Do Not Track header in our privacy docs.

All of our reporting features have been updated to support this change as well – you will now see how many subscribers have enabled tracking, and anonymous tracking records are marked clearly on reports.

Fundamentally though, this is all about putting control in the hands of the subscriber, where it belongs, which is one of the principles of privacy by design, also part of GDPR.

Meanwhile, all of this has been delivered on top of a major low-level update to our PHP code base: all of our code now runs in "strict types" mode, and we are taking full advantage of the strong typing features of PHP 7.4 – we will be looking at our migration to PHP 8.0 next.

We have also been paying attention to performance – you'll find things are even snappier than they were already: better caching, better compression, smaller code sizes, and better use of HTTP/2, while maintaining our customary levels of security.

Connectivity issues, and some good stuff too

Today we have experienced some network connectivity issues. Our hosting provider's data centre provider (Equinix) was very late in posting details of a power supply problem that degraded network connectivity for most of today, so if you thought things were slow, that's why.

In better news, we've not been posting here much because we've not had any problems to report! We have upgraded all our application servers to PHP 7.4, and everything is faster than ever. We have rolled out our multi-account management system (great for agencies running multiple brands, companies with branches and subsidiaries, universities with many departments), alongside shareable, pre-paid, per-message billing. As usual, we are right on top of our privacy controls, providing unparalleled data protection and privacy for both you and your subscribers.

Hey, and Bill & Ted 3 is coming out soon too! Party on, dudes!

System update - New servers, PHP 7.3, IPv6

We rolled out some behind-the-scenes changes last weekend:

• New servers! They're faster than the old ones.
• Now running on PHP 7.3 for improved performance, reliability, and a few new features.
• Improved security - now using the samesite flag on authentication cookies.
• New servers are IPv6-only! If you're on IPv4, you still have access via our hosting provider's gateway.
• Lots of small tweaks & fixes.

Enjoy the upgrades!

System update - more improvements and fixes

Yet more goodies for you!

New privacy policy

Our new privacy policy is now live; there is also an account-specific version that's used on your subscribe pages, and anywhere else your subscribers interact with our pages.

Privacy improvements

Our normal and subscriber login pages are now entirely cookie-free on first hit, and only set session cookies if you actually log in.

The subscriber portal no longer asks for cookie consent as no non-essential, non-session cookies are set at all, and access is not subject to our T&Cs as it's a statutory requirement.

We've tightened our Content Security Policy headers even further, gaining us an excellent set of results on this privacy checker.

Our strict-origin-when-cross-origin referrer policy means only our domain is passed in referrer headers, and then only to secure pages; it's important that the full referrer URL is not leaked. In the event that a public page we host contains personal data - such as on personalised web versions which rely on unguessable URLs - outbound links must not point directly back to the page via the referrer header. This is why we don't use such pages by default, and have always used a referrer policy that does not leak the full URL.

Privacy notices that mention Do Not Track now show whether the current browser has that option set.

Template zip imports completed

As we mentioned last time, we support importing templates from zip files; this has now been completed, and now includes support for bundled images in an images folder. The docs have been updated to show how to use that.

Templates now default to using HTML5 instead of the previous HTML4.01/transitional doctype.

RSS

Our RSS feed reader code is now smaller and faster, and should be more reliable into the bargain. In case you didn't know, we can do all kinds of cool things with RSS, meaning you don't need to duplicate effort in creating templates if you're already writing blog posts, or if your ecommerce system provides RSS feeds of special offers, etc.

Interface improvements

Drop-down menus now make full use of available screen height - this is especially useful on long menus such as those used for time zones and template tags.

Corruption in content using the UTF-8 character set (i.e. all of it!) is now filtered much more reliably, and corruption resulting from mixing ISO-8859-1 and UTF-8 character sets in the same content is now dealt with automatically.

The syntax check indicator on the send page no longer defaults to showing an error!

Logging out manually now says that's what you did, not that your connection timed out.

Callbacks

We have added a new "delete" event to our callback system, meaning you will receive a notification on your callback URL when a subscriber chooses to delete their own data (a requirement of GDPR), allowing you to keep your own systems up to date with respect to your users' preferences.

More lists in subscriber portal

In the subscriber data access portal, we now show subscriptions to lists that are not marked as visible so that subscribers may choose to unsubscribe from them. We'd like to remind you that mailing list names can be shown to subscribers, so it helps to give lists meaningful names, and also to make use of our separate public and internal list names. One of our competitors had an embarrassing incident relating to unexpectedly visible list names, and we'd like you to avoid the same fate!

Á la prochaine!

System update - privacy enhancements

We've been rolling out numerous small updates over the last few months, and we've just pushed a big one. It's mainly about our data protection configuration for GDPR and ePR compliance. Smartmessages has always had a very strong policy on data protection, and this means we have not had to change anything fundamental for GDPR (that's why you're not seeing any of those silly "reconsenting" emails from us), however, we have improved some smaller things to enhance our compliance.

Support for Do Not Track

The biggest change is that we now support "Do Not Track " and anonymous tracking. If a subscriber opens a message we send, or clicks a link in a message, those requests are served by Smartmessages, and normally we record these in full, including the identity of the subscriber (something which is mentioned at the point of sign-up so subscribers are aware of this before they subscribe - see below). If a subscriber has the "Do Not Track" setting enabled in their browser, we will still record that an open or click has occurred, and which mailshot that it happened in, but we do not record their identity. This will mean that you see some new stats in your mailshot reports for anonymous opens and clicks, and subscribers making use of this feature will not appear in "Hot List" reports.

Enhanced Data Subject Access Request support

Under GDPR and earlier data protection law, anyone that you store data about can request to see, amend, and delete the data that you store about them. These are called DSARs. It's extremely rare for web apps to have any integrated support for DSARs, but we introduced built-in support in 2005. Anyone whose data is used by Smartmessages (whether as an account holder or list subscriber) can log in and see the data that is stored about them, and amend or delete it at will, as is their right.

Improved data retention implementation

We have deleted low-level data after 6 months for many years, but there were some places where user data was kept unnecessarily, particularly in archived mailshots (ones more than 6 months old). We have made some internal changes to make it easier for us to delete data held in logs and archived mailshots, either due to expiration or DSARs.

Subscription page privacy policy

We have clarified key items of our privacy policy on our standard subscribe and landing pages, right where it's needed most. You can see in in action on our own subscribe form. If you host your own subscribe forms, you need to present these same options to your potential subscribers - to skimp on that means that even double-opt-in subscriptions will be invalid since transparency of processing is a requirement under GDPR.

Landing page improvements

Our default landing/preferences page — a simple destination to manage multiple subscriptions and basic data collection — has had a cleanup, making the layout more compact and easier to use on mobile devices, and also easier to customise by providing more ID and class selectors for your custom CSS to target.

Gravatar privacy proxy

Previously we made direct use of the Gravatar service (operated by Wordpress) to provide avatars for subscribers. Doing this leaks IP addresses of the subscribers to a US-based entity without explicit permission, and we don't like that, so we implemented a proxy service that means that subscriber avatars are served via our own servers, in a way that means that Gravatar is never contacted by subscribers directly, and their IPs are never revealed. This was the only remaining external service that could handle subscriber data, so now we can be certain that data is shared with nobody except  Smartmessages account holders, who are the data controllers for subscriber data. Yes - we're now entirely free of tracking cookies and scripts.

Enhanced Content Security Policy

We have strengthened our content security policy (a technical feature in HTTP) substantially. This mainly applies to the smm.im domain that we use for open & click tracking, and for serving images. The new configuration now means browsers will reject anything served from this domain that's not an image. This helps us stay off malware scanners - if someone should ever manage to upload, for example, a malicious javascript file that ends up served from this domain, browsers will refuse to load it. Our CSP has been tightened on the rest of our sites too, and that may interfere with things that rely on privacy contraventions, such as Facebook "like" buttons. We also no longer leak data through HTTP referrer headers - some other ESPs had serious issues with this, but we were never exposed to that. This will not affect mailings as normal HTTP links continue to work just fine.

You're welcome to test our domains at any time, using tools like securityheaders.com and Qualys SSL labs. Should you find a security issue that you would like to report to us privately, please use our standard security.txt file. Of course you should run the same tests on our competition too!

Importing mailing lists

We've always supported importing mailing lists, but one very common aspect of exported lists is a lack of information about the origin of the subscription, and precisely when it occurred. We have supported the export of this information in our list exports for many years, however, we have not supported it on import. That's now changed, and we are now able to import IP, timestamp, referrer URL, and user agent strings used by subscribers (at the point of subscription confirmation) from imported records. Imported records that contain a valid public IP and timestamp will now be marked as having completed a double-opt-in process. This is specifically compatible with the format used by Mailchimp, making migration to Smartmessages even easier. Documentation on our export and import formats has been updated to match.

Importing templates

Importing templates has been an important feature since the beginning, but to date it's been limited to copy/paste, or importing from a URL. We now support importing from local files, and specifically to import from zip files containing HTML & plain text files. This is a common format used by third-party email template creation tools, and also used in exports from various other ESPs. We automatically apply format conversions so that templates designed for other ESPs can work as expected - though of course you should always test before committing to a big send. We will be adding the ability to import images linked to these templates automatically as well.

Improved Excel report generation

You won't see much visible difference, but the system used for generating Excel-format reports has been overhauled, switching to a new PHPSpreadsheet implementation, which should be faster and more reliable.

Migration to PHP 7.1 complete

All our services are now running on at least PHP 7.1, and in some parts, 7.2, helping both security and performance. Work on migration to PHP 7.3 and MySQL 8.0 has already begun.

We also switched this status blog to use HTTPS. Woohoo!

I think that's quite enough to be getting on with, but there is more to come! As always, if you would like to ask us anything, contact us.

System update in progress

Smartmessages will be having several short outages over the next few hours while we are deploying some major changes. More news later.

Smartmessages performance boost

Today we're happy to announce a big performance boost to smartmessages.net. We've migrated our whole platform to PHP 7, so you'll see a huge boost in page load speeds (which were pretty good to start with!) and email send rates.

We've also improved our TLS configuration (still an A+ rating on SSL Labs!) with the addition of ChaCha20/Poly1305 cipher suites, which give massively improved performance on low-power devices such as phones.

We hope you like it!

Database update

Yesterday we ran into some database issues that resulted in the system being very slow. We had partly commissioned a new database server, so rather than fixing the problem on the old server, we worked around it by completing the configuration of the new server and deploying it into production.
This accomplished what we hoped - a massive performance improvement across the board.

We apologies for the slow performance yesterday, but hope you enjoy the enormous speed boost you've got today!

Achievement unlocked: HTTP/2 deployed!

We're happy to announce that we've completed deployment of HTTP/2 support across all our services. In addition to our previous support for SSL everywhere and IPv6, we now complete the set with HTTP/2. Expect a fair performance boost on all web page interactions, especially on mobiles.

If you're running a very old browser, you won't see any difference - but there's never been a better time to upgrade!

Big technical update

We've just rolled out a pretty hefty technical upgrade to our back-end systems.

• Switched our web servers from apache to nginx (faster)
• Updated from PHP 5.4 to 5.6 (faster, better)
• Enabled IPv6 (faster, better on modern networks)
• Updated to SPDY 3.1 (faster)
• Enabled the HSTS header (safer) - we now get an A+ on SSLLabs tests.

You should find things work exactly as before, just faster and more securely!

The HSTS header means that we now always serve everything over SSL encryption, including mailshot web versions and images. If you are hosting your own mailshot images on an insecure site, this may mean that recipients viewing a web version are either not shown images or see a security warning about insecure content from your site. To fix this, either upgrade your site to use SSL (we can help with that), or move your image hosting onto our servers.

Major upgrade, scheduled downtime

As announced to all our subscribers this week, there will be downtime this Saturday evening while we deploy a major new update featuring a massive upgrade to our templating system. Hopefully you're all busy doing nice things like watching the World Cup. Keep an eye on this post for updates.

• 2014-06-14 21:02 Upgrade process started, scheduled sending paused.
• 2014-06-15 04:18 New app deployed, but database migrations taking longer than expected.
• 2014-06-15 10:10 Everything is available again, some template conversions still in progress.
• 2014-06-15 14:22 Conversions complete.
• 2014-06-15 21:02 Some teething troubles dealt with!

A new-look Smartmessages

Last weekend we rolled out a complete overhaul of our user interface. For the most part it was a revamp of the underlying systems that provide our web interface, so it won't come as too much of a shock to our long-term users, but there are hundreds of small visual and usability tweaks throughout the site.

So what's changed under the hood?

• Switch to HTML5 based on html5boilerplate
• A new consistent CSS framework built on Twitter Bootstrap 2 and LESS
• Responsive layouts for improved usability on small screens
• A complete overhaul of template structuring using Smarty 3's inheritance
• Switch from Prototype.js to jQuery (1.7.2), requiring a rewrite of all Javascript
• Removal of nearly all images
• Use of vector artwork and fonts (with FontAwesome)
• Javascript graphs courtesy of Morris.js
• YSlow optimisation
• SPDY protocol support

We're hitting a score of 98 on Yslow - we would be at 100 if Google set longer expiries on their web fonts! We actually had to report bugs in Yslow as we were being marked down for using SPDY! Replacing images with font glyphs generally means that things look much cleaner and sharper on high-resolution devices such as iPad 3, plus it helps reduce the number of trips to the server.
Combined with SPDY support in browsers that support it (Firefox 11+ and Chrome), this makes for a much snappier experience overall. SPDY is a good fit for Smartmessages as we deliver everything over HTTPS anyway.

All of these changes took about a month to implement, and they improve our ability to change things in future, so we've now got a great base on which to build the new features we're planning!

We hope you like it.

New features: RSS and more!

We're very happy to announce some big new features:
RSS
Many customers have asked for this, and it's finally here! We've had internal suport for it for years, but now it's out in the open. You can now build templates that dynamically import content from an RSS feed at send time, and you can format the overall feed and the stories within it separately. This is  great way to shorten the amount of time it takes to prepare a newsletter as you don't have to refomat content from elsewhere - just point it at your blog/CMS and have it pull in your latest stories automatically. Docs on how to use it are on our wiki.
Social network integration
We now have tags for Facebook 'like', Google+ '+1', Twitter 'tweet' and LinkedIn 'share' buttons. Just drop the tags into your templates and the rest will happen automatically, complete with open graph tags, allowing the networks to find important details about your mailings easily. You'll see a new section in reports for the stats you've got on social network activity. There are quite a few considerations when using social network buttons, so please read the docs.
Revamped callbacks system
Our callbacks system has had a major overhaul, has some new events, selective event registration, more consistent parameters, better handling of failures, and now supports retrying if your system is down. Of course there is updated documentation to go with it.
Gravatars
Gravatars allow to you to obtain personalised icons associated with an email address via gravatar.com. If a user has set their own icon, it will be used, otherwise it will generate a unique one from their address. For example, here's the one for info@smartmessages.net:

and here's one for an address that has no predefined icon (and is thus generated automatically):

We've been using them in our admin interface for some time, but now you can use them in your templates too.
There are lots of other small tweaks throughout the system, and as always, there's plenty more to come!

New bounce report

When you look at a report for one of your mailshots, you'll find a new link to a bounce report for it. This shows a more detailed breakdown of the kind of bounces that you've received, and the top 20 domains responsible for them.

Scheduled maintenance Aug 13th

We will be conducting some maintenance tasks on the evening of Satuday 13th August. Smartmessages services will be unavailable for several hours starting from 7pm UK time.

More new features!

We've just rolled out some more new features,  improvements and bug-fixes:

• Automatically import template images - no more boring image uploading!
• You can now delete unsent mailshots
• JavaScript library updates
• Faster preview images and web versions
• Improved stats on contacts page
• Update to the latest WYSIWYG editor
• Editor config tweaks to make images work better in Outlook (height, width and align attributes)
• New tooltips!
• More detail in the image report (file sizes, dimensions)
• Fixed several issues in Excel campaign reports

We hope you like the changes, and please let us know if you run into any trouble, of if we just made your day!

New Smartmessages web site

We're very happy to roll out our new smartmessages web site today. It's got lots more info than before, deeper content, it's much prettier and it's easier for us to maintain and update. We hope you like it!

Interface update

We rolled out some small interface changes today.

• Most obvious is a new font (Ubuntu), which will tie in with our soon-to-be-released web site.
• The login page now redirects straight to the home page if you're already logged in.
• Switched to monospaced fonts in places where you'll be editing HTML code.
• Several clean-ups and minor fixes for Javascript, especially in IE.
• More space to work on the templates page.
• Gravatar icons in a few places.

Scheduled maintenance May 4th - updated

We are performing a major software upgrade on our mail servers on the evening of Wednesday May 4th, starting from 6pm UK time and lasting for 2-3 hours. The Smartmessages web interface will remain available, but no mailshots will be sent during the maintenance window.

Update: I'm sorry to report that our software upgrades were curtailed by a major hardware failure last night. We're working on a fix.

Update: We've now provisioned new servers to take over the failed servers and all services are now running normally. Many apologies for the delay to today's sends.