Friday 22 June 2018

System update - more improvements and fixes

Yet more goodies for you!

New privacy policy

Our new privacy policy is now live; there is also an account-specific version that's used on your subscribe pages, and anywhere else your subscribers interact with our pages.

Privacy improvements

Our normal and subscriber login pages are now entirely cookie-free on first hit, and only set session cookies if you actually log in.

The subscriber portal no longer asks for cookie consent as no non-essential, non-session cookies are set at all, and access is not subject to our T&Cs as it's a statutory requirement.

We've tightened our Content Security Policy headers even further, gaining us an excellent set of results on this privacy checker.

Our strict-origin-when-cross-origin referrer policy means only our domain is passed in referrer headers, and then only to secure pages; it's important that the full referrer URL is not leaked. In the event that a public page we host contains personal data - such as on personalised web versions which rely on unguessable URLs - outbound links must not point directly back to the page via the referrer header. This is why we don't use such pages by default, and have always used a referrer policy that does not leak the full URL.

Privacy notices that mention Do Not Track now show whether the current browser has that option set.

Template zip imports completed

As we mentioned last time, we support importing templates from zip files; this has now been completed, and now includes support for bundled images in an images folder. The docs have been updated to show how to use that.

Templates now default to using HTML5 instead of the previous HTML4.01/transitional doctype.


Our RSS feed reader code is now smaller and faster, and should be more reliable into the bargain. In case you didn't know, we can do all kinds of cool things with RSS, meaning you don't need to duplicate effort in creating templates if you're already writing blog posts, or if your ecommerce system provides RSS feeds of special offers, etc.

Interface improvements

Drop-down menus now make full use of available screen height - this is especially useful on long menus such as those used for time zones and template tags.

Corruption in content using the UTF-8 character set (i.e. all of it!) is now filtered much more reliably, and corruption resulting from mixing ISO-8859-1 and UTF-8 character sets in the same content is now dealt with automatically.

The syntax check indicator on the send page no longer defaults to showing an error!

Logging out manually now says that's what you did, not that your connection timed out.


We have added a new "delete" event to our callback system, meaning you will receive a notification on your callback URL when a subscriber chooses to delete their own data (a requirement of GDPR), allowing you to keep your own systems up to date with respect to your users' preferences.

More lists in subscriber portal

In the subscriber data access portal, we now show subscriptions to lists that are not marked as visible so that subscribers may choose to unsubscribe from them. We'd like to remind you that mailing list names can be shown to subscribers, so it helps to give lists meaningful names, and also to make use of our separate public and internal list names. One of our competitors had an embarrassing incident relating to unexpectedly visible list names, and we'd like you to avoid the same fate!

Á la prochaine!

Monday 14 May 2018

System update - privacy enhancements

We've been rolling out numerous small updates over the last few months, and we've just pushed a big one. It's mainly about our data protection configuration for GDPR and ePR compliance. Smartmessages has always had a very strong policy on data protection, and this means we have not had to change anything fundamental for GDPR (that's why you're not seeing any of those silly "reconsenting" emails from us), however, we have improved some smaller things to enhance our compliance.

Support for Do Not Track

The biggest change is that we now support "Do Not Track " and anonymous tracking. If a subscriber opens a message we send, or clicks a link in a message, those requests are served by Smartmessages, and normally we record these in full, including the identity of the subscriber (something which is mentioned at the point of sign-up so subscribers are aware of this before they subscribe - see below). If a subscriber has the "Do Not Track" setting enabled in their browser, we will still record that an open or click has occurred, and which mailshot that it happened in, but we do not record their identity. This will mean that you see some new stats in your mailshot reports for anonymous opens and clicks, and subscribers making use of this feature will not appear in "Hot List" reports.

Enhanced Data Subject Access Request support

Under GDPR and earlier data protection law, anyone that you store data about can request to see, amend, and delete the data that you store about them. These are called DSARs. It's extremely rare for web apps to have any integrated support for DSARs, but we introduced built-in support in 2005. Anyone whose data is used by Smartmessages (whether as an account holder or list subscriber) can log in and see the data that is stored about them, and amend or delete it at will, as is their right.

Improved data retention implementation

We have deleted low-level data after 6 months for many years, but there were some places where user data was kept unnecessarily, particularly in archived mailshots (ones more than 6 months old). We have made some internal changes to make it easier for us to delete data held in logs and archived mailshots, either due to expiration or DSARs.

Subscription page privacy policy

We have clarified key items of our privacy policy on our standard subscribe and landing pages, right where it's needed most. You can see in in action on our own subscribe form. If you host your own subscribe forms, you need to present these same options to your potential subscribers - to skimp on that means that even double-opt-in subscriptions will be invalid since transparency of processing is a requirement under GDPR.

Landing page improvements

Our default landing/preferences page — a simple destination to manage multiple subscriptions and basic data collection — has had a cleanup, making the layout more compact and easier to use on mobile devices, and also easier to customise by providing more ID and class selectors for your custom CSS to target.

Gravatar privacy proxy

Previously we made direct use of the Gravatar service (operated by Wordpress) to provide avatars for subscribers. Doing this leaks IP addresses of the subscribers to a US-based entity without explicit permission, and we don't like that, so we implemented a proxy service that means that subscriber avatars are served via our own servers, in a way that means that Gravatar is never contacted by subscribers directly, and their IPs are never revealed. This was the only remaining external service that could handle subscriber data, so now we can be certain that data is shared with nobody except  Smartmessages account holders, who are the data controllers for subscriber data. Yes - we're now entirely free of tracking cookies and scripts.

Enhanced Content Security Policy

We have strengthened our content security policy (a technical feature in HTTP) substantially. This mainly applies to the domain that we use for open & click tracking, and for serving images. The new configuration now means browsers will reject anything served from this domain that's not an image. This helps us stay off malware scanners - if someone should ever manage to upload, for example, a malicious javascript file that ends up served from this domain, browsers will refuse to load it. Our CSP has been tightened on the rest of our sites too, and that may interfere with things that rely on privacy contraventions, such as Facebook "like" buttons. We also no longer leak data through HTTP referrer headers - some other ESPs had serious issues with this, but we were never exposed to that. This will not affect mailings as normal HTTP links continue to work just fine.

You're welcome to test our domains at any time, using tools like and Qualys SSL labs. Should you find a security issue that you would like to report to us privately, please use our standard security.txt file. Of course you should run the same tests on our competition too!

Importing mailing lists

We've always supported importing mailing lists, but one very common aspect of exported lists is a lack of information about the origin of the subscription, and precisely when it occurred. We have supported the export of this information in our list exports for many years, however, we have not supported it on import. That's now changed, and we are now able to import IP, timestamp, referrer URL, and user agent strings used by subscribers (at the point of subscription confirmation) from imported records. Imported records that contain a valid public IP and timestamp will now be marked as having completed a double-opt-in process. This is specifically compatible with the format used by Mailchimp, making migration to Smartmessages even easier. Documentation on our export and import formats has been updated to match.

Importing templates

Importing templates has been an important feature since the beginning, but to date it's been limited to copy/paste, or importing from a URL. We now support importing from local files, and specifically to import from zip files containing HTML & plain text files. This is a common format used by third-party email template creation tools, and also used in exports from various other ESPs. We automatically apply format conversions so that templates designed for other ESPs can work as expected - though of course you should always test before committing to a big send. We will be adding the ability to import images linked to these templates automatically as well.

Improved Excel report generation

You won't see much visible difference, but the system used for generating Excel-format reports has been overhauled, switching to a new PHPSpreadsheet implementation, which should be faster and more reliable.

Migration to PHP 7.1 complete

All our services are now running on at least PHP 7.1, and in some parts, 7.2, helping both security and performance. Work on migration to PHP 7.3 and MySQL 8.0 has already begun.

We also switched this status blog to use HTTPS. Woohoo!

I think that's quite enough to be getting on with, but there is more to come! As always, if you would like to ask us anything, contact us.

Sunday 13 May 2018

System update in progress

Smartmessages will be having several short outages over the next few hours while we are deploying some major changes. More news later.