Scheduled maintenance

Our hosting provider is performing some hardware upgrades on the servers we run on, so there will be an outage tonight of several hours. Since this is the middle of the night in Europe, this should not affect many of our customers or their subscribers.

It's privacy time!

As you've no doubt gathered, at Smartmessages we are very keen on privacy and preserving the rights of subscribers. Today adds an important privacy upgrade: tracking consent.

When someone consents for us to send them email for marketing purposes, we are required to be able to demonstrate that they did so with full transparency (GDPR's accountability requirement) and via an explicit, informed opt-in mechanism (a requirement of PECR / EPD rather than GDPR). So when we gather someone'e email address for the purpose of sending them messages, that is all that we are asking them for; at no point are they asking to be tracked, and historically, nor have we asked them if it's ok if we do. This is a clear contravention of the purpose limitation. That changes today. All of our subscribe forms now include a tracking consent checkbox (with polite copy!), and if a subscriber does not check it, they are not tracked. As simple as that!

We have long allowed account holders to do without user tracking (indeed, it is turned off altogether by default), and we have also always honoured "do not track" requests from browsers. Today's addition is to request opt-in consent for tracking of opens and clicks as part of the subscribe process. The reason for adding this is quite straightforward: the law requires it. GDPR's principles include this requirement for purpose limitation:

"Personal data shall be collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes"

This means that we can't take someone's data for one thing, and then use it for something else, in this case, we can't take someone's email address so that we can then send them mail, and then use it for tracking.

Then the principle of data minimisation applies:

"Personal data shall be adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed"

Taking a minimal approach (as this says), the only thing we require in order to send them email is their email address. We can ask for further data or processing permissions (such as tracking), but it's not specifically needed, so we require consent and be disabled by default.

You can read more about how exactly our tracking of opens and clicks works, and how they interact with the account-wide tracking setting and the Do Not Track header in our privacy docs.

All of our reporting features have been updated to support this change as well – you will now see how many subscribers have enabled tracking, and anonymous tracking records are marked clearly on reports.

Fundamentally though, this is all about putting control in the hands of the subscriber, where it belongs, which is one of the principles of privacy by design, also part of GDPR.

Meanwhile, all of this has been delivered on top of a major low-level update to our PHP code base: all of our code now runs in "strict types" mode, and we are taking full advantage of the strong typing features of PHP 7.4 – we will be looking at our migration to PHP 8.0 next.

We have also been paying attention to performance – you'll find things are even snappier than they were already: better caching, better compression, smaller code sizes, and better use of HTTP/2, while maintaining our customary levels of security.

Connectivity issues, and some good stuff too

Today we have experienced some network connectivity issues. Our hosting provider's data centre provider (Equinix) was very late in posting details of a power supply problem that degraded network connectivity for most of today, so if you thought things were slow, that's why.

In better news, we've not been posting here much because we've not had any problems to report! We have upgraded all our application servers to PHP 7.4, and everything is faster than ever. We have rolled out our multi-account management system (great for agencies running multiple brands, companies with branches and subsidiaries, universities with many departments), alongside shareable, pre-paid, per-message billing. As usual, we are right on top of our privacy controls, providing unparalleled data protection and privacy for both you and your subscribers.

Hey, and Bill & Ted 3 is coming out soon too! Party on, dudes!

System update - New servers, PHP 7.3, IPv6

We rolled out some behind-the-scenes changes last weekend:

• New servers! They're faster than the old ones.
• Now running on PHP 7.3 for improved performance, reliability, and a few new features.
• Improved security - now using the samesite flag on authentication cookies.
• New servers are IPv6-only! If you're on IPv4, you still have access via our hosting provider's gateway.
• Lots of small tweaks & fixes.

Enjoy the upgrades!

System update - more improvements and fixes

Yet more goodies for you!

New privacy policy

Our new privacy policy is now live; there is also an account-specific version that's used on your subscribe pages, and anywhere else your subscribers interact with our pages.

Privacy improvements

Our normal and subscriber login pages are now entirely cookie-free on first hit, and only set session cookies if you actually log in.

The subscriber portal no longer asks for cookie consent as no non-essential, non-session cookies are set at all, and access is not subject to our T&Cs as it's a statutory requirement.

We've tightened our Content Security Policy headers even further, gaining us an excellent set of results on this privacy checker.

Our strict-origin-when-cross-origin referrer policy means only our domain is passed in referrer headers, and then only to secure pages; it's important that the full referrer URL is not leaked. In the event that a public page we host contains personal data - such as on personalised web versions which rely on unguessable URLs - outbound links must not point directly back to the page via the referrer header. This is why we don't use such pages by default, and have always used a referrer policy that does not leak the full URL.

Privacy notices that mention Do Not Track now show whether the current browser has that option set.

Template zip imports completed

As we mentioned last time, we support importing templates from zip files; this has now been completed, and now includes support for bundled images in an images folder. The docs have been updated to show how to use that.

Templates now default to using HTML5 instead of the previous HTML4.01/transitional doctype.

RSS

Our RSS feed reader code is now smaller and faster, and should be more reliable into the bargain. In case you didn't know, we can do all kinds of cool things with RSS, meaning you don't need to duplicate effort in creating templates if you're already writing blog posts, or if your ecommerce system provides RSS feeds of special offers, etc.

Interface improvements

Drop-down menus now make full use of available screen height - this is especially useful on long menus such as those used for time zones and template tags.

Corruption in content using the UTF-8 character set (i.e. all of it!) is now filtered much more reliably, and corruption resulting from mixing ISO-8859-1 and UTF-8 character sets in the same content is now dealt with automatically.

The syntax check indicator on the send page no longer defaults to showing an error!

Logging out manually now says that's what you did, not that your connection timed out.

Callbacks

We have added a new "delete" event to our callback system, meaning you will receive a notification on your callback URL when a subscriber chooses to delete their own data (a requirement of GDPR), allowing you to keep your own systems up to date with respect to your users' preferences.

More lists in subscriber portal

In the subscriber data access portal, we now show subscriptions to lists that are not marked as visible so that subscribers may choose to unsubscribe from them. We'd like to remind you that mailing list names can be shown to subscribers, so it helps to give lists meaningful names, and also to make use of our separate public and internal list names. One of our competitors had an embarrassing incident relating to unexpectedly visible list names, and we'd like you to avoid the same fate!

Á la prochaine!

System update - privacy enhancements

We've been rolling out numerous small updates over the last few months, and we've just pushed a big one. It's mainly about our data protection configuration for GDPR and ePR compliance. Smartmessages has always had a very strong policy on data protection, and this means we have not had to change anything fundamental for GDPR (that's why you're not seeing any of those silly "reconsenting" emails from us), however, we have improved some smaller things to enhance our compliance.

Support for Do Not Track

The biggest change is that we now support "Do Not Track " and anonymous tracking. If a subscriber opens a message we send, or clicks a link in a message, those requests are served by Smartmessages, and normally we record these in full, including the identity of the subscriber (something which is mentioned at the point of sign-up so subscribers are aware of this before they subscribe - see below). If a subscriber has the "Do Not Track" setting enabled in their browser, we will still record that an open or click has occurred, and which mailshot that it happened in, but we do not record their identity. This will mean that you see some new stats in your mailshot reports for anonymous opens and clicks, and subscribers making use of this feature will not appear in "Hot List" reports.

Enhanced Data Subject Access Request support

Under GDPR and earlier data protection law, anyone that you store data about can request to see, amend, and delete the data that you store about them. These are called DSARs. It's extremely rare for web apps to have any integrated support for DSARs, but we introduced built-in support in 2005. Anyone whose data is used by Smartmessages (whether as an account holder or list subscriber) can log in and see the data that is stored about them, and amend or delete it at will, as is their right.

Improved data retention implementation

We have deleted low-level data after 6 months for many years, but there were some places where user data was kept unnecessarily, particularly in archived mailshots (ones more than 6 months old). We have made some internal changes to make it easier for us to delete data held in logs and archived mailshots, either due to expiration or DSARs.

Subscription page privacy policy

We have clarified key items of our privacy policy on our standard subscribe and landing pages, right where it's needed most. You can see in in action on our own subscribe form. If you host your own subscribe forms, you need to present these same options to your potential subscribers - to skimp on that means that even double-opt-in subscriptions will be invalid since transparency of processing is a requirement under GDPR.

Landing page improvements

Our default landing/preferences page — a simple destination to manage multiple subscriptions and basic data collection — has had a cleanup, making the layout more compact and easier to use on mobile devices, and also easier to customise by providing more ID and class selectors for your custom CSS to target.

Gravatar privacy proxy

Previously we made direct use of the Gravatar service (operated by Wordpress) to provide avatars for subscribers. Doing this leaks IP addresses of the subscribers to a US-based entity without explicit permission, and we don't like that, so we implemented a proxy service that means that subscriber avatars are served via our own servers, in a way that means that Gravatar is never contacted by subscribers directly, and their IPs are never revealed. This was the only remaining external service that could handle subscriber data, so now we can be certain that data is shared with nobody except  Smartmessages account holders, who are the data controllers for subscriber data. Yes - we're now entirely free of tracking cookies and scripts.

Enhanced Content Security Policy

We have strengthened our content security policy (a technical feature in HTTP) substantially. This mainly applies to the smm.im domain that we use for open & click tracking, and for serving images. The new configuration now means browsers will reject anything served from this domain that's not an image. This helps us stay off malware scanners - if someone should ever manage to upload, for example, a malicious javascript file that ends up served from this domain, browsers will refuse to load it. Our CSP has been tightened on the rest of our sites too, and that may interfere with things that rely on privacy contraventions, such as Facebook "like" buttons. We also no longer leak data through HTTP referrer headers - some other ESPs had serious issues with this, but we were never exposed to that. This will not affect mailings as normal HTTP links continue to work just fine.

You're welcome to test our domains at any time, using tools like securityheaders.com and Qualys SSL labs. Should you find a security issue that you would like to report to us privately, please use our standard security.txt file. Of course you should run the same tests on our competition too!

Importing mailing lists

We've always supported importing mailing lists, but one very common aspect of exported lists is a lack of information about the origin of the subscription, and precisely when it occurred. We have supported the export of this information in our list exports for many years, however, we have not supported it on import. That's now changed, and we are now able to import IP, timestamp, referrer URL, and user agent strings used by subscribers (at the point of subscription confirmation) from imported records. Imported records that contain a valid public IP and timestamp will now be marked as having completed a double-opt-in process. This is specifically compatible with the format used by Mailchimp, making migration to Smartmessages even easier. Documentation on our export and import formats has been updated to match.

Importing templates

Importing templates has been an important feature since the beginning, but to date it's been limited to copy/paste, or importing from a URL. We now support importing from local files, and specifically to import from zip files containing HTML & plain text files. This is a common format used by third-party email template creation tools, and also used in exports from various other ESPs. We automatically apply format conversions so that templates designed for other ESPs can work as expected - though of course you should always test before committing to a big send. We will be adding the ability to import images linked to these templates automatically as well.

Improved Excel report generation

You won't see much visible difference, but the system used for generating Excel-format reports has been overhauled, switching to a new PHPSpreadsheet implementation, which should be faster and more reliable.

Migration to PHP 7.1 complete

All our services are now running on at least PHP 7.1, and in some parts, 7.2, helping both security and performance. Work on migration to PHP 7.3 and MySQL 8.0 has already begun.

We also switched this status blog to use HTTPS. Woohoo!

I think that's quite enough to be getting on with, but there is more to come! As always, if you would like to ask us anything, contact us.

System update in progress

Smartmessages will be having several short outages over the next few hours while we are deploying some major changes. More news later.