A Modern Museum Web Stack
AbstractThe Museum of Applied Arts and Sciences launched their new Wordpress website in December 2014. Before development started, the team conducted extensive research on current best practises for modern web development. This How-To session details the results and application of our research. It is split up into the main three components of a web stack - infrastructure, backend and frontend. This session will be conducted by Kaho Cheung (Senior Digital Developer/Producer) and Rowan Stenhouse (Senior Digital Developer/Producer) from the Digital Studio team.
Keywords: stack, wordpress, bedrock, ansible, kitchen-ci, gulp
A Modern Museum Web Stack
The landscape of the internet and the web has changed massively in the past few years. The challenge facing so many institutions with small development teams is how to develop and maintain modern web applications with limited time and resources. The Museum of Applied Arts and Sciences (MAAS) recently conducted research to improve almost every aspect of our web environment and we would like to share this journey with the wider museum community.
All the technologies listed in this paper are open source projects to some degree. The Museum’s web team has a strong focus on open source and consider it a prerequisite for all our projects. We have found that these solutions are often more flexible and dependable than their closed source counterparts. They allow the Museum’s small development team to rapidly develop innovative solutions and to “stand on the shoulders of giants”.
This paper is broken down into three main sections – infrastructure, backend and frontend. Each section provides a general overview of how we modernised our development practises. For areas of interest, we have provided links for further reading.
The software that underlies a web application is a regularly forgotten – although it is a critical component in a web application. In many organisations, the job of setting up the server is left to a system administrator or a VPS provider who has little understanding of the web application. More effort is often expended on giving the server a pet name than ensuring it is optimised to run code. The end result is the web application cannot take full advantage of the hardware it is running on. Developers also have a poor understanding of how to address performance, security, and stability issues when they arise.
In the past the Museum has utilised a traditional Linux, Apache, MySQL and PHP (LAMP) stack as it is easy to maintain and well understood. It is the go-to setup for many open source applications, has excellent community support and a proven track record. The shortcomings of such a setup become apparent when you run complex and database intensive applications in high load production environments. The end result is application response times suffer during high demand and end users quickly become frustrated. MAAS has experienced performance issues with our web applications at peak times such as school holidays and special events. At the beginning of 2015 the team started an investigation into how we could improve our infrastructure in three key areas: performance, security and stability.
In terms of performance, our goal was to reduce the page load times and also be able to handle a higher level of site visitors. It is our strong belief that the responsiveness of a website is a key user experience indicator and studied how responsiveness could be improved through both server optimisation and frontend/backend improvements.
Web application security is vitally important these days. To keep on top of security issues, the team required systems that were easy to patch and keep up to date as hackers often exploit vulnerabilities before they are have been updated. Keeping our web user’s information secure while in transmission was also very important to us so. We studied SSL/TLS best practices with an aim to provide HTTPS for all our sites that handle user information. Google now uses HTTPS as a ranking signal in Google search as part of their “HTTPS everywhere” (Bahajji & Illyes, 2014) initiative, giving us another reason to implement it across our web applications.
Infrastructure As Code
The key issue was how to rapidly implement changes to our infrastructure while maintaining application stability. As a result, one of the major concepts we introduced to the Museum this year is infrastructure as code (Sitakange, 2014). Instead of viewing server software and configuration as monolithic, we treat it the same as a web application, something that can be improved upon and evolve with time.
Defining infrastructure as code allows the build process of servers to be stored in repositories instead of being kept in forgotten documentation or even worse, someone’s head. It also lowers the bar for many developers who see server configuration and management complex and difficult to understand. Once the infrastructure they use everyday is displayed to them in the form of code, the undertaking of making additions and improvements is far more approachable.
The process of attempting such a task has been made much easier in recent years with the rise of tools such as Docker, Puppet, Ansible and Chef that make provisioning and configuring servers very approachable for the average developer.
After evaluating the various tools available, the Museum decided on Chef, as it has strong community support, a simple way of encrypting sensitive server configuration variables (database passwords, SSL private keys, etc) and tight integration with Amazon Web Services through Opsworks. In addition, the Chef Development Kit (ChefDK) comes with an application called Kitchen CI allowing us to test most aspects of our infrastructure before we use it in a production environment.
Kitchen CI enables the Museum to rapidly build and test new server software and configurations. This facilitates a more ambitious approach to our application stack, including the addition of new technologies like Google’s SPDY protocol and Facebook’s HipHop Virtual Machine. Testing a Chef build with Kitchen CI can be as simple as confirming a particular software package has been installed on the server to testing detailed performance characteristics of the system. Another great feature of Chef is data bags, a tool used to encrypt sensitive information about the server.
Because Chef is largely platform agnostic, it has also has given us the flexibility to choose where to deploy our web applications. The same Chef build can be provisioned on the Museum’s own virtual environment, a third party VPS provider or Amazon Web Services (AWS), depending on the requirements of a particular project. When using a VPS or AWS, a VM is provided which includes a preconfigured operating system, whereas our own in-house virtual environment (vSphere) and local development environment (VirtualBox) does not. To fill that gap we utilise a tool called Veewee that allows us to easily build clean installs from an operating system install (iso file) giving us the same consistency you would expect from AWS.
Along with Kitchen CI’s suite of tests we utilise a collection of quality, ‘free to use’ web services that allow us to tune various aspects of application stack.
Redbot is a simple to use tool for testing server response headers and is especially useful for checking that headers correctly for caching and compression.
SSL Server Test
Implementing HTTPS correctly can often be a complex and difficult task for novice developers. Qualys SSL Labs provides an excellent tool for confirming your HTTPS configuration is correct and that it prevents techniques used to subvert HTTP encryption such as heartbleed and BEAST. Qualys also provides easy to understand guides for implementing HTTPS on a variety of different web servers. Utilising the information available from Qualys has allowed the Museum to implement advanced HTTPS features such as Perfect Forward Secrecy and Strict Transport Security.
Pagespeed is a great tool for testing a variety of performance characteristics of your application. Although Pagespeed’s main focus is on front-end performance improvements, it does offer insights into how aspects of the infrastructure is performing.
Developing code on our new infrastructure also required a new approach to our own local development environments. We could not longer utilise local web servers like XAMPP and MAMP because of the vast differences between local and production environments. Due to differing PHP versions and other configuration options, code that worked on local machines would not reliably behave in the same manner on the production server.
To address this we started using the virtualisation software Vagrant. This program allows us to easily configure and run multiple virtual environments on our local machines using popular virtualisation applications such as VirtualBox or VMware Fusion. Consequently, we have gone from a team of developers running bespoke and often customised local environments, to having consistent and easy to duplicate environments which can be destroyed and rebuilt in as little as 20 minutes on modern hardware.
Vagrant makes running local copies of our web applications so easy that we are even considering setting it up on our content producers’ machines giving them a sandbox environment to test new content ideas.
We have now built a variety of new systems using these techniques including our main website, maas.museum. With the ability to build and test server infrastructure in place, we have been able to implement better security practices, improve the reliability of our applications and decrease application response times by an order of magnitude. The great thing about this approach is that it is only the a starting point. We will continue to develop our infrastructure with the same enthusiasm as the applications we run on top of them.
Diagram 2: Iteration and testing of our systems has taken us from the application stack on the left to the one on the right
In an environment where a web application is expected to be available 24/7, deploying code to a web server has often been a stressful event. Deployment frequently requires a carefully crafted sequence of events to ensure the application is still accessible while pushing new features and bug fixes to the site. The issue of code deployment has only become more pressing as the demand for quick turnarounds of new features has increased, so traditional approaches to code deployment have become unwieldy and prone to error.
Every developer has a story about a deployment that has gone wrong. Having great developers and good deployment practices is no guarantee that everything will go smoothly. What the Museum wanted to achieve was being able to make code deployments in a consistent and repeatable fashion, to record deployments made to a web application and track performance differences between deployments.
The Museum approached this task by comparing modern deployment tools including the Python based Ansible and Fabric, along with Ruby’s Capistrano. All of these tools have a rich feature set allowing rollback of deployments, front-end build processes (e.g. Grunt, Gulp), package management and deployment notifications.
We ended up choosing Capistrano because of its flexibility, ease of use and integration with our WordPress boilerplate, Bedrock. Capistrano makes it simple to deploy code to multiple environments in a consistent manner and to perform other tasks such as cache invalidation and logging of individual deployments. What Capistrano (and other deployment tools for that matter) cannot provide is tracking the performance characteristics of each deployment. This is where the web service New Relic comes into play.
New Relic is a real time application monitoring tool that the Museum uses to track application errors and performance. Having this information available to us has been invaluable in catching bugs that escape other means of testing. In conjunction with Capistrano, it allows us to make informed decisions about deployments and measure all aspects of our application stack.
While deploying code will always involve a certain aspect of finger crossing, the Museum has been able to place a reliable mechanism for rolling out new features while ensuring minimum disruption to our end users.
Like many institutions, we use a content management system (CMS) to allow members of staff to publish content to the web. WordPress is our CMS of choice as it is intuitive, has great community support and is extremely extensible. The downside is it misses out on some modern features that other applications take for granted, such as dependency management, deployment strategies and separation of configuration from code.
Bedrock WordPress Boilerplate
In our research, we discovered the Bedrock boilerplate brought many of these modern application framework features to WordPress. Built and maintained by Roots, this team of full stack developers studiously keep the project up to date with the latest web technologies.
The following features provide a good overview of Bedrock’s advantages over the stock standard WordPress install.
Many modern programming languages use dependency management to leverage open source code and manage complex code libraries. Composer is really the only game in town for modern PHP dependency management.
Bedrock comes with a base configuration file for Composer. This JSON file is used to download and update plugins, themes and even WordPress itself with one command. This makes it very easy to roll back to previous versions if an incompatibility is found.
Perhaps the biggest advantage of Composer is that WordPress core plugins and themes do not have to be uploaded into a Git repository. This is best practise as it keeps the repository free of vendor code and avoids dependency hell.
Using Composer also makes a particular WordPress site easily transportable. One member of the team can edit the JSON file to upgrade a plugin and then commit the change into the repository. Other members can then pull in the new changes, resulting in everyone using the same version on their local machines. Compared to manual upgrades in WordPress, Composer prevents mismatched WordPress and plugin versions across the team.
This portability is also important for deployments, in fact Composer is a requirement when deploying to a server using Capistrano. Instead of uploading WordPress files, Capistrano will trigger Composer to run on the server and install it along with other PHP dependencies. This process bypasses the local machine as the server does all the work in grabbing external code.
Updated File and Folder Structure
The standard folder structure is familiar to most WordPress developers:
Bedrock makes significant changes to this structure as illustrated below:
The biggest structure changes include the WordPress folder moved two levels down in /web/wp and the uploads folder relocated from wp-content to web/app. This results in the entire WordPress core being separate from themes, plugins and uploaded content. In this case, Bedrock considers WordPress to be a type of dependency. By explicitly ensuring that it is isolated, code integrity is maintained as working files cannot leak into the core.
The other important change relates to the WordPress configuration file (wp-config.php). Despite it containing sensitive server data, it usually sits exposed in the root folder of a typical WordPress website. In Bedrock, the web root of the website points to /web, where wp-config.php is now located. However it is no longer a security issue as it references all configuration variables from outside of the web root in /config. These variables are further defined by environment using a tool called Dotenv.
Configuration variables change between environments and storing them in code repositories is cumbersome and poses a security risk. Following one of the tenets of the 12 Factor App (http://12factor.net/), Bedrock uses a small utility called dotenv to store environment variables such as database details, domain specific information and caching configurations. Dotenv was originally built in Ruby but was ported to PHP in 2013 and has allowed us to manage our variables in a flexible and portable manner.
With this in mind, the next stage was to research the best way to build a future proof WordPress theme. Rather than extending an existing theme or building one from scratch, we decided to find an appropriate starter theme that incorporated all the latest trends in modern frontend development. Once again we returned to the Roots group and found they had built a starter theme called Sage that met all our requirements.
Unlike most WordPress themes, Sage has barely any visual styling, instead providing a modern platform for frontend and theme development.
Current MAAS WordPress websites that use Sage include:
Don’t Repeat Yourself (DRY)
Most WordPress themes have multiple template files that each load a header and footer file. While this is easy to use and understand, it does create a lot unnecessary code duplication (aka WET – Write Everything Twice).
Similar to Root’s Bedrock custom folder structure, Sage eschews the standard theme structure by funnelling all pages through one file called base.php. The header, footer and even the sidebar are only loaded in one file. This ensures that everything is DRY as possible.
As frontend development has grown in complexity, new tools have been built to help automate repetitive tasks like managing dependencies and minifying script files. Sage leverages Node.js and its vast ecosystem to create a fast and efficient build process.
Extending CSS: SASS vs LESS
The main purpose of CSS is to style HTML pages. In its current form it is not considered a programming language as it doesn’t have standard programming features such as variables, conditionals and functions. However, SASS and LESS have become popular ways of extending CSS to include these features.
Although LESS and SASS are very similar, it has become pretty clear in the last year that SASS has more momentum. To illustrate, the latest version of Twitter Bootstrap (4.0 Alpha) has announced that they are switching from LESS to SASS (Bootstrap Team, 2015). In anticipation of this, the museum decided during the research stage to work with SASS.
Currently, browsers only recognise CSS files, not SASS (.scss) or LESS (.less) files. This is where Gulp comes in to preprocess and convert the file to standard CSS. Gulp can be run manually so when a change is made, a build command is executed and the browser is refreshed to view the changes. Unfortunately, this process gets tedious very quickly as there could be over a hundred changes throughout the day.
Luckily, Gulp comes with the watch function where it will detect file save changes, automatically compile source files (CSS, JS, HTML etc) and inject the changes instantly into the browser. This feature is a huge improvement on a typical manual workflow and is accomplished using the Gulp plugin BrowserSync.
BrowserSync and Gulp can do all of this, but also track navigation across multiple browser windows. This will even work across devices in the same network as the development computer. This means that while a development site is viewed, every page click will be reflected across all connected devices. In addition to navigation tracking, scrolling information is also tracked. We found these features particularly useful for building responsive websites. Any code updates could be viewed instantly in multiple device sizes without any manual refreshing.
One interesting outcome of technologies like LiveReload and BrowserSync is that in some circumstances, the design mockup stage of website building can be skipped. Rather than experimenting with a colour scheme or typography in Photoshop, the combination of SASS and BrowserSync can lead to quicker results. This is particularly true for designing responsive websites because graphics programs are unable to replicate the variety of browser sizes required for real world application.
Building layouts is usually aided by a grid framework, such as Twitter Bootstrap. While Bootstrap comes pre-installed with Sage, we researched other alternatives and decided to go a different route.
Susy Grid vs Twitter Bootstrap
Our research led us to Susy, a SASS based flexible grid system that can handle as many columns as necessary. We also chose Susy because it is not downloaded by the browser at all, in contrast to Bootstrap’s sizeable grid library download. Susy is essentially a set of mathematical functions to help work out column widths, gutters and other layout properties.
From five columns, to six and even seven, Susy only requires a simple command in the SASS stylesheet to make it work (eg. span(2 of 10), span(2 of 12), span(1 of 7)). This also differs greatly from Bootstrap’s method, where column widths are set in HTML as specially named classes (eg. col-md-6).
The other disadvantage to Bootstrap is the need to include layout information in HTML. This ties the content strongly to styling and makes it difficult to decouple from Bootstrap in the future. After working with Susy, we found that we could rapidly prototype complicated layouts with as many custom breakpoints as needed.
With our HTML free of Bootstrap code and a blank canvas for our starter theme, we took the opportunity to review our Museum-wide CSS naming conventions. We knew that our flagship site would have a long life span, so getting CSS right the first time would pay off dividends in the future.
Popular CSS methodologies include:
- Object Oriented CSS (OOCSS)
- Scalable and Modular Architecture for CSS (SMACSS)
- Block Element Modifier (BEM)
We eventually settled on BEM as its component based system worked well with our theme design and Gulp/SASS build process. In the BEM convention, all main components such as the header, content and footer are called Blocks.
Items within the Block are identified as Elements while Modifiers are used to make minor changes to a Block or Element. Part of the convention is to always use lowercase and a single dash between words. This differentiates from the double dash used for Modifiers. Element names must be prefixed with the parent Block, using two underscores between them. The following diagram illustrates an example use case.
- .header (Block)
- .header__menu (Element), .header__menu–light (Modifier)
- .body-content (Block)
- .body-content__main (Element)
- .body-content__sidebar (Element)
- .footer (Block)
In BEM, everything is namespaced and has a unique class, so descendent selectors are never used. While there is extra markup, giving a unique class to every element has improved maintenance. We found that classes are easier to understand, behave in a predictable manner and avoids specificity wars.
A Word on IE8 Support
During 2014, MAAS upgraded all Museum computers from aging boxes running Windows XP to new systems on Windows 7. The upside to this was that Internet Explorer 8 was no longer in use as the Windows 7 came pre-installed with IE9. After our analytics revealed that only a minute proportion of users were using IE8 and that Windows XP had passed its end-of-life status, the Museum decided to not support this browser.
Dropping support for IE8 was also necessary due to its dated support for SSL cipher suites. IE8 lacks support for TLS 1.2 and none of its ciphers offer real world perfect forward secrecy.
It is hoped this research paper has presented worthwhile information about all parts of a Museum website stack. In some ways, we have only scratched the surface as infrastructure, backend and frontend are subjects worthy of research papers on their own.
A key point is the need to continually iterate all parts of the stack and not be married to a particular technology. In fact, this paper should be considered a living document as we are already embarking on new research to further improve our Museum stack.
Dan Collins, Head of Digital and Media, Museum of Applied Arts and Sciences.
Bahajji, Z and Illyes, G. (2014). Google Online Security Blog: HTTPS as a ranking signal. Consulted August 26, 2015. Available: http://googleonlinesecurity.blogspot.com/2014/08/https-as-ranking-signal_6.html
Bootstrap Team. (2015). Bootstrap 4 alpha – Bootstrap Blog. Consulted 26 Aug. 2015. Available: http://blog.getbootstrap.com/2015/08/19/bootstrap-4-alpha/
Gerchev, I. (2014). 5-most-popular-frontend-frameworks-compared – SitePoint. Consulted 25 August 2015. Available: http://www.sitepoint.com/5-most-popular-frontend-frameworks-compared/
Sitakange, J. (2014). Infrastructure as Code: A Reason to Smile | ThoughtWorks. Consulted 25 August 2015. Available: http://www.thoughtworks.com/insights/blog/infrastructure-code-reason-smile
. "A Modern Museum Web Stack." MWA2015: Museums and the Web Asia 2015. Published September 1, 2015. Consulted .