Introduction
The final part of our series of WPY microsite technical posts, this will cover the implementation of the Digital Media and TS development teams’ first use of AEM Content Services – functionality that integrates our current traditional CMS (Adobe AEM) with decoupled front-end JavaScript applications like this WPY React/NextJS application. This is a more modern web application architecture which aims to separate web page presentation (the ‘head’) from the underlying content stored in a CMS.
The rise of JavaScript frameworks to develop decoupled web applications (often called the ‘Jamstack’ https://jamstack.org/what-is-jamstack/) is an example of a common engineering strategy to separate increasingly complex software systems into more manageable components and services (areas of expertise).
A recap of related topics from the previous posts in the series:
In Part 1 (https://naturalhistorymuseum.blog/2020/07/28/recreating-wildlife-photographer-of-the-year-online-part-1-introduction-and-technical-approach/) – I mentioned how the site’s content is split between a database driven API (the WPY API) and AEM Content Services, which delivers the supporting content created by our content authors.
In Part 2 (https://naturalhistorymuseum.blog/2020/08/11/recreating-wildlife-photographer-of-the-year-online-part-2-key-site-features/ ) – we covered some of the different WPY site routes like the /gallery and /competition hub sections. Depending on the route, either the WPY API or the AEM Content Service act as the page ‘generators’. Both are APIs, but only our Content Service data is (currently) CMS editable, in effect making this a hybrid ‘headless’/database-driven site.
With online digital needing to do so much more for both the user and the business – combining traditional authored articles with more product/developer-designed templates, incorporating content like personalised or interactive experiences and upsells – we still need to ensure that there is enough authoring flexibility for our content authors to promote the Museum’s message and tell their stories.
Adobe AEM Content Services overview
The Museum has been using Adobe Experience Manager (AEM) as its core website CMS since 2015. It’s a traditional CMS model with content and presentation all delivered together via the same (monolithic) application. When AEM version 6.4 was released in 2018, Adobe introduced the new Content Services functionality, in response to the general shift from traditional tightly coupled web page generation to a decoupled approach.
Content Services is functionality that allows an AEM developer to build an API to expose the JSON version of the same CMS content previously only available as the final HTML web page sent to a user’s browser. JSON data is far more suitable for programmatic sorting, grouping and manipulation than HTML; it allows content to become more re-useable in different contexts. Additionally, by taking the presentation layer out of AEM, the front-end developer is free to utilise any architecture or tools they need to build the presentation layer of different digital products, without worrying if they can be easily integrated into the existing AEM codebase. Web technology and optimisations move at a rapid rate, driven by the competitive nature of digital marketing, and it’s easier to leverage them in a decoupled software environment.
Key to this new AEM functionality is the use of Content Fragments – snippets of content that can be updated and re-used independently of specific AEM pages. AEM content can be a mix of an authored sequence of components (i.e. article content) or standalone fragments (i.e. content snippets or promotions). We’ve been using Content Fragments for a while on our main site, and they’re particularly powerful when coupled with a headless approach as they become a building block for a content as a service system. A single piece of content or information can be authored once in the existing CMS but appear in very different end user channels and contexts (website articles, social media, ecommerce platforms, gallery applications or apps etc.).
Content Service JSON data of shop and donation content fragments used on WPY gallery pages
NextJS gallery page generated by WPY API data, with the separate AEM content fragment on the right
Designing the Content Service
When we started the WPY project with our partner agency, Clearleft, a lot of the design work and initial component architecture in the NextJS front end application had already progressed before the AEM Content Service development started in earnest.
This was acceptable as arguably the core of the site (the gallery) is generated by the WPY API, which was already well formed at this point. We could architect the gallery pages and knew which sections would need to be served by AEM. The competition hub pages similarly had designs and we had some initial front-end components built out for them, but we knew these were less final as they would be generated by the AEM Content Service data structure.
Meeting with our AEM developer Alistair, we specified a list of AEM components that would match the front-end React components already in design. Discussion here was relatively quick and informal, we agreed core functionality like the basic fields that would make up the component data as well as their intended use. Development then continued relatively independently, with regular catchups to discuss progress and further details for sticking points.
In effect we hadn’t decided the specific JSON data structure (the interface) we needed for our AEM-driven components at this first stage, sticking to using dummy data when we needed to progress the design.
This meant Alistair could implement as he needed, meeting the guidelines of the editing functionality and component structure we’d agreed together, he was using the content service functionality for the first time as well.
Additionally we were keen to understand how our existing editing experience of WYSIWYG content authoring (What You See Is What You Get), that all the content authors were used to might need to adapt with the move to headless, since crucially, the front-end code would no longer automatically be within AEM to drive the WYSIWYG editor.
Content editing approach
For this project we took the route of emulating the design of the equivalent React front-end components within AEM, implementing HTML output of the Content Service components and a manual import of the relevant CSS to provide a visually similar version in the editing process. The main aim being to give visual context to the editors as they make their changes. This was important as we didn’t have a way at the time of letting them easily preview their content changes in the final web app without publishing the content.
In this project’s context, with very simple editor-driven templates and only one other API, our emulation approach worked well. I’ll give my thoughts on this approach going forward in a future article, as content authoring approach and its impact on web application architecture is a complex topic.
The other side – Front-end API transforms
Early on in this process we decided to use the concept of API data transforms within the NextJS web app. This proved to be fundamental when integrating a separately developed API to a front-end, as it allows for separation of content structure from a UI component’s data requirements.
The idea of the data transform layer is that the API data (whether WPY or content service) is requested by the web app and then the structure of that data is reconfigured to fit with what the UI components expect before being passed on to them. It allows the front-end developer to design simpler, more consistent and more reusable components.
For example, the API source data may only supply the required combination of content for a front-end component in separate requests or structures – this would make each component complicated as it would need to filter through large amounts of different data to get what it needs. This approach abstracts that away, so it’s done in a separate layer in the code, leaving the component to focus only on displaying the UI.
Some code snippets to describe the process:
For example, instead of the React banner component needing to reference ‘x[‘:items’].root[‘:items’].banner.title’ to display it’s title, it just needs to use ‘title’.
The design component can therefore be made independent of the underlying content structure, making it simpler and more reusable, vital when building flexible but consistent presentation layers across many different types of product.
Freeform headless content
The above approach with the homepage example works well when dealing with fixed and known source data – for example, we always know the content will have a Banner component and a collection of key date fragments.
But what happens with more free-form page content like articles and edited content, where different combinations of content will be created by the editor? We might have some paragraphs, images and CTAs in a different order, or none at all. This was what we needed to solve for our supporting ‘content pages’ in the community section of the site, which are entirely authored by the AEM editor as headless pages.
The solution was the castAEMToReactComponents function and the concept of a ‘dynamic’ React wrapper component. This allows the front-end component list to match the generation and order of AEM components, which gives us the ability to render an entirely AEM-authored page structure, instead of relying on developer-controlled page layout with predefined components.
Key to this is that the AEM Content Service data structure includes both the AEM ‘resource’ type and the structures they are laid out in by the author within AEM.
This data is used to map AEM components to React components, with the front-end developer free to transform them as the product’s requirements evolve.
Freeform AEM headless output
The castAEMToReactComponents function ‘pre-generates’ a structured list (matching the editor’s layout) of React components from the headless data during the transform layer, this is then simply rendered at the UI layer.
The benefits of the transform approach are numerous, the UI components (if suitably designed and planned) can be reused across many web apps that use different underlying data sources, the data source can start off as dummy data during development before being changed to an API or CMS source, all without modifying the components’ design and implementation.
Authoring new pages
As briefly mentioned in the Part 2 article, NextJS gives us the option of dynamic URL routing, which means it can dynamically generate an appropriate page response for any URL – instead of a developer having to implement an individual template for every url in the site (i.e. /competition/young-wpy), this is instead handled dynamically. In WPY the homepage and gallery templates are pre-defined, but any other URLs requested from the NextJS app will be handled by this dynamic route, which maps the requested URL to the matching AEM content service it needs.
e.g. https://www.nhm.ac.uk/wpy/competition/young-wpy/rules will automatically request its data from https://www.nhm.ac.uk/content/wpy/en/home/competition/young-wpy/rules.model.json
This means that an editor can create (or unpublish) a new content service page within AEM and that page is immediately available on the decoupled site automatically. Additionally, the content service data can list all child pages, which gives us the ability to have automated site navigation and sitemaps as well.
Link handling
The final part of the headless functionality I’ll cover here is how we handled links from the AEM editorial content. Key to this is understanding how links need to work in a JavaScript controlled SPA (Single Page Application). Whilst a standard <a href> HTML link tag can be used, for internal links in an SPA, a special component (in this case NextJS’s <Link> component) needs to be used instead.
What this component does is tell the web app to perform an internal JavaScript route change instead of a full web page reload. Here, the web app loads only what it needs to display the content requested by the link (usually loading only new API data and any code not yet downloaded), without clearing and recreating the whole webpage environment again every time, as in a traditional website. The URL in the address bar however still changes so the new ‘page’ is still link-able, trackable and shareable.
This is at the heart of the in-session performance benefits modern JavaScript frameworks provide. It also means a user’s current session can be kept in ‘state’ across different URLs, without resorting to complex backend setups or cookies to keep track of a user’s progress throughout a multi-page experience, giving the option for far more app-like or joined up web experiences.
Coming back to how I dealt with links created in the headless AEM content, the editor can only create standard <a href> elements in their AEM content, so I needed a way to convert any internal web app links to their equivalent NextJS Link before displaying the content.
My solution here was an AEMContentLinkReplacer utility function, called during the transform layer described earlier.
This function is used on any AEM content that may contain <a href> links. It performs a parsing function that converts the standard HTML elements in the content into their JavaScript equivalents (JSX). It then looks for any <a href> HTML elements and replaces them directly with the NextJS Link component, before returning the list of JSX components for display. The resolveLink function is used to detect any links intended to be internal (we agreed a convention with the content authors that any links that start with /wpy are internal app routes) so that they are configured in the appropriate way.
The final result being we have a mechanism for editors to add Single Page App links in the separate editorial content, maintaining the in-session performance benefits of the NextJS framework.
Summary
The solutions described in this series of posts gave us a good start in exploring headless CMS functionality. We can see how page authoring can work independently of the front end but still directly allow for new and updated content and pages in a decoupled web application. This allows the front-end JavaScript developers to focus on optimisation and new features with more technical freedom, with less fear of impacting the backend architecture or the editing experience for the authors.
Over the past year the Digital Media and TS technical teams have developed four new but very different web products with this decoupled ‘Jamstack’ approach (WPY site, new Ticketing platform, Visit planner and Digital nature journal), and there’s several far-reaching questions and challenges that still need to be met – both on the headless content authoring approach described here, as well as accompanying development advances such as automated testing and continuous code deployments. Adobe itself is also continuously evolving AEM’s headless functionality. I’ll be covering our progress on these in an upcoming post – it’s always exciting to discuss the possibilities, capabilities and efficiencies new engineering approaches can bring.