April 19, 2024

In a current mission, we had been tasked with designing how we’d substitute a
Mainframe system with a cloud native software, constructing a roadmap and a
enterprise case to safe funding for the multi-year modernisation effort
required. We had been cautious of the dangers and potential pitfalls of a Huge Design
Up Entrance, so we suggested our shopper to work on a ‘simply sufficient, and simply in
time’ upfront design, with engineering in the course of the first part. Our shopper
appreciated our method and chosen us as their associate.

The system was constructed for a UK-based shopper’s Information Platform and
customer-facing merchandise. This was a really complicated and difficult activity given
the scale of the Mainframe, which had been constructed over 40 years, with a
number of applied sciences which have considerably modified since they had been
first launched.

Our method relies on incrementally transferring capabilities from the
mainframe to the cloud, permitting a gradual legacy displacement moderately than a
“Huge Bang” cutover. In an effort to do that we would have liked to determine locations within the
mainframe design the place we might create seams: locations the place we will insert new
habits with the smallest attainable modifications to the mainframe’s code. We are able to
then use these seams to create duplicate capabilities on the cloud, twin run
them with the mainframe to confirm their habits, after which retire the
mainframe functionality.

Thoughtworks had been concerned for the primary 12 months of the programme, after which we handed over our work to our shopper
to take it ahead. In that timeframe, we didn’t put our work into manufacturing, however, we trialled a number of
approaches that may assist you to get began extra rapidly and ease your personal Mainframe modernisation journeys. This
article offers an outline of the context by which we labored, and descriptions the method we adopted for
incrementally transferring capabilities off the Mainframe.

Contextual Background

The Mainframe hosted a various vary of
providers essential to the shopper’s enterprise operations. Our programme
particularly centered on the information platform designed for insights on Shoppers
in UK&I (United Kingdom & Eire). This explicit subsystem on the
Mainframe comprised roughly 7 million traces of code, developed over a
span of 40 years. It supplied roughly ~50% of the capabilities of the UK&I
property, however accounted for ~80% of MIPS (Million directions per second)
from a runtime perspective. The system was considerably complicated, the
complexity was additional exacerbated by area tasks and issues
unfold throughout a number of layers of the legacy atmosphere.

A number of causes drove the shopper’s resolution to transition away from the
Mainframe atmosphere, these are the next:

  1. Adjustments to the system had been sluggish and costly. The enterprise subsequently had
    challenges retaining tempo with the quickly evolving market, stopping
  2. Operational prices related to operating the Mainframe system had been excessive;
    the shopper confronted a industrial danger with an imminent value enhance from a core
    software program vendor.
  3. While our shopper had the required ability units for operating the Mainframe,
    it had confirmed to be laborious to search out new professionals with experience on this tech
    stack, because the pool of expert engineers on this area is restricted. Moreover,
    the job market doesn’t supply as many alternatives for Mainframes, thus folks
    aren’t incentivised to discover ways to develop and function them.

Excessive-level view of Shopper Subsystem

The next diagram reveals, from a high-level perspective, the varied
parts and actors within the Shopper subsystem.

The Mainframe supported two distinct sorts of workloads: batch
processing and, for the product API layers, on-line transactions. The batch
workloads resembled what is often known as an information pipeline. They
concerned the ingestion of semi-structured knowledge from exterior
suppliers/sources, or different inside Mainframe techniques, adopted by knowledge
cleaning and modelling to align with the necessities of the Shopper
Subsystem. These pipelines integrated varied complexities, together with
the implementation of the Id looking logic: in the UK,
not like the US with its social safety quantity, there isn’t any
universally distinctive identifier for residents. Consequently, corporations
working within the UK&I need to make use of customised algorithms to precisely
decide the person identities related to that knowledge.

The web workload additionally introduced important complexities. The
orchestration of API requests was managed by a number of internally developed
frameworks, which decided this system execution circulate by lookups in
datastores, alongside dealing with conditional branches by analysing the
output of the code. We should always not overlook the extent of customisation this
framework utilized for every buyer. For instance, some flows had been
orchestrated with ad-hoc configuration, catering for implementation
particulars or particular wants of the techniques interacting with our shopper’s
on-line merchandise. These configurations had been distinctive at first, however they
possible grew to become the norm over time, as our shopper augmented their on-line

This was applied by means of an Entitlements engine which operated
throughout layers to make sure that clients accessing merchandise and underlying
knowledge had been authenticated and authorised to retrieve both uncooked or
aggregated knowledge, which might then be uncovered to them by means of an API

Incremental Legacy Displacement: Ideas, Advantages, and

Contemplating the scope, dangers, and complexity of the Shopper Subsystem,
we believed the next rules could be tightly linked with us
succeeding with the programme:

  • Early Danger Discount: With engineering ranging from the
    starting, the implementation of a “Fail-Quick” method would assist us
    determine potential pitfalls and uncertainties early, thus stopping
    delays from a programme supply standpoint. These had been:
    • Consequence Parity: The shopper emphasised the significance of
      upholding end result parity between the prevailing legacy system and the
      new system (You will need to observe that this idea differs from
      Characteristic Parity). Within the shopper’s Legacy system, varied
      attributes had been generated for every shopper, and given the strict
      business laws, sustaining continuity was important to make sure
      contractual compliance. We would have liked to proactively determine
      discrepancies in knowledge early on, promptly deal with or clarify them, and
      set up belief and confidence with each our shopper and their
      respective clients at an early stage.
    • Cross-functional necessities: The Mainframe is a extremely
      performant machine, and there have been uncertainties {that a} resolution on
      the Cloud would fulfill the Cross-functional necessities.
  • Ship Worth Early: Collaboration with the shopper would
    guarantee we might determine a subset of essentially the most important Enterprise
    Capabilities we might ship early, guaranteeing we might break the system
    aside into smaller increments. These represented thin-slices of the
    general system. Our objective was to construct upon these slices iteratively and
    incessantly, serving to us speed up our general studying within the area.
    Moreover, working by means of a thin-slice helps scale back the cognitive
    load required from the crew, thus stopping evaluation paralysis and
    guaranteeing worth could be constantly delivered. To attain this, a
    platform constructed across the Mainframe that gives higher management over
    purchasers’ migration methods performs a significant function. Utilizing patterns similar to
    Darkish Launching and Canary
    Launch would place us within the driver’s seat for a clean
    transition to the Cloud. Our objective was to realize a silent migration
    course of, the place clients would seamlessly transition between techniques
    with none noticeable impression. This might solely be attainable by means of
    complete comparability testing and steady monitoring of outputs
    from each techniques.

With the above rules and necessities in thoughts, we opted for an
Incremental Legacy Displacement method together with Twin
Run. Successfully, for every slice of the system we had been rebuilding on the
Cloud, we had been planning to feed each the brand new and as-is system with the
identical inputs and run them in parallel. This enables us to extract each
techniques’ outputs and test if they’re the identical, or at the very least inside an
acceptable tolerance. On this context, we outlined Incremental Twin
as: utilizing a Transitional
Structure to help slice-by-slice displacement of functionality
away from a legacy atmosphere, thereby enabling goal and as-is techniques
to run quickly in parallel and ship worth.

We determined to undertake this architectural sample to strike a stability
between delivering worth, discovering and managing dangers early on,
guaranteeing end result parity, and sustaining a clean transition for our
shopper all through the period of the programme.

Incremental Legacy Displacement method

To perform the offloading of capabilities to our goal
structure, the crew labored carefully with Mainframe SMEs (Topic Matter
Specialists) and our shopper’s engineers. This collaboration facilitated a
simply sufficient understanding of the present as-is panorama, by way of each
technical and enterprise capabilities; it helped us design a Transitional
Structure to attach the prevailing Mainframe to the Cloud-based system,
the latter being developed by different supply workstreams within the

Our method started with the decomposition of the
Shopper subsystem into particular enterprise and technical domains, together with
knowledge load, knowledge retrieval & aggregation, and the product layer
accessible by means of external-facing APIs.

Due to our shopper’s enterprise
objective, we recognised early that we might exploit a serious technical boundary to organise our programme. The
shopper’s workload was largely analytical, processing largely exterior knowledge
to supply perception which was bought on to purchasers. We subsequently noticed an
alternative to separate our transformation programme in two components, one round
knowledge curation, the opposite round knowledge serving and product use instances utilizing
data interactions as a seam. This was the primary excessive stage seam recognized.

Following that, we then wanted to additional break down the programme into
smaller increments.

On the information curation facet, we recognized that the information units had been
managed largely independently of one another; that’s, whereas there have been
upstream and downstream dependencies, there was no entanglement of the datasets throughout curation, i.e.
ingested knowledge units had a one to at least one mapping to their enter recordsdata.

We then collaborated carefully with SMEs to determine the seams
throughout the technical implementation (laid out beneath) to plan how we might
ship a cloud migration for any given knowledge set, ultimately to the extent
the place they might be delivered in any order (Database Writers Processing Pipeline Seam, Coarse Seam: Batch Pipeline Step Handoff as Seam,
and Most Granular: Data Characteristic
). So long as up- and downstream dependencies might change knowledge
from the brand new cloud system, these workloads might be modernised
independently of one another.

On the serving and product facet, we discovered that any given product used
80% of the capabilities and knowledge units that our shopper had created. We
wanted to discover a totally different method. After investigation of the best way entry
was bought to clients, we discovered that we might take a “buyer section”
method to ship the work incrementally. This entailed discovering an
preliminary subset of consumers who had bought a smaller proportion of the
capabilities and knowledge, lowering the scope and time wanted to ship the
first increment. Subsequent increments would construct on high of prior work,
enabling additional buyer segments to be minimize over from the as-is to the
goal structure. This required utilizing a unique set of seams and
transitional structure, which we talk about in Database Readers and Downstream processing as a Seam.

Successfully, we ran a radical evaluation of the parts that, from a
enterprise perspective, functioned as a cohesive complete however had been constructed as
distinct parts that might be migrated independently to the Cloud and
laid this out as a programme of sequenced increments.

We’re releasing this text in installments. Future installments will
describe the totally different sorts of seams that we established in our work.

To search out out after we publish the subsequent installment subscribe to the
web site’s
RSS feed, Martin’s
Mastodon feed, or
X (Twitter) stream.