Ab Initio Web Site

Contents
Toc...............................................................................................................................................................1
Metadata.....................................................................................................................................................1
WHY IT’S SO DIFFICULT.............................................................................................................1
HOW AB INITIO CAN HELP.........................................................................................................2
Data Archiving.............................................................................................................................................3
EARLY SOLUTIONS.......................................................................................................................3
EXTRACTING THE DATA WHILE PROVIDING EASY ACCESS..........................................4
A COST SAVINGS BONUS.............................................................................................................4
Blowout Preventer......................................................................................................................................4
TAKING A NEW APPROACH.......................................................................................................5
WHEN THE WORST HAPPENS....................................................................................................6
Sub-ledger Complexity................................................................................................................................6
WHAT DIDN’T WORK...................................................................................................................7
WHAT DID WORK..........................................................................................................................7
Time Tested.................................................................................................................................................8
GROWING PAINS............................................................................................................................8
LOOKING TO THE FUTURE........................................................................................................9
Mobile Internet Security..............................................................................................................................9
REGULATORS STEP IN.................................................................................................................9
ACTING QUICKLY.......................................................................................................................10
A SUCCESSFUL SOLUTION........................................................................................................10
Realtime platforms....................................................................................................................................10
FIGURING OUT THE POSSIBILITIES.....................................................................................................11
WORKING TOGETHER........................................................................................................................11
IMPLEMENTATION.............................................................................................................................12
Government Use.......................................................................................................................................12
TOO MANY DIFFERENT SYSTEMS.......................................................................................................13
THE IMPORTANCE OF METADATA.....................................................................................................13
REALIZING A VISION AND GETTING RESULTS.....................................................................................13
Metadata Challenge at a Global Bank.......................................................................................................14
 GETTING A HANDLE ON THINGS............................................................................................14
UP AND RUNNING........................................................................................................................15
Use Case at Health Care Company............................................................................................................15
STORY ONE: THE COMPLEXITIES OF MEMBER MANAGEMENT.........................................................16
STORY TWO: THE COMPLEXITIES OF CODE CONVERSION.................................................................17
Usecase at a Telcom..................................................................................................................................18
GETTING TO THE FOUNDATION.............................................................................................19
THE NITTY GRITTY....................................................................................................................19
NO NEED TO COMPROMISE.....................................................................................................20
Use case at a Consumer Goods Company.................................................................................................20
EARLY CONSIDERATIONS...................................................................................................................20
RAPID DEVELOPMENT AND A NEW SOLUTION..................................................................................21
LOOKING AHEAD................................................................................................................................21
ETL usecase................................................................................................................................................22
A FIRST ATTEMPT...............................................................................................................................22
CONVERTING THE CODE....................................................................................................................22
LOOKING AHEAD................................................................................................................................23
Usecase at CreditCard Company...............................................................................................................23
EARLY ATTEMPTS...............................................................................................................................24
BUILDING THE NEW DATA BACKBONE...............................................................................................24
SUCCESSFUL RESOLUTION.................................................................................................................25
Transportation Company Use Case...........................................................................................................25
AN INEFFECTIVE SYSTEM EXPENSIVE TO MAINTAIN..........................................................................26
A NEW KID IN TOWN.........................................................................................................................26

Metadata
It is common for users to underestimate the challenge of building enterprise data warehouses
(EDWs).
Take, for comparison purposes, the process of building an airplane. While an airplane may have
hundreds of thousands of parts, each airplane coming off the assembly line is approximately the
same as the previous one. The parts are predesigned and manufactured to fit together. The
architecture of the plane is well known (two wings, a fuselage, landing gear…), and at design
time, the engineering needed to choose sizes and shapes and materials is all known. All airplanes
fly through air, and the properties of air on Earth are well documented. So as hard as it is to build
airplanes, it is a relatively repeatable task that has predictable results.

This is not so for EDWs. Each EDW is different from all others. Each has a completely different
set of inputs and outputs, and different methods for putting all the data together into an
understandable structure. In an attempt to create conformity and reuse, customers have moved in
the direction of using “industry-standard models”. While this sounds like a good idea, this is akin
to saying that airplanes, helicopters, and rocket ships can all be manufactured from a single
master blueprint. Yes, they’re all conceptually the same, but it’s the details that kill.

WHY IT’S SO DIFFICULT

And kill they do. In the EDW world, the details start with requirements gathering: The industry-
standard model is morphed into a physical model. The physical model is turned into
specifications. The specifications get turned into spreadsheets of rules and mappings. The
spreadsheets get sent offshore where they are turned into code. The code is sent back where,
often for the first time, it is run against actual data. Surprise, surprise: the code doesn’t match the
data, and the team gets to start over. And over, and over again. This process is like starting the
manufacture of an airplane before knowing whether it is going to fly through air or water. Even
with the best architects and engineers, errors are going to be common and costly.

To compound the problem, because there are so many streams of data coming into and going out
of the EDW, the EDW is deconstructed into a large number of “feeds,” and an army of people
are employed to work separately on each one. To manage the large number of people, each feed
is further deconstructed into distinct steps, and a different person is typically assigned to each
step. The result is an enormous number of handoffs from one person to the next, and very little
reuse since each feed is developed separately. All this multiplies the time needed to build an
EDW. And time equals money. Using the standard approach with standard technologies, EDWs
are very expensive to build.

HOW AB INITIO CAN HELP

But there is a better way.

Ab Initio has attacked this problem by applying “first principles” thinking and the results are
startling. Instead of a feed taking 3 to 4 months to implement, the Ab Initio approach takes 2 to 3
weeks, and sometimes just days! Ab Initio calls this approach building a metadata driven
warehouse (MDW).

An MDW is a collection of reusable Ab Initio-based applications that have the ability to process
data from the source file all the way to the consolidated EDW model. This includes all the feed-
processing complexity that is usually built by hand: file checking, data enrichments, filtering,
validation, cleansing, key management, history management, aggregation, mapping, archiving,
recovery, and model loading. Each one of these EDW activities has been abstracted to a level
where the same template application can be used on a wide variety of feeds simply by changing
the associated metadata.

By design, the MDW enables an analyst to specify and test the processing for a feed at the
beginning of the cycle – usually without any development involvement at all. This approach is
based on using metadata to drive the system. This is the same metadata that the analyst would
normally have to specify in a document, such as the target data model, file formats, keys,
mapping rules, and so on. Instead of writing a document and passing it along to a development
team, the analyst, using the MDW, can enter the metadata directly, and then run and test the
resulting application immediately on real data, avoiding the long and costly development and test
iterations. The savings in time are enormous, and the quality of the resulting system is much
higher since it passes through many fewer hands.

The Ab Initio technology was designed, from the beginning, to support this approach. From the
user’s perspective, the savings begin with the construction of the MDW and the benefits are
ongoing. For example, all the entered metadata is held in Ab Initio’s metadata repository, called
the Enterprise Meta>Environment (EME), providing the necessary management capabilities,
such as version control, data lineage, impact analysis, data quality, and access security. The EME
gives management an integrated view into their systems that they would never have otherwise.
They can ask questions about how fields in reports were calculated, tracing all the way back to
operational systems. They can quickly answer questions about how data propagates around the
system – answers that are necessary for auditors and for predicting maintenance efforts. And
they can get a handle on the quality of their data, and how that quality impacts downstream
systems. In short, management can finally understand the systems they have paid to build and
that were previously black boxes.

Ab Initio has helped many customers build MDWs. And while each Ab Initio-built MDW is able
to meet the specific and differing needs of those customers, the results are repeatable and
predictable.

Data Archiving
Data archiving. Yawn. Even the very definition is downright boring:

“Data archiving is the process of moving data that is no longer actively used to a separate data
storage device for long-term retention. Data archives consist of older data that is still important
and necessary for future reference. Data archives are indexed and have search capabilities so that
data can be easily located and retrieved.” –Whatis.com

Having been around for more than 25 years, data archiving is definitely not one of those “tech
buzz words”. It doesn’t compete for mindshare like social networking and iPads. And it doesn’t
make for stimulating dinner party conversation. But in the last few years data archiving has
become an increasingly important topic for IT departments. So why is it interesting now?

Well, there are two aspects. First, the need for data archiving has increased significantly as a
result of the step change in regulation and the desire of businesses to retain more data and make
it accessible for much longer. Second, the variety and amount of data makes everything more
challenging from a technical perspective – setting up the processes to capture the data from
different files (for example, XML) and databases, physically storing the data and then providing
fast, ad-hoc access to it. This may not sound like much, but if these needs are not met, a business
can be shut down.

A large broadcasting company faced just these types of challenges. In their case, the amount of
data being added each month to their core business systems, such as CRM (customer relationship
management) and billing, was causing operational issues – billing runs were taking too long;
customer service representatives were experiencing poor response times while on the telephone
with a customer, etc. In short, data archiving issues were directly impacting the company from
both a service perspective and a financial perspective. Resolving them was high on the business
agenda.

EARLY SOLUTIONS

Naturally, the first potential solution the company explored was to add hardware to the existing
operational systems and thereby remove the need to archive any data at all. However, given the
amount of data being added to the databases each month, this would have meant a continual
series of expensive upgrades over time with limited benefits. The problem was in the internal
architecture of the databases holding the data: they just could not scale in a linear manner. After
much testing, it was concluded that the only practical and cost-effective answer was simply to
reduce the size of the large database tables. While required by law to store a full 7 years of data,
the broadcasting company needs to keep only 18 months worth in the operational systems. The
rest can be held in an archive. The vast majority of the data (billions of records) are concentrated
in a few large “fact” tables, such as a billing or contact events. By archiving this data and
removing it from the original operational systems, the analysis showed that significant
performance and response-time benefits could be achieved.

There was a downside to this solution, however. Many thousands of users within the company
still needed access to all of the data in real-time, and many of these were customer service
representatives who demanded sub-second response times. This was made more challenging by
the fact that another set of users wanted to be able to combine data from the archive with other
data held in both operational and business intelligence systems – a sophisticated “federated
query”.

The combination of these technical challenges meant that there simply wasn’t an off-the-shelf
package available. The only company that could solve all aspects of their archiving requirements
was Ab Initio.

EXTRACTING THE DATA WHILE PROVIDING EASY ACCESS

Using the Ab Initio Co>Operating System, the company's developers built parallel applications
called “graphs” to unload the billions of records to be archived from their operational databases.
The extracted data was written in parallel to an Ab Initio Indexed Compressed Flat File (ICFF)
system. ICFF systems not only can load data many times faster than traditional databases, but
they also require only a small fraction of the disk to hold the data, which is ideal for an archive.
To gain access to these storage devices, Ab Initio provided both a web service interface and an
ANSI-standard SQL interface for ad-hoc queries. The SQL interface also supports federated
queries across the archive and the operational databases. In summary: an end-to-end solution.

A COST SAVINGS BONUS

The project has been extremely successful for the broadcasting company. They have
significantly improved the performance and responsiveness of their operational systems, and at
the same time have sub-second access to terabytes of archived data. Given that the majority of
their core data has now been moved from Tier-1 storage to compressed data held on inexpensive
disk, it has also delivered real cost savings for the business.

Perhaps there is some sizzle in the old topic of data archiving, after all!

Blowout Preventer
Until 2010, who had ever heard of a “blowout preventer”? With the huge oil spill in the Gulf of
Mexico, however, there was news story after news story about how blowout preventers are the
crucial safety wall between safe, clean oil recovery and ecological disaster. The blowout
preventer is a several-hundred-ton box of machinery that sits on top of a well on the ocean floor.
When everything is fine, the blowout preventer allows oil to flow continuously from the well
into the recovery pipe. When things go wrong, the blowout preventer is supposed to activate
enormous hydraulic rams that can squeeze the 2-foot pipe shut – and even cut it in two.
Activating the blowout preventer is obviously not an everyday event. It has to work, or else…

Fortunately, drilling disasters are a rare occurrence. Unlike failures in business data-processing
systems, which are common: power goes out and the backups don’t turn on, machines crash even
though they are redundant, disk drives fail even though they are in RAID arrays, someone pushes
the wrong button even though they have been trained not to do that, bad data gets sucked into a
system and gums up the works, databases jam and refuse to load data, networks get congested
and transactions get dropped… When these systems go bad, if you are lucky, things just stop.
But if you are unlucky, like the oil well disaster in the Gulf, data spews out of the data pipes and
makes a horrible mess! Then the operations people have to shut down the systems and try to
recover whatever data they can. And restarting operations can be dangerous as it can lead to
another blowout, more loss of data and more mess.

Software developers don’t like to use the word “blowout” because it scares people – but that’s
what a failure usually is. It’s a blowout! The data gets sprayed all over the place, and pieces of
the system may be lying dead or disabled on the floor.
Software engineers have been wrestling with this type of problem for decades. But building fail-
safe systems is extremely difficult because each one is different, and the “software blowout
preventers” are custom designed (often by people who don’t really understand). The associated
costs are high, and so software engineers are inclined to cut corners (how many times have you
heard, “It’s not supposed to fail”?). Testing these systems tends to be done haphazardly due to
the lack of understanding as to why failures occur. Even worse, a high degree of robustness is
usually directly in conflict with other needs for the data-processing system, such as high
performance – you can have one or the other, but not both at the same time. Since one can’t
prove that a system is robust, but one can easily measure delivery of business goals such as
performance, robustness is usually something that is pushed off until later.

TAKING A NEW APPROACH

Ab Initio realized, as a core principle, that the only way to prevent these data-processing
blowouts was to take a different and comprehensive approach. Ab Initio engineers vowed that
they would not build any software until they had figured out how to design in mechanisms –
from the beginning – that were both robust and easy to use, and that would not impact
performance. This approach is built into Ab Initio’s software so that users can focus on their
business requirements and be confident that they will not have disabling blowouts.

How does this work? First, think of an Ab Initio application as a series of processing steps
connected together through pipes that flow data. Data flows at high rates through these pipes
from one step to the next. Sometimes a step will have multiple incoming pipes and/or multiple
outgoing pipes. These pipes eventually connect to data storage units or other processing systems
that accept or produce streams of data. These interconnected systems of processing steps and
data pipes can be extremely large – much larger and more complex than a New York City
subway map!

The Ab Initio checkpointing mechanism consists of interlocking valves on these data pipes. In
general there is a set of valves on the inputs and outputs to the overall processing system. There
may also be valves at strategic points in the processing system. In addition, there may be data
reservoirs at the valves (checkpoints) to capture a copy of the data before it passes into the next
section of the system. While some of these valves (checkpoints) may be specified by users,
others are automatically positioned by the Ab Initio software. Most of these valves are
implemented in Ab Initio’s software, though some are in other software technologies like
databases and message queues. The key is that all these valves are connected to a central
controller – the Ab Initio Co>Operating System – that operates their opening and closing in a
carefully synchronized manner.

The Co>Operating System is designed for high-performance parallel and distributed processing,
and to be incredibly robust in the face of failure. So while large amounts of data are flowing
through its pipes at high speed – over networks and across servers and connected to all kinds of
external systems – it has to be vigilantly on the lookout for a blowout.

WHEN THE WORST HAPPENS

Because the Co>Operating System has been carefully opening and closing the valves, if a
blowout occurs the loss is limited to the data flowing between valves. And because the
Co>Operating System keeps a copy in its reservoirs of any data that might be at risk, and
because it knows exactly how much data has flowed into and out of the system, no data is ever
actually lost. Once the cause of the failure has been fixed, the Co>Operating System can be
restarted and it will automatically resume the applications at the right places. It will refill the data
pipes with the right data (from the reservoirs), and everything will be as if the failure had never
occurred. And that’s what everyone wants to hear: blowouts happen, but nobody gets injured and
no data is lost, and pumping resumes as soon as the pipes are fixed. What more can you ask for?

PS: Lots of details were omitted from this description of checkpointing. Stuff like 2-phase
commit, checkpoint triggers, message blips, transaction managers, and the XA protocol. Ab
Initio sweats the details so that you don’t have to. The good news is: It just works.

Sub-ledger Complexity
Some applications are hard to build. That was the experience of a very large multinational
insurance company with their reinsurance sub-ledger application.

Reinsurance is simple in principle. The goal is for the underwriter to share the risks and
premiums of the policies it writes with other underwriters. By spreading the risk around, the
damages of a very large claim have limited impact on any one insurance company, and the
policy holders can be assured that their claims will be paid in full.

The process of reinsurance involves a primary underwriter entering into reinsurance “treaties”
with other underwriters. These treaties have complex terms describing how the primary insurer
will share costs and premiums with the other insurers. Sounds simple, but things get complicated
fast. First, the other end of a treaty may not be another insurer but rather another treaty that may
connect to yet more treaties and eventually to other insurers. This can make it extremely difficult
to calculate the effects of any one treaty. Second, the business people who define these treaties
can be quite creative in inventing new terms, and their expectation is that the IT systems will
quickly implement whatever they come up with, after the fact.

The purpose of the reinsurance sub-ledger application is to calculate the total costs and premiums
that are to be shared with each reinsurance partner, as well as to determine profitability for the
business people who create and sell the treaties. It is essential that everything be line-item
accountable and traceable; otherwise, the reinsurance partners may deny claims made to them.

WHAT DIDN’T WORK

The story at this industry-leading insurance company is a common one in the industry. They had
attempted to build their reinsurance sub-ledger application more than once using standard coding
technology and large teams of developers. These were hugely expensive and time-consuming
undertakings, and each time, the number and complexity of the rules foiled the project. Each
time, the development team had difficulty implementing, testing, and verifying the complex rules
using standard coding. Their productivity was insufficient not only to build the application but
also to keep up with new rules coming from the business (after all, the business didn’t stop while
the app was being built). Inevitably, the project would be canceled and the business team would
go back to using spreadsheets and manual labor. It was a disaster.

WHAT DID WORK

Finally, a senior VP stuck his neck out and said, “I can get this done with Ab Initio.” Given the
company’s history with other technologies, this was a bold undertaking. Normally, a project of
this nature could take a large team of developers anywhere from 5 to 7 years to implement. In
this instance, the customer put together a team of developers, much smaller this time, and Ab
Initio provided 2 consultants.

Ab Initio projects often go beyond conventional limits when it comes to data volumes,
transaction rates and interoperability. In this case, the volumes were very modest (even the
largest insurance companies don’t process that many claims), and all the data was
straightforward and readily available. The challenge was dealing with the large number of rules
and their complexity, and dealing with business requirements that were changing dynamically.
High productivity and complexity management are key Ab Initio strengths that were vital on this
project.

The project lasted just 18 months, during which the team implemented and tested about 15,000
rules. Critical to the success of the project, and an ongoing business requirement, is Ab Initio’s
ability to graphically trace logical computations from final results all the way back to the original
inputs. In this way, all the results are auditable – if in doubt, the users can display all logic that
the data went through with a click. The application finally gives full itemization on all
information going through all the treaties. Now, when a bill is submitted to a reinsurance partner,
there are no more questions about what has to be paid. No more spreadsheets and no more
manual work.

The gentleman who stuck out his neck still has his head on his shoulders... and even a new title
to show for his efforts.

Time Tested
Very little of the Internet infrastructure of 10 years ago is still in place today. Usage patterns
have changed dramatically, and volumes have exploded. Nevertheless, one of the systems in
place at one of the core Internet players has survived the test of time – and this system was built
with Ab Initio.

The company is one of the primary players in advertising on the Internet. In order to target ads in
the most effective way possible, it is necessary for the company to track all views of all ads by
everyone on the Internet and to the extent possible, to have an understanding of which web sites
everyone has visited. Naturally, as an advertiser, you would want to focus pet food ads on
someone with a history of visiting animal-related sites, and kitchen equipment ads on someone
known to research recipes or kitchen design. However, you would not want to inundate someone
with too many of the same kinds of ads or they lose their effectiveness. Nor would you want a
competitor’s ads to be running alongside yours. Finally, and this is what matters to revenue,
tabulating which ads are shown to which people drives the billing system for advertisers. All this
needs to be done in near-real-time because the faster you can react to changes in user behavior,
the more effective the ads can be.

Calculating this usage information may not seem especially challenging, until you consider the
scale. We’re talking about many billions of events each day, and all that information about each
Internet user must somehow be maintained. To collect the data, each user is tagged with a unique
identifier (the infamous “cookies”), which is read every time the user visits a web page that
carries an advertisement. The system keeps track of those visits, tracking the specific web sites
and also which ads have been served up to the person’s screen. Each ad is classified in a number
of ways, and so a total user experience is calculated. The goal is to deliver the right ads to the
right people at the right time.

GROWING PAINS

At the time of the first Internet explosion (circa 2000), online advertising was taking off. The
company’s original system had been built at the dawn of the modern Internet era and was already
remarkably scalable. Unfortunately, it had a hard limit – it would not handle more than 2 billion
ads a day. The system had been designed to run across a limited number of servers, and making
it bigger would require a complete rewrite. Faced with rapidly expanding Internet volumes, the
company realized that hand coding would simply be too time consuming – they had no choice
but to consider alternatives.

There was but one alternative: Ab Initio. The Ab Initio Co>Operating System is totally scalable,
and it enables complex calculations to be expressed in a graphical manner that is much easier
than hand coding. In a matter of just 2 months, Ab Initio demonstrated a system that could not
only do everything the existing hand-coded system could do but could handle even more
complex logic. (And the hand-coded system was the culmination of 2 years of development by
lots of smart people.) Moreover, from the start, the Ab Initio application was completely scalable
with no upper bound. The clock was ticking, so the customer went into high gear to deploy the
solution before their existing platform hit the wall. The new application went online just as they
were about to reach the limits of the old system.

LOOKING TO THE FUTURE

That was over ten years ago. Today’s volume is tens of billions of events per day and is still
rapidly climbing. Over these ten years, this customer has been through three different hardware
platforms. The one constant has been Ab Initio. The same application, with only minor changes
to meet new business requirements, is still in production today. The same Ab Initio
Co>Operating System has enabled this application to scale by a factor of 20 without change and
to span several generations of computer hardware.
When something is done right, it withstands the test of time.

Mobile Internet Security

Since the advent of 3G networks, mobile operators around the world have been creating new
products and services to take advantage of the significant increase in network bandwidth. With
smartphones and laptops gobbling up all that 3G (and now 4G) bandwidth, high-speed data
connectivity has been a huge hit.

All this growth has come with some unexpected challenges. A well-publicized risk has been
alarmingly high bills resulting from inadvertent over-use. A customer roaming in a foreign
country can—Oops—start a YouTube download, a misstep that ends up costing hundreds of
Euros. Widely reported stories of customers receiving unexpected bills for tens of thousands of
Euros in a single month helped coin the phrase “bill shock.” Stories like these spread like
wildfire.

“Bill shock” isn’t just an issue for customers. When a customer refuses to pay a massive bill, the
operator ends up eating the lost revenue. But even worse, many more customers are afraid to use
data services for fear of a nasty surprise in the mail. This is equally bad for customers and
operators.

REGULATORS STEP IN

Regulators in Europe were quick to respond. Now all operators must notify customers before
they cross a cost threshold on their usage, and customers must be able to choose a voluntary cap
of €50 per month. And, it had to be done in a handful of months.

At first glance, this looks easy. But it means that each operator has to actively monitor, price and
manage all the incremental data usage for all their customers in real-time—figuring out that a
customer crossed the threshold on the previous day is too late. This is extremely challenging—it
involves combining large amounts of complex usage data with rapidly changing pricing
information.

ACTING QUICKLY

Recognizing the size of the challenge and the limited time to address it, one of the world’s
largest mobile operators turned to Ab Initio. A proof of concept (POC) exercise was rapidly
undertaken. In just 2 weeks, Ab Initio field staff built a system to collect real-time data-usage
records directly from the network switch (GGSN) in an ASN.1 format and using the industry
standard GTP’ protocol. This massive volume of usage records is aggregated continuously.
When usage exceeds a set threshold, the system sends an SMS message to the customer.

These aggregations are hardly as simple as adding up the number of downloaded bytes. Not only
can operators change their tariffs on demand, customers can switch pricing plans in the middle of
the day, too! Continuously changing usage against continuously changing pricing information,
24×7, 365 days per year is a real challenge in any technology.

While the POC was not the complete system (how could it be in just 2 weeks?), the speed with
which it was built convinced the operator that Ab Initio was their only viable approach. As a
result, Ab Initio was invited to help complete the end-to-end production system.

A SUCCESSFUL SOLUTION

The system went live soon after and is highly effective. No more “bill shock,” no more bad
press, and no more lost revenue.

This customer has many significant Ab Initio deployments across the globe for applications
ranging from mediation to data retention and business intelligence. Not only have they have been
able to save money while increasing agility, but they can move ahead confidently, knowing that
there are no more “shocks” in store.

Realtime platforms

With trillions of dollars worth of business being traded every day in the financial markets, there
can be no doubt that both the financial system and global businesses rely on the efficient
operation of our major exchanges... they are truly mission-critical to the economy in every sense.

At the heart of these exchanges is a trade-management system which accepts trade data in real-
time from the trading platform and provides several key services:

 Validation of the trades

 Execution of complex rules to convert the trades into a granular standardized form
 Management of a datastore of daily trades
 Provision of messaging services for exchange members, enabling them to query and update
trade information
 Publishing of trade history for auditing purposes
 Daily trade clearing

Undertaking to replace such an important system can never be taken lightly. Indeed, one of the
world’s main exchanges had been enhancing and maintaining their system for about 20 years.
Although they had known for some time that it needed replacing, the cost and risk of a rewrite in
a low-level language such as C++ or Java had always been prohibitive. And high-level languages
could not provide the necessary performance or scalability. Eventually, however, driven by the
need to replace obsolete underlying hardware and software, combined with the need to overcome
important scaling and flexibility constraints, the exchange decided it was time to reimplement
their trade-management system using new technology.
FIGURING OUT THE POSSIBILITIES

Naturally, an extensive evaluation of technologies was undertaken. To provide confidence in the

decision and to help build the business case, the exchange set Ab Initio the challenge of
replicating some of the key elements of the trade-management system in a Proof of Concept
(POC). The aim was to determine if the technology could:

 Keep up with peak trading volumes, both current and estimated far into the future
 Provide a resilient architecture for 24×7 operation with no missed messages, ever
 Handle the proprietary TCP/IP-based messaging protocol used for all communication
 Provide a large increase in productivity, a requirement for the business case

Many POCs have clear-cut goals, such as: “We must get modules A, B, and C implemented in no
more than 3 weeks; the system has to process X gigabytes per hour with turnaround for
individual messages of no more than 80 milliseconds; it has to run on less than 10 CPUs; and it
has to meet all operability and robustness requirements.” The POC with the exchange had all of
these and one additional, critical one: the existing development team, steeped in traditional
programming languages, had to want to switch to Ab Initio.

WORKING TOGETHER

The joint Ab Initio/exchange team developed and implemented a number of new ideas and
approaches during the POC. They figured out the sizing of the hardware platforms needed to
meet the response and latency times. While a number of server configurations were tested, the
efficiency of the Ab Initio infrastructure meant that, in the end, only 3 cores of a modern Linux
blade were needed to handle a critical part of this international market infrastructure. One can
imagine the faces at the exchange when they learned that a small server was sufficient to run it!

The joint customer/Ab Initio team built an architecture that could support all the flexibility
needed to handle changing market circumstances: new exchange products, different regulatory
frameworks, and other industry-wide initiatives. Toward this end, a modular service-based
architecture using Ab Initio’s publish/subscribe capabilities was adopted. This not only allowed
the application to be decomposed in a logical manner, but also created a resilient architecture
with processing distributed across multiple servers. As one would hope, all aspects of the system
were designed to ensure transactional integrity, and specific effort went into ensuring that the
application provides the real-time performance required of a modern exchange.

All the numerical and architectural requirements of the POC were met in a matter of just a
month. The toughest requirement, getting the exchange team on board with Ab Initio, became
the easiest to meet: the development team leader declared that he would be ten times more
productive in Ab Initio than in a standard third-generation language; and he had the backing of
his team, based on their own first-hand experience.

IMPLEMENTATION
With the POC finished, the key business owners agreed upon an internal project plan, including
schedules and costs. The development team provided conservative estimates for the development
and test activities – at least 40% faster with Ab Initio for the whole effort – to make sure they
would be successful and leave room for the unforeseen. They were secretly salivating at the
prospect of getting the project done even faster and under budget, and being hailed as heroes.

The system went live in 2010 and is now being deployed to further exchanges around the world.
And best of all, the business owners have a new relationship with their IT development team.
They can count on systems being done faster than they could have hoped for previously, and
these systems have a degree of robustness and flexibility that will allow the business to grow
without impediments.

Government Use
Some problems are so big that just getting a grip on understanding what the problem is can be an
enormous challenge. One of the biggest challenges at one of the world’s largest government
agencies is financial management. Financial management touches everything: all priorities
ultimately get settled according to the funds they are allocated. It sounds simple, right? Higher-
priority activities get more money than lower-priority ones. But what if you don’t really know
how much money is being spent on any one activity? Without that critical piece of information,
there’s no real basis for your decision-making and it becomes fuzzy. Such is life at this agency.

The agency has a vision for solving this problem, and it requires an integrated government
transaction clearinghouse. If all financial transactions were to go through a single clearinghouse,
and if that clearinghouse understood the details of all the transactions, then it would be possible
to figure out where the money is actually going. It would be possible to optimize expenditures,
so that more – possibly lots more – could be done with less.

This organization has been working toward realizing this vision for many years, but the problem
is so big that it can’t be solved in one step, or even in one giant leap. They have already set up a
number of transaction clearinghouses. Grossly simplified, these clearinghouses pass transactions
back and forth between different agencies to get things purchased and to make sure those
purchases are actually delivered.

TOO MANY DIFFERENT SYSTEMS

But these clearinghouses have been designed with a “come-as-you-are party” mentality. In a
normal commercial organization, a central governing body will often dictate the interfaces
between systems, even those that cross organizational boundaries. If the target were a toga party,
then everyone would have to wear togas, like it or not. However, in this organization, each
department has a great deal of autonomy, with different systems and often very different security
requirements. It is neither possible to reach into the different systems to get the data one wants,
nor to dictate how the data will come dressed for your system.

For a transaction clearinghouse, the result of this come-as-you-are party mentality is that
everyone comes to the party with a different set of incompatible interfaces and data, and it is the
clearinghouse’s responsibility to make it all mesh together. One of the largest transaction
clearinghouses in this entity has been growing in this manner for years. Unfortunately, it has
reached the limits of the well-known EAI (enterprise application integration) product it has been
using.

THE IMPORTANCE OF METADATA

The core issue with the EAI technology is that it has no sense of “metadata”. The interface rules
are specified by analysts on paper, and then coders turn the specs into programs that are plugged
into the EAI framework. The EAI framework has no idea what is going on in the rules. The rules
are coded in third generation programming languages and are therefore opaque to all but the
original coder (assuming the original coder can even remember what he did). There is virtually
no opportunity for reuse, and it is almost impossible to impose standards. This was all right at the
beginning of the effort, but as the system has grown larger, it has effectively collapsed under its
own weight.

Luckily, the systems integrator (SI) responsible for this transaction clearinghouse saw the
collapse coming way before it happened and initiated a search for a new technology – one that
would handle the transaction volume and complexity, that would coexist with the old system
while it was being phased out, and that would not just capture metadata but also accept metadata
as a specification mechanism. After an exhaustive search, the SI realized that Ab Initio’s
Co>Operating System was the answer.

REALIZING A VISION AND GETTING RESULTS

With the shift to the Co>Operating System, the SI has had to rediscover many of the existing
rules, because none of the people who originally implemented them were around. Further, many
shortcuts had been taken in the original system because there had not been a standard for the
specification of a transaction. In many cases, the original system just passed a transaction along
without having any idea of what was in it. That was clearly unacceptable and is no longer
happening in the new system. Each interface has been redesigned with new standards in mind.
Ab Initio also helped by providing automated translation capabilities from the old rules
technology to the new.

After much previous investment and no success, this agency is finally getting the results it had
always wanted. It is beginning to be able to analyze transactions to associate total cost of
ownership with specific capital items. It is beginning to be able to discern expenditures with
different activities (R&D, maintenance, operations…). It is starting to be able to answer prosaic
questions like “How much does it cost to build a XXX?”

And when you’re dealing with financials on a monumental scale, that’s a very important
question to have answered.
Metadata Challenge at a Global Bank
The word “metadata” usually sends a shiver down the spines of senior IT people. Academia has
been touting its potential benefits for more than 20 years, but the truth is that practically no
metadata project has delivered on its lofty promises. However, the implementation of an Ab
Initio-based metadata-management capability at a global banking organization has really
demonstrated that things have finally changed...

To set the scene, this global bank took over the back-office processing of a number of asset-
management companies. Although this was effective from a business perspective, the IT
department ended up with multiple systems running different technology stacks, all performing
similar activities. The challenge for the bank was to deliver on the anticipated economies of scale
by consolidating everything into a new processing system. However, with many tens of
thousands of interfaces and reports across the different systems, this was a much more complex
task than had been imagined. The analysis effort alone was a sizable and costly piece of work
that affected program timelines and delayed the realization of any benefits that the new system
would deliver.

Much of the analysis phase of the project was a metadata challenge: to relate data about the
existing systems to the new system, it was important to understand how data was processed and
how it flowed through the different systems. All of this had to be done while both the existing
and new systems were continually changing.

GETTING A HANDLE ON THINGS

Initial attempts at the analysis were carried out using spreadsheets, but it quickly became clear
that the process was unmanageable. The number of spreadsheets was growing exponentially, and
this limited the bank’s ability to check for completeness and accuracy. They soon realized that a
sophisticated metadata-management system was their only hope.

Up to this point, no Ab Initio products were being used within that division, but after intense
analysis of metadata technologies, Ab Initio’s Enterprise Meta>Environment (EME) technology
was selected as the foundation for the analysis work.

The bank proceeded to capture metadata about all their legacy applications. This metadata in turn
drove the new application architecture, new interfaces and reports, and a data dictionary. As a
result, the bank was able to fully support the analysts to:

 Understand existing interfaces and reports through viewing and exploring the lineage of
individual data items in the existing systems
 Capture data requirements
 Map data requirements to a standard data dictionary
 Validate the design through visualization in the EME and determine completeness and
accuracy through the associated reporting system
  Automatically generate key outputs: interface design specifications and reports

Much of the metadata required to support this process was captured automatically from non-Ab
Initio technologies, including databases, modeling tools, reporting environments and “ETL”
products. Additionally, more than 1,000 SQL stored procedures were parsed to automatically
extract embedded data-transformation rules and mappings to further enrich the metadata picture.

UP AND RUNNING

The system was operational within a couple of months, at which point the bank saw an
immediate leap in productivity. The analysts needed much less time to understand existing
functionality. Specifying new requirements was much faster. They had completely eliminated the
specification writing step from the development lifecycle, and the specifications that were
automatically generated were consistent, complete, and generally of better quality.

According to the bank, interface and report definition now requires 80% less effort than was
required with the previous process. Furthermore, with hundreds of thousands in savings each
month, the metadata solution paid for itself within 3 months and enabled the critical path of the
program to contract by 6 months. The bank also now has visibility and control of the program
together with real agility – the team can assess and activate a wide range of change projects far
faster than ever before.

In the end, the academics were right... there IS enormous benefit in metadata, but only if you
have a sound vision and the deep technology to support that vision.

Use Case at Health Care Company

Ab Initio projects come in many shapes and sizes. For many, the business and technical
objectives are relatively clear, enabling a game plan to go from start to finish and meet the
objectives. That doesn’t mean that a project is simple or easy – it can be massive and complex.
But like building a suspension bridge, while it may be big and difficult to do, you know pretty
well what the start and end points are and what has to be surmounted.

And then there are other kinds of projects: projects with more general goals, for which success
can only be described in clear terms once those goals have been reached. Such projects are not
like construction efforts with knowable end points. Rather, they are journeys.

Health care management in the US is in a huge state of flux, and no one knows where it will all
end up. Ab Initio is helping a number of large health care providers navigate aspects of this
journey. What follows is an account of two such efforts.

STORY ONE: THE COMPLEXITIES OF MEMBER MANAGEMENT

One big topic for health care providers is member management. Obviously, healthy members are
more profitable than less healthy ones. But health care providers don’t usually get to choose their
members. And if they tried to, they could understandably cause an uproar.

Ab Initio is helping one health care provider (which we shall call "HCP") take a positive
approach to this problem. First, make sure healthy members stay with HCP. Second, help healthy
members stay healthy. Third, identify members who could be using lower-cost generic drugs and
make them aware of that option. Fourth, aggressively steer somewhat less-healthy members into
programs that will identify medical issues sooner (when costs are lower) rather than later (when
costs are higher). And finally, for members incurring significant out-of-pocket expenses,
determine whether Medicare Part B would be a better choice; and if so, steer them there. If
Medicare Part B is better for the member, chances are it will also lower costs for HCP.

While these goals are somewhat nebulous – there is no single or obvious way to achieve them –
Ab Initio is helping HCP investigate and implement a variety of IT approaches. These efforts
center around information capture, integration, analysis, rules definition, and finally, rules
execution in systems that interact with the members.

To begin, Ab Initio has helped establish a framework for defining and executing large numbers
of complex rules. These rules are applied to information for each member. The rules categorize
the members (healthy, less healthy, healthy but…), and within these categories members are
flagged for various actions.

A “member” can be far more complex than just a single individual – it might include entire
families. The categorization of members changes as HCP better understands its customer base.
And ideas about the kinds of activities it can provide its members are ever-evolving.

Some ideas work and some don’t, but the only way to find out what works is to build something
and try it out. The faster you can try out ideas, the faster you can figure out which ones will
succeed. The Ab Initio framework, as constructed for HCP, allows HCP to pull complex data
about individuals, put it together into “members”, and then apply complex sets of rules. And it
can do all of this in very small amounts of time, allowing for hugely productive what-if
experiments.

The rules and data are so complex that understanding results can be a challenge. The Ab Initio
Business Rules Environment and Enterprise Meta>Environment (EME) allow the business users
to try out rules and immediately see the results, and to trace the computation of the results all the
way back to the original data.

Additionally, the complexity of the overall problem means that being able to graphically see how
the system makes decisions is critical to the business users. Once rule changes have been defined
and initial testing has been done, the new rules need to be run over the entire member base for a
complete analysis. The Ab Initio Co>Operating System’s raw performance is so high that not
only can test runs be done quickly, but HCP is also able to run all of their rules across all their
member data on a daily basis.
As a consequence of deploying the new system, HCP has recognized that it can make even better
decisions by pulling in ever more data from their disparate systems. The business has recognized
that their strategy now depends on truly pulling together all the information they have into a
single place; and that means substantially expanding their data warehouse, which is also built
and maintained with Ab Initio software.

STORY TWO: THE COMPLEXITIES OF CODE CONVERSION

Ab Initio is helping another health care provider with a different kind of journey, the replacement
of ICD version 9 codes with ICD version 10 codes. ICD stands for "International Statistical
Classification of Diseases and Related Health Problems" and these codes are used pervasively
across the health care industry. The ICD10 codes represent all the same diseases and health
problems as the ICD9 codes, but with far greater detail. For example, there is a single ICD9 code
for “open wound of shoulder and upper arm”, but there are ICD10 codes for further describing
the wound as “superficial”, “open”, “crush”, “amputation”, “dislocation/sprain/strain”, and
“other”. These codes are of great interest to health care providers because they are used for many
purposes, including reimbursements. How codes are assigned can make a big difference to the
bottom line.

For this provider, the first step in this journey is to update all applications that use ICD9 codes to
use ICD10. There are about 1,000 ICD9 codes, but close to 50,000 ICD10 codes. Ouch. This is
obviously an enormous task. Ab Initio’s expertise with metadata helps the provider survey all of
its datasets as well as all of its applications’ source code. The Ab Initio Data Profiler can analyze
large numbers of large datasets to figure out which fields of these datasets appear to have ICD9
codes, and which other data elements appear to be related or relevant to ICD. And Ab Initio’s
EME code-parsing capabilities can scan application source code for key verbs, field names,
phrases, and the like.

Ab Initio helped the provider put all this information together into the EME and then built a
workflow process so that humans could inspect each potential “hit” in context and determine if it
is indeed valid.

At the same time that identification of all uses of the ICD9 codes was being performed, data-
processing applications were built to pull together information about each claim to automatically
convert ICD9 codes to ICD10. There is often sufficient other information in the datasets that
when combined with ICD9 can yield the right ICD10. However, this is very specific to each
dataset, and so there can be many rules for the conversion and the conversion rules can be very
complex. As with the earlier HCP example, getting these rules right is an enormous challenge, so
Ab Initio set up a framework for the business experts to be able to interactively specify, test and
deploy their conversion rules.

As part of building conversion rules, the provider had to take into account the potential financial
implications of the conversion mappings it had chosen. There is no such thing as the “correct”
mapping of ICD9 to ICD10 – other data in a member’s claim can substantially change the
mapping – and yet these codes are used to calculate reimbursements. So the mapping strategy
can have a major impact on the bottom line.
Imagine doing a complete conversion effort and only then finding out that the choice of codes
would reduce reimbursements by 10%. Unfortunately, that is mostly what is happening in the
industry. For this health care provider, Ab Initio enables bulk conversions to be done very
rapidly, and these are applied to large claims datasets to analyze the financial impact of the
conversion rules. Rather than waiting till the end of the conversion project, as is typical, the
business can steer its conversion process to make sure that it is at least revenue neutral.

While no one can claim to know where the health care system will end up a few years from now,
these are but two examples of how Ab Initio – its people and its technology – is actively helping
its customers navigate this long and difficult journey.

Usecase at a Telcom
We all know what happens to the plumbing in an old house: When it is first installed, it’s state of
the art. But over time, as the needs of the occupants grow and the house is expanded, more
plumbing is grafted onto the old. Then building codes change, so some of the pipes have to be
refitted. As the house ages, some pipes begin to leak and more plumbing is replaced. Then there
are the old lead pipes, which constrains the use of certain taps. The kitchen and the bathrooms
are updated, so a new hookup to the water main is necessary for the increased demand.
Eventually, almost all of the pipes have been impacted in one way or another.

If the data coursing through the veins of your business were like this, you would certainly say
“enough!” and order a rearchitecture and reimplementation. Or would you? Sadly, if you are a
network operator in the mobile telecom industry, instead you probably spent your time “bolting
on” one new system after another to keep up with mergers and market forces. You would have
ended up with a whole range of different and incompatible technologies: multiple network
devices, billing, CRM, and reporting systems... to name just a few. And the complexity,
inefficiency and cost would be eating you alive today. Just like the old house with leaky
plumbing.

“But does it have to be like this?” a large mobile operator asked of Ab Initio. “Yes” if you work
with standard technologies and methodologies. “No” if you work with Ab Initio. So this
customer gave Ab Initio the green light to dig into its systems.

GETTING TO THE FOUNDATION

Getting started, Ab Initio uncovered the usual smorgasbord of technologies. Holding the whole
thing together was a large amount of handcrafted C++ code. Ultimately, in this case, the root of
the problem was the mediation architecture—its lack of flexibility meant that changing upstream
and downstream systems would require even more custom C++ code, and this encouraged all
kinds of work-around solutions. The mediation engine was also incapable of keeping up with the
data volumes, hundreds of millions of call detail records (CDRs) per day, which meant that it
took days for the data to wind its way through the environment.
So Ab Initio set the following architectural principles for the redesign:

 Decode CDRs once only, saving processing time and resources

 Write data to disk once only
 Promote reuse of application code (develop once, use many times)
 Future-proof against record format and application changes
 Provide faster and more reliable application delivery time
 Ensure metadata availability covering end-to-end processing
 Deliver much lower latency from network switch to target systems

If you are not a geek, you don’t need to read further—the project was a success! Of course, Ab
Initio wouldn’t have it any other way. From inception through pre-production testing to go-live
took less than 12 months for this business critical system. Today, the mediation system not only
processes the full daily volume on a fraction of the previous hardware, the latency is measured in
minutes rather than days—which means that the operational and reporting systems are tracking
the business in near-real-time. When the management decides it needs the data in real-real-time,
the system as is will support that too. Finally, due to the “metadata-driven” flexibility of the new
mediation engine, the bottlenecks in the upstream and downstream systems were quickly
resolved and much of the custom C++ code was thrown away.

THE NITTY GRITTY

Now, for the geeks amongst you, here are some of the technical details.

Ab Initio approached this challenge by designing a single generic CDR mediation system that is
able to take the numerous different input stream formats (such as voice calls, data records,
multimedia, etc.) and produce decoded, de-duplicated and enriched CDRs. These CDRs are then
published in Ab Initio queues, ready for consumption by the various downstream systems.

By publishing decoded and de-duplicated data in Ab Initio queues, the team met the architectural
principles of decoding data once and writing data once. These queues are low-latency, parallel
devices that support multiple independent subscribers (readers). As a result, data is retained in
the queue until all subscribers have consumed the records, thus ensuring that only one copy of
the data is required to be stored in recoverable media.

The mediation system now handles changes to data formats and processing logic for each input
data stream without requiring large development projects. This was achieved by designing the
mediation system to be “metadata-driven.” As a result, the specific requirements for each data
stream are decoupled from the generic mediation processing flow, which means all data streams
are processed in a consistent manner—and specific formats and rules for a data stream are loaded
dynamically when required.

NO NEED TO COMPROMISE

Traditionally, there is a compromise between flexibility and performance, and between speed of
implementation and implementation longevity. The Ab Initio solution, built on top of Ab Initio’s
Co>Operating System, defies these traditions. Volumes are on their way to doubling, and yet
upstream and downstream systems have the flexibility they need to adapt. And new development
efforts are being undertaken at speeds not previously seen, while simplifying and reducing
maintenance.

At Ab Initio, innovation trumps tradition.

Use case at a Consumer Goods Company

A very large consumer products goods company (CPG) had a problem that was core to
integrating its different product lines—a common problem in the industry. While the nature of
the business around each product line is similar, they grew up as separate entities—essentially
separate businesses—each with its own independent infrastructure. Each business would pick up
its raw food materials, ship them to factories where finished packaged food goods were
produced, ship those to warehouses and distribution centers, and ultimately deliver them to
stores. Unfortunately, each business had its own transportation network, so even though their
trucks were often visiting the same locations, those trucks were often unnecessarily empty. As
they have one of the largest trucking fleets in the world, the wasted trips compounded quickly.
As a result, the CPG was leaving a lot of money on the table.

EARLY CONSIDERATIONS

At first, the CPG considered building a new application from scratch using Java. But as the
design work began, it became apparent that the complexity would turn this into an ultra-large
project that would take a long time and carry a high degree of risk. Each day that went by
without a solution had a quantifiable cost to the business, but the cost of a failure in the system
would be even more enormous. Without the right kind and quantity of goods in the stores,
revenue would take a big hit. So they turned to Ab Initio to solve this problem.

There were many requirements for the new application. It would have to be fully integrated into
each of the very large CPG’s divisions. It would have to interface with the ordering systems for
each. It would have to interpret and process various details of the trucks, factories, warehouses,
distribution centers, and stores. It would have to understand aspects of the goods that related to
transportation. It would have to enable independent truckers to bid on planned shipments. It
would have to connect to all the places where trucks might pick up goods in order to generate
bills of lading as well as directions to the crews regarding which goods should be put on which
trucks. It would have to feed information to a third-party state-of-the-art optimization engine and
communicate the resulting optimized plan to all relevant parts of this global company. In other
words, this application would be literally in the middle of everything. And because this business
runs 7×24, it would need to be a real-time 7×24 operational system.

RAPID DEVELOPMENT AND A NEW SOLUTION

One would think building such an application for one of the world’s largest businesses would
require a large effort and a lot of people. But for Ab Initio, that isn’t necessary, even on large
applications. A single Ab Initio consultant was able to help with architecture, implementation,
and all the necessary on-the-job training for a small team of the customer’s own application
developers, none of whom had used Ab Initio software before.

When it came to interfacing to the customer’s enterprise message bus (in this case, supported by
a common messaging product), Ab Initio once again defied expectations. The standard approach
is to break down an application into many small programs, each of which reads and writes
messages from the bus. All the small programs have to work together in a consistent manner or
else the whole will not survive a system failure. Because the responsibility for ensuring
robustness lies with the system architects and developers, this often translates into significant
amounts of additional coding and testing. And, as is often the case, if the team doesn’t anticipate
all failure modes, well, you know what happens.

Ab Initio architecture, however, provides robustness features to handle all kinds of failures. As a
consequence, the application was reduced to a small number of “graphs” – graphical renditions
of the business process and logic. All processes and logic were built entirely in Ab Initio, without
resorting to traditional coding. The system was inherently robust because the underlying
technology had robustness already built in.

Start to finish, this project was completed in less than one year. And the customer was extremely
surprised (and gratified) to discover that once the system was in production, it just worked.
Normally, new systems go through a lot of “teething pain”. However, because the application
had been substantially simplified with Ab Initio, and because failure handling was built into the
underlying technology, the normal challenges melted away.

LOOKING AHEAD

This very large CPG has deployed the new transportation-optimization system across all of its
divisions and has reaped tens of millions of dollars in savings. The project’s manager, since
promoted to SVP, likes to point out that in CIO status meetings, while his peers are bogged down
discussing thorny issues with their systems, he gets to talk about his next project, which he is
building with Ab Initio, naturally.

ETL usecase
The stock exchange business is in a high state of flux: new markets are opening around the
world, new competitors are jumping in, and new technologies are changing everything. In order
to remain one of the premier stock exchanges in the world, a customer embarked on an
aggressive IT modernization effort, choosing Ab Initio for critical parts of their systems.
A simplified view of this customer’s data processing is that there are two halves: the front is the
trading platform that supports the trade floor and computer trading. The back is known as the
“post-trade” processing platform. The back half does everything from settling trades to market
surveillance to interfacing with all the exchange members and the clearing houses. The post-
trade platform is where the heavy computation happens, and this customer is reimplementing all
of it in Ab Initio.

A FIRST ATTEMPT

The legacy post-trade system dates back 20 years and consists of more than a million lines of
Cobol code. An increasingly outdated behemoth, it is not only difficult to maintain and enhance,
but, complicating matters, the original coders are retiring. This means that every change to this
platform is a big deal, and naturally the business people are frustrated that they can’t get the
functionality they want.

In an effort to lower its costs, the exchange decided to move the post-trade system to less
expensive Unix servers. Their first try was to use Unix rehosting software for Cobol. In
principle, this should have been easy: just take the Cobol code and relocate it to the Unix servers,
where it would run with no changes. This would not modernize the code, but it would achieve a
key goal of getting off the mainframe and onto much cheaper Unix servers.

But even after more than a year and a large expenditure of effort, the results were not pretty. Too
much code required adjusting, the adjustments required significant testing, and in the end the
resulting code didn’t run fast enough. The environment was extremely complex and hard to
manage, and there were reliability issues. Plus, the output was still Cobol, so none of the other
business goals were addressed.

CONVERTING THE CODE

The good news is the exchange discovered Ab Initio. Ab Initio has technology that will
automatically translate 70% to 90% of a large Cobol application to Ab Initio’s graphically driven
Co>Operating System. Instead of code that is hard to read and maintain, the entire million-plus
lines of Cobol were translated into easy-to-understand graphical images and business rules.
Because of the general-purpose nature of the Co>Operating System, not one piece of the
resulting system required external coding, shell scripts, or stored procedures.

The entire effort of converting the million-plus lines of Cobol to Ab Initio took just 10 months
from start to finish. The sections of the code that could not be automatically translated
(mainframe data structures like VSAM that don’t exist on Unix, and arbitrary GOTO statements
that create infamous “spaghetti” code) were handed off to a small offshore team. The total end-
to-end productivity was more than 1,000 lines of Cobol per person per day to get from the
original code to QA.

Another critical requirement was interfacing with the trading message bus. This bus can burst at
rates of hundreds of thousands of messages per second. Needless to say, every message has to
get through, and in the proper order. To make this possible, the customer had built an elaborate
infrastructure in Cobol. As a result, the customer’s IT professionals were concerned about
moving forward. Having been burned before by software companies with exaggerated marketing
claims, they were naturally skeptical that Ab Initio could not only translate and run millions of
lines of Cobol code but also process data at rates of hundreds of thousands of messages per
second.

But Ab Initio does not exaggerate. It doesn’t have to. Not only was the customer’s complicated
and unwieldy code successfully translated and put into production in just 10 months, the Ab
Initio applications were benchmarked at many hundreds of thousands of messages per second.
And this was easy because of Ab Initio’s built-in robustness.

LOOKING AHEAD

Now that the post-trade platform has been rehosted to Ab Initio, this customer is able to keep its
costs lower while still responding to the needs of the business far more aggressively than before.
Plus, it can do so in full confidence that its systems will scale in a robust manner to whatever
level is needed.

From this, even more ambitious projects are under way.

Usecase at CreditCard Company

Ab Initio gets called upon to solve big problems, and this is an example of a big one. A number
of years ago, one of the largest world-wide credit card networks was running out of horsepower
in its data backbone. Transaction volumes were climbing steadily, and it was just a small matter
of time before the system would no longer keep up. In addition, because the backbone had been
cobbled together over many years as the business had grown, it consisted of a large number of
complex point-to-point connections and interfaces. The result was that no one aspect of the
backbone could be changed without touching all the elements. And the business was desperate
for more flexibility.

EARLY ATTEMPTS

This company tried everything to remedy the situation. It rearchitected some of its major
mainframe systems. It moved some systems off the mainframes. It brought in a well-known
enterprise application integration (EAI) product and started writing loads of C++ code.
Ultimately, none of this solved the problem. Remediating individual systems bought time but
didn’t solve the underlying problem. Furthermore, the EAI product and C++ code were not only
extremely time-consuming to deploy, they didn’t scale. What was worse, the more they tried to
make them scale, the bigger the problem they had with dropping transactions. Naturally, in this
business, you can’t drop transactions.
This is where Ab Initio came in. At first, it was difficult for this customer to believe that a
software product existed that would allow them to (1) implement all of their complex logic on
complex data in a graphical manner, (2) scale systems to whatever rate they might ever need, (3)
run equally well on mainframe and Unix platforms, and (4) build robust systems that would
never lose data. It sounded so good that their chief architect kept saying “Stop making all those
claims! If you do half of what you claim to do, that would be a miracle!” From an Ab Initio
perspective, it wasn’t a miracle – just good engineering.

But before going forward, the customer required, and Ab Initio insisted on, an end-to-end Proof-
of-Concept. In a matter of just a few months, this customer and Ab Initio put together a small
team that built the majority of the data backbone and benchmarked it at a rate of many tens of
thousands of transactions per second on a medium-sized mainframe. This was several times
faster than the actual requirement, and it handled all the complex data that flowed through the
system. The customer decided to go forward.

BUILDING THE NEW DATA BACKBONE

Over the next 6 months, this small team replumbed all of this customer’s systems to get all credit
and debit card transaction authorizations through the new data backbone. Many dozens of back
end systems were involved, and they came in every possible flavor: some were on mainframes
and some on Unix, some were real-time and some were batch, some were very high volumes and
some were trickle feeds, some took data in legacy formats and some wanted XML. This was
done so quickly because of Ab Initio’s ability to handle complex data and complex business
logic, which in turn allowed connections to legacy systems that did not require modifications to
those applications.

Given the centrality of the data backbone to this customer’s systems, the company was extremely
conservative in its deployment, testing in every possible way to ensure that the new backbone
would be resilient in the face of inevitable failures of the environment. Ab Initio’s checkpointing
and guaranteed message delivery are core capabilities that were heavily exercised during this
testing. The system was so robust that when it finally went live, the worst thing that happened to
the deployment team was that their beverage cooler sprang a leak. That was a mess to be cleaned
up!

Since then, this customer has reimplemented many of its back end systems with Ab Initio. These
systems include merchant billing, fraud detection, loyalty cards, large-scale data retention for
customer service, and everything having to do with business intelligence and reporting.

As for the data backbone, transaction rates have approximately doubled since initial deployment
on this customer’s mainframes. Rather than doubling the mainframe platform, an expensive
proposition, the customer took advantage of Ab Initio’s capability of running applications
distributed across multiple servers with different operating systems. They did so by relocating
much of the data backbone’s computational workload from the mainframes to less-expensive
Unix servers simply by changing configuration settings – the applications themselves were not
changed at all.
SUCCESSFUL RESOLUTION

This company has successfully deployed the Co>Operating System across applications of all
types, from operational systems to business intelligence. To support all these users and
applications, they have taken extensive and full advantage of Ab Initio’s strengths – its
technology, its people, and its deep customer-focused culture.

Transportation Company Use Case

You know how perfectly nice, capable families can end up in a rickety old house where
everything is patched up and patched together? The pattern is common: they buy a problematic
house hoping to overcome its limitations by investing their time and money. Eventually things
snowball, the problems become overwhelming, the house no longer works for them, and they
need to move. But having invested so heavily in their present home, they have limited funds for
the next one, and so they end up with another fixer-upper with a different set of problems. This
cycle repeats, except each time the expense and effort of dealing with the inadequacies increases,
making the next jump that much more difficult.

And it’s not just the money. People grow accustomed to the routines associated with upkeep, and
they become attached to the things they have worked on so hard for so long. They may not even
be able to imagine that something better exists; and so instead of looking for alternatives, they
keep right on patching.

Well, the same thing happens – and all too often – with IT systems. The only difference is that
entire companies are involved and so the costs are that much bigger. But when, from time to
time, new people join the team, having a fresh new perspective in the mix can open people’s
eyes to what they have been doing. Such was the situation at a very large multinational
transportation company: their outdated billing and invoicing system – the gateway through which
all revenue comes into the company – was no longer working effectively for them.

AN INEFFECTIVE SYSTEM EXPENSIVE TO MAINTAIN

This customer had originally built its own billing system. But after experiencing high growth
rates for many years, the company finally farmed it out to a well-known systems integrator who
reworked it multiple times, costing the company many tens of millions of dollars. While some
aspects of the system were improved, ultimately the changes made it harder for the company to
manage and extend. So they spent most of their time going in circles, fixing the same kinds of
problems over and over again.

Over time, the weight of maintaining the system became overwhelming. With millions of items
needing to be processed nightly, the system simply could not keep up. This directly impacted
revenue and was highly visible. The cost of maintaining the system was also significant – 100
full-time people were required to deal with the application problems that cropped up each day!
Nevertheless, this company kept patching up the system for years because it could not imagine
doing something different, or that something different could be better. And the staff that worked
on the system didn’t want to contemplate a world that did not require them.

A NEW KID IN TOWN

Until a new senior manager showed up. This manager knew from experience what could be
accomplished with Ab Initio and wanted to rebuild the system from scratch. He knew there was
something different, something better out there. But resistance was fierce. All kinds of excuses
and arguments were made: “But how do we know it will work properly?”, “But the software is
too expensive”, “But we’ll have to retrain everyone”, “We can’t afford any mistakes”, “We can’t
throw away our investment in the current system, it took so long to build”, “How do we know
Ab Initio will help us?”, “But we don’t have anyone who knows Ab Initio”.... The manager,
having been through it before, forged ahead anyway.

Six months. Four developers. Two Ab Initio consultants. That’s all it took. The entire application
was replicated, tested, and put into production. The new version was verified to generate exactly
the same results as the old system, minus the bugs. Because of Ab Initio’s built-in scalability, all
the invoices were now being processed each night, which had a noticeable positive effect on cash
flow. And equally important, the cost of the software was less than what the customer was
already spending on licenses for a well-known processing utility, a utility that was rendered
obsolete by the Ab Initio software. And instead of requiring a staff of 100 to maintain the system
on an ongoing basis, the robustness and ease-of-use of the software means that only 15 are
needed. The others were freed up to do other tasks of far greater value to the business.

It can be hard to get people to vacate their rickety old houses. But when they move into
something better, they will never want to go back.