Upsolver+Snowflake_at_____________Centene.pdf

Santona Tuli | Director of Data
Alexander Adam | Manager, Cloud Platform Engineering
February 23, 2023
Powering Centene’s real-time
enterprise data analytics
with Upsolver

Manager, Cloud Platform Engineering
Santona Tuli
Director of Data
Upsolver
Alexander Adam
Centene Corporation

AGENDA • Intro to Upsolver
High quality, observable data ingestion at scale
• Intro to Centene
Largest Medicaid managed care organization
• Centene data architecture
A perfect match for Upsolver and Snowflake
• Why Centene chose Upsolver
Benefits of managing data quality at ingestion

About Upsolver

Data quality at ingestion
A cloud data ingestion platform, purpose-built to overcome the quality
challenges posed by complex data (streams, files, CDC).
Certifications Partnerships

High quality, observable data at work
Over 10,000 production pipelines
Processing petabytes per day
Across multiple industries in enterprise, midsize, and startups

The compounding cost of quality issues
Ingestion Transformation Consumption
$
DISASTER
DIFFICULT FIX
SIMPLE FIX Analytics engineer rebuilds
multiple pipelines and
downstream dependencies
Data engineer fixes a
single ingestion job
Stale, incorrect, or
corrupt data used in
numerous analytics
and ML use cases
$
$
Incident
occurs
Data lifecycle
Quality
Issue
Impact

Source data may arrive out-of-order
and include duplicates
Transform models need to be really
complex to handle different cases
Sources & targets use different data
types and naming conventions
Dependent models break,
columns disappear or are repeated,
data are cast incorrectly
Source systems’ outputs
change without warning
Unexpected or empty values in
important fields ruin analytics
Root cause Downstream impact
Issues at ingestion wreck havoc later

ExactSync
● exactly-once delivery
● strong data ordering
Transformations can focus on data,
rather than error-handling
Auto schema evolution
Resolves type conversion and
column naming violations
No schema-related breakage or
manual evolution work
Real-time data observability
Continuous statistics per field for
monitoring and retrospection
Quick detection dramatically
reduces data issues and cost to fix
Upsolver solution Benefits
How can we prevent it?

Significant increase in data volume
Significant number of NULLs in important fields
Haven’t been updated recently
Newly added columns
Not what I expected
Outage!! - unexpected drop in events
Real-time data observability

1
2
3
Getting started with

Quality
Observability
Simpler models,
fresher data
Model
Sources Ingest Stage Transform Deliver
Reference architecture

Lakehouse
Data
warehouse
Operational
databases
dbt CLI SDK
SQL
GUI
ExactSync Auto Schema
Evolution
Real-time Data
Observability
Streams
Operational
Databases
Object Stores
Upsolver in the stack

Upsolver in any stack

TESTIMONIALS
Upsolver plays a crucial part in our core
data infrastructure, and the team has
proven to be a reliable partner that’s
been committed to our success from
day one.
Amit Attias | CTO, Bigabid
With Upsolver, I could handle massive
amounts of streaming data and see
real results, in a fraction of the time I
thought it would take.
Guy Levy-Yurista | ex CSO, Sisense
Upsolver is solving a problem
at scale that I don’t want to
think about. The fact that
it just works and people
don’t complain is a win.
Yuji Xie | Analytics Lead, Gem
Upsolver has saved thousands of engineering
hours and significantly reduced total cost of
ownership, enabling us to invest in our
hypergrowth rather than data pipelines.
Seva Feldman | VP of R&D, Ironsource (Unity)

Upsolver is like the “easy button” for Snowflake.
We ingest data from our Kafka streams, process it
as necessary for different use cases, and deliver it,
all while observing how our schema and data are
changing in real time.
Alexander Adam
Cloud Platform Engineering Manager, Centene

Meet our client
Centene Corporation

Centene Corporation in a snapshot
• Purpose: Transforming the health of the
community, one person at a time.
• Mission: Better health outcomes at lower costs.
Centene provides access to high-quality
healthcare, innovative programs and a wide range
of health solutions that help families and
individuals get well, stay well, and be well.
• By the numbers:
○ 74,300 employees
○ Covering all 50 States
○ 27.1 Million Members
○ Fortune #26

How Centene uses data
We use data to help provide care and solutions for our members.
Our data lake receives data from many different source systems related to:
Claims, Member, Provider, Authorization, Corporate Operations, Health Plan, etc.
Different business domain teams leverage data to improve an array of solutions, such as:
• Processing authorizations on the latest information about a member or their care.
• Validating prescriptions by combining pharmacy system data and member data.
We have a strict cloud security model so we can protect our member data.

Data platform team
Our group’s goal is to ingest data from source systems and ship relevant data to
domain-driven data stores, depending on different criteria and SLAs.
• Information in the source systems get updated throughout the day.
• Data get streamed through Kafka, processed by Upsolver, and pushed into Snowflake.
• Data from different sources are transformed and combined into larger,
domain-informed data sets for downstream operations by business partners.
Transform
Extract Load Operate

Platform enablement
Moving from on-prem to cloud, simply adding headcount to the platform team didn’t scale:
• Our domain-driven use cases are highly specialized.
• With our previous setup, it was taking too long to onboard and train data engineers.
We needed:
• A platform that allows different teams to self-serve their relevant data.
• An enablement team for setting up processes and establishing usage patterns,
providing templates, documentation, and how-to guides on data flow patterns.
• A solution for data platform engineers, domain data engineers, and data analysts.
We found a simple solution in Upsolver → “the easy button”

Data ingestion for Snowflake at Centene

Why Centene chose Upsolver
Simplification of workflows
• Quick way to ingest and flatten JSON data, with
metrics on schema drift.
• Fan out source feed to multiple Snowflake databases
with different transformation requirements.
• Eliminated the development complexities of
hand-coding Glue/PySpark jobs.
• Avoid pitfalls such as rouge Glue jobs that may
unpredictably drive up costs.
• Require less access provisioning to AWS account.

Why Centene chose Upsolver
Putting quality, governance, and security first
• Data never leaves Centene’s cloud infrastructure.
• We can build monitoring based on the metadata found
within the system tables in Upsolver.
• Clusters auto-scaling significantly reduces costs.
• Enables cost attribution to teams based on usage.

How Upsolver stood out
In one word, simplicity.
• Ease of setting up ingestion from Kafka to Snowflake.
• Creates a responsive staging table.
• Live metrics and metadata on fields as data flow.
Added more than 100 data pipelines
Onboarded 45 active users
… in two months

Business impact in a short time
• With Upsolver, cloud data pipeline creation went from quarters to a single sprint.
• A platform for subject matter expert data engineers to self-serve their data through
the lake to EDWs that abstracts the complexities of creating an ETL pipeline.
• A lot less overhead on infrastructure management compared to other tools, while still
providing flexibility and customization of infrastructure.
Time
Cost
Ops

Q&A

THANK YOU!

Upsolver+Snowflake_at_____________Centene.pdf

Recommended

More Related Content

Similar to Upsolver+Snowflake_at_____________Centene.pdf (20)

Recently uploaded (20)

Upsolver+Snowflake_at_____________Centene.pdf