3 Pitfalls Everyone Should Avoid with Cloud Data

Mar 11, 2024Download as PPTX, PDF0 likes102 views

The daily hype is all around you. From cloud native, multicloud, to hybrid cloud, this is the path to your digital future. The choices you make as a developer does not preclude the daily work of enhancing your customer's experience and agile delivery of your applications. With all this delivery and infrastructure, there is a lot of data generated when engaging with any cloud experience. Regulatory and compliance pressures force us to store audit and observability data. Understanding the pitfalls around the collection, storage, and maintenance of your cloud data can mean the difference between bankruptcy and success with our cloud native strategy. Let us take you on a journey, looking closely at the decisions you are making as a developer delivering and dealing with monitoring your applications. Join us for an hour of power, where real customer experiences are used to highlight the three top lessons learned as their developers transitioned their data needs into cloud native environments. Key Takeaways: Attendees to this session will gain insights into the data explosion that is part of the large scale cloud native world. Real customer experiences are used to highlight the three top lessons learned as their developers transitioned their data needs into cloud native environments.

chronosphere.io
chronosphere.io
3 Pitfalls
Eric D. Schabell
Director Evangelism
@ericschabell{@fosstodon.org}
Everyone Should Avoid with Cloud Data

chronosphere.io
Understand your
cloud costs?

chronosphere.io
De Persgroep
(Based on audience question from ‘Power to Innovate talk)
“Why don’t you go 100% to cloud for hosting your news group
assets?”
The quoted price for hosting was OK, but the bandwidth
quotes were off the charts.
Shortly thereafter, during the Paris attacks (Charlie Hebdo)
the French people crashed all local news sites. The Walloon
(FR Belgium) sites hosted by De Persgroep received an extra
1.2 million unique visitors…. (810K BE, 450K NL).

chronosphere.io
Observability across
entire landscape?

chronosphere.io
“It’s remarkable how common
this situation is, where an
organization is paying more for
their observability data, than
they do for their production
infrastructure.”

chronosphere.io
Data complexity
Experiment:
- Hello World app on 4 node
Kubernetes cluster with
Tracing, End User Metrics
(EUM), Logs, Metrics
(containers / nodes)
- 30 days == +450 GB

chronosphere.io
The struggle is real
“I don't yet collect spans/traces because I can hardly get our devs to care about basic metrics, let alone traces.”
“This is a large enterprise with approx. 1000 developers. Cultivating a culture of engineering that cares about
availability is a challenge that we need to solve alongside any technical implementations.”

chronosphere.io
https://go.chronosphere.io/2023-observability-report.html
87%
of engineers say using
cloud native architectures
have increased the
complexity of discovering
and troubleshooting
incidents

chronosphere.io
10 hours
on average, per week, trying
to triage and understand
incidents - a quarter
of a 40 hour work week

chronosphere.io
33%
said those issues disrupted
their personal life
39%
admitting they are
frequently stressed out

chronosphere.io
22%
said they want to quit

chronosphere.io
Cloud Native Observability
at Scale

chronosphere.io
3. FinOps the crucial Ops

chronosphere.io
Know the cost of
observability
metrics data?

chronosphere.io
Dedicated FinOps
“By 2023, 80% of organizations
using cloud services will
establish a dedicated FinOps
function to automate policy-
driven observibility and
optimization of cloud
resources to maximize value.”
-- Source: IDC 2022

chronosphere.io
What should be
the #1 item on
cloud wishlist?

chronosphere.io
1. … ??? …
2. Determine your goals of migrating data to the cloud
3. Assess your current situation
4. Select the right cloud migration partner
5. Create your business case for the cloud
6. Select the type of cloud environment needed – public, private, hybrid or hybrid-multi?
7. Determine the specific cloud components necessary
8. Choose the right cloud provider
9. Plan the cloud approach
10. Execute the migration
11. Observability of the production environment

Questions?
Eric D. Schabell
Director Evangelism
@ericschabell{@fosstodon.org}

The daily hype is all around you. From cloud, multicloud, to hybrid cloud, this is the path to your digital future. These choices you’ve got to make don’t preclude the daily work of enhancing your customer's experience and agile delivery of those applications. With all this delivery and infrastructure, there is a lot of data to consider when engaging with any cloud experience. Regulatory and compliance pressures force us to store audit and observability data. Understanding the pitfalls around the collection, storage, and maintenance of our data can mean the difference between bankruptcy and success with our cloud strategy. Let us take you on a journey, looking closely at the decisions being made for delivering and dealing with monitoring those applications. Join us for an hour of power, where real customer experiences are used to highlight the three top lessons learned as they transitioned their data needs into cloud environments.

3 Pitfalls Everyone Should Avoid with Cloud Native ObservabilityEric D. Schabell

Are you looking at your organization's efforts to enter or expand into the cloud native landscape and feeling a bit daunted by the vast expanse of information surrounding cloud native observability? When you're moving so fast with agile practices across your DevOps, SRE's, and platform engineering teams, it's no wonder this can seem a bit confusing. Unfortunately, the choices being made have a great impact on both your business, your budgets, and the ultimate success of your cloud native initiatives. That hasty decision up front leads to big headaches very quickly down the road. In this talk, I'll introduce the problem facing everyone with cloud native observability followed by 3 common mistakes that I'm seeing organizations make and how you can avoid them!

DZone webinar - Shift left ObservabilityEric D. Schabell

Whether you’re an enterprise migrating to cloud-native or born in the cloud, most of today’s APM and Observability tools don’t support how your engineers and DevOps teams need to develop, deploy, and support their software. Observability needs to shift left and reflect the modern way companies organize their development teams and their vital interdependencies. Chronosphere is the only vendor addressing the unique requirements for observability in a cloud-native world. Join this webinar to learn: • What cloud native observability is and how it is different from the promises made by traditional cloud APM and observability vendors • How to use cloud-native observability to do more “Dev” and less “Ops” so you can dramatically improve developer and engineer workflows and productivity • How to make on-call shifts less stressful so that your engineers aren’t getting burned out

Shift left ObservabilityEric D. Schabell

Whether you’re an enterprise migrating to cloud native or born in the cloud, most of today’s APM and Observability tools don’t support how your engineers and DevOps teams need to develop, deploy, and support their software. Observability needs to shift left and reflect the modern way companies organize their development teams and their vital interdependencies. Chronosphere is the only vendor addressing the unique requirements for observability in a cloud native world. Join this webinar to learn: - What cloud native observability is and how it is different from the promises made by traditional cloud APM and observability vendors - How to use cloud native observability to do more “Dev” and less “Ops” so you can dramatically improve developer and engineer workflows and productivity - How to make on-call shifts less stressful so your engineers aren’t getting burned out

Trajectory 2022 - Shifting Cloud Native Observability to the LeftEric D. Schabell

The emergence of cloud native environments has changed our platform world. Not only are you coding, but expected to deliver, maintain, and support all aspects of the platform. This change means the Ops in DevOps has been overshadowing why we got into development in the first place. There is hope for us yet if we can just manage to shift left to close the gap in our time spent on non-operational activities. How do we do that, you ask? It’s imperative that developers are given not only the organizational structure to succeed, and even more critical is the ability to reduce time on triaging failures in their deployments. Attendees will be provided with insights into DevOps cloud native tips and tricks to spending more time on projects that matter and less on call. Join us in learning how you can shift left with your DevOps activities to bring more joy to your days.

KCD Porto: Choose Your Own Adventure - Cloud Naive Observability PitfallsEric D. Schabell

Data Tribology: Overcoming Data Friction with Cloud AutomationIan Foster

Will the Cloud be your disaster, or will Cloud be your disaster recovery?Livingstone Advisory

Making real sense of enterprise Cloud computing in the context of your business is not always a trivial task. The volume, diversity and intensity of opinions on what cloud can do for your organization are relentless, as are the pressures to lower IT costs, speed up implementations, simplify enterprise IT and deliver more value in your own organizations. Shifting your mission critical systems to the cloud presents a formidable range of challenges for many organizations, least of which the potential loss of control over your disaster recovery capability. Conversely, keeping your enterprise IT systems where you can see them, and using the cloud to manage your backups and disaster recovery may appear to run counter to the prevailing perception that the cloud is the ultimate destination for all IT systems. In this presentation, Rob Livingstone will be covering off some of the key considerations of disaster recovery planning in the hybrid cloud environment and how, paradoxically, cloud could either be the cause of your disaster or has the potential to save you from one. He will be offering practical insights and tips on how you should approach the cloud when it comes to planning for the worst so that you come out looking your best.

Checking the pulse of your cloud native architectureEric D. Schabell

The daily choices you make as an engineer when shipping code contributes to the feedback loop. In cloud native environments a surprising amount of data is generated from the application layer down to infrastructure and along the delivery path. Regulatory and compliance pressures force us to store audit and observability data. Understanding the pressures on our engineering teams around the collection, storage, and maintenance of your cloud data can mean the difference between successful teams and burnout. Let us take you on a journey, looking closely at the current state of observability based on a recent research conducted with 500 cloud native engineers and find out what it’s like to be in the trenches.

Red Hat Summit 2018 - 3 pitfalls everyone should avoid with hybrid multicloudEric D. Schabell

The daily hype is all around you. From cloud, hybrid cloud, to hybrid multicloud, you’re told this is the way to ensure a digital future for your business. These choices you’ve got to make don’t preclude the daily work of enhancing your customer's experience and agile delivery of those applications. Let us take you on a journey, looking closely at what hybrid multicloud means for your business, the decisions being made about delivering applications, and dealing with legacy applications, likely the most important resources to your business. Join us for an hour of power, where real customer experiences are used to highlight the three top lessons learned as they transitioned into hybrid multicloud environments. https://docs.google.com/presentation/d/1_MFHY3YJyvz-qrJfOZKDJ_dwTLhd2WxpQGFWxd34r5s

“Real Time Machine Learning Architecture and Sentiment Analysis Applied to Fi...Quantopian

Presented at QuantCon Singapore 2016, Quantopian's quantitative finance and algorithmic trading conference, November 11th. The vast proliferation of data related to the financial industry introduces both new opportunities and challenges to quantitative investors. These challenges are often due to the nature of big data and include: volume, variety, and velocity. In this talk, Dr. Cheng will take the audience on a tour of the “big-data production line” in InfoTrie and show how the financial news collected from various and customizable sources are transformed into quantitative signals in a real-time manner. The talk will touch on various kind of topics like sentiment analysis, entity detection, topic classification, and big-data tools.

Choose Your Own Adventure - Cloud Native Observability PitfallsEric D. Schabell

Industry4.0 IoT Vincent Thavonekham - Azure Day UkraineFactoVia

Vincent Thavonekham discusses how Azure Cloud fits with OPC UA, IoT Edge, and data science technologies. He presents two real-world cases where OPC UA and IoT Edge were used: 1) an IoT and energy optimization case using sensors and an AI optimizer, and 2) a manufacturing Industry 4.0 case using sensors, analytics, and insights. He then explains how standards like OPC UA are important for connecting diverse industrial devices and systems in an interoperable way.

Observability For You and Me with OpenTelemetryEric D. Schabell

Are you interested in dipping your toes in the cloud native observability waters, but as an engineer you are not sure where to get started with tracing problems through your microservices and application landscapes on Kubernetes? Then this is the session for you, where we take you on your first steps in an active open-source project that offers a buffet of languages, challenges, and opportunities for getting started with telemetry data. The project is called openTelemetry, but before diving into the specifics, we’ll start with de-mystifying key concepts and terms such as observability, telemetry, instrumentation, cardinality, percentile to lay a foundation. After understanding the nuts and bolts of observability and distributed traces, we’ll explore the openTelemetry community; its Special Interest Groups (SIGs), repositories, and how to become not only an end-user, but possibly a contributor.We will wrap up with an overview of the components in this project, such as the Collector, the OpenTelemetry protocol (OTLP), its APIs, and its SDKs. Attendees will leave with an understanding of key observability concepts, become grounded in distributed tracing terminology, be aware of the components of openTelemetry, and know how to take their first steps to an open-source contribution! Key Takeaways: Open source, vendor neutral instrumentation is an exciting new reality as the industry standardizes on openTelemetry for observability. OpenTelemetry is on a mission to enable effective observability by making high-quality, portable telemetry ubiquitous. The world of observability and monitoring today has a steep learning curve and in order to achieve ubiquity, the project would benefit from growing our contributor community.

How to Wrestle Your Observability Data Demons and Win!Eric D. Schabell

The document discusses the challenges of managing observability data growth in cloud native environments. It notes that observability data is increasing faster than infrastructure and 71% of companies are concerned about the rate of growth. It then discusses Chronosphere's solution, which allows users to understand data growth, control what data is kept and at what resolution, reducing data volumes by an average of 48%. Case studies show how Chronosphere helped Abnormal Security and DoorDash improve reliability, performance, and reduce costs of their observability systems while managing rapid data growth.

VMblog - 2018 Containers Predictions from 16 Industry Expertsvmblog

Find out what's going on in the world of #containers in 2018. Read #predictions from 16 of the industry's leading experts to learn more about Docker, Kubernetes, Microservices and more! Hear from industry thought leaders from companies like Datadog, Hedvig, ManageEngine, Mirantis, Red Hat, SUSE and more. Make sure to also read the more than 280+ other expert predictions from technologies across virtualization, cloud computing, hyperconverged, IoT, security, etc. here: http://bit.ly/2DQi2OT

Situation Normal, FOWA DublinSimon Wardley

The document discusses the evolution of cloud computing from its early conceptualization to its current form. It explores how cloud computing has progressed from an undefined concept to widespread ubiquity due to increasing demand and continuous improvements by suppliers. Key factors that have driven this transition include the commoditization of infrastructure, the delivery of software and platforms as standardized services, and a shift towards viewing these resources as utilities rather than custom-built products.

Mary Barnsdale article about Fog Computing for CiscoMary Barnsdale

Fog computing is a new networking paradigm that extends cloud computing to the edge of the network, enabling data processing to occur closer to sensors and devices. It involves using small computing devices with storage and networking capabilities at the edge of the network, rather than sending all data to the cloud. This allows for real-time response and processing of data from the billions of devices that will be connected as part of the Internet of Everything. Cisco researchers are developing the architecture for fog computing to help support the growth of smart grids, smart cities, and other applications requiring low latency processing of data from devices.

Cloud Computing - Fundamentals and ApplicationsEmiliano Reynares

1) Cloud computing allows on-demand access to configurable computing resources like servers and networks that can be provisioned with minimal management effort. However, some see it as proprietary systems that increase costs over time. 2) Cloud computing provides scalable resources over the internet as a service, but concerns include reliability, availability, security, and lack of customization. 3) Fog computing extends cloud computing to the edge of networks by performing storage and processing near data sources like IoT devices to enable low latency applications. It provides scalable services across geographically distributed devices.

SRECon EU 2023 - Three Phases to Better Observability OutcomesEric D. Schabell

We all want to have better business outcomes for our organizations solutions, such as faster remediation of problems, easier problem detection, greater revenue generation, happier customers, and engineering teams that can remain focused on delivering more business value. The problem with the popular three pillars (metrics, logs, tracing) is that you are talking about technology aspects and not about solutions. It's like talking about the tools in a mechanics toolbox used to make your convertible run again, instead of focusing on the blue smoke coming out of the exhaust, the rising engine temperature, and using that data to quickly remediate the problem by replacing the seals to prevent oil leaking in the engine. Let’s quickly tour the phases that lead to better outcomes and get our focus back on effective observability goals. Key takeaways - Modern cloud native observability needs three guiding phases to provide better outcomes, not tooling. Based on article: https://www.schabell.org/2022/09/o11y-guide-cloud-native-observability-needs-phases.html

MundiOCRE | Open Clouds for Research Environments

OCRE Workshop: Shaping the Earth Observation Services Market for Research. Session 3: Presentations from DIAS and eoMALL. This workshop aims to bring the EO service providers closer to the research community, capture their needs and develop fit for purpose EO services. The event will be the 4th OCRE Requirements Gathering Workshop. Researchers and Earth Observation Service Providers will be asked to provide inputs to help us shape OCRE's tender. The OCRE project aims to provide the first end-to-end instance of organised, large-scale market pull for EO services in Europe. These services will be provided for free to EU researchers through the European Open Science Cloud. To ensure that the services meet the actual needs of the research community we invite both the demand and the supply side, to share their views and engage in a productive dialogue. Our aim is to capture the needs of EU researchers and inform the EO service providers so that they make available services that effectively address them. We will also explain how the OCRE process will work, how the different stakeholders should be involved and how to make the most of the foreseen benefits.

Optimizing Observability Spend: MetricsEric D. Schabell

Are you collecting just about every metric under the sun and the kitchen sink too? Understanding the cost of collecting metrics and the usefulness of those metrics is the only way to scale in a cloud native world. You can’t get away with just collecting everything as you grow. Your observability teams need to make decisions about what to collect, what to drop, what to aggregate, and still be able to alert, triage, remediate, and do their root cause analysis on a daily basis. Gain immediate insights into high cost data (DPPS), when to drop time series data, and how to determine when the value of that data is at its lowest. Session includes a recorded demo video of it in action.

Cloud costs: my 2 centsRightScale

This document discusses cloud computing costs and how they are changing IT spending. It notes that global IT spending in 2012 was $3.5 trillion. It then provides background on the author and the development of PlanForCloud. It explains what a cloud is from an infrastructure perspective. It discusses how cloud providers have massive data centers containing thousands of servers. It outlines the various models of cloud computing like Infrastructure as a Service. It explains that PlanForCloud helps users estimate costs and compare deployment options and cloud providers. It provides examples of how companies have rapidly scaled infrastructure on public clouds. It discusses the various pricing models from different cloud providers and how prices can change over time. It emphasizes that while cost is important, other factors like security,

Research topics for EON Realty's Research Grant Program (RGP) v16Senthilkumar R

The document proposes several research topics for EON Reality's grant program that are aligned with their interests in augmented, virtual and merged reality technology. Topic areas include integrating AR/VR environments with IoT sensor data through digital twins; using AR for safety training and alerts; automating annotation and knowledge portal generation; studying large-scale AR deployment in education; crowdsourcing data to build a knowledge metaverse; methods for cultural heritage digitization; and enhancing tourism experiences with AR. Each topic provides example use cases and potential applications.

Conquering Disaster Recovery Challenges and Out-of-Control Data with the Hybr...actualtechmedia

Technologies to Build Hybrid Clouds on Public-Private InfrastructuresHelix Nebula The Science Cloud

This document discusses building hybrid clouds that connect public research organizations and commercial cloud services. It proposes using a network to link these resources to provide computing power for research communities in a cost-effective way while ensuring trust, control, and long-term preservation of data. The European Open Science Cloud aims to establish such a multi-tenant, multi-provider cloud infrastructure through public-private partnerships and by addressing challenges like interoperability and seamless elasticity across clouds.

Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...Eric D. Schabell

It's time you stopped letting your telemetry data pressure your budgets and get in the way of solving issues with agility! No more I say! Take back control of your telemetry data as we guide you through the open source project Fluent Bit. Learn how to manage your telemetry data from source to destination using the pipeline phases covering collection, parsing, aggregation, transformation, and forwarding from any source to any destination. Buckle up for a fun ride as you learn by exploring how telemetry pipelines work, how to set up your first pipeline, and exploring several common use cases that Fluent Bit helps solve. All this backed by a self-paced, hands-on workshop that attendees can pursue at home after this session (https://o11y-workshops.gitlab.io/workshop-fluentbit).

Observability-as-a-Service: When Platform Engineers meet SREsEric D. Schabell

Monitoring the behavior of a system is essential to ensuring its long-term effectiveness. However, managing an end-to-end observability stack can feel like stepping into quicksand, without a clear plan you’re risking sinking deeper into system complexities. In this talk, we’ll explore how combining two worlds—developer platforms and observability—can help tackle the feeling of being off the beaten cloud native path. We’ll discuss how to build paved paths, ensuring that adopting new developer tooling feels as seamless as possible. Further, we’ll show how to avoid getting lost in the sea of telemetry data generated by our systems. Implementing the right strategies and centralizing data on a platform ensures both developers and SREs stay on top of things. Practical examples are used to map out creating your very own Internal Developer Platform (IDP) with observability integrated from day 1.

More Related Content

Similar to 3 Pitfalls Everyone Should Avoid with Cloud Data (20)