Big Data on EC2: Mashing Technology in the Cloud

Nov 19, 20091 like656 views

This document discusses how a startup serving widgets on popular online publications scaled their infrastructure using Amazon Web Services to handle spikes in traffic from over 1 billion users sharing over 10 billion URLs. They used a hub-and-spoke architecture with components like Cascading, Amazon Elastic MapReduce, and AsterData to analyze user sharing patterns in a cost-effective and horizontally scalable way.

Big Data on EC2: Mashing Technology in the Cloud

ShareThis

Paco Nathan, Data Insights Team

ScaleCamp 2009-06-09

Scenario…

✤ Given: >10^6 publishers, >10^9 users, >10^10 urls

✤ Early-stage start-up, < 25 people, seeking to minimize spend on Ops
and capex for data centers

✤ Serving widgets on page views for popular online publications: ESPN,
HuffPost, FOX, CS Monitor, CBS Marketwatch, Wired, TechCrunch, etc.

✤ Spikes in popularity of stories leads to elastic demands throughout
the system architecture: serving API, logging, DW, BI, etc.

✤ Business needs to improve user experience by analyzing how people
share online content

ScaleCamp 2009-06-09

System Architecture

✤ 100% infrastructure based on Amazon AWS

✤ Each component designed for cost-effective, horizontal scale-out

✤ AsterData: infrastructure based on a “hub-and-spoke” pattern of batch
jobs and data consolidation

✤ Cascading: abstraction layer for tying together system components

✤ Batch jobs run on Amazon Elastic MapReduce and AsterData SQL/MR

✤ Vertical search based on Bixo and Katta

ScaleCamp 2009-06-09

Cascading

✤ Syntax is for humans, APIs are for software

✤ Cascading defines apps in terms of functions applied to data flows,
incorporates end-points; whereas MR is coarse, key/values are brittle

✤ Direct benefits to team’s responsibilities, process, deliverables

✤ Also consider impact on team size, composition, staffing, training

✤ Ideal for provisioning/monitoring elastic resources, such as EMR

✤ Great potential to allow for migrating apps

✤ (see also: Cascading talk @ Hadoop Summit tomorrow, 6pm)

ScaleCamp 2009-06-09

Amazon Elastic MapReduce

✤ Can scale-out wide and select different instance types at launch

✤ Reads input from S3, writes output to S3, optionally copies logs to S3

✤ Excellent command line tools make dev/test/debug cycle more efficient

✤ Risks for DIY Hadoop clusters: time+cost to acquire leases, data locality,
etc., — EMR resolves these to make large apps more cost-effective

✤ Simplifies needs for Ops — which can be quite difficult and expensive
to staff for Big Data apps, and pose a risk to scale-out strategy

✤ See the Cascading examples: LogAnalyzer for CloudFront and Multitool

ScaleCamp 2009-06-09

AsterData nCluster Cloud Edition

✤ Scalable, fault-tolerant, relational database — directly in the cloud

✤ Takes only minutes to go from idea to a prototype which scales well;
accessible by data scientists who have less coding background

✤ Use of SQL unions on map phase, plus other SQL primitives for shuffle
and reduce phases, allows for complex MR workflows

✤ Leveraging SQL primitives during shuffle and reduce phases allows for
automated optimizations

✤ Contributed code among developer community, implementing libraries
for popular algorithms based on In-Database MapReduce

ScaleCamp 2009-06-09

ShareThis as Case Study

See also: ShareThis case study, "Cascading"
by Chris K Wensel, in…

ScaleCamp 2009-06-09

Contacts:
http://sharethis.com
@pacoid on Twitter
http://cascading.org
http://asterdata.com/product/ncluster_cloud.php
http://aws.amazon.com/elasticmapreduce
http://github.com/emi/bixo
http://katta.sourceforge.net
http://www.hadoopbook.com

ScaleCamp 2009-06-09

This document summarizes MemSQL's geospatial database capabilities and features. It discusses how always-on connectivity and ubiquitous mobile devices have enabled new transportation-based geospatial applications using real-time taxi and rideshare data. It provides examples using New York taxi data, including demonstrations of MemSQL's Supercar and Zoomdata's TaxiStats applications that perform real-time geospatial analytics on streaming transportation data.

IronSource Atom - Redshift - Lessons LearnedIdan Tohami

Building big data applications on AWS by Ran TesslerIdan Tohami

Ran Tessler discusses building big data applications on AWS. He outlines architectural principles like using a decoupled data bus to store, process and retrieve data and answers. He demonstrates using AWS services like Kinesis Firehose for data collection, EMR with Spark and Hive for processing, and Redshift and QuickSight for analysis and visualization. Tessler references the Lambda architecture and its batch, speed and serving layers that can be implemented using AWS services like S3, DynamoDB, ElastiCache and Lambda.

Tech Thursday - Beer & DevOps 24.11.Nebula Oy

This document outlines an agenda for a DevOps event on auto-scaling. The agenda includes welcome remarks, presentations on auto-scaling use cases and a Kontena demo. There is also a scheduled auto-scale demo showing how resource groups can be provisioned and scaled up/down automatically based on metrics and policies in OpenStack Heat. The document argues that auto-scaling can be simplified using standardized metrics and workloads that scale out horizontally rather than vertically. It notes auto-scaling can be implemented in a few hours and at very low cost using services like Datadog to monitor metrics and trigger scaling policies.

AWS_ac_ra_loganalysis_11Kashif .T Ibrahim

This document describes an architecture for analyzing large-scale web logs generated by applications hosted on Amazon Web Services. The architecture utilizes Amazon Elastic MapReduce to process log files stored in Amazon S3 using Hadoop. Results are stored in an Amazon RDS database. EC2 spot instances provide additional flexible capacity for log analysis jobs. Amazon CloudFront also generates logs that can be analyzed in this system.

Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016Coburn Watson

The document summarizes Netflix's cloud capacity planning operations. It notes that Netflix tracks over 2,400 applications running on 80 instance types deployed across 30 zones and 60 accounts in AWS. To provide transparency and actionable insights at scale, Netflix built interactive dashboards and focuses on deviations and trends. Its tool called Picsou comprehensively monitors Netflix's current cloud footprint, forecasts future capacity needs, and optimizes on-demand and reserved capacity for cost and reliability. Going forward, Picsou will provide capacity alerts, forecasts, and recommendations to optimize reserved capacity usage.

Stream Processing with Kafka in Uber, Danny Yuan confluent

- The document discusses Uber's use of stream processing to enable real-time analytics and complex event processing over streaming data from its global ridesharing marketplace. - Key applications include real-time OLAP, detecting patterns in event streams, and supply positioning to monitor marketplace health. - The architecture uses Apache Kafka for event collection, Apache Samza for event processing, and storage and visualization applications. It addresses challenges of processing large-scale, real-time geo-temporal data streams.

Building a real-time, scalable and intelligent programmatic ad buying platformJampp

PowerStream: Propelling Energy Innovation with Predictive Analytics SingleStore

This document discusses a presentation about MemSQL PowerStream, a product for predicting the global health of wind turbines. The presentation covers renewable energy news stories, introduces PowerStream, demonstrates high-speed data ingestion and predictive analytics using MemSQL and Spark, and shows how SQL queries can be pushed down to MemSQL for faster processing. It concludes with a question and answer section.

High availability, real-time and scalable architecturesJampp

Modeling the Smart and Connected City of the Future with Kafka and SparkSingleStore

- Modeling the Smart and Connected City of the Future with Kafka and Spark discusses using Kafka, Spark, and MemSQL to build a real-time data pipeline for a hypothetical "MemCity" that captures data from 1.4 million households. - The document outlines the components of the "Real-Time Trinity" - Kafka for a high-throughput message queue, Spark for data transformation, and MemSQL for real-time data serving and analytics. - It also introduces MemSQL Streamliner, which is designed to simplify the creation of real-time data pipelines through a graphical interface and one-click deployment of integrated Apache Spark clusters.

Processing 19 billion messages in real time and NOT dying in the processJampp

Razorfish - Amazon EMR usecaseguestda111d9

Big problems Big Data, simple solutionsClaudio Pontili

Multi objective tasks scheduling algorithm for cloud computing throughput opt...Shakas Technologies

The document proposes a multi-objective task scheduling algorithm to improve throughput in cloud computing datacenters. The algorithm aims to optimally schedule tasks to virtual machines based on multiple criteria like execution time, cost, and bandwidth in order to better utilize resources and reduce costs without violating service level agreements. The algorithm was simulated using CloudSim and results demonstrated improved performance and higher throughput compared to methods that schedule based on single criteria.

Change Data Capture - Scale by the Bay 2019Petr Zapletal

The document discusses change data capture (CDC) in distributed systems. CDC allows capturing changes to data and notifying actors so they can react accordingly. Some considerations for CDC include reliability, scalability, performance, consistency, and fault tolerance. Common strategies involve using timestamps, triggers, diffs, or log scraping. CDC has various use cases such as caching, search indexing, offline processing, and simplifying monolithic applications. Popular frameworks for implementing CDC include pg2k4j, Maxwell, AWS Database Migration Service, SpinalTap, DataBus, and Debezium.

Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Coburn Watson

Netflix carefully plans holiday capacity to ensure availability while maintaining efficiency. A cross-functional team works with engineering teams to understand capacity needs through bottoms-up and tops-down modeling. They prepare for spikes by coordinating large service migrations to optimize "trough borrowing" and manage failover capacity. The goal is to fulfill their charter of ensuring efficient, available cloud capacity so engineering can prioritize innovation and availability.

Howtomakeyourown gi sdashboardGeoMedeelel

1. The document describes how to create a COVID-19 dashboard using SuperMap software. It provides steps to set up SuperMap iServer, iServer Datastore, and iPortal. 2. It explains how to prepare coronavirus and base map data, then visualize the data on a map using DataViz. Components like charts, indicators can then be added to the map to create a dashboard using Map Dashboard. 3. The COVID-19 dashboard can then be published for public access on web and mobile browsers.

0 supermapproductsintroductionGeoMedeelel

SuperMap GIS 10i is a cloud native, cross-platform GIS technology system that includes products for SDK, servers, clients, and desktop use. The document introduces the SuperMap GIS 10i technology system and product family, and provides details about the SuperMap iDesktop Java and .NET desktop GIS platforms, describing their abilities to produce and manage spatial data, represent spatial information, perform spatial analysis and data mining, and enable 2D/3D GIS and custom application development.

Practical Experiences With ArcGIS ServerWisconsin Land Information Association

02 supermapiclientforjavascriptintroductionGeoMedeelel

SuperMap Software Co., Ltd. introduces their JavaScript client library called SuperMap iClient for JavaScript. The library provides client-side GIS application development capabilities and acts as a unified JavaScript client for consuming SuperMap's cloud services and online maps. The document compares several popular open-source JavaScript mapping libraries such as Leaflet, OpenLayers, and MapboxGL that can be used with SuperMap iClient for visualization and mapping. Examples are provided showing how different mapping and visualization libraries can be combined.

01 supermapiportaloverviewGeoMedeelel

SuperMap iPortal is a GIS resource portal platform that is responsible for publishing GIS services from SuperMap iServer. It allows users to integrate, discover, share, and manage various GIS resources. Key features include resource integration from heterogeneous GIS servers, discovery of resources through fuzzy search and filtering, sharing of resources at different access levels, and multi-level management of resources based on organizational structure and permissions. The portal also supports customization and creation of web applications and data apps.

Autonomous analytics on streaming dataClaudiu Barbura

This document describes an autonomous analytics platform that allows users to analyze streaming data. The platform uses a unified big data technology stack including Spark, Cassandra, Hadoop, Kafka and Elasticsearch. It has a cloud-agnostic architecture and supports multiple machine learning frameworks. The platform includes a Domain Specific Language (DSL) that allows power users to create full data pipelines and analytics workflows with a few lines of code. It also includes a DSL Workbench for interactively building, editing and publishing analytical pipelines. Additionally, the document introduces "Auto Curious", which harnesses user interactions to autonomously discover insights and compose DSL commands through a question graph interface.

01 supermapiserverintroductionGeoMedeelel

SuperMap iServer is a GIS server software that provides various 2D/3D GIS services through a service-oriented architecture. It offers GIS, map, spatial analysis, network analysis, and other services. SuperMap iServer can be deployed on Windows, Linux, and other platforms. It includes management capabilities and supports development of customized GIS services and diverse client applications including web, mobile, and desktop.

Scalable crawling with Kafka, scrapy and spark - November 2021Max Lapan

This document describes the architecture of a scalable crawling system used by Scoutbee to crawl thousands of company websites weekly. It uses Kafka to store and transfer crawled data for high throughput and low latency. Scrapy is used for distributed crawling scaled at the domain level. Spark is used for data processing pipelines to decrease data latency. Crawled data is stored long-term in S3 for efficient retrieval and reuse in machine learning pipelines. The system fulfills the contradictory requirements of fast, inexpensive crawling at scale.

Cloud computing's truly open silver lining: OpenStackAsociatia ProLinux

1) OpenStack is an open-source cloud computing platform that provides on-demand access to computing power and storage and is capable of scaling as needed. 2) It includes OpenStack Compute (Nova) for automatically creating and managing large groups of virtual private servers and OpenStack Object Storage (Swift) for creating redundant, scalable object storage. 3) OpenStack is hypervisor agnostic and works with technologies like Xen, KVM, VMware ESXi, and Hyper-V.

Spark Summit West 2017: Real-Time Image Recognition with MemSQL and SparkSingleStore

Ahead of the curve with EcommerceRahul Singh

This webinar presentation covered how eCommerce is being disrupted by changing consumer expectations and the mobile revolution. It discussed how Magento is well-positioned for businesses of any size to keep up with these changes by providing a customizable, scalable platform. It also included a case study of a manufacturing company that migrated from an inflexible vendor solution to Magento Enterprise Edition in order to gain control of their platform and meet the needs of their growing business into the future. The presentation concluded with contact information for OSSCube if attendees had any other questions.

Social Media for Atlantic Canada MarketingMediaBadger

Social media for brands.pdfAnjanette Delgado

This document discusses how brands can use social media for marketing and customer relationship management. It notes that social media allows for two-way interaction between brands and customers. The document then provides statistics on the growth and usage of various social media platforms like Twitter, YouTube, Facebook, LinkedIn and MySpace. It discusses how social media platforms continuously evolve based on user demands. The document advocates that brands can benefit from social media by empowering customers to spread positive experiences about brands to large networks, which can significantly help sales and brand perception.

More Related Content

What's hot (19)

PowerStream: Propelling Energy Innovation with Predictive Analytics SingleStore

High availability, real-time and scalable architecturesJampp

Modeling the Smart and Connected City of the Future with Kafka and SparkSingleStore

Processing 19 billion messages in real time and NOT dying in the processJampp

Razorfish - Amazon EMR usecaseguestda111d9

Big problems Big Data, simple solutionsClaudio Pontili

Multi objective tasks scheduling algorithm for cloud computing throughput opt...Shakas Technologies

Change Data Capture - Scale by the Bay 2019Petr Zapletal

Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Coburn Watson

Howtomakeyourown gi sdashboardGeoMedeelel

0 supermapproductsintroductionGeoMedeelel

Practical Experiences With ArcGIS ServerWisconsin Land Information Association

02 supermapiclientforjavascriptintroductionGeoMedeelel

01 supermapiportaloverviewGeoMedeelel

Autonomous analytics on streaming dataClaudiu Barbura

01 supermapiserverintroductionGeoMedeelel

Scalable crawling with Kafka, scrapy and spark - November 2021Max Lapan

Cloud computing's truly open silver lining: OpenStackAsociatia ProLinux

Spark Summit West 2017: Real-Time Image Recognition with MemSQL and SparkSingleStore

PowerStream: Propelling Energy Innovation with Predictive Analytics SingleStore

High availability, real-time and scalable architecturesJampp

Modeling the Smart and Connected City of the Future with Kafka and SparkSingleStore

Processing 19 billion messages in real time and NOT dying in the processJampp

Razorfish - Amazon EMR usecaseguestda111d9

Big problems Big Data, simple solutionsClaudio Pontili

Multi objective tasks scheduling algorithm for cloud computing throughput opt...Shakas Technologies

Change Data Capture - Scale by the Bay 2019Petr Zapletal

Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Coburn Watson

Howtomakeyourown gi sdashboardGeoMedeelel

0 supermapproductsintroductionGeoMedeelel

Practical Experiences With ArcGIS ServerWisconsin Land Information Association

02 supermapiclientforjavascriptintroductionGeoMedeelel

01 supermapiportaloverviewGeoMedeelel

Autonomous analytics on streaming dataClaudiu Barbura

01 supermapiserverintroductionGeoMedeelel

Scalable crawling with Kafka, scrapy and spark - November 2021Max Lapan

Cloud computing's truly open silver lining: OpenStackAsociatia ProLinux

Spark Summit West 2017: Real-Time Image Recognition with MemSQL and SparkSingleStore

Viewers also liked (20)

Ahead of the curve with EcommerceRahul Singh

Social Media for Atlantic Canada MarketingMediaBadger

Social media for brands.pdfAnjanette Delgado

Canadaalbinonunez

Women and Social Media in CanadaAyelet Baron

Women are highly engaged with social media globally and in Canada. They spend more time online than men and are more likely to visit social networking sites. In North America, 91% of women use social media compared to 87.5% of men. Social media allows for conversations and relationship building across different groups. It has become an important way for sharing information and recommendations. While social media gives women a voice, their influence depends on being strategic with using collective online communities.

How Canadians Shopguest970d5

1) The aging Canadian population and changing household structures are challenging for retailers as shopping behaviors change. 2) Quick trips make up over half of all grocery shopping trips but account for a smaller portion of total spending, while general stock-up trips are less frequent but represent more total spending. 3) Shopping responsibilities are often shared or fall to women, with the presence of children impacting trip types and missions from quick replenishments to larger stock-up trips.

CanadaeconomicsNorth Gwinnett Middle School

Canada has a free enterprise economy where individuals and groups control businesses rather than the government. It is one of the world's top food producers due to its fertile soil, waterways, climate, and equipment. Canada has a large industrial economy based on its natural resources like oil, coal, gas, and minerals. The government plays a more direct role in the economy by providing universal healthcare and regulating industries. Today, the economy is focused more on services than manufacturing. Industries, jobs, and people are located near coasts, the US border, and areas with oil and coal deposits.

Maglev trainsSelf-employed

Maglev trains use electromagnetic forces to both levitate above the track and propel the train forward, eliminating friction and allowing for high speeds. The trains are powered by batteries and linear generators, making them more energy efficient and environmentally friendly than traditional trains. While offering a smooth, safe ride, the high construction costs of maglev systems have caused some governments to hesitate in adopting the technology on a large scale.

Wind power forecasting accuracy and uncertainty in FinlandVTT Technical Research Centre of Finland Ltd

This document discusses wind power forecasting accuracy and uncertainty in Finland. It analyzes forecast errors and imbalance costs based on real data from over 20 wind power sites in Finland from 2010-2012. The key findings are: 1) Aggregating production from multiple sites can significantly reduce forecast errors and imbalance costs, lowering costs by up to 60%. 2) Combining forecasts from different weather models, such as averaging them, can improve accuracy by up to 6%. 3) Quantifying the uncertainty of forecasts through probability intervals is important as wind power penetration increases. Aggregation also decreases the variation and uncertainty of forecasts.

Kristof Coussement - The Debate: the Future of (Big) Data Analytics SoftwareBAQMaR

The document discusses trends in analytical software such as big data, data science, and the growing popularity and use of the R programming language. It provides an overview of Revolution Analytics as the leading commercial provider of software and services based on the open source R language. The presentation discusses challenges in big data analysis and how tools like R and Revolution R can help address these challenges by enabling faster analytics on larger and more varied data sources.

Animatronics PortfolioBrian Jaecker-Jones

Brian Jaecker-Jones has extensive experience designing and building animatronic creatures and puppets for films, TV, and live performances. Some of his past works include a chipmunk and lizard with moving jaws and eyes, robotic aliens and wolves with many axes of motion, and bats with flapping wings. He is skilled at designing complex mechanical systems that achieve realistic movement within tight production timelines.

Histmojurassicpark 090807051516-phpapp01ajeesh n

Animatronics were first developed for Disney studios in 1954 for the film 20,000 Leagues Under the Sea. The first use of audio-animatronics was in Walt Disney's Enchanted Tiki Room at Disneyland, which opened in 1963. Animatronics involve mechanized creatures that make imaginary animals appear alive through movement. The process of creating animatronics involves designing, sculpting, model making, armature fabrication, costuming, programming, and putting everything together to make the creatures move in a lifelike manner.

eMarketer Webinar: Demographics in Canada—Age-based Digital BehaviorsEMARKETER

Stan winston & animatronicsJason Seon-gyu Park

Animatronics is the use of electronics and robotics in mechanised puppets to simulate life. Animatronics are mainly used in movie making, but also in theme parks and other forms of entertainment. Its main advantages to CGI and stop motion is that it is not a simulation of reality, but rather physical objects moving in real time in front of the camera. The technology behind animatronics has become more advanced and sophisticated over the years, making the puppets even more realistic and lifelike.

AnimatronicsPLANB1

Maglev trains pptLokesh Choudhary

The document discusses magnetic levitation (Maglev) trains. It begins by defining Maglev as using magnetic levitation to suspend, guide, and propel trains using magnets. It then explains the basic principles of levitation, propulsion, and lateral guidance that Maglev trains use to operate at high speeds. This includes using magnets to levitate the train 10 cm above the track and linear motors in the guideway to propel the train electromagnetically. The document also discusses the technologies, merits, and demographics of existing and planned Maglev systems around the world.

The Impact of Big Data On Marketing Analytics (UpStream Software)Revolution Analytics

Ultra wideband technology (UWB)Mustafa Khaleel

Ultra Wide-Band Technology (UWB) is a short-range, high-bandwidth communications technology that can be used for data transfer, imaging, and localization applications. UWB operates by transmitting very short pulses across a wide frequency band with low power. Key applications of UWB include high-speed wireless communications and high-resolution radar and imaging systems. Standardization efforts have developed standards for UWB personal area networks, and UWB offers advantages like high data rates and secure transmission, but also faces limitations from its low-power emissions.

Canada country presentation_presentationTeresa Maroto Ramos

Wireless Application Protocol pptgo2project

WAP (Wireless Application Protocol) is a standard for accessing information on mobile devices like phones. It uses WML (Wireless Markup Language) instead of HTML. WAP has a layered architecture including application, session, security, transaction, and transport layers. It allows access to the internet from mobile devices in a standardized way and is supported by most phone manufacturers and wireless networks.

Ahead of the curve with EcommerceRahul Singh

Social Media for Atlantic Canada MarketingMediaBadger

Social media for brands.pdfAnjanette Delgado

Canadaalbinonunez

Women and Social Media in CanadaAyelet Baron

How Canadians Shopguest970d5

CanadaeconomicsNorth Gwinnett Middle School

Maglev trainsSelf-employed

Wind power forecasting accuracy and uncertainty in FinlandVTT Technical Research Centre of Finland Ltd

Kristof Coussement - The Debate: the Future of (Big) Data Analytics SoftwareBAQMaR

Animatronics PortfolioBrian Jaecker-Jones

Histmojurassicpark 090807051516-phpapp01ajeesh n

eMarketer Webinar: Demographics in Canada—Age-based Digital BehaviorsEMARKETER

Stan winston & animatronicsJason Seon-gyu Park

AnimatronicsPLANB1

Maglev trains pptLokesh Choudhary

The Impact of Big Data On Marketing Analytics (UpStream Software)Revolution Analytics

Ultra wideband technology (UWB)Mustafa Khaleel

Canada country presentation_presentationTeresa Maroto Ramos

Wireless Application Protocol pptgo2project

Similar to Big Data on EC2: Mashing Technology in the Cloud (20)

Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...DATAVERSITY

Mainframes continue to perform mission-critical transaction processing and contain massive amounts of core business data. But digital transformation initiatives and cloud computing have created both opportunities and challenges for unlocking and utilizing this data. Qlik and AWS will share some of the proven strategies from successful customer deployments across a range of different mainframe to cloud use cases, including legacy application modernization, data analytics, and data migrations. In this presentation, you will learn how to: • Replicate very large volumes of mainframe data in real-time to the cloud • Automate the creation of analytics-ready data lakes and data warehouses • Achieve a 30% reduction in cost of compute

AWS Start-Up Tour 2009 / ShareThisPaco Nathan

ShareThis is a social content sharing platform that allows users to easily share online content. It works with many major publishers and must scale out its services when popular stories go viral to keep up with demand. As an early stage startup with limited resources, ShareThis uses AWS to horizontally scale its architecture including API services, loggers, data warehouse, and business intelligence tools. This allows the small team to build a high level of service entirely in the cloud and efficiently handle spikes in traffic. Key benefits of AWS include the ability to scale out without rewriting code, using S3 for backups, and enabling developers to test in their own sandbox environments.

Cloud Computing Realities - Getting past the hype and setting your cloud stra...Compuware APM

Companies are increasingly demanding that Web applications "move to the cloud" to reign in IT costs, reduce server sprawl and perhaps most importantly, help to ensure that your infrastructure is tuned to deliver an exceptional end-user experience for your customers. The challenge is to reap those benefits while ensuring top performance, keeping IT operations and development on the same page, and delivering enterprise level capabilities and scalability. Join 3 cloud computing experts Forrester Principal Analyst, James Staten; Savvis’ Chief Technology Officer, Bryan Doerr; and Gomez’s Chief Technology Officer, Imad Mouline as they discuss the cloud landscape, application performance in the cloud and successful cloud adoption strategies. What you will learn: * How to determine which applications are best suited for cloud deployments * A game plan for cloud adoption for the next 90 days and beyond * How to use Platform as a Service (PaaS) and Infrastructure as a Service (IaaS) delivery models to test more efficiently and better leverage internal computing resources * Which techniques can improve your lifecycle management of cloud based applications * Best practices to ensure optimum end-user performance of your cloud environment

Apache Cassandra: NoSQL in the enterprisejbellis

This document discusses Apache Cassandra and how it enables real-time analytics on large datasets. It provides examples of how Netflix, Backupify, Ooyala, and Formspring use Cassandra for its scalability, performance, and flexibility. The document also outlines how DataStax Enterprise unifies real-time and analytic processing to allow complex queries on both live and historical data without the complexity of traditional Hadoop deployments.

ESA and the CloudNetcetera

Cloud Computing ...changes everythingLew Tucker

Lew Tucker discusses the rise of cloud computing and its impact. He defines various cloud service models like SaaS, PaaS, and IaaS. Tucker analogizes the shift to cloud computing from individual data centers generating their own power to today's electrical grid. Major drivers of cloud computing include the growth of web APIs and massive amounts of user-generated data. Tucker outlines how cloud computing changes what developers can access and how applications are designed and scaled.

Leveraging Mainframe Data for Modern Analyticsconfluent

The document provides an overview of leveraging mainframe data for modern analytics using Attunity Replicate and Confluent streaming platform powered by Apache Kafka. It discusses the history of mainframes and data migration, how Attunity enables real-time data migration from mainframes, the Confluent streaming platform for building applications using data streams, and how Attunity and Confluent can be combined to modernize analytics using mainframe data streams. Use cases discussed include query offloading and cross-system customer data integration.

Zeller Edm Summit Agile Deployment Of Predictive AnalyticsRonald.Ramos

The document discusses leveraging predictive analytics and cloud computing. It outlines using the R programming language for predictive model development, PMML for model integration and deployment, and Amazon EC2 for scalable predictive model execution in the cloud. The overall process involves developing predictive models with R, exporting them to PMML, deploying them on EC2 using a platform like ADAPA for real-time predictive analytics in a scalable, standards-based way.

Cloud Computingwebscale

The document provides a comparison of Amazon AWS, Google App Engine, and Sun Project Caroline cloud computing platforms. It discusses their offerings such as hardware as a service, platform as a service, and software as a service. Amazon AWS provides extensive infrastructure services while Google App Engine focuses on its APIs and big data capabilities. Project Caroline is a research project aiming to provide programmatic control of distributed resources on a large shared grid.

Giga Spaces Alternative To GAE_JavaOne 09Amnon Raviv

The document discusses an alternative PaaS solution to Google App Engine for Java applications called GigaSpaces. It summarizes the key differences between AWS and GAE, challenges with enterprise applications on the cloud, and how GigaSpaces addresses these challenges by providing a PaaS platform on top of AWS. This provides more flexibility than GAE while still offering ease of use. A case study is presented of a risk analytics company that used GigaSpaces to build a scalable platform on AWS for running Monte Carlo simulations.

AWS Summit 2018 SummaryAshish Mrig

The document discusses several new features of AWS services including AWS Athena, AWS Glue, AWS Auto Scaling, machine learning services, and data services. Some key points include: - AWS Athena released new JDBC/ODBC drivers that are faster and support Active Directory access control. It also introduced Athena Work Groups for access control and cost management. - AWS Glue now supports Hive, Spark, and Presto. It includes features like crawlers for automatic data discovery and job bookmarks. - AWS Auto Scaling added predictive scaling using machine learning to forecast resource needs. - AWS SageMaker is a full managed service for building, training, and deploying machine learning models using various frameworks.

Cloud Architecture - Multi Cloud, Edge, On-PremiseAraf Karsh Hamid

Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Am...Yahoo Developer Network

1) Amazon Elastic MapReduce enables customers to easily process vast amounts of data by launching Hadoop clusters across AWS infrastructure. 2) It provides features for managing, monitoring, and debugging Hadoop jobs and clusters without the operational complexities of Hadoop. 3) New features were announced that provide more flexibility for enterprises including expanding and shrinking running clusters, using spot instances to reduce costs, and additional support options.

GigaSpaces - Getting Ready For The Cloudgigaspaces

Mr Nati Shalom, Founder and CTO of GigaSpaces Nati is responsible for defining the technology roadmap and the direction of GigaSpaces products as they relate to standards adaptations, architecture, and product design. He has more than 10 years of experience with distributed technology and architecture namely CORBA, Jini, J2EE, Grid and SOA. He has been working for the past ten years with some of the leading Israeli companies, such as ECI, Comverse, BMC, Elisra, Rafael, and Amdocs. He has led the development of the first Reverse BID exchange in the Israeli Yellow Pages. He previously worked with IONA, and was responsible for the penetration of their products and technology, to most of the leading ISV's in Israel. As the Head of the Israeli Grid consortium, Mr. Shalom is recognized as a software visionary and industry leader, he is a frequent presenter at industry conferences and is actively involved in evangelizing Space Based Architecture, Data Grid patterns, and Cloud Computing. ______________________________________________________________________________________________________________ Topic - Getting Ready for the Cloud – Technology In this session Nati will describe what is the latest developments in the industry on cloud computing, and where he feels this will be going. He will also share his experience on how to design and deploy enterprise applications in a cloud/grid computing platform, what to take into account while developing or deploying applications on the cloud, and demonstrate how to transition applications to run on the Cloud without needing to completely re-architect them. Standard Application Servers as we've known them only partially address enterprises' needs for scalability. As a result, a new class of application servers has emerged, focused on massive scalability. In this session, we will explore some of the common characteristics of these servers while looking at how to migrate an existing Java EE web app to a scale-out application server, relatively seamlessly. Included is a 10-minute demo on turning an existing tier-based application into a tierless scaled out application running on the Amazon EC2 Cloud. In this live demo session, we will also use the cloud-based environment to demonstrate how you can add dynamic scaling, self healing and improved performance with almost no changes to your code.

Giga Spaces Getting Ready For The Cloudchzesin

The document discusses how traditional middleware is not well-suited for cloud computing and the benefits of using a scale-out application platform like GigaSpaces to build cloud-ready applications. It provides examples of companies like BazuMedia and Miwok Airways that are using GigaSpaces to build scalable applications that can easily expand to meet demand spikes and run cost-effectively on cloud infrastructure like Amazon EC2.

Best Practices for Building Open Source Data LayersIBMCompose

Azure Data Explorer deep dive - review 04.2020Riccardo Zamana

Modern Data Science Lifecycle with ADX & Azure This document discusses using Azure Data Explorer (ADX) for data science workflows. ADX is a fully managed analytics service for real-time analysis of streaming data. It allows for ad-hoc querying of data using Kusto Query Language (KQL) and integrates with various Azure data ingestion sources. The document provides an overview of the ADX architecture and compares it to other time series databases. It also covers best practices for ingesting data, visualizing results, and automating workflows using tools like Azure Data Factory.

Successful Cloud Orchestration with RightScale CMPRightScale

As cloud use explodes, it’s important to develop a strategy for orchestrating and automating across all your cloud and non-cloud infrastructure environments. However, the use of the term orchestration has become overloaded and can refer to many different types of tools. Learn about the different types of orchestration automation tools and the role a cloud management platform (CMP) can play as an “orchestrator of orchestrators.”

Confluent_AWS_ImmersionDay_Q42023.pdfAhmed791434

This document outlines an agenda for a webinar on building secure, event-driven microservices with Confluent Cloud on AWS. The agenda includes presentations on building modern streaming analytics with Confluent on AWS, event streaming made easy with Confluent, and a lab on building end-to-end streaming data pipelines with Confluent Cloud. The hosts for the webinar are Ahmed Zamzam from Confluent and Nuno Barreto from AWS.

Build real-time streaming data pipelines to AWS with Confluentconfluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...DATAVERSITY

AWS Start-Up Tour 2009 / ShareThisPaco Nathan

Cloud Computing Realities - Getting past the hype and setting your cloud stra...Compuware APM

Apache Cassandra: NoSQL in the enterprisejbellis

ESA and the CloudNetcetera

Cloud Computing ...changes everythingLew Tucker

Leveraging Mainframe Data for Modern Analyticsconfluent

Zeller Edm Summit Agile Deployment Of Predictive AnalyticsRonald.Ramos

Cloud Computingwebscale

Giga Spaces Alternative To GAE_JavaOne 09Amnon Raviv

AWS Summit 2018 SummaryAshish Mrig

Cloud Architecture - Multi Cloud, Edge, On-PremiseAraf Karsh Hamid

Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Am...Yahoo Developer Network

GigaSpaces - Getting Ready For The Cloudgigaspaces

Giga Spaces Getting Ready For The Cloudchzesin

Best Practices for Building Open Source Data LayersIBMCompose

Azure Data Explorer deep dive - review 04.2020Riccardo Zamana

Successful Cloud Orchestration with RightScale CMPRightScale

Confluent_AWS_ImmersionDay_Q42023.pdfAhmed791434

Build real-time streaming data pipelines to AWS with Confluentconfluent

More from George Ang (20)

Wrapper induction construct wrappers automatically to extract information f...George Ang

Wrapper induction is a technique to automatically generate wrappers to extract information from web sources. It involves learning extraction rules from labeled examples to construct a wrapper as a finite state machine or set of delimiters. Two main wrapper induction systems are WIEN, which defines wrapper classes including LR, and STALKER, which uses a more expressive model with extraction rules and landmarks to handle structure hierarchically. Remaining challenges include selecting informative examples, generating label pages automatically, and developing more expressive models.

Opinion mining and summarizationGeorge Ang

This document summarizes a tutorial given by Bing Liu on opinion mining and summarization. The tutorial covered several key topics in opinion mining including sentiment classification at the document and sentence level, feature-based opinion mining and summarization, comparative sentence extraction, and opinion spam detection. The tutorial provided an overview of the field of opinion mining and abstraction as well as summaries of various approaches to tasks such as sentiment classification using machine learning methods and feature scoring.

Huffman codingGeorge Ang

The document provides an overview of Huffman coding, a lossless data compression algorithm. It begins with a simple example to illustrate the basic idea of assigning shorter codes to more frequent symbols. It then defines key terms like entropy and describes the Huffman coding algorithm, which constructs an optimal prefix code from the frequency of symbols in the data. The document discusses how the algorithm works, its advantages in achieving compression close to the source entropy, and some limitations. It also covers applications of Huffman coding like image compression.

Do not crawl in the dust  different ur ls similar textGeorge Ang

The document describes the DustBuster algorithm for discovering DUST rules - rules that transform one URL into another URL that contains similar content. The algorithm takes as input a list of URLs from a website and finds valid DUST rules without requiring any page fetches. It detects likely DUST rules based on a large support principle and small buckets principle. It then eliminates redundant rules and validates the remaining rules using a sample of URLs to identify rules that transform URLs with similar content. Experimental results on logs from two websites show that DustBuster is able to discover DUST rules that can help improve crawling efficiency.

大规模数据处理的那些事儿George Ang

腾讯大讲堂02 休闲游戏发展的文化趋势George Ang

腾讯大讲堂03 qq邮箱成长历程George Ang

腾讯大讲堂04 im qqGeorge Ang

腾讯大讲堂05 面向对象应对之道George Ang

腾讯大讲堂06 qq邮箱性能优化George Ang

腾讯大讲堂07 qq空间George Ang

腾讯大讲堂08 可扩展web架构探讨George Ang

腾讯大讲堂09 如何建设高性能网站George Ang

The document discusses techniques for optimizing front-end web performance. It provides examples of how much time is spent loading different parts of top websites, both with empty caches and full caches. The "performance golden rule" is that 80-90% of end-user response time is spent on the front-end. The document also outlines Yahoo's 14 rules for performance optimization, which include making fewer HTTP requests, using content delivery networks, adding Expires headers, gzipping components, script and CSS placement, and more.

腾讯大讲堂01 移动qq产品发展历程George Ang

腾讯大讲堂10 customer engagementGeorge Ang

腾讯大讲堂11 拍拍ce工作经验分享George Ang

腾讯大讲堂14 qq直播(qq live) 介绍George Ang

腾讯大讲堂15 市场研究及数据分析理念及方法概要介绍George Ang

腾讯大讲堂16 产品经理工作心得分享George Ang

Wrapper induction construct wrappers automatically to extract information f...George Ang

Opinion mining and summarizationGeorge Ang

Huffman codingGeorge Ang

Do not crawl in the dust  different ur ls similar textGeorge Ang

大规模数据处理的那些事儿George Ang

腾讯大讲堂02 休闲游戏发展的文化趋势George Ang

腾讯大讲堂03 qq邮箱成长历程George Ang

腾讯大讲堂04 im qqGeorge Ang

腾讯大讲堂05 面向对象应对之道George Ang

腾讯大讲堂06 qq邮箱性能优化George Ang

腾讯大讲堂07 qq空间George Ang

腾讯大讲堂08 可扩展web架构探讨George Ang

腾讯大讲堂09 如何建设高性能网站George Ang

腾讯大讲堂01 移动qq产品发展历程George Ang

腾讯大讲堂10 customer engagementGeorge Ang

腾讯大讲堂11 拍拍ce工作经验分享George Ang

腾讯大讲堂14 qq直播(qq live) 介绍George Ang

腾讯大讲堂15 市场研究及数据分析理念及方法概要介绍George Ang

腾讯大讲堂16 产品经理工作心得分享George Ang

Recently uploaded (20)

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

Splunk Security Update | Public Sector Summit Germany 2025Splunk

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts! 📕 Agenda Welcome & Introductions Orchestrator API Overview Exploring the Swagger Interface Test Manager API Highlights Streamlining Automation & Testing with APIs (Demo) Q&A and Open Discussion Perfect for developers, testers, and automation enthusiasts! 👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/ This session streamed live on April 29, 2025, 18:00 CET. Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.

Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex

Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how: • Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules. • Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance. • Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity. • Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications. • Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market. With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications. Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family

Procurement Insights Cost To Value Guide.pptxJon Hansen

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes