This was a talk, largely on Kamaelia & its original context given at a Free Streaming Workshop in Florence, Italy in Summer 2004. Many of the core
concepts still hold valid in Kamaelia today
Caffe - A deep learning framework (Ramin Fahimi)irpycon
Caffe is a deep learning framework. It is used for tasks like visual recognition using neural networks and deep learning techniques. Caffe uses plain text configuration files called prototxt to define neural network architectures and hyperparameters. It also supports distributed training on GPUs for large datasets. Caffe provides pre-trained models and tools to load, fine-tune, and publish new models for tasks like image classification and object detection.
The document discusses the application layer in the OSI model and DNS. It explains that the application layer provides services to end users through programs and interacts directly with them. It also describes DNS, including that it translates user-friendly domain names to IP addresses, allowing users to access resources by name instead of numerical address. DNS uses a hierarchical name space with domains, zones, primary/secondary name servers, and root servers to distribute its database around the network.
Hadoop Summit 2014 - San Jose - Introduction to Deep Learning on HadoopJosh Patterson
As the data world undergoes its cambrian explosion phase our data tools need to become more advanced to keep pace. Deep Learning has emerged as a key tool in the non-linear arms race of machine learning. In this session we will take a look at how we parallelize Deep Belief Networks in Deep Learning on Hadoop’s next generation YARN framework with Iterative Reduce. We’ll also look at some real world examples of processing data with Deep Learning such as image classification and natural language processing.
Optimize Performance of I/O-intensive Java applications Using Zero CopyIndicThreads
This session explains how you can improve performance of I/O-intensive Java™ applications through a technique called zero copy. Zero copy lets you to avoid redundant data copies between intermediate buffers and reduces number of context switches between user and kernel space.
Background: Many applications [Web Servers, FTP-like services] serve a significant amount of static content, which amounts to reading data off of a disk and writing the exact data back to the response socket. Each time data traverses the user-kernel boundary; it must be copied, which consumes CPU cycles and memory bandwidth. Fortunately, we can eliminate these copies through a technique called — zero copy.
The Java class libraries support zero copy on Linux and UNIX systems through the transferTo() method in java.nio.channels.FileChannel.
Session Agenda: The session will initially focus on “Zero Copy” concept and its relevance in Data transfer applications. The traditional approach of transferring data b/w processes using File and Socket I/O will be explained in detail. It will demonstrate the overhead incurred when using traditional copy semantics, and will show how transferTo() achieves better performance. transferTo() API brings down the time 65% compared to traditional approach.
Summary
The Session demonstrates performance advantages of using transferTo() compared to the traditional approach. Intermediate buffer copies — even those hidden in the kernel — can have a measurable cost. In applications that do a great deal of copying of data between channels, zero-copy technique can offer a significant performance improvement.
Colorspace: Useful For More Than Just Color? - SF Video Tech Meetup - 27 May ...Derek Buitenhuis
This document discusses using colorspaces for more than just representing color, specifically for image and video compression purposes. It provides:
1) A brief history of colorspaces used in compression like YIQ, YUV, and YCbCr and how they were designed more for compression than accurate color representation.
2) Current uses of color transforms in formats like JPEG-XR and JPEG-XL that use colorspaces like YCoCg and XYB specifically designed for compression rather than color accuracy.
3) Potential future uses of reversible color transforms like YCoCg-R, reversible KLT-based transforms, and new proposed spaces like those in the "Alphabet Soup" section that aim to further optimize for compression.
This document discusses planning, optimizing, and troubleshooting DHCP in a network. It covers creating a DHCP plan by designing DHCP infrastructures, scope reservations, options, and security. To optimize performance, the document recommends monitoring DHCP and adjusting the lease duration if the server is overloaded. Troubleshooting tools like Network Monitor, DHCP Audit Log, and IPConfig can help identify client-side, server-side, or infrastructure problems.
Talk given at internal Vimeo lunch talks with an intro to JPEG / image compression. There is a codebase that goes along with this, but it is not public yet, unfortunately.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/09/a-practical-guide-to-implementing-ml-on-embedded-devices-a-presentation-from-the-chamberlain-group/
Nathan Kopp, Principal Software Architect for Video Systems at the Chamberlain Group, presents the “Practical Guide to Implementing ML on Embedded Devices” tutorial at the May 2021 Embedded Vision Summit.
Deploying machine learning onto edge devices requires many choices and trade-offs. Fortunately, processor designers are adding inference-enhancing instructions and architectures to even the lowest cost MCUs, tools developers are constantly discovering optimizations that extract a little more performance out of existing hardware, and ML researchers are refactoring the math to achieve better accuracy using faster operations and fewer parameters.
In this presentation, Kopp takes a high-level look at what is involved in running a DNN model on existing edge devices, exploring some of the evolving tools and methods that are finally making this dream a reality. He also takes a quick look at a practical example of running a CNN object detector on low-compute hardware.
This document provides information about an assignment for the course "Network Programming and Administration". It includes details like the course code, title, assignment number, maximum marks, weightage, and due dates. The assignment has 4 questions worth 80 marks total. An additional 20 marks are for a viva voce. Question 1 asks about IPv6 and includes a sample solution. Question 2 includes subquestions about sliding window protocols, TCP/IP protocols in the OSI model, and LAN network types. Question 3 asks about HTTP and includes sample request methods and statuses.
Tom Peters, Software Engineer, Ufora at MLconf ATL 2016MLconf
Say What You Mean: Scaling Machine Learning Algorithms Directly from Source Code: Scaling machine learning applications is hard. Even with powerful systems like Spark, Tensor Flow, and Theano, the code you write has more to do with getting these systems to work at all than it does with your algorithm itself. But it doesn’t have to be this way!
In this talk, I’ll discuss an alternate approach we’ve taken with Pyfora, an open-source platform for scalable machine learning and data science in Python. I’ll show how it produces efficient, large scale machine learning implementations directly from the source code of single-threaded Python programs. Instead of programming to a complex API, you can simply say what you mean and move on. I’ll show some classes of problem where this approach truly shines, discuss some practical realities of developing the system, and I’ll talk about some future directions for the project.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/08/tinyml-isnt-thinking-big-enough-a-presentation-from-perceive/
Steve Teig, CEO of Perceive, presents the “TinyML Isn’t Thinking Big Enough” tutorial at the May 2021 Embedded Vision Summit.
Today, TinyML focuses primarily on shoehorning neural networks onto microcontrollers or small CPUs but misses the opportunity to transform all of ML because of two unfortunate assumptions: first, that tiny models must make significant performance and accuracy compromises to fit inside edge devices, and second, that tiny models should run on CPUs or microcontrollers.
Regarding the first assumption, information-theoretic considerations would suggest that principled compression (vs., say, just replacing 32-bit weights with 8-bit weights) should make models more accurate, not less. For the second assumption, CPUs are saddled with an intrinsically power-inefficient memory model and mostly serial computation, but the evident parallelism of neural networks naturally leads to high-performance, power-efficient, massively parallel inference hardware. By upending these assumptions, TinyML can revolutionize all of ML–and not just inside microcontrollers.
Knowledge Network is a public broadcaster in British Columbia that relies on an automated, file-based workflow using Telestream products like Vantage and Pipeline to ingest, transcode, and deliver programming. Vantage transcodes content into various formats for broadcast and online distribution. It has replaced their tape-based system, improving efficiency. Vantage allows them to process multiple files simultaneously and handle content in various formats from international distributors. The automated workflow is vital to their operations.
Kaz Sato, Evangelist, Google at MLconf ATL 2016MLconf
Machine Intelligence at Google Scale: Tensor Flow and Cloud Machine Learning: The biggest challenge of Deep Learning technology is the scalability. As long as using single GPU server, you have to wait for hours or days to get the result of your work. This doesn’t scale for production service, so you need a Distributed Training on the cloud eventually. Google has been building infrastructure for training the large scale neural network on the cloud for years, and now started to share the technology with external developers. In this session, we will introduce new pre-trained ML services such as Cloud Vision API and Speech API that works without any training. Also, we will look how TensorFlow and Cloud Machine Learning will accelerate custom model training for 10x – 40x with Google’s distributed training infrastructure.
XMPP can provide a flexible and scalable solution for real-time push notifications across devices and platforms. ProcessOne offers an XMPP-based push platform as a service to enable reliable delivery of notifications to users. Case studies demonstrate how the platform supports use cases like radio program updates, social media feeds, and mobile applications. ProcessOne's expertise in XMPP pubsub helps make these services highly scalable and able to support new features over time.
Let's Be HAV1ng You - London Video Tech October 2019Derek Buitenhuis
Talk I have at the October 2019 London Video Tech meetup covering a few of the many AV1 coding tools (old and new), a small rant on some AV1 tests, and some graphs.
Video: <upload pending>
Cartographer, or Building A Next Generation Management Frameworkansmtug
Dr. Bobby Krupczak's slides about the Cartographer management agent and the underlying XMP management framework. Presented at the February 10, 2009 meeting of the Atlanta Network and Systems Management Technical User Group (ANSMTUG).
5 maximazing networkcapacity_v4-jorge_alvaradoSSPI Brasil
This document discusses how to maximize network capacity through bandwidth optimization and data compression techniques. It provides an agenda that covers defining wireless link optimization, maximizing network capacity for internet access, VPN networks, UDP traffic, corporate applications, and cellular backhaul. Specific scenarios and case studies are presented where XipLink's optimization solutions have reduced bandwidth usage by 18-60% for various application types including internet, VPNs, VoIP, video surveillance, and file transfers. The solutions provide a typical return on investment of less than 4 months.
The document outlines a syllabus for a computer networks course taught by Usha Barad. The syllabus covers 5 topics: 1) introduction to computer networks and the Internet, 2) application layer, 3) transport layer, 4) network layer, and 5) link layer and local area networks. It also lists recommended reference books for the course.
Keras Tutorial For Beginners | Creating Deep Learning Models Using Keras In P...Edureka!
** AI & Deep Learning Training: https://www.edureka.co/ai-deep-learning-with-tensorflow ** )
This Edureka Tutorial on "Keras Tutorial" (Deep Learning Blog Series: https://goo.gl/4zxMfU) provides you a quick and insightful tutorial on the working of Keras along with an interesting use-case! We will be checking out the following topics:
Agenda:
What is Keras?
Who makes Keras?
Who uses Keras?
What Makes Keras special?
Working principle of Keras
Keras Models
Understanding Execution
Implementing a Neural Network
Use-Case with Keras
Coding in Colaboratory
Session in a minute
Check out our Deep Learning blog series: https://bit.ly/2xVIMe1
Check out our complete Youtube playlist here: https://bit.ly/2OhZEpz
Follow us to never miss an update in the future.
Instagram: https://www.instagram.com/edureka_learning/
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
The document provides an overview and agenda for an Amazon Deep Learning presentation. It discusses AI and deep learning at Amazon, gives a primer on deep learning and applications, provides an overview of MXNet and Amazon's investments in it, discusses deep learning tools and usage, and provides two application examples using MXNet on AWS. It concludes by discussing next steps and a call to action.
Title
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTorch + XGBoost + Airflow + MLflow + Spark + Jupyter + TPU
Video
https://youtu.be/vaB4IM6ySD0
Description
In this workshop, we build real-world machine learning pipelines using TensorFlow Extended (TFX), KubeFlow, and Airflow.
Described in the 2017 paper, TFX is used internally by thousands of Google data scientists and engineers across every major product line within Google.
KubeFlow is a modern, end-to-end pipeline orchestration framework that embraces the latest AI best practices including hyper-parameter tuning, distributed model training, and model tracking.
Airflow is the most-widely used pipeline orchestration framework in machine learning.
Pre-requisites
Modern browser - and that's it!
Every attendee will receive a cloud instance
Nothing will be installed on your local laptop
Everything can be downloaded at the end of the workshop
Location
Online Workshop
Agenda
1. Create a Kubernetes cluster
2. Install KubeFlow, Airflow, TFX, and Jupyter
3. Setup ML Training Pipelines with KubeFlow and Airflow
4. Transform Data with TFX Transform
5. Validate Training Data with TFX Data Validation
6. Train Models with Jupyter, Keras/TensorFlow 2.0, PyTorch, XGBoost, and KubeFlow
7. Run a Notebook Directly on Kubernetes Cluster with KubeFlow
8. Analyze Models using TFX Model Analysis and Jupyter
9. Perform Hyper-Parameter Tuning with KubeFlow
10. Select the Best Model using KubeFlow Experiment Tracking
11. Reproduce Model Training with TFX Metadata Store and Pachyderm
12. Deploy the Model to Production with TensorFlow Serving and Istio
13. Save and Download your Workspace
Key Takeaways
Attendees will gain experience training, analyzing, and serving real-world Keras/TensorFlow 2.0 models in production using model frameworks and open-source tools.
Related Links
1. PipelineAI Home: https://pipeline.ai
2. PipelineAI Community Edition: http://community.pipeline.ai
3. PipelineAI GitHub: https://github.com/PipelineAI/pipeline
4. Advanced Spark and TensorFlow Meetup (SF-based, Global Reach): https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup
5. YouTube Videos: https://youtube.pipeline.ai
6. SlideShare Presentations: https://slideshare.pipeline.ai
7. Slack Support: https://joinslack.pipeline.ai
8. Web Support and Knowledge Base: https://support.pipeline.ai
9. Email Support: [email protected]
This document provides an agenda for a presentation on deep learning with TensorFlow. It includes:
1. An introduction to machine learning and deep networks, including definitions of machine learning, neural networks, and deep learning.
2. An overview of TensorFlow, including its architecture, evolution, language features, computational graph, TensorBoard, and use in Google Cloud ML.
3. Details of TensorFlow hands-on examples, including linear models, shallow and deep neural networks for MNIST digit classification, and convolutional neural networks for MNIST.
Convolutional Neural Networks at scale in Spark MLlibDataWorks Summit
Jeremy Nixon will focus on the engineering and applications of a new algorithm built on top of MLlib. The presentation will focus on the methods the algorithm uses to automatically generate features to capture nonlinear structure in data, as well as the process by which it’s trained. Major aspects of that are the compositional transformations over the data, convolution, and distributed backpropagation via SGD with adaptive gradients and an adaptive learning rate. Applications will look into how to use convolutional neural networks to model data in computer vision, natural language and signal processing. Details around optimal preprocessing, the type of structure that can be learned, and managing its ability to generalize will inform developers looking to apply nonlinear modeling tools to problems that they face.
Probabilistic Approach to Provisioning of ITV - By Amos_KohnAmos Kohn
This white paper discusses a probabilistic approach to provisioning network and computing resources for delivering interactive TV. It develops a proprietary spreadsheet model to estimate the costs and benefits of deploying an interactive TV streaming processor. The model is based on analyzing user behavior, data packaging into MPEG streams, required bit rates, forward and return network paths, processing needs, and financial projections to calculate return on investment.
High-quality point clouds have recently gained interest as an emerg- ing form of representing immersive 3D graphics. Unfortunately, these 3D media are bulky and severely bandwidth intensive, which makes it difficult for streaming to resource-limited and mobile de- vices. This has called researchers to propose efficient and adaptive approaches for streaming of high-quality point clouds.
In this paper, we run a pilot study towards dynamic adaptive point cloud streaming, and extend the concept of dynamic adaptive streaming over HTTP (DASH) towards DASH-PC, a dynamic adaptive bandwidth-efficient and view-aware point cloud streaming system. DASH-PC can tackle the huge bandwidth demands of dense point cloud streaming while at the same time can semantically link to human visual acuity to maintain high visual quality when needed. In order to describe the various quality representations, we pro- pose multiple thinning approaches to spatially sub-sample point clouds in the 3D space, and design a DASH Media Presentation Description manifest specific for point cloud streaming. Our initial evaluations show that we can achieve significant bandwidth and performance improvement on dense point cloud streaming with minor negative quality impacts compared to the baseline scenario when no adaptations is applied.
The document discusses Huffman coding, which is a lossless data compression algorithm that uses variable-length codes to encode symbols based on their frequency of occurrence. It begins with definitions of Huffman coding and related terms. It then describes the encoding and decoding processes, which involve constructing a Huffman tree based on symbol frequencies and traversing the tree to encode or decode data. An example is provided that shows the full process of constructing a Huffman tree for a sample frequency table and determining the Huffman codes, average code length, and total encoded length.
IMPROVING IPV6 ADDRESSING TYPES AND SIZEIJCNCJournal
This document discusses proposed modifications to IPv6 addressing types and address size. It suggests that multicast addressing can mimic anycast and limited broadcast addressing, making those types unnecessary. It also proposes reducing the IPv6 address size from 128-bits to decrease packet overhead, while ensuring the new size supports future internet growth. A formula is presented to predict IP address exhaustion dates for different address sizes based on current usage and population projections.
Braxton McKee, CEO & Founder, Ufora at MLconf NYC - 4/15/16MLconf
Say What You Mean: Scaling Machine Learning Algorithms Directly from Source Code: Scaling machine learning applications is hard. Even with powerful systems like Spark, Tensor Flow, and Theano, the code you write has more to do with getting these systems to work at all than it does with your algorithm itself. But it doesn’t have to be this way!
In this talk, I’ll discuss an alternate approach we’ve taken with Pyfora, an open-source platform for scalable machine learning and data science in Python. I’ll show how it produces efficient, large scale machine learning implementations directly from the source code of single-threaded Python programs. Instead of programming to a complex API, you can simply say what you mean and move on. I’ll show some classes of problem where this approach truly shines, discuss some practical realities of developing the system, and I’ll talk about some future directions for the project.
The document discusses the development of a pure peer-to-peer computing system using socket programming. It aims to facilitate parallel computation of complex tasks by distributing work across available peers in a network. This allows heavier calculations to be performed faster by utilizing otherwise idle processing resources. The system is designed to remove scalability and security issues while managing tasks through administrator, query manager, task dispatcher, and processor groups. A literature review found that decentralized peer-to-peer systems like Freenet and GNUtella provide benefits like failure tolerance, efficiency and cost effectiveness.
Solutions for Exercises: Distributed Systems 5th Edition by Coulouris & Dolli...industriale82
Explore distributed computing concepts with solutions to exercises from Distributed Systems 5th Edition. Learn about middleware, scalability, and fault tolerance. A key resource for CS students and practitioners.
This document provides information about an assignment for the course "Network Programming and Administration". It includes details like the course code, title, assignment number, maximum marks, weightage, and due dates. The assignment has 4 questions worth 80 marks total. An additional 20 marks are for a viva voce. Question 1 asks about IPv6 and includes a sample solution. Question 2 includes subquestions about sliding window protocols, TCP/IP protocols in the OSI model, and LAN network types. Question 3 asks about HTTP and includes sample request methods and statuses.
Tom Peters, Software Engineer, Ufora at MLconf ATL 2016MLconf
Say What You Mean: Scaling Machine Learning Algorithms Directly from Source Code: Scaling machine learning applications is hard. Even with powerful systems like Spark, Tensor Flow, and Theano, the code you write has more to do with getting these systems to work at all than it does with your algorithm itself. But it doesn’t have to be this way!
In this talk, I’ll discuss an alternate approach we’ve taken with Pyfora, an open-source platform for scalable machine learning and data science in Python. I’ll show how it produces efficient, large scale machine learning implementations directly from the source code of single-threaded Python programs. Instead of programming to a complex API, you can simply say what you mean and move on. I’ll show some classes of problem where this approach truly shines, discuss some practical realities of developing the system, and I’ll talk about some future directions for the project.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/08/tinyml-isnt-thinking-big-enough-a-presentation-from-perceive/
Steve Teig, CEO of Perceive, presents the “TinyML Isn’t Thinking Big Enough” tutorial at the May 2021 Embedded Vision Summit.
Today, TinyML focuses primarily on shoehorning neural networks onto microcontrollers or small CPUs but misses the opportunity to transform all of ML because of two unfortunate assumptions: first, that tiny models must make significant performance and accuracy compromises to fit inside edge devices, and second, that tiny models should run on CPUs or microcontrollers.
Regarding the first assumption, information-theoretic considerations would suggest that principled compression (vs., say, just replacing 32-bit weights with 8-bit weights) should make models more accurate, not less. For the second assumption, CPUs are saddled with an intrinsically power-inefficient memory model and mostly serial computation, but the evident parallelism of neural networks naturally leads to high-performance, power-efficient, massively parallel inference hardware. By upending these assumptions, TinyML can revolutionize all of ML–and not just inside microcontrollers.
Knowledge Network is a public broadcaster in British Columbia that relies on an automated, file-based workflow using Telestream products like Vantage and Pipeline to ingest, transcode, and deliver programming. Vantage transcodes content into various formats for broadcast and online distribution. It has replaced their tape-based system, improving efficiency. Vantage allows them to process multiple files simultaneously and handle content in various formats from international distributors. The automated workflow is vital to their operations.
Kaz Sato, Evangelist, Google at MLconf ATL 2016MLconf
Machine Intelligence at Google Scale: Tensor Flow and Cloud Machine Learning: The biggest challenge of Deep Learning technology is the scalability. As long as using single GPU server, you have to wait for hours or days to get the result of your work. This doesn’t scale for production service, so you need a Distributed Training on the cloud eventually. Google has been building infrastructure for training the large scale neural network on the cloud for years, and now started to share the technology with external developers. In this session, we will introduce new pre-trained ML services such as Cloud Vision API and Speech API that works without any training. Also, we will look how TensorFlow and Cloud Machine Learning will accelerate custom model training for 10x – 40x with Google’s distributed training infrastructure.
XMPP can provide a flexible and scalable solution for real-time push notifications across devices and platforms. ProcessOne offers an XMPP-based push platform as a service to enable reliable delivery of notifications to users. Case studies demonstrate how the platform supports use cases like radio program updates, social media feeds, and mobile applications. ProcessOne's expertise in XMPP pubsub helps make these services highly scalable and able to support new features over time.
Let's Be HAV1ng You - London Video Tech October 2019Derek Buitenhuis
Talk I have at the October 2019 London Video Tech meetup covering a few of the many AV1 coding tools (old and new), a small rant on some AV1 tests, and some graphs.
Video: <upload pending>
Cartographer, or Building A Next Generation Management Frameworkansmtug
Dr. Bobby Krupczak's slides about the Cartographer management agent and the underlying XMP management framework. Presented at the February 10, 2009 meeting of the Atlanta Network and Systems Management Technical User Group (ANSMTUG).
5 maximazing networkcapacity_v4-jorge_alvaradoSSPI Brasil
This document discusses how to maximize network capacity through bandwidth optimization and data compression techniques. It provides an agenda that covers defining wireless link optimization, maximizing network capacity for internet access, VPN networks, UDP traffic, corporate applications, and cellular backhaul. Specific scenarios and case studies are presented where XipLink's optimization solutions have reduced bandwidth usage by 18-60% for various application types including internet, VPNs, VoIP, video surveillance, and file transfers. The solutions provide a typical return on investment of less than 4 months.
The document outlines a syllabus for a computer networks course taught by Usha Barad. The syllabus covers 5 topics: 1) introduction to computer networks and the Internet, 2) application layer, 3) transport layer, 4) network layer, and 5) link layer and local area networks. It also lists recommended reference books for the course.
Keras Tutorial For Beginners | Creating Deep Learning Models Using Keras In P...Edureka!
** AI & Deep Learning Training: https://www.edureka.co/ai-deep-learning-with-tensorflow ** )
This Edureka Tutorial on "Keras Tutorial" (Deep Learning Blog Series: https://goo.gl/4zxMfU) provides you a quick and insightful tutorial on the working of Keras along with an interesting use-case! We will be checking out the following topics:
Agenda:
What is Keras?
Who makes Keras?
Who uses Keras?
What Makes Keras special?
Working principle of Keras
Keras Models
Understanding Execution
Implementing a Neural Network
Use-Case with Keras
Coding in Colaboratory
Session in a minute
Check out our Deep Learning blog series: https://bit.ly/2xVIMe1
Check out our complete Youtube playlist here: https://bit.ly/2OhZEpz
Follow us to never miss an update in the future.
Instagram: https://www.instagram.com/edureka_learning/
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
The document provides an overview and agenda for an Amazon Deep Learning presentation. It discusses AI and deep learning at Amazon, gives a primer on deep learning and applications, provides an overview of MXNet and Amazon's investments in it, discusses deep learning tools and usage, and provides two application examples using MXNet on AWS. It concludes by discussing next steps and a call to action.
Title
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTorch + XGBoost + Airflow + MLflow + Spark + Jupyter + TPU
Video
https://youtu.be/vaB4IM6ySD0
Description
In this workshop, we build real-world machine learning pipelines using TensorFlow Extended (TFX), KubeFlow, and Airflow.
Described in the 2017 paper, TFX is used internally by thousands of Google data scientists and engineers across every major product line within Google.
KubeFlow is a modern, end-to-end pipeline orchestration framework that embraces the latest AI best practices including hyper-parameter tuning, distributed model training, and model tracking.
Airflow is the most-widely used pipeline orchestration framework in machine learning.
Pre-requisites
Modern browser - and that's it!
Every attendee will receive a cloud instance
Nothing will be installed on your local laptop
Everything can be downloaded at the end of the workshop
Location
Online Workshop
Agenda
1. Create a Kubernetes cluster
2. Install KubeFlow, Airflow, TFX, and Jupyter
3. Setup ML Training Pipelines with KubeFlow and Airflow
4. Transform Data with TFX Transform
5. Validate Training Data with TFX Data Validation
6. Train Models with Jupyter, Keras/TensorFlow 2.0, PyTorch, XGBoost, and KubeFlow
7. Run a Notebook Directly on Kubernetes Cluster with KubeFlow
8. Analyze Models using TFX Model Analysis and Jupyter
9. Perform Hyper-Parameter Tuning with KubeFlow
10. Select the Best Model using KubeFlow Experiment Tracking
11. Reproduce Model Training with TFX Metadata Store and Pachyderm
12. Deploy the Model to Production with TensorFlow Serving and Istio
13. Save and Download your Workspace
Key Takeaways
Attendees will gain experience training, analyzing, and serving real-world Keras/TensorFlow 2.0 models in production using model frameworks and open-source tools.
Related Links
1. PipelineAI Home: https://pipeline.ai
2. PipelineAI Community Edition: http://community.pipeline.ai
3. PipelineAI GitHub: https://github.com/PipelineAI/pipeline
4. Advanced Spark and TensorFlow Meetup (SF-based, Global Reach): https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup
5. YouTube Videos: https://youtube.pipeline.ai
6. SlideShare Presentations: https://slideshare.pipeline.ai
7. Slack Support: https://joinslack.pipeline.ai
8. Web Support and Knowledge Base: https://support.pipeline.ai
9. Email Support: [email protected]
This document provides an agenda for a presentation on deep learning with TensorFlow. It includes:
1. An introduction to machine learning and deep networks, including definitions of machine learning, neural networks, and deep learning.
2. An overview of TensorFlow, including its architecture, evolution, language features, computational graph, TensorBoard, and use in Google Cloud ML.
3. Details of TensorFlow hands-on examples, including linear models, shallow and deep neural networks for MNIST digit classification, and convolutional neural networks for MNIST.
Convolutional Neural Networks at scale in Spark MLlibDataWorks Summit
Jeremy Nixon will focus on the engineering and applications of a new algorithm built on top of MLlib. The presentation will focus on the methods the algorithm uses to automatically generate features to capture nonlinear structure in data, as well as the process by which it’s trained. Major aspects of that are the compositional transformations over the data, convolution, and distributed backpropagation via SGD with adaptive gradients and an adaptive learning rate. Applications will look into how to use convolutional neural networks to model data in computer vision, natural language and signal processing. Details around optimal preprocessing, the type of structure that can be learned, and managing its ability to generalize will inform developers looking to apply nonlinear modeling tools to problems that they face.
Probabilistic Approach to Provisioning of ITV - By Amos_KohnAmos Kohn
This white paper discusses a probabilistic approach to provisioning network and computing resources for delivering interactive TV. It develops a proprietary spreadsheet model to estimate the costs and benefits of deploying an interactive TV streaming processor. The model is based on analyzing user behavior, data packaging into MPEG streams, required bit rates, forward and return network paths, processing needs, and financial projections to calculate return on investment.
High-quality point clouds have recently gained interest as an emerg- ing form of representing immersive 3D graphics. Unfortunately, these 3D media are bulky and severely bandwidth intensive, which makes it difficult for streaming to resource-limited and mobile de- vices. This has called researchers to propose efficient and adaptive approaches for streaming of high-quality point clouds.
In this paper, we run a pilot study towards dynamic adaptive point cloud streaming, and extend the concept of dynamic adaptive streaming over HTTP (DASH) towards DASH-PC, a dynamic adaptive bandwidth-efficient and view-aware point cloud streaming system. DASH-PC can tackle the huge bandwidth demands of dense point cloud streaming while at the same time can semantically link to human visual acuity to maintain high visual quality when needed. In order to describe the various quality representations, we pro- pose multiple thinning approaches to spatially sub-sample point clouds in the 3D space, and design a DASH Media Presentation Description manifest specific for point cloud streaming. Our initial evaluations show that we can achieve significant bandwidth and performance improvement on dense point cloud streaming with minor negative quality impacts compared to the baseline scenario when no adaptations is applied.
The document discusses Huffman coding, which is a lossless data compression algorithm that uses variable-length codes to encode symbols based on their frequency of occurrence. It begins with definitions of Huffman coding and related terms. It then describes the encoding and decoding processes, which involve constructing a Huffman tree based on symbol frequencies and traversing the tree to encode or decode data. An example is provided that shows the full process of constructing a Huffman tree for a sample frequency table and determining the Huffman codes, average code length, and total encoded length.
IMPROVING IPV6 ADDRESSING TYPES AND SIZEIJCNCJournal
This document discusses proposed modifications to IPv6 addressing types and address size. It suggests that multicast addressing can mimic anycast and limited broadcast addressing, making those types unnecessary. It also proposes reducing the IPv6 address size from 128-bits to decrease packet overhead, while ensuring the new size supports future internet growth. A formula is presented to predict IP address exhaustion dates for different address sizes based on current usage and population projections.
Braxton McKee, CEO & Founder, Ufora at MLconf NYC - 4/15/16MLconf
Say What You Mean: Scaling Machine Learning Algorithms Directly from Source Code: Scaling machine learning applications is hard. Even with powerful systems like Spark, Tensor Flow, and Theano, the code you write has more to do with getting these systems to work at all than it does with your algorithm itself. But it doesn’t have to be this way!
In this talk, I’ll discuss an alternate approach we’ve taken with Pyfora, an open-source platform for scalable machine learning and data science in Python. I’ll show how it produces efficient, large scale machine learning implementations directly from the source code of single-threaded Python programs. Instead of programming to a complex API, you can simply say what you mean and move on. I’ll show some classes of problem where this approach truly shines, discuss some practical realities of developing the system, and I’ll talk about some future directions for the project.
The document discusses the development of a pure peer-to-peer computing system using socket programming. It aims to facilitate parallel computation of complex tasks by distributing work across available peers in a network. This allows heavier calculations to be performed faster by utilizing otherwise idle processing resources. The system is designed to remove scalability and security issues while managing tasks through administrator, query manager, task dispatcher, and processor groups. A literature review found that decentralized peer-to-peer systems like Freenet and GNUtella provide benefits like failure tolerance, efficiency and cost effectiveness.
Solutions for Exercises: Distributed Systems 5th Edition by Coulouris & Dolli...industriale82
Explore distributed computing concepts with solutions to exercises from Distributed Systems 5th Edition. Learn about middleware, scalability, and fault tolerance. A key resource for CS students and practitioners.
Serhiy Kalinets "Embracing architectural challenges in the modern .NET world"Fwdays
For more than decade .NET has been used primarily in enterprise software development. We all remember intranet deployment, IIS, SQL Server, N-tier applications and so on. The toolset (Visual Studio, SQL Management Studio, IIS Management snap-in etc) seemed to be set in stone as well as architecture (controllers, services, repositories). .NET people were isolated from other folks, who were using clusters, containers, clouds, and Linux.
However, adoption of clouds during few past years, the release of .NET Core made much more choices available to developers. It turned out that traditional way of building application is not that efficient from many viewpoints, including costs, time, performance or robustness. It happens because the environment has been changed and many assumptions are not still relevant.
In this talk, we will discuss what and why has been changed and how to deal with that. What are new requirements for our applications? What are new services available, and how to use them wisely? And finally, how should we design our applications to be cost-effective, competitive and have a lot of fun working with .NET Core.
The document discusses the architecture of an online survey platform. It covers goals of performance, availability, and scalability. It describes using services like Amazon Web Services for storage, computing, and content delivery. It also discusses optimizing application design, database separation, coding best practices, and automated testing to achieve goals. Video hosting and online analytical reporting features are explained.
This document discusses changes to Hyper-V virtualization from Windows Server 2008 to 2012. Key changes include the ability to share virtual hard disks between VMs, improved quality of service controls, and more robust resource sharing between host and guest systems. The new features make Hyper-V more reliable and scalable for server virtualization needs over the next 2-3 years.
What is DPI? How can it be used effectively? What are the different use cases and requirements for such products? We discuss this and the methodologies needed to properly evaluate the DPI functionality of network devices under the demanding network conditions in which they will be deployed.
http://nsslabs.com/DPI
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...Nati Shalom
Twitter is a good example for next generation real-time web applications, but building such an application imposes challenges such as handling an every growing volume of tweets and responses, as well as a large number of concurrent users, who continually *listen* for tweets from users (or topics) they follow. During this session we will review some of the key design principles addressing these challenges, including alternatives *NoSQL* alternatives and blackboard patterns. We will be using Twitter as a use case, while learning how to apply these to any real-time we application
Introduction to requirement of microservicesAvik Das
We are talking about microservices. It is a pattern to resolve the complexity of the system those need to process a high amount of data within a short period.
Financial lose may occur on implementation of this pattern for an application of limited complexity in the initial phase. Initial phases have a learning curve to understand the relation and behavior of domain entities.
Small and medium companies lean this during development. Large companies can allocate additional times for documentation and correction on design phases for a reasonable long period. So, sometimes it is good to start with a monolithic architecture and grow with the achievement of the company then migrate to microservices.
The document discusses the technical teams at Tuenti and their work developing various products and services. It covers their frontend, backend, and systems teams and some of the challenges they face in building large-scale, high-performance applications and services to support millions of users. It also provides specifics on their development of Tuenti's instant messaging platform using open-source technologies and Erlang.
This document discusses several distributed computing systems:
1) DNS is a distributed system that maps domain names to IP addresses using a hierarchical naming structure and caching DNS servers for efficiency.
2) BOINC is a volunteer computing platform that uses over a million computers worldwide for distributed applications like disease research. It provides incentives and verifies results to prevent cheating.
3) PlanetLab is a research network with over 700 servers globally that allows testing new distributed systems at large scales under realistic conditions. It isolates projects using virtualization and trust relationships.
This document discusses strategies for handling large amounts of data in web applications. It begins by providing examples of how much data some large websites contain, ranging from terabytes to petabytes. It then covers various techniques for scaling data handling capabilities including vertical and horizontal scaling, replication, partitioning, consistency models, normalization, caching, and using different data engine types beyond relational databases. The key lessons are that data volumes continue growing rapidly, and a variety of techniques are needed to scale across servers, datacenters, and provide high performance and availability.
CH02-Computer Organization and Architecture 10e.pptxHafizSaifullah4
This document discusses computer performance and benchmarking. It covers several topics related to improving computer performance, including designing for performance, microprocessor speed techniques, improvements in chip organization and architecture like multicore processors, and issues that limit further increases in clock speed. It also discusses Amdahl's Law, Little's Law, and ways to measure computer performance, including various types of means to calculate benchmark results. SPEC benchmarks are mentioned as examples of widely used benchmark programs.
DM Radio Webinar: Adopting a Streaming-Enabled ArchitectureDATAVERSITY
Architecture matters. That's why today's innovators are taking a hard look at streaming data, an increasingly attractive option that can transform business in several ways: replacing aging data ingestion techniques like ETL; solving long-standing data quality challenges; improving business processes ranging from sales and marketing to logistics and procurement; or any number of activities related to accelerating data warehousing, business intelligence and analytics.
Register for this DM Radio Deep Dive Webinar to learn how streaming data can rejuvenate or supplant traditional data management practices. Host Eric Kavanagh will explain how streaming-first architectures can relieve data engineers from time-consuming, error-prone processes, ideally bidding farewell to those unpleasant batch windows. He'll be joined by Kevin Petrie of Attunity, who will explain why (with real-world story successes) streaming data solutions can keep the business fueled with trusted data in a timely, efficient manner for improved business outcomes.
This document presents a case study comparing a traditional single-node approach and a cloud-based approach for analyzing a large dataset of over 150 million domain names to determine which are hosted by SoftLayer. The single-node approach ran on a single server and took approximately 300 hours to complete at a cost of $102.67. A cloud-based approach using multiple servers in parallel could complete the task much faster and potentially at a lower overall cost by leveraging elastic computing resources in the cloud.
The document provides details of a proposed network solution for ACME Inc. that will allow 70 users to work productively from the company's 3-story office. Key aspects include:
- Implementing Active Directory, file/print services, and a company intranet to centralize management and sharing of files and communications.
- Dividing the network into subnets for different floors/departments and assigning IP addresses and devices.
- Specifying the required hardware, software, and licenses including laptops, desktops, servers, networking equipment, and applications.
- Outlining the conceptual network design with remote and on-site clients connecting through a firewall, VPN server, and other servers.
-
This is the course that was presented by James Liddle and Adam Vile for Waters in September 2008.
The book of this course can be found at: http://www.lulu.com/content/4334860
UnConference for Georgia Southern Computer Science March 31, 2015Christopher Curtin
I presented to the Georgia Southern Computer Science ACM group. Rather than one topic for 90 minutes, I decided to do an UnConference. I presented them a list of 8-9 topics, let them vote on what to talk about, then repeated.
Each presentation was ~8 minutes, (Except Career) and was by no means an attempt to explain the full concept or technology. Only to wake up their interest.
The document discusses embracing concurrency for simpler code. It notes that hardware is becoming massively concurrent, providing an opportunity. While concurrency is viewed as hard, the fundamental problem may be lack of proper tools. Imperative languages often overlook concurrency as a core concept. A variety of desktop and media applications could benefit from a concurrent approach. The document advocates using concurrent components that communicate via messages while keeping data private. It also discusses software transactional memory and different perspectives in APIs for concurrent systems. Finally, it presents examples of using pipelines and graphlines as part of a concurrency domain specific language.
This was the Kamaelia Tutorial at Europython. It goes from basics - ie building a mini-kamaelia from scratch, through to a file multicaster, through a video recording application all the walk through to a multiuser bulletin board system.
Embracing concurrency for fun utility and simpler codekamaelian
The document discusses embracing concurrency for simpler code. It notes that hardware is becoming more concurrent, but most programming languages and tools treat concurrency as difficult. The Kamaelia project aims to make concurrency easy and usable for novice and advanced developers alike through fundamental control structures and messaging between components. Examples shown include using pipelines, graphlines, servers, and backplanes to build concurrent applications in a simple way.
This document guides the reader through building a system where a user connects to a server over a secure connection and receives a sequence of JSON-encoded objects. It begins by introducing the ServerCore component and shows how to fill its protocol handler factory hole. It then demonstrates creating a stackedjson protocol handler using a pipeline of components like PeriodicWakeup, Chooser, and MarshallJSON. This protocol securely transmits JSON data chunks to clients like a ConsoleEchoer. It discusses how the client-side mirrors the server components to receive and display the messages.
Sharing Data and Services Safely in Concurrent Systems using Kamaeliakamaelian
Kamaelia is generally a \"shared nothing\" architecture, but occasionally you *really* need to share data explicitly. When you do, you need to constrain how you share data and careful about how you advertise services. This is the first presentation done on the facilities that exist in Kamaelia to support this.
This presentation was given at Pycon UK 2008, Birmingham uk. Lots of good feedback was had during the q&a and an updated & improved version will be posted at some point in the relatively near future.
Practical concurrent systems made simple using Kamaeliakamaelian
This talk was given at Pycon UK 2008 in Birmingham.
This presentation aims to teach you how to get started with Kamaelia, building a variety of systems, as well as walking through the design and implementation of some systems built in the last year. Systems built over the past year include tools for dealing with spam (greylisting), through database modelling, video & image transcoding for a youtube/flickr type system, paint programs, webserving, XMPP, games, and a bunch of other things.
This presentation aims to show people that they already know how to deal with concurrency.
It argues that if we have the tools for large scale concurrency (mashups) and small scale (hardware) that midrange (normal apps) can be done in a similar way, using existing tools.
This is done by showing useful systems that have been produced in this manner using existing tools. ie from existing practice, not theory
During the actual presentation I also talked about Kamaelia projects created by novice programmers of varying ability which show high levels of concurrency.
These include: previewing PVR content on mobiles, multicast island joining, as-live streaming using bit torrent, Open GL based user interfaces & integration, seaside style webserving, speex based secure phone, IRC/IM systems, a shakespeare script player, and games tools.
Other systems created include Atom/RSS routing, memcached integration, P2P whiteboarding (with audio + mixing), gesture recognition, presentation tools, a kids development environment, topology visualisation tools, database modelling etc.
This presentation was given at Python North West. It explains a complete Kamaelia application for greylisting which was written specifically to eliminate my personal spam problem. It walks through the code as well (though that's best looked at with the code side by side!)
Open Source at the BBC: When, Why, Why not & Howkamaelian
This talk was given at Linux World 2006. It covers 3 aspects of open source at the BBC - use, extension & origination through the 4 lenses of when, why, why not & how. It focusses entirely on pragmatics in all cases. The style is Lessig style. A write up on the text can be found here: http://tinyurl.com/yd4j2y
This was an invited talk at Open Source Forum Russia in April 2005. It covers open source at the BBC from the perspective of "why use open source?" "what sort of stuff gets used?" "what has the BBC released as open source & why?" open source vs open standards
This talk was part tongue in cheek, part serious, but entirely fun and given twice as a lightning talk - once at Europython & once at the ACCU python uk 05. It presents a generic python like language parser which does actually work. Think of it as an alternative to brackets in Lisp!
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaeliakamaelian
This presentation on Kamaelia at Euro OSCON 2006, and specifically focusses
on a particular system - Kamaelia Macro which is essentially a system for
timeshifting pretty much everything.
This talk was given at Pycon UK 07. It's actually a thin wrapper around the
Kamaelia Mini Axon tutorial which can be found here:
http://kamaelia.sourceforge.net/MiniAxon/
In this talk I talked about how,in the Kamaelia project, we manage the dilemma of encouraging innovation and creativity in a project whilst maintaining an engineered solution. Why? Because we find it allows a high level of creative freedom, whilst also providing a path through to a high level of confidence in the reliabilty of the final code.
This was a talk on how to build systems with Kamaelia given at Pycon UK. It
goes through from basics through to building a swarming P2P live radio
system.
The document discusses free and open source software. It begins by defining free software as software that users have the freedom to use, study, distribute, and change. It notes that free software is also known as libre software or open source software. The document outlines several advantages of free software such as giving users control, reducing costs, using open standards, sustainability, skills development, and improved security and quality. It also briefly discusses some potential disadvantages like smaller installed bases and issues of compatibility with proprietary software. Overall, the document presents an overview of the key concepts around free and open source software.
This talk was the keynote talk at the EBU's Seminar on Open Source Oct 1st, 2nd 2007. http://www.ebu.ch/en/technical/opensource/
The video referenced is IBM's "Prodigy" advert, which can be found here: http://youtube.com/watch?v=q5Kp1Q39VwI
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc
Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency.
This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data.
Attendees will learn:
- Consumer awareness around data brokers and what consumers are doing to limit data collection
- How businesses assess third-party vendors and their consent management operations
- Where business preparedness needs improvement
- What these trends mean for the future of privacy governance and public trust
This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.
HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/
HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client.
Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience.
In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including
- Accessing the console
- Locating and interpreting log files
- Accessing the data folder within the browser’s cache (using OPFS)
- Understand the difference between single- and multi-user scenarios
- Utilizing Client Clocking
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55
We’re bringing the TDX energy to our community with 2 power-packed sessions:
🛠️ Workshop: MuleSoft for Agentforce
Explore the new version of our hands-on workshop featuring the latest Topic Center and API Catalog updates.
📄 Talk: Power Up Document Processing
Dive into smart automation with MuleSoft IDP, NLP, and Einstein AI for intelligent document workflows.
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity
Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts!
📕 Agenda
Welcome & Introductions
Orchestrator API Overview
Exploring the Swagger Interface
Test Manager API Highlights
Streamlining Automation & Testing with APIs (Demo)
Q&A and Open Discussion
Perfect for developers, testers, and automation enthusiasts!
👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/
This session streamed live on April 29, 2025, 18:00 CET.
Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...Alan Dix
Talk at the final event of Data Fusion Dynamics: A Collaborative UK-Saudi Initiative in Cybersecurity and Artificial Intelligence funded by the British Council UK-Saudi Challenge Fund 2024, Cardiff Metropolitan University, 29th April 2025
https://alandix.com/academic/talks/CMet2025-AI-Changes-Everything/
Is AI just another technology, or does it fundamentally change the way we live and think?
Every technology has a direct impact with micro-ethical consequences, some good, some bad. However more profound are the ways in which some technologies reshape the very fabric of society with macro-ethical impacts. The invention of the stirrup revolutionised mounted combat, but as a side effect gave rise to the feudal system, which still shapes politics today. The internal combustion engine offers personal freedom and creates pollution, but has also transformed the nature of urban planning and international trade. When we look at AI the micro-ethical issues, such as bias, are most obvious, but the macro-ethical challenges may be greater.
At a micro-ethical level AI has the potential to deepen social, ethnic and gender bias, issues I have warned about since the early 1990s! It is also being used increasingly on the battlefield. However, it also offers amazing opportunities in health and educations, as the recent Nobel prizes for the developers of AlphaFold illustrate. More radically, the need to encode ethics acts as a mirror to surface essential ethical problems and conflicts.
At the macro-ethical level, by the early 2000s digital technology had already begun to undermine sovereignty (e.g. gambling), market economics (through network effects and emergent monopolies), and the very meaning of money. Modern AI is the child of big data, big computation and ultimately big business, intensifying the inherent tendency of digital technology to concentrate power. AI is already unravelling the fundamentals of the social, political and economic world around us, but this is a world that needs radical reimagining to overcome the global environmental and human challenges that confront us. Our challenge is whether to let the threads fall as they may, or to use them to weave a better future.
How Can I use the AI Hype in my Business Context?Daniel Lehner
𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨?
Everyone’s talking about AI but is anyone really using it to create real value?
Most companies want to leverage AI. Few know 𝗵𝗼𝘄.
✅ What exactly should you ask to find real AI opportunities?
✅ Which AI techniques actually fit your business?
✅ Is your data even ready for AI?
If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies
In late April 2025, a significant portion of Europe, particularly Spain, Portugal, and parts of southern France, experienced widespread, rolling power outages that continue to affect millions of residents, businesses, and infrastructure systems.
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB
I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada
Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next.
Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/
Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.
Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell
With expertise in data architecture, performance tracking, and revenue forecasting, Andrew Marnell plays a vital role in aligning business strategies with data insights. Andrew Marnell’s ability to lead cross-functional teams ensures businesses achieve sustainable growth and operational excellence.
What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat
The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.
Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo
From predictive maintenance to robotic automation, AI is driving the future of manufacturing. But without high-quality annotated data, even the smartest models fall short.
Discover how data annotation services are powering accuracy, safety, and efficiency in AI-driven manufacturing systems.
Precision in data labeling = Precision on the production floor.
Semantic Cultivators : The Critical Future Role to Enable AIartmondano
By 2026, AI agents will consume 10x more enterprise data than humans, but with none of the contextual understanding that prevents catastrophic misinterpretations.
Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next.
Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/
Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.
1. Scaling BBC Streaming Concepts, Current Research, Future Goals, Michael Sparks BBC R&D [email_address]
2. Streaming at the BBC Peak is around 40-50K concurrent streams Both live & on-demand Small audience by BBC standards Want to enable larger services, cost effectively Real, because it's cross platform BBC prefers open standards/platforms
3. Caveat! Very mixed talk! Lots relating to current work Some blue sky/where next I'm aiming for high content ratio – means I might go fast Slides contain: lots of text since mixed audience; illustrative code to show key points – not actually from main codebase
4. Caveat! No really, this might be death by powerpoint Even though powerpoint never came near this presentation, but hey
5. What is Scalability? Practical definition: Predictability and affordability when systems get large Not just concerned with unit efficiency However unit efficiency becomes important when you have scarce resources/big numbers
6. Common Techniques Server Software: At operating system level, optimise threading, processes, schedulers Not portable by definition In applications build their own concurrency primitives State machines, event wheels, reactors Can easily obfuscate the system
7. Areas Under Research Alternative concurrency models Modern language primitives Alternative composition models The world is filled with concurrent systems relatively simple to understand: TVs, Radios Clocks, Roads, Companies Unix pipelines
8. Scaling & Software Cost Writing out own testbed server platform Unencumbered Licensing: (choice of LGPL, MPL, GPL) To encourage take up of any experimental protocols, as per RFC Allows embedding in PVRs and clients
9. Scaling & Software Cost Dirac Was in prototype stage when MPEG 4 costs initially bandied about (initially prohibitively high) Cost free, high quality codec Theora was pre-alpha stage at this point Dirac and Theora complement each other, the more open codecs, the better
10. Scaling Network Systems Get the network to distribute your content We've been working to get multicast deployed by UK ISPs to allow us to serve larger audiences at better quality for lower cost Having our own server allows integration with effective P2P Long way off, but that's a goal
11. Kamaelia Kamaelia is the name of our in-house testbed media server project Uses a rather different architecture from other network servers This talk covers it in as much detail as is practical in an hour
12. Other aspects to be covered Protocol scalability RTP is well designed for VoIP, not aimed primarily at unidirectional streaming This causes problems Is Streaming the right thing? What sort of protocol do we really need? Social aspects of architecture and enabling participation.
13. Scaling BBC Streaming 20 million homes, 700,000 hours of video/audio (approx) Everyone watching something different? Problems: Bandwidth, number of servers, physical space, legal issues, costs Will be here for long term if a success What problem does streaming solve?
14. Bandwidth issues Digital TV: 20000000*5Mbits = 100Tbit/s Quality fairly predictable Comfortable DSL: 20000000*370Kbit/s = 7.4 Tbit/s Quality dependent on codec/content No single location will work
15. Number of Servers Assume servers are capable of flatlining their network card. (Rare) Assume 1Gbit/s NICs 7.4Tbit/s / 1Gbit/s = 7400 servers
16. Physical Space Requirements 7400 servers 1U servers, 42U racks – 176 racks ~ 300 sq m Archive would be about 15,000 Terabytes Would fill a small building? How practical? This is without redundancy – bad, numbers are underestimates
17. Legal Issues We don't own all rights outright Rights need tracking and to be protected Sample issues: Geographical distribution agreements, repurposing for internet, background music, background pictures, finding people Copyright will expire though – what then? NB. I am not a lawyer, this is not legal advice!
18. Scaling Internet Costs Software fees often related to audience size/number of servers Can we grow service by 3 orders of magnitude? Software fees increasing 1000 fold would be welcomed by vendors, but cripple ability to make programmes
19. Here to Stay BBC TV is 5 decades old (or so) BBC Radio is 8 decades old Maintenance is key issue, designing for the long term Fortran, Lisp, Cobol all 4-5 decades old and still used due to long term systems Where are the proprietary languages of then?
20. Hidden Assumptions Linear scalability at worst case Moore's law & its variations continue to operate in our favour Neither are guaranteed. If these fail, we still need to deal with it. Need to plan for outlandish optimisation ideas.
21. Measuring Cost Lots of literature about measuring algorithm and system cost 3 key results Same cost no matter size of problem Cost is proportional to problem size (or cheaper) Unit cost increases as problem size increases Fourth important metric is scarce resouce Cost/resource is defined as CPU cycles, memory, storage, money, etc
22. License & Standards “Costs” F/OSS is fixed software cost for any problem size Proprietary software is category 2 Custom/bespoke software might be category 3 Open standards are between 1& 2 generally. (MPEG4 is 2, HTTP is 1) Category 1 is best for consumers
23. Protocol Cost – Example ICP Internet Cache Protocol is used by web caches to query others for misses Each request runs risk of 2s timeout 2 caches - 1 request to the other cache 10 caches - 9 requeststo the other caches n caches - n-1 requests to the other caches As you grow service, you add caches Unit cost per request goes up with service growth
24. Protocol Cost: Example RTCP Feedback mechanism in RTP Scarce resource issue Multicast session of 20M homes 20M*128bits alone. If reporting every 5 minutes = 8Mbit/s being sent to everyone If drop this down to 10% of dialup bandwidth, you get one report per user a week (useless)
25. So What? Bandwidth costs will be offset if others redistribute our content Better for us to figure out how to do this than someone else Software costs will be offset of costs are closer to that of open source Working on the this could help with the first point
26. What are we working on? 2 main projects are working on scaling distribution Kamaelia is focused on the software aspects, and looking at architectures for long term scaling, not short term win We set up private multicast peering with ISPs for the Olympics. We leave the latter for a different time
27. Server Architectures Lots of concurrent connections Scalability often driven by concurrency choice OS supported concurrency (threads/processes) Manual concurrency handling (events etc) Language concurrency primitves Last two are logically equivalent
28. OS Concurrency Used by Apache & Inetd Scales fairly well - on some operating systems better than others Code remains relatively clear looking With high levels of concurrency context switching becomes a problem when dealing cross platform Ruled out because it restricts OS, and portability
29. Manual Concurrency Most commonly used approach for building scalable servers Unless our approach pans out, use this for your servers! Often state machine/event based Problem is this is back to one step above “goto” Ruled out because it makes joining & maintaining project potentially a lot harder
30. Language Based Concurrency Not many languages support this approach Hence not commonly used Like manual concurrency risks lock up due to being co-operative multitasking Allows your code to remain single threaded Chosen because it simplifies joining & maintaining the project if provide a sensible framework for management
31. Kamaelia: Architecture Single threaded, select based concurrency through language support Concurrency wrapped up in components Components only communicate via named boxes. Boxes are connected via linkages. Components do not know who they are communicating with Same idea as: Unix pipes, CSP, async hardware, hardware systems
32. Python Generators 1 These are used as the base concurrency mechanism. example: def fib(base1, base2): while True: yield base2 base1,base2 = base2, base1+base2 usage: >>> fibs=fib(0,1) >>> for i in range(10): print fibs.next(), ... 1 1 2 3 5 8 13 21 34 55
33. Python Generators 2 More complex usage >>> fibs=[fib(0,i) for i in range(10)] >>> for i in range(10): ... print [fibs[i].next() for x in fibs] ... [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] [2, 2, 4, 6, 10, 16, 26, 42, 68, 110] ... snip ... [9, 9, 18, 27, 45, 72, 117, 189, 306, 495] Note the inherent concurrency in generators
34. Python Generators 3 Look like normal functions, but actually return generator objects yield x rather than return x Calling gen.next() advances gen to the next yield, which returns value to caller Essentially allows a “return and continue” Multiple initialised generators are allowed, and essentially run concurrently
35. Generators – Issues, Solutions No means of communication – only globals and initial arguments Can only yield to immediate caller Does mean that generators generally stay simple – which can some things No default scheduler system Solution: give generators context by wrapping them inside a class
36. Wrapping Generators Many ways, this is simplest, we use a slightly more complex approach. Utility functon: import copy def wrapgenerator(bases=object, **attrs): def decorate(func): class statefulgenerator(bases): __doc__ = func.__doc__ def __init__(self,*args): super(statefulgenerator, self) __init__(*args) self.func=func(self,*args) for k in attrs.keys(): self.__dict__[k] = copy.deepcopy(attrs[k]) self.next=self.__iter__().next def __iter__(self): return iter(self.func) return statefulgenerator return decorate
37. Wrapping Generators - 2 Suppose we wish to allow the following behaviour: We communicate only with named queues We allow components to nest, communicating only with their named queues We can encapsulate this in a class and then use our utility function to apply it to the generator
38. Sample Behaviour Details aren't important, just the fact this is simple to code: class com(object): def __init__(_, *args): # Default queues _.queues = {"inbox":[],"control":[], "outbox":[], "signal":[]} def send(_,box,obj): _.queues[box].append(obj) def dataReady(_,box): return len(_.queues[box])>0 def recv(_, box): # NB. Exceptions aren't caught X=_.queues[box][0] del _.queues[box][0] return X
39. Sample Wrapped Generator @wrapgenerator(com) def forwarder(self): "Simple data forwarding generator" while 1: if self.dataReady("inbox"): self.send("outbox",self.recv("inbox")) elif self.dataReady("control"): if self.recv("control") == "shutdown": break yield 1 self.send("signal","shutdown") yield 0
40. Essentials of Kamaelia's Components Generators with named in/out-boxes Linkages join outboxes to inboxes A Scheduler A postman for delivery of messages along linkages Runs independently of components A co-ordinating assistant tracker
41. Kamaelia: Building Communication Linkages are tied to the substructure of a component, and are formed via a method call. The method call registers the linkage with the local postman. Example: self.link((socket_handler,"outbox"), (protocol_handler,"inbox")) self.link((protocol_handler,"outbox"), (socket_handler,"inbox")) Note we link component/box to component/box
42. Scheduling Generators Our (simplified) scheduler is trivial: while(self.running): for mprocess in self.threads: if mprocess: try: result = mprocess.next() if self.newProcesses(result): self.activateProcesses(result): except StopIteration: self.shutdownProcess(result)
43. Postman Logic The postman is a generator One per component. It gets shutdown when it's component is shutdown Logic is trivial while 1: for link in self.linkages: if link.dataToMove(): link.moveData() yield 1
44. Co-ordinating Assistant Tracker Provides book-keeping and service lookup Services are (component, inbox) pairs Consider biological systems Nervous system is point to point Non-specific comms is useful for certain tasks. Biological systems use hormones for this purpose.
45. Writing Servers Using Generators We have components for: Listening for new connections Handling active connections Protocol handling Checking that network connections for activity This can all be wrapped up as a component itself. Code that follows lacks error checking
46. Note regarding Code Examples Please don't try to read everything! I'm going to largely whisk past them The idea is to show the common structure that results and reuse it encourages Furthermore it's designed to show that each subsection essentially remains looking single threaded despite an active server being highly concurrent
48. Listening for connections 2 Handle closed connection messages if self.dataReady("_csa_feedback"): data = self.recv("_csa_feedback") if isinstance( data, socketShutdown): theComponent,sock = self.closeSocket(data) self.send( shutdownCSA(self,(theComponent,sock)), "signal") self.removeChild(theComponent)
49. Listening for Connections 3 Handle new connection messages if self.dataReady("DataReady"): data = self.recv("DataReady") CSA = self.mkConnectedSocket(self.listener) self.send(_ki.newCSA(self, CSA), "protocolHandlerSignal") self.addChildren(CSA) self.link((CSA, "FactoryFeedback"), (self,"_csa_feedback")) self.send(newCSA(CSA, (CSA,CSA.socket)), "signal") return CSA
50. Listening for Connections 4 Everything on one slide: (too small to read probably) @wrapgenerator(component, queues=["DataReady", "_csa_feedback","protocolHandlerSignal", "signal"]) def TCPServer(self, listenport): self.listenport = listenport self.listener,junk = makeTCPServerPort(listenport, maxlisten=5) service, newlycreated = selectorComponent.getSelectorService(self.tracker) if newlycreated: self.addChildren(newSelector) self.link((self, "signal"),service) self.send(newServer(self, (self,self.listener)), "signal") yield newComponent(*(self.children)) while 1: if self.dataReady("_csa_feedback"): data = self.recv("_csa_feedback") if isinstance( data, socketShutdown): theComponent,sock = self.closeSocket(data) self.send(shutdownCSA(self, (theComponent,sock)), "signal") self.removeChild(theComponent) if self.dataReady("DataReady"): data = self.recv("DataReady") CSA = self.createConnectedSocket(self.listener) self.send(_ki.newCSA(self, CSA), "protocolHandlerSignal") self.addChildren(CSA) self.link((CSA, "FactoryFeedback"),(self,"_csa_feedback")) self.send(newCSA(CSA, (CSA,CSA.socket)), "signal") return CSA yield 1
51. Handling active connections @wrapgenerator(component, queues=["DataReady", "DataSend", "control", "outbox", "FactoryFeedback","signal"]) def ConnectedSocketAdapter(self, listenport): self.socket = listensocket while self.running: if self.dataReady("DataReady"): data = self.recv("DataReady") socketdata = _saferecv(self.socket, 1024) if (socketdata): self.send(socketdata, "outbox") if self.dataReady("DataSend"): data = self.recv("DataSend") _safesend(self.socket, data) if self.dataReady("control"): data = self.recv("control") if isinstance(data, producerFinished): self.send(socketShutdown(self,self.socket), "FactoryFeedback") self.send(socketShutdown(self), "signal")
52. Protocol handling Depends on protocol. This implements an echo protocol. Note: we've seen this today! @wrapgenerator(component) def EchoProtocol(self): "Simple data forwarding generator" while 1: if self.dataReady("inbox"): self.send("outbox",self.recv("inbox")) elif self.dataReady("control"): if self.recv("control") == "shutdown": break yield 1 self.send("signal","shutdown") yield 0
53. Select Based Connection Tracking @wrapgenerator(AdaptiveCommsComponent, queues=["inbox", "control","notify","outbox","signal"]) def selectorComponent Initialisation. (not covered, not particularly interesting) while 1: handle notify messages readables, writeables, excepts = select.select( self.readersockets, self.writersockets, [],0) for sock in readables: passBackResult = handle readable socket (sock) if passBackResult: yield passBackResult for sock in writeables: passBackResult = handle writeable socket (sock) if passBackResult: yield passBackResult
54. Connection Tracking 2 handle notify messages if self.dataReady("notify"): msg = self.recv("notify") (managingComponent, sock) = msg.object if shutdown(message): self.writersockets.remove(sock) else: if message.handlesWriting(): self.writersockets.append(sock) else: self.readersockets.append(sock) self.wireInComponent((managingComponent, sock))
56. Building a Generic Server The example that follows is that of a generic server. You provide a protocol component and port It runs a server on that port passing over all data to the supplied protocol handler This means to write a server, you just implement the protocol You don't have to worry about transport at all
57. Building a Generic Server 2 @wrapgenerator(AdaptiveCommsComponent, queues=["_oobinfo"]) def SimpleServer(self, protocol, port=1601): self.protocolClass = protocol myPLS = _ic.TCPServer(listenport=port) self.link((myPLS,"protocolHandlerSignal"), (self,"_oobinfo")) self.addChildren(myPLS) yield newComponent(myPLS) while 1: result = handle messages received via _oobinfo if result: yield result
58. Building a Generic Server 3 handle messages received via _oobinfo if self.dataReady("_oobinfo"): data = self.recv("_oobinfo") if newConnectedSocketAdaptor(data): return self.handleNewCSA(data) if socketShutdownMessage(data): self.handleClosedCSA(data)
59. Building a Generic Server 4 handling closed connections def handleClosedCSA(self,data): CSA = data.caller inboxes, outboxes, pHandler = \ self.retrieveTrackedResourceInformation(CSA) self.send("shutdown",outboxes[0]) self.removeChild(CSA) self.removeChild(pHandler) self.deleteOutbox(outboxes[0]) self.ceaseTrackingResource(CSA)
60. Building a Generic Server 5 handling closed connections def handleNewCSA(self, data): CSA = data.object pHandler = self.protocolClass() pHandlerShutdownOutbox= self.addOutbox( "protocolHandlerShutdownSignal") self.trackResourceInformation(CSA, [], [pHandlerShutdownOutbox], pHandler) self.addChildren(CSA,pHandler) self.link((CSA,"outbox"),(pHandler,"inbox")) self.link((pHandler,"outbox"),(CSA,"DataSend")) self.link((CSA,"signal"),(self,"_oobinfo")) self.link((self,pHandlerShutdownOutbox), (pHandler, "control")) if "signal" in pHandler.Outboxes: self.link((pHandler,"signal"),(CSA, "control")) return newComponent(CSA,pHandler)
61. Interlude The code on the preceding slides is the core code for a scalable server which can accept more or less any protocol This makes it ideal and simple for playing with protocols That's the idea behind the system – a little more work to make things easy for people Essentially the entire architecture is plugins
62. Implementing A TCP Client The real question is how much reuse can we get in a network client ? Lets try!
63. Implementing A TCP Client 2 @wrapgenerator(AdaptiveCommsComponent, queues=["_oobinfo"]) def TCPClient(self,host,port): message = None try: sock = socket.socket(AF_INET, SOCK_STREAM); yield 0.3 sock.setblocking(0); yield 0.6 while not Safe Connect (self,sock,(self.host, self.port)): yield 1 yield newComponent(* Setup CSA (self,sock)) while Wait CSA Close (self): self.pause() ; yield 2 result = sock.shutdown(1) ; yield 3 sock.close() ; yield 4 except socket.error, e: message = e self.send(message, "signal")
65. Implementing A TCP Client 4 Safe Connect (not so safe – error handling removed(!)...) def safeConnect(self, sock, *sockArgsList): try: sock.connect(*sockArgsList); self.connecting,self.connected=0,1 return True except socket.error, socket.msg: (errno, errmsg) = socket.msg.args if (errno==EALREADY or errno==EINPROGRESS): self.connecting=1 return False raise socket.msg
66. Implementing A TCP Client 5 Wait CSA Close if self.dataReady("_socketFeedback"): message = self.recv("_socketFeedback") if socketShutdownMessage(message): return False return True
67. Notes About the TCP Client Not really that much simpler than a TCP server really Corrollary: servers don't have to be hard Note we reused without any changes: ConnectedSocketAdaptor Selector Small components encourages reuse rather than duplication
68. Some Key Results Individual components : Simple. Complex systems via controlled composition Replacing select with poll == simple Replacing TCP with UDP similarly limits knockon changes Single layer yield encourages small focussed components
69. Some Key Results Protocol implementation can be done independent of network code Provides a toolset for experimenting with protocols, transports Very similar feel to building unix pipelines
70. Less Obvious Stuff Owning the code allows us to ship the server in the client Lightweight nature makes writing clients this way simple Merging code from other projects written in python is relatively simple: Add in a number of yields in strategic points Add in support for communication/control
71. More Less Obvious Stuff Interesting possibility: Client tuner utility with embedded RTSP/RTP server and Bit Torrent client. Cursory looks suggest that the component nature makes this a realistic possibility. High decoupling: parallelism across threads, CPUs, machines practical. Resulting structure lends itself to optimisation in hardware .
72. Kamaelia: Ongoing Work Lots of things to do: Integration of RTP/RTSP components Stream oriented UDP Implement better a/v delivery protocol Optimisation of bottlenecks (linkages) Use python features to remove cruft Investigate Twisted “ integration” with Lots more!
73. RTP is not a Streaming Protocol RTP isn't designed for unidirectional streaming. Does somethings that break hard in streaming Misses out key things streaming protocols do Stream thinning Stream indexing Define a mechanism for going over TCP.
74. RTP For Streaming Everyone (?) who does open standards streaming and doesn't do HTTP does RTP Why? Because it's the only open standard that comes close to the requirements So let's look at streaming
76. What problem does streaming solve? Audio/Video delivery? Quickest way of delivery? Digital Restrictions Management - preventing unauthorised copies?
77. What problem does streaming solve? Audio/Video delivery? No, HTTP downloading handles that fine Quickest way of delivery? No, again, HTTP Digital Restrictions Management - preventing unauthorised copies? No – a lot of streaming is over HTTP
78. Characteristics of streaming? Capped bitrate connection Expected to stay at that capped bitrate Play as you recieve On demand, play now. Time oriented seeking Fine grained, but not conceptual - why not act, chapter, content oriented?
79. What problem does streaming solve? OK, an answer: Timely delivery of content down pipes normally too small to take it That's why we have capped bit rates That's why we use compression. Timely means people want to watch now, rather than wait for higher quality
80. Observations High compression, packet loss == unwatchable video Buffering common to mask transport errors Devices have lots of local store If bitrate of video < capacity of client link, why not switch to download faster and large local cache model? ie Actually switch protocols?
81. Observations 2 If bitrate >> capacity of link - why deny access - which not allow switch to maximum speed download to local cache model ? ie allow people to choose quality or timeliness and then act accordingly What about letting multiple source feed a single stream? RTP doesn't really fit these models...
82. Good Transport Protocol Aspects Content agnostic - you can send anything Fully defines all aspects of usage Simple to use Changing content doesn't require understanding the content structure P2P is good at this HTTP is good at this RTP sucks at this
83. RTP Problems (for streaming) Half a transport protocol Doesn't define how to send anything Defines protocol profiles for lots of content types Lots of content types missing containers Vorbis, Speex, Theora, Dirac, ... Vorbis has an (expired?) IETF draft Not anyone's "fault" - protocol issue Doesn't define framing over TCP
84. RTP Problems (for streaming) Time based, not content based RTCP has issues large multicast groups. This is due to RTP not really having the concept of unidirectional sessions Multicast has similar issues generally Or you don't implement the protocol Lots of features not needed in streaming Designed for a/v conferencing
85. P2P Issues Unknown quantity really. No, really. Remember we're after how to serve ~ 7Tbit/s How many 256Kbit/s DSL uplinks? Can't do it just from the edge Can't do it just from the centre Legitimate Bit Torrent use – eg linux distros has proven moderate sized audience can work Large numbers grabbing lots of things?
86. P2P Issues 2 How do ensure producers of art get paid? We all want the "next Matrix"? We all want the "next Really Cool Film/Song" Films are a real problem – not a one man job, nor really suitable for open source style Current approaches aren't working Do we really want people breaking laws? Do we really want to terrify the **AA down roads no-one wants to go?
87. P2P Issues 3 Copyright infringement isn't really a P2P issue P2P makes it simpler because P2P is an effective user controlled distribution system I grew up with Star Wars & Superman as a kid My kids get LOTR & Spiderman P2P has the power to ensure that your kids don't get their epics
88. P2P Issues 4 It's a social responsibility issue. We can choose to define protocols that will adhere to rights. Systems that look at creative commons licenses embedded in content are a GREAT start Maybe new protocols could consider this?
89. Need for simpler Media Storage RTP has to understand every media wrapper, and often codec details Users can ask via RTSP for specific time indexes. This must be satisfied. Parsing thousands of files at once is VERY expensive Darwin streaming server hints the content Patented by apple...
90. Simpler Media Storage Rather than leave content unchanged, why not stuff it into a standard format? Like usenet servers do Like XML repositories Like DNS defines Having a shared approach makes reuse simpler Note, I'm talking specifically for server use
91. Simpler Server Storage One obvious approach: Define in header “this is maximum bit rate”, “this is time granularity for index points” Instantly have mapping of bytes and time Pad chunks for VBR Content agnostic Could provide semantic -> time mapping Very similar to IFF formats (19 years old)
92.
93. Need for a New Streaming Protocol Wish list (partial): Transmission between a/v caches Transmission mode migration Downloading should allow swarming behaviour Similarly for streaming where practical Stream thinning Semantic indexing as well as time Mechanisms for arbitrary content distribution
94. Need for a New Streaming Protocol Wish list (partial): (cont) Provide mechanisms for arbitrary content distribution Mechanisms similar to those in Scattercast would be useful when combined with P2P for a self organising media distribution network Too large a topic for this talk
95. Copyright License This presentation is part of the Kamaelia Project, and was presented at Firenze World Vision: Free Streaming 2004 session, organised by the Media Innovation Unit of Firenze Technologia As such it and the code it contains may be used under the terms of the LGPL (v2.1), MPL (v1.1), or GPL (v2.0) http://www.opensource.org/licenses/mozilla1.1.php http://www.gnu.org/copyleft/lesser.html http://www.gnu.org/licenses/gpl.html (This is due to the large code element in this presentation)