SlideShare a Scribd company logo
Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Projects
Testing, CI Gating & Community
Fast Feedback: The Challenge
of Integration Projects
Nikolas Hermanns
Jose Lausuch
Outline
• Challenges of the SDNVPN project
• XCI patchset verification
• Virtual testing setup
• Functest test challenge
Integration Project (e.g. SDNVPN)
ODL SDNVPN Fuel
Apex
• Consume many different
Project
• Different release dates
• Different Bug tracking
• Different source code handling
• Different programming language
• Etc.
Challenge of Integration Projects (e.g. ODL in OPNFV)
ODL-Master
time
ODL-Release
OPNFV-Integration
•Bug finding
ODL-Release
Bug1
ODL-Stable
Bug2
Bug1 Bug2
ODL-Master
x
• Bug 1 found and
solved
• Bug 2 found but not
valid for master
anymore
• Bug 2 was not worth
the effort
• Bug 3 will not be
found until next
release
ODL is consumed too late!
Bug3
Challenge of Integration Projects (e.g. ODL in OPNFV)
Developer A
Bug 1
time
ODL-Release
ODL-Release
Bug1
Integrator A
• Dev A expert in Part of
ODL
• Int A expert in Fuel but
doesn’t know much about
ODL code base
• Int A needs to find and
understand the bug!
• Int A need to talk to Dev A
and explain the found bug
• Might be Dev A does not
even remember
Solutions
• Solution 1: More test cases in consumed project
• Integrator needs to convince and explain the test to Developer
• Test case might be too E2Eish to be created in consumed project
• Solution 2: XCI patchset verification
• Automated integration (Consume latest state)
• Fast feedback (reducing from months to < 1 hour)
• No interaction from integrator
New
Patchset
(Commit)
Create Virtual
Testing Platform
Download/Install
new artifact
Use standard
testing framework
e.g. Functest
Build ODL
artifact
10 min 15 min 35 min
Unit test
Installer Project
e.g. Apex
Snapshot
5 min
Positive -> Merge commit
Negative -> Prevent merge
Investigate the
logs/Find bugs
ODL OPNFV
Feedback
Bare Metal (48 threads/256 GB RAM/500 GB SSD)
vJenkins
slave
vController vCompute
vCompute
vCompute
Thread pool – 8 vCPU
 10 GB Diskspace
 40 GB RAM
 Own OVS networks
vJenkins
slave
vController vCompute
vController vCompute
Snapshot
Controller
vController
vController
vController
Snapshot
Compute
vCompute
vCompute
vCompute
~ 100 GB Diskspace
x6
• RAM and CPU limiting factor
• Very low disk IO cause no
installation process
• ~2 mins until Openstack/ODL
services up
• Easy exchange of snapshot
Using the Snapshot
vController
vController
vController
Snapshot
Controller
vController
vController
vController
~ 100 GB Diskspace
>1GB disk consumption per vHost
• Using the base concept of qemu-img
CPU pinning
• 4
• 3 8 vCPUs ? -> YES! – sharing is the key.
• 1
time
vJenkins
slave
vController
vCompute
CPU-Load
Jenkins-
Slave
running
8 vCPU
Start
vCmp
vCtrl
Started
vCmp
vCtrl
Testing
started
Testing
more
API
Under-
provisioned
Cpu pinning
• Other vPod do not get effected
Networking
• OVS for complete
isolation
• vPods using same
IPs so every vPod
has its own
networks
• Jenkins-slave for
Gateway doing NAT
vJenkins
slave
vController vCompute
Bare metal
host
admin
storage
private
public
host
NAT
internet
Bare Metal (48 threads/256 GB RAM/500 GB SSD)
vJenkins
slave
vController vCompute
vCompute
vCompute
Thread pool – 8 vCPU
 10 GB Diskspace
 40 GB RAM
 Own OVS networks
vJenkins
slave
vController vCompute
vController vCompute
Snapshot
Controller
vController
vController
vController
Snapshot
Compute
vCompute
vCompute
vCompute
~ 100 GB Diskspace
x6
• RAM and CPU limiting factor
• Very low disk IO cause no
installation process
• ~2 mins until Openstack/ODL
services up
• Easy exchange of snapshot
Example of application: Functest
FUNCTEST
pep8
unit
tests
pylint
docs
py35
py27
test new
code on
OPNFV
deployment
Example of application: Functest
• Functest bug might break CI
• Needs a way to test each change before merging
• Fast feedback to the commit owner
• Easy to validate framework changes on Functest
• Can be tested on a basic nosdn-nofeature scenario
Example of application: Functest
Challenges:
• Changes on Feature tests  need specific scenario deployed
• Increase verification time of each patchset ( + time to build a
docker image)
• Identify where the change is (framework, test case, …) to trigger
an appropriate test case
Thank You
Nikolas Hermanns Nikolas.Hermanns@ericsson.com
Jose Lausuch jose.lausuch@ericsson.com
Ad

More Related Content

What's hot (20)

Demo how to efficiently evaluate nf-vi performance by leveraging opnfv testi...
Demo  how to efficiently evaluate nf-vi performance by leveraging opnfv testi...Demo  how to efficiently evaluate nf-vi performance by leveraging opnfv testi...
Demo how to efficiently evaluate nf-vi performance by leveraging opnfv testi...
OPNFV
 
Meetup 23 - 02 - OVN - The future of networking in OpenStack
Meetup 23 - 02 - OVN - The future of networking in OpenStackMeetup 23 - 02 - OVN - The future of networking in OpenStack
Meetup 23 - 02 - OVN - The future of networking in OpenStack
Vietnam Open Infrastructure User Group
 
The Open vSwitch and OVN Projects
The Open vSwitch and OVN ProjectsThe Open vSwitch and OVN Projects
The Open vSwitch and OVN Projects
LinuxCon ContainerCon CloudOpen China
 
See what happened with real time kvm when building real time cloud pezhang@re...
See what happened with real time kvm when building real time cloud pezhang@re...See what happened with real time kvm when building real time cloud pezhang@re...
See what happened with real time kvm when building real time cloud pezhang@re...
LinuxCon ContainerCon CloudOpen China
 
Software-defined migration how to migrate bunch of v-ms and volumes within a...
Software-defined migration  how to migrate bunch of v-ms and volumes within a...Software-defined migration  how to migrate bunch of v-ms and volumes within a...
Software-defined migration how to migrate bunch of v-ms and volumes within a...
OPNFV
 
My network functions are virtualized, but are they cloud-ready
My network functions are virtualized, but are they cloud-readyMy network functions are virtualized, but are they cloud-ready
My network functions are virtualized, but are they cloud-ready
OPNFV
 
Get a Taste of 1 k+ Nodes by a Handful of Servers
Get a Taste of 1 k+ Nodes by a Handful of Servers Get a Taste of 1 k+ Nodes by a Handful of Servers
Get a Taste of 1 k+ Nodes by a Handful of Servers
LinuxCon ContainerCon CloudOpen China
 
Deploy TOSCA Network Functions Virtualization (NFV) Workloads in OpenStack
Deploy TOSCA Network Functions Virtualization (NFV) Workloads in OpenStackDeploy TOSCA Network Functions Virtualization (NFV) Workloads in OpenStack
Deploy TOSCA Network Functions Virtualization (NFV) Workloads in OpenStack
Sahdev Zala
 
Supercomputing by API: Connecting Modern Web Apps to HPC
Supercomputing by API: Connecting Modern Web Apps to HPCSupercomputing by API: Connecting Modern Web Apps to HPC
Supercomputing by API: Connecting Modern Web Apps to HPC
OpenStack
 
OpenStack High Availability
OpenStack High AvailabilityOpenStack High Availability
OpenStack High Availability
Jakub Pavlik
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User Group
HungWei Chiu
 
Ovn vancouver
Ovn vancouverOvn vancouver
Ovn vancouver
Mason Mei
 
OVN: Scaleable Virtual Networking for Open vSwitch
OVN: Scaleable Virtual Networking for Open vSwitchOVN: Scaleable Virtual Networking for Open vSwitch
OVN: Scaleable Virtual Networking for Open vSwitch
mestery
 
Running Neutron at Scale - Gal Sagie & Eran Gampel - OpenStack Day Israel 2016
Running Neutron at Scale - Gal Sagie & Eran Gampel - OpenStack Day Israel 2016Running Neutron at Scale - Gal Sagie & Eran Gampel - OpenStack Day Israel 2016
Running Neutron at Scale - Gal Sagie & Eran Gampel - OpenStack Day Israel 2016
Cloud Native Day Tel Aviv
 
IP Virtual Server(IPVS) 101
IP Virtual Server(IPVS) 101IP Virtual Server(IPVS) 101
IP Virtual Server(IPVS) 101
HungWei Chiu
 
[OpenInfra Days Korea 2018] Day 2 - E6: "SONA: ONOS SDN Controller 기반 OpenSta...
[OpenInfra Days Korea 2018] Day 2 - E6: "SONA: ONOS SDN Controller 기반 OpenSta...[OpenInfra Days Korea 2018] Day 2 - E6: "SONA: ONOS SDN Controller 기반 OpenSta...
[OpenInfra Days Korea 2018] Day 2 - E6: "SONA: ONOS SDN Controller 기반 OpenSta...
OpenStack Korea Community
 
KVM Enhancements for OPNFV
KVM Enhancements for OPNFVKVM Enhancements for OPNFV
KVM Enhancements for OPNFV
OPNFV
 
Being Brave: Deploying OpenStack from Master
Being Brave: Deploying OpenStack from MasterBeing Brave: Deploying OpenStack from Master
Being Brave: Deploying OpenStack from Master
OPNFV
 
iptables and Kubernetes
iptables and Kubernetesiptables and Kubernetes
iptables and Kubernetes
HungWei Chiu
 
Opensource approach to design and deployment of Microservices based VNF
Opensource approach to design and deployment of Microservices based VNFOpensource approach to design and deployment of Microservices based VNF
Opensource approach to design and deployment of Microservices based VNF
Michelle Holley
 
Demo how to efficiently evaluate nf-vi performance by leveraging opnfv testi...
Demo  how to efficiently evaluate nf-vi performance by leveraging opnfv testi...Demo  how to efficiently evaluate nf-vi performance by leveraging opnfv testi...
Demo how to efficiently evaluate nf-vi performance by leveraging opnfv testi...
OPNFV
 
See what happened with real time kvm when building real time cloud pezhang@re...
See what happened with real time kvm when building real time cloud pezhang@re...See what happened with real time kvm when building real time cloud pezhang@re...
See what happened with real time kvm when building real time cloud pezhang@re...
LinuxCon ContainerCon CloudOpen China
 
Software-defined migration how to migrate bunch of v-ms and volumes within a...
Software-defined migration  how to migrate bunch of v-ms and volumes within a...Software-defined migration  how to migrate bunch of v-ms and volumes within a...
Software-defined migration how to migrate bunch of v-ms and volumes within a...
OPNFV
 
My network functions are virtualized, but are they cloud-ready
My network functions are virtualized, but are they cloud-readyMy network functions are virtualized, but are they cloud-ready
My network functions are virtualized, but are they cloud-ready
OPNFV
 
Deploy TOSCA Network Functions Virtualization (NFV) Workloads in OpenStack
Deploy TOSCA Network Functions Virtualization (NFV) Workloads in OpenStackDeploy TOSCA Network Functions Virtualization (NFV) Workloads in OpenStack
Deploy TOSCA Network Functions Virtualization (NFV) Workloads in OpenStack
Sahdev Zala
 
Supercomputing by API: Connecting Modern Web Apps to HPC
Supercomputing by API: Connecting Modern Web Apps to HPCSupercomputing by API: Connecting Modern Web Apps to HPC
Supercomputing by API: Connecting Modern Web Apps to HPC
OpenStack
 
OpenStack High Availability
OpenStack High AvailabilityOpenStack High Availability
OpenStack High Availability
Jakub Pavlik
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User Group
HungWei Chiu
 
Ovn vancouver
Ovn vancouverOvn vancouver
Ovn vancouver
Mason Mei
 
OVN: Scaleable Virtual Networking for Open vSwitch
OVN: Scaleable Virtual Networking for Open vSwitchOVN: Scaleable Virtual Networking for Open vSwitch
OVN: Scaleable Virtual Networking for Open vSwitch
mestery
 
Running Neutron at Scale - Gal Sagie & Eran Gampel - OpenStack Day Israel 2016
Running Neutron at Scale - Gal Sagie & Eran Gampel - OpenStack Day Israel 2016Running Neutron at Scale - Gal Sagie & Eran Gampel - OpenStack Day Israel 2016
Running Neutron at Scale - Gal Sagie & Eran Gampel - OpenStack Day Israel 2016
Cloud Native Day Tel Aviv
 
IP Virtual Server(IPVS) 101
IP Virtual Server(IPVS) 101IP Virtual Server(IPVS) 101
IP Virtual Server(IPVS) 101
HungWei Chiu
 
[OpenInfra Days Korea 2018] Day 2 - E6: "SONA: ONOS SDN Controller 기반 OpenSta...
[OpenInfra Days Korea 2018] Day 2 - E6: "SONA: ONOS SDN Controller 기반 OpenSta...[OpenInfra Days Korea 2018] Day 2 - E6: "SONA: ONOS SDN Controller 기반 OpenSta...
[OpenInfra Days Korea 2018] Day 2 - E6: "SONA: ONOS SDN Controller 기반 OpenSta...
OpenStack Korea Community
 
KVM Enhancements for OPNFV
KVM Enhancements for OPNFVKVM Enhancements for OPNFV
KVM Enhancements for OPNFV
OPNFV
 
Being Brave: Deploying OpenStack from Master
Being Brave: Deploying OpenStack from MasterBeing Brave: Deploying OpenStack from Master
Being Brave: Deploying OpenStack from Master
OPNFV
 
iptables and Kubernetes
iptables and Kubernetesiptables and Kubernetes
iptables and Kubernetes
HungWei Chiu
 
Opensource approach to design and deployment of Microservices based VNF
Opensource approach to design and deployment of Microservices based VNFOpensource approach to design and deployment of Microservices based VNF
Opensource approach to design and deployment of Microservices based VNF
Michelle Holley
 

Similar to Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Projects (20)

Neutron CI Run on Docker
Neutron CI Run on DockerNeutron CI Run on Docker
Neutron CI Run on Docker
Hirofumi Ichihara
 
Java-light-speed NebraskaCode.pdf
Java-light-speed NebraskaCode.pdfJava-light-speed NebraskaCode.pdf
Java-light-speed NebraskaCode.pdf
RichHagarty
 
2014 11-05 hpcac-kniep_christian_dockermpi
2014 11-05 hpcac-kniep_christian_dockermpi2014 11-05 hpcac-kniep_christian_dockermpi
2014 11-05 hpcac-kniep_christian_dockermpi
QNIB Solutions
 
Использование AzureDevOps при разработке микросервисных приложений
Использование AzureDevOps при разработке микросервисных приложенийИспользование AzureDevOps при разработке микросервисных приложений
Использование AzureDevOps при разработке микросервисных приложений
Vitebsk Miniq
 
MoldCamp - multidimentional testing workflow. CIBox.
MoldCamp  - multidimentional testing workflow. CIBox.MoldCamp  - multidimentional testing workflow. CIBox.
MoldCamp - multidimentional testing workflow. CIBox.
Andrii Podanenko
 
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
STePINForum
 
DrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration ToolboxDrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration Toolbox
Andrii Podanenko
 
Cross Community CI project
Cross Community CI projectCross Community CI project
Cross Community CI project
Victor Morales
 
To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On PremTo Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
CloudOps2005
 
Talk on PHP Day Uruguay about Docker
Talk on PHP Day Uruguay about DockerTalk on PHP Day Uruguay about Docker
Talk on PHP Day Uruguay about Docker
Wellington Silva
 
Secrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on KubernetesSecrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on Kubernetes
Bruno Borges
 
Kubernetes at NU.nl (Kubernetes meetup 2019-09-05)
Kubernetes at NU.nl   (Kubernetes meetup 2019-09-05)Kubernetes at NU.nl   (Kubernetes meetup 2019-09-05)
Kubernetes at NU.nl (Kubernetes meetup 2019-09-05)
Tibo Beijen
 
Enable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunEnable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zun
heut2008
 
Taking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and DecideTaking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and Decide
Bret Fisher
 
Taking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and DecideTaking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and Decide
Docker, Inc.
 
Detailed Introduction To Docker
Detailed Introduction To DockerDetailed Introduction To Docker
Detailed Introduction To Docker
nklmish
 
Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications
OpenEBS
 
PuppetConf 2016: Changing the Engine While in Flight – Neil Armitage, VMware
PuppetConf 2016: Changing the Engine While in Flight – Neil Armitage, VMwarePuppetConf 2016: Changing the Engine While in Flight – Neil Armitage, VMware
PuppetConf 2016: Changing the Engine While in Flight – Neil Armitage, VMware
Puppet
 
Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community
 
Docking postgres
Docking postgresDocking postgres
Docking postgres
rycamor
 
Java-light-speed NebraskaCode.pdf
Java-light-speed NebraskaCode.pdfJava-light-speed NebraskaCode.pdf
Java-light-speed NebraskaCode.pdf
RichHagarty
 
2014 11-05 hpcac-kniep_christian_dockermpi
2014 11-05 hpcac-kniep_christian_dockermpi2014 11-05 hpcac-kniep_christian_dockermpi
2014 11-05 hpcac-kniep_christian_dockermpi
QNIB Solutions
 
Использование AzureDevOps при разработке микросервисных приложений
Использование AzureDevOps при разработке микросервисных приложенийИспользование AzureDevOps при разработке микросервисных приложений
Использование AzureDevOps при разработке микросервисных приложений
Vitebsk Miniq
 
MoldCamp - multidimentional testing workflow. CIBox.
MoldCamp  - multidimentional testing workflow. CIBox.MoldCamp  - multidimentional testing workflow. CIBox.
MoldCamp - multidimentional testing workflow. CIBox.
Andrii Podanenko
 
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
STePINForum
 
DrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration ToolboxDrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration Toolbox
Andrii Podanenko
 
Cross Community CI project
Cross Community CI projectCross Community CI project
Cross Community CI project
Victor Morales
 
To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On PremTo Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
CloudOps2005
 
Talk on PHP Day Uruguay about Docker
Talk on PHP Day Uruguay about DockerTalk on PHP Day Uruguay about Docker
Talk on PHP Day Uruguay about Docker
Wellington Silva
 
Secrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on KubernetesSecrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on Kubernetes
Bruno Borges
 
Kubernetes at NU.nl (Kubernetes meetup 2019-09-05)
Kubernetes at NU.nl   (Kubernetes meetup 2019-09-05)Kubernetes at NU.nl   (Kubernetes meetup 2019-09-05)
Kubernetes at NU.nl (Kubernetes meetup 2019-09-05)
Tibo Beijen
 
Enable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunEnable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zun
heut2008
 
Taking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and DecideTaking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and Decide
Bret Fisher
 
Taking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and DecideTaking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and Decide
Docker, Inc.
 
Detailed Introduction To Docker
Detailed Introduction To DockerDetailed Introduction To Docker
Detailed Introduction To Docker
nklmish
 
Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications
OpenEBS
 
PuppetConf 2016: Changing the Engine While in Flight – Neil Armitage, VMware
PuppetConf 2016: Changing the Engine While in Flight – Neil Armitage, VMwarePuppetConf 2016: Changing the Engine While in Flight – Neil Armitage, VMware
PuppetConf 2016: Changing the Engine While in Flight – Neil Armitage, VMware
Puppet
 
Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community
 
Docking postgres
Docking postgresDocking postgres
Docking postgres
rycamor
 
Ad

More from OPNFV (20)

How to Reuse OPNFV Testing Components in Telco Validation Chain
How to Reuse OPNFV Testing Components in Telco Validation ChainHow to Reuse OPNFV Testing Components in Telco Validation Chain
How to Reuse OPNFV Testing Components in Telco Validation Chain
OPNFV
 
Energy Audit aaS with OPNFV
Energy Audit aaS with OPNFVEnergy Audit aaS with OPNFV
Energy Audit aaS with OPNFV
OPNFV
 
Hands-On Testing: How to Integrate Tests in OPNFV
Hands-On Testing: How to Integrate Tests in OPNFVHands-On Testing: How to Integrate Tests in OPNFV
Hands-On Testing: How to Integrate Tests in OPNFV
OPNFV
 
Storage Performance Indicators - Powered by StorPerf and QTIP
Storage Performance Indicators - Powered by StorPerf and QTIPStorage Performance Indicators - Powered by StorPerf and QTIP
Storage Performance Indicators - Powered by StorPerf and QTIP
OPNFV
 
Big Data for Testing - Heading for Post Process and Analytics
Big Data for Testing - Heading for Post Process and AnalyticsBig Data for Testing - Heading for Post Process and Analytics
Big Data for Testing - Heading for Post Process and Analytics
OPNFV
 
Upstream Testing Collaboration
Upstream Testing Collaboration Upstream Testing Collaboration
Upstream Testing Collaboration
OPNFV
 
Enabling Carrier-Grade Availability Within a Cloud Infrastructure
Enabling Carrier-Grade Availability Within a Cloud InfrastructureEnabling Carrier-Grade Availability Within a Cloud Infrastructure
Enabling Carrier-Grade Availability Within a Cloud Infrastructure
OPNFV
 
Learnings From the First Year of the OPNFV Internship Program
Learnings From the First Year of the OPNFV Internship ProgramLearnings From the First Year of the OPNFV Internship Program
Learnings From the First Year of the OPNFV Internship Program
OPNFV
 
OPNFV and OCP: Perfect Together
OPNFV and OCP: Perfect TogetherOPNFV and OCP: Perfect Together
OPNFV and OCP: Perfect Together
OPNFV
 
The Return of QTIP, from Brahmaputra to Danube
The Return of QTIP, from Brahmaputra to DanubeThe Return of QTIP, from Brahmaputra to Danube
The Return of QTIP, from Brahmaputra to Danube
OPNFV
 
Distributed vnf management architecture and use-cases
Distributed vnf management  architecture and use-casesDistributed vnf management  architecture and use-cases
Distributed vnf management architecture and use-cases
OPNFV
 
Securing your nfv and sdn integrated open stack cloud- challenges, use-cases ...
Securing your nfv and sdn integrated open stack cloud- challenges, use-cases ...Securing your nfv and sdn integrated open stack cloud- challenges, use-cases ...
Securing your nfv and sdn integrated open stack cloud- challenges, use-cases ...
OPNFV
 
Challenge in asia region connecting each testbed and poc of distributed nfv ...
Challenge in asia region  connecting each testbed and poc of distributed nfv ...Challenge in asia region  connecting each testbed and poc of distributed nfv ...
Challenge in asia region connecting each testbed and poc of distributed nfv ...
OPNFV
 
Challenges in testing for composite vim platforms
Challenges in testing for composite vim platformsChallenges in testing for composite vim platforms
Challenges in testing for composite vim platforms
OPNFV
 
Crossing the river by feeling the stones from legacy to cloud native applica...
Crossing the river by feeling the stones  from legacy to cloud native applica...Crossing the river by feeling the stones  from legacy to cloud native applica...
Crossing the river by feeling the stones from legacy to cloud native applica...
OPNFV
 
OPNFV with 5G Applications
OPNFV with 5G ApplicationsOPNFV with 5G Applications
OPNFV with 5G Applications
OPNFV
 
Test and perspectives on nfvi from china unicom sdn nfv lab
Test and perspectives on nfvi from china unicom sdn nfv labTest and perspectives on nfvi from china unicom sdn nfv lab
Test and perspectives on nfvi from china unicom sdn nfv lab
OPNFV
 
Automatic Integration, Testing and Certification of NFV in China Mobile
Automatic Integration, Testing and Certification of NFV in China MobileAutomatic Integration, Testing and Certification of NFV in China Mobile
Automatic Integration, Testing and Certification of NFV in China Mobile
OPNFV
 
Faster, Higher, Stronger – Accelerating Fault Management to the Next Level
Faster, Higher, Stronger – Accelerating Fault Management to the Next LevelFaster, Higher, Stronger – Accelerating Fault Management to the Next Level
Faster, Higher, Stronger – Accelerating Fault Management to the Next Level
OPNFV
 
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.ioFast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.io
OPNFV
 
How to Reuse OPNFV Testing Components in Telco Validation Chain
How to Reuse OPNFV Testing Components in Telco Validation ChainHow to Reuse OPNFV Testing Components in Telco Validation Chain
How to Reuse OPNFV Testing Components in Telco Validation Chain
OPNFV
 
Energy Audit aaS with OPNFV
Energy Audit aaS with OPNFVEnergy Audit aaS with OPNFV
Energy Audit aaS with OPNFV
OPNFV
 
Hands-On Testing: How to Integrate Tests in OPNFV
Hands-On Testing: How to Integrate Tests in OPNFVHands-On Testing: How to Integrate Tests in OPNFV
Hands-On Testing: How to Integrate Tests in OPNFV
OPNFV
 
Storage Performance Indicators - Powered by StorPerf and QTIP
Storage Performance Indicators - Powered by StorPerf and QTIPStorage Performance Indicators - Powered by StorPerf and QTIP
Storage Performance Indicators - Powered by StorPerf and QTIP
OPNFV
 
Big Data for Testing - Heading for Post Process and Analytics
Big Data for Testing - Heading for Post Process and AnalyticsBig Data for Testing - Heading for Post Process and Analytics
Big Data for Testing - Heading for Post Process and Analytics
OPNFV
 
Upstream Testing Collaboration
Upstream Testing Collaboration Upstream Testing Collaboration
Upstream Testing Collaboration
OPNFV
 
Enabling Carrier-Grade Availability Within a Cloud Infrastructure
Enabling Carrier-Grade Availability Within a Cloud InfrastructureEnabling Carrier-Grade Availability Within a Cloud Infrastructure
Enabling Carrier-Grade Availability Within a Cloud Infrastructure
OPNFV
 
Learnings From the First Year of the OPNFV Internship Program
Learnings From the First Year of the OPNFV Internship ProgramLearnings From the First Year of the OPNFV Internship Program
Learnings From the First Year of the OPNFV Internship Program
OPNFV
 
OPNFV and OCP: Perfect Together
OPNFV and OCP: Perfect TogetherOPNFV and OCP: Perfect Together
OPNFV and OCP: Perfect Together
OPNFV
 
The Return of QTIP, from Brahmaputra to Danube
The Return of QTIP, from Brahmaputra to DanubeThe Return of QTIP, from Brahmaputra to Danube
The Return of QTIP, from Brahmaputra to Danube
OPNFV
 
Distributed vnf management architecture and use-cases
Distributed vnf management  architecture and use-casesDistributed vnf management  architecture and use-cases
Distributed vnf management architecture and use-cases
OPNFV
 
Securing your nfv and sdn integrated open stack cloud- challenges, use-cases ...
Securing your nfv and sdn integrated open stack cloud- challenges, use-cases ...Securing your nfv and sdn integrated open stack cloud- challenges, use-cases ...
Securing your nfv and sdn integrated open stack cloud- challenges, use-cases ...
OPNFV
 
Challenge in asia region connecting each testbed and poc of distributed nfv ...
Challenge in asia region  connecting each testbed and poc of distributed nfv ...Challenge in asia region  connecting each testbed and poc of distributed nfv ...
Challenge in asia region connecting each testbed and poc of distributed nfv ...
OPNFV
 
Challenges in testing for composite vim platforms
Challenges in testing for composite vim platformsChallenges in testing for composite vim platforms
Challenges in testing for composite vim platforms
OPNFV
 
Crossing the river by feeling the stones from legacy to cloud native applica...
Crossing the river by feeling the stones  from legacy to cloud native applica...Crossing the river by feeling the stones  from legacy to cloud native applica...
Crossing the river by feeling the stones from legacy to cloud native applica...
OPNFV
 
OPNFV with 5G Applications
OPNFV with 5G ApplicationsOPNFV with 5G Applications
OPNFV with 5G Applications
OPNFV
 
Test and perspectives on nfvi from china unicom sdn nfv lab
Test and perspectives on nfvi from china unicom sdn nfv labTest and perspectives on nfvi from china unicom sdn nfv lab
Test and perspectives on nfvi from china unicom sdn nfv lab
OPNFV
 
Automatic Integration, Testing and Certification of NFV in China Mobile
Automatic Integration, Testing and Certification of NFV in China MobileAutomatic Integration, Testing and Certification of NFV in China Mobile
Automatic Integration, Testing and Certification of NFV in China Mobile
OPNFV
 
Faster, Higher, Stronger – Accelerating Fault Management to the Next Level
Faster, Higher, Stronger – Accelerating Fault Management to the Next LevelFaster, Higher, Stronger – Accelerating Fault Management to the Next Level
Faster, Higher, Stronger – Accelerating Fault Management to the Next Level
OPNFV
 
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.ioFast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.io
OPNFV
 
Ad

Recently uploaded (20)

Why Orangescrum Is a Game Changer for Construction Companies in 2025
Why Orangescrum Is a Game Changer for Construction Companies in 2025Why Orangescrum Is a Game Changer for Construction Companies in 2025
Why Orangescrum Is a Game Changer for Construction Companies in 2025
Orangescrum
 
Automation Techniques in RPA - UiPath Certificate
Automation Techniques in RPA - UiPath CertificateAutomation Techniques in RPA - UiPath Certificate
Automation Techniques in RPA - UiPath Certificate
VICTOR MAESTRE RAMIREZ
 
Maxon CINEMA 4D 2025 Crack FREE Download LINK
Maxon CINEMA 4D 2025 Crack FREE Download LINKMaxon CINEMA 4D 2025 Crack FREE Download LINK
Maxon CINEMA 4D 2025 Crack FREE Download LINK
younisnoman75
 
EASEUS Partition Master Crack + License Code
EASEUS Partition Master Crack + License CodeEASEUS Partition Master Crack + License Code
EASEUS Partition Master Crack + License Code
aneelaramzan63
 
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Dele Amefo
 
Meet the Agents: How AI Is Learning to Think, Plan, and Collaborate
Meet the Agents: How AI Is Learning to Think, Plan, and CollaborateMeet the Agents: How AI Is Learning to Think, Plan, and Collaborate
Meet the Agents: How AI Is Learning to Think, Plan, and Collaborate
Maxim Salnikov
 
Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...
Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...
Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...
Ranjan Baisak
 
Avast Premium Security Crack FREE Latest Version 2025
Avast Premium Security Crack FREE Latest Version 2025Avast Premium Security Crack FREE Latest Version 2025
Avast Premium Security Crack FREE Latest Version 2025
mu394968
 
Exploring Wayland: A Modern Display Server for the Future
Exploring Wayland: A Modern Display Server for the FutureExploring Wayland: A Modern Display Server for the Future
Exploring Wayland: A Modern Display Server for the Future
ICS
 
The Significance of Hardware in Information Systems.pdf
The Significance of Hardware in Information Systems.pdfThe Significance of Hardware in Information Systems.pdf
The Significance of Hardware in Information Systems.pdf
drewplanas10
 
Douwan Crack 2025 new verson+ License code
Douwan Crack 2025 new verson+ License codeDouwan Crack 2025 new verson+ License code
Douwan Crack 2025 new verson+ License code
aneelaramzan63
 
How to Optimize Your AWS Environment for Improved Cloud Performance
How to Optimize Your AWS Environment for Improved Cloud PerformanceHow to Optimize Your AWS Environment for Improved Cloud Performance
How to Optimize Your AWS Environment for Improved Cloud Performance
ThousandEyes
 
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Andre Hora
 
Explaining GitHub Actions Failures with Large Language Models Challenges, In...
Explaining GitHub Actions Failures with Large Language Models Challenges, In...Explaining GitHub Actions Failures with Large Language Models Challenges, In...
Explaining GitHub Actions Failures with Large Language Models Challenges, In...
ssuserb14185
 
Top 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docxTop 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docx
Portli
 
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Lionel Briand
 
Who Watches the Watchmen (SciFiDevCon 2025)
Who Watches the Watchmen (SciFiDevCon 2025)Who Watches the Watchmen (SciFiDevCon 2025)
Who Watches the Watchmen (SciFiDevCon 2025)
Allon Mureinik
 
PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025
mu394968
 
Download YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full ActivatedDownload YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full Activated
saniamalik72555
 
Secure Test Infrastructure: The Backbone of Trustworthy Software Development
Secure Test Infrastructure: The Backbone of Trustworthy Software DevelopmentSecure Test Infrastructure: The Backbone of Trustworthy Software Development
Secure Test Infrastructure: The Backbone of Trustworthy Software Development
Shubham Joshi
 
Why Orangescrum Is a Game Changer for Construction Companies in 2025
Why Orangescrum Is a Game Changer for Construction Companies in 2025Why Orangescrum Is a Game Changer for Construction Companies in 2025
Why Orangescrum Is a Game Changer for Construction Companies in 2025
Orangescrum
 
Automation Techniques in RPA - UiPath Certificate
Automation Techniques in RPA - UiPath CertificateAutomation Techniques in RPA - UiPath Certificate
Automation Techniques in RPA - UiPath Certificate
VICTOR MAESTRE RAMIREZ
 
Maxon CINEMA 4D 2025 Crack FREE Download LINK
Maxon CINEMA 4D 2025 Crack FREE Download LINKMaxon CINEMA 4D 2025 Crack FREE Download LINK
Maxon CINEMA 4D 2025 Crack FREE Download LINK
younisnoman75
 
EASEUS Partition Master Crack + License Code
EASEUS Partition Master Crack + License CodeEASEUS Partition Master Crack + License Code
EASEUS Partition Master Crack + License Code
aneelaramzan63
 
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Salesforce Data Cloud- Hyperscale data platform, built for Salesforce.
Dele Amefo
 
Meet the Agents: How AI Is Learning to Think, Plan, and Collaborate
Meet the Agents: How AI Is Learning to Think, Plan, and CollaborateMeet the Agents: How AI Is Learning to Think, Plan, and Collaborate
Meet the Agents: How AI Is Learning to Think, Plan, and Collaborate
Maxim Salnikov
 
Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...
Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...
Proactive Vulnerability Detection in Source Code Using Graph Neural Networks:...
Ranjan Baisak
 
Avast Premium Security Crack FREE Latest Version 2025
Avast Premium Security Crack FREE Latest Version 2025Avast Premium Security Crack FREE Latest Version 2025
Avast Premium Security Crack FREE Latest Version 2025
mu394968
 
Exploring Wayland: A Modern Display Server for the Future
Exploring Wayland: A Modern Display Server for the FutureExploring Wayland: A Modern Display Server for the Future
Exploring Wayland: A Modern Display Server for the Future
ICS
 
The Significance of Hardware in Information Systems.pdf
The Significance of Hardware in Information Systems.pdfThe Significance of Hardware in Information Systems.pdf
The Significance of Hardware in Information Systems.pdf
drewplanas10
 
Douwan Crack 2025 new verson+ License code
Douwan Crack 2025 new verson+ License codeDouwan Crack 2025 new verson+ License code
Douwan Crack 2025 new verson+ License code
aneelaramzan63
 
How to Optimize Your AWS Environment for Improved Cloud Performance
How to Optimize Your AWS Environment for Improved Cloud PerformanceHow to Optimize Your AWS Environment for Improved Cloud Performance
How to Optimize Your AWS Environment for Improved Cloud Performance
ThousandEyes
 
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Andre Hora
 
Explaining GitHub Actions Failures with Large Language Models Challenges, In...
Explaining GitHub Actions Failures with Large Language Models Challenges, In...Explaining GitHub Actions Failures with Large Language Models Challenges, In...
Explaining GitHub Actions Failures with Large Language Models Challenges, In...
ssuserb14185
 
Top 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docxTop 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docx
Portli
 
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Lionel Briand
 
Who Watches the Watchmen (SciFiDevCon 2025)
Who Watches the Watchmen (SciFiDevCon 2025)Who Watches the Watchmen (SciFiDevCon 2025)
Who Watches the Watchmen (SciFiDevCon 2025)
Allon Mureinik
 
PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025
mu394968
 
Download YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full ActivatedDownload YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full Activated
saniamalik72555
 
Secure Test Infrastructure: The Backbone of Trustworthy Software Development
Secure Test Infrastructure: The Backbone of Trustworthy Software DevelopmentSecure Test Infrastructure: The Backbone of Trustworthy Software Development
Secure Test Infrastructure: The Backbone of Trustworthy Software Development
Shubham Joshi
 

Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Projects

  • 2. Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Projects Nikolas Hermanns Jose Lausuch
  • 3. Outline • Challenges of the SDNVPN project • XCI patchset verification • Virtual testing setup • Functest test challenge
  • 4. Integration Project (e.g. SDNVPN) ODL SDNVPN Fuel Apex • Consume many different Project • Different release dates • Different Bug tracking • Different source code handling • Different programming language • Etc.
  • 5. Challenge of Integration Projects (e.g. ODL in OPNFV) ODL-Master time ODL-Release OPNFV-Integration •Bug finding ODL-Release Bug1 ODL-Stable Bug2 Bug1 Bug2 ODL-Master x • Bug 1 found and solved • Bug 2 found but not valid for master anymore • Bug 2 was not worth the effort • Bug 3 will not be found until next release ODL is consumed too late! Bug3
  • 6. Challenge of Integration Projects (e.g. ODL in OPNFV) Developer A Bug 1 time ODL-Release ODL-Release Bug1 Integrator A • Dev A expert in Part of ODL • Int A expert in Fuel but doesn’t know much about ODL code base • Int A needs to find and understand the bug! • Int A need to talk to Dev A and explain the found bug • Might be Dev A does not even remember
  • 7. Solutions • Solution 1: More test cases in consumed project • Integrator needs to convince and explain the test to Developer • Test case might be too E2Eish to be created in consumed project • Solution 2: XCI patchset verification • Automated integration (Consume latest state) • Fast feedback (reducing from months to < 1 hour) • No interaction from integrator
  • 8. New Patchset (Commit) Create Virtual Testing Platform Download/Install new artifact Use standard testing framework e.g. Functest Build ODL artifact 10 min 15 min 35 min Unit test Installer Project e.g. Apex Snapshot 5 min Positive -> Merge commit Negative -> Prevent merge Investigate the logs/Find bugs ODL OPNFV Feedback
  • 9. Bare Metal (48 threads/256 GB RAM/500 GB SSD) vJenkins slave vController vCompute vCompute vCompute Thread pool – 8 vCPU  10 GB Diskspace  40 GB RAM  Own OVS networks vJenkins slave vController vCompute vController vCompute Snapshot Controller vController vController vController Snapshot Compute vCompute vCompute vCompute ~ 100 GB Diskspace x6 • RAM and CPU limiting factor • Very low disk IO cause no installation process • ~2 mins until Openstack/ODL services up • Easy exchange of snapshot
  • 10. Using the Snapshot vController vController vController Snapshot Controller vController vController vController ~ 100 GB Diskspace >1GB disk consumption per vHost • Using the base concept of qemu-img
  • 11. CPU pinning • 4 • 3 8 vCPUs ? -> YES! – sharing is the key. • 1 time vJenkins slave vController vCompute CPU-Load Jenkins- Slave running 8 vCPU Start vCmp vCtrl Started vCmp vCtrl Testing started Testing more API Under- provisioned
  • 12. Cpu pinning • Other vPod do not get effected
  • 13. Networking • OVS for complete isolation • vPods using same IPs so every vPod has its own networks • Jenkins-slave for Gateway doing NAT vJenkins slave vController vCompute Bare metal host admin storage private public host NAT internet
  • 14. Bare Metal (48 threads/256 GB RAM/500 GB SSD) vJenkins slave vController vCompute vCompute vCompute Thread pool – 8 vCPU  10 GB Diskspace  40 GB RAM  Own OVS networks vJenkins slave vController vCompute vController vCompute Snapshot Controller vController vController vController Snapshot Compute vCompute vCompute vCompute ~ 100 GB Diskspace x6 • RAM and CPU limiting factor • Very low disk IO cause no installation process • ~2 mins until Openstack/ODL services up • Easy exchange of snapshot
  • 15. Example of application: Functest FUNCTEST pep8 unit tests pylint docs py35 py27 test new code on OPNFV deployment
  • 16. Example of application: Functest • Functest bug might break CI • Needs a way to test each change before merging • Fast feedback to the commit owner • Easy to validate framework changes on Functest • Can be tested on a basic nosdn-nofeature scenario
  • 17. Example of application: Functest Challenges: • Changes on Feature tests  need specific scenario deployed • Increase verification time of each patchset ( + time to build a docker image) • Identify where the change is (framework, test case, …) to trigger an appropriate test case

Editor's Notes

  • #7: And yes odl is a great product but as any other project it has bugs But lets assume there are only 2 bugs in the release (I know hard assumption) Bug1 easy one change on stable and on master Bug2 Only valid on stable