SlideShare a Scribd company logo
BigQueue
A Big, Fast and Persistent
Queue
    - by William
    http://bulldog2011.github.com/
Feature Highlights
   Fast
         enqueue and dequeue are close to O(1) direct memory access.
   Big
         Only limited by available disk space.
   Persistent
         Data is persisted on disk and is crash resistant.
   Reliable
         OS will be responsible for message persistence even your process crashes.
   Realtime
         Produced messages will be immediately visible to consumers
   Flexible Queue Semantics
         Consume once queue, fanout queue, can even consume by index
   Memory-efficient
         Automatic pagging & swapping algorithm, only most recently accessed data is kept in memory.
   Thread-safe
         Multiple threads can concurrently enqueue and dequeue without data corruption.
   Simple & Light-weight
         Current library jar is less than 40K.
Performance Highlights

   In concurrent producing and consuming case,
    average throughput is around 166MBps.
   In sequential producing then consuming case,
    average throughput is around 333MBps.
Design – Logical View

   Looks just like a big array or a circular array
Design – Consume Once and Fanout
Semantics Support
Design – Physical View : Paged Index +
Data File
Design – Components View
Design – Dynamic View :
Memory Mapped Sliding Window
Concurrency

   Produce(or append) is synchronized in the
    queue implementation
   Consume(or read) is already thread-safe
Simple Interface

   Creation
    IBigQueue bigQueue = new BigQueueImpl("d:/bigqueue/tutorial",
    "demo");
   Enqueue
     for(int i = 0; i < 10; i++) {
        String item = String.valueOf(i);
        bigQueue.enqueue(item.getBytes());
    }
   Dequeue
    for(int i = 0; i < 5; i++) {
      String item = new String(bigQueue.dequeue());
    }
   Peek
     byte[] data = bigQueue.peek();
Fanout Queue
   Creation
    IFanOutQueue foQueue = new FanOutQueueImpl("d:/tutorial/fanout-queue", "demo");
   Enqueue
     for(int i = 0; i < 10; i++) {
       String log = "log-" + i;
       foQueue.enqueue(log.getBytes());
    }
   Fanout 1 Dequeue
    String fanoutId1 = "realtime";
    while(!foQueue.isEmpty(fanoutId1)) {
        String item = new String(foQueue.dequeue(fanoutId1));
        System.out.println(item);
    }
   Fanout 2 Dequeue
    String fanoutId2 = "offline";
    while(!foQueue.isEmpty(fanoutId2)) {
        String item = new String(foQueue.dequeue(fanoutId2));
        System.out.println(item);
    }
    Fanout 1 and Fanout 2 consuming are independent
Fanout Queue Semantics
Use Case 1 :
Log Collecting & Consuming
Use Case 2 :
Big Data Sorting
        Queue Only Algorithm:
    1.     Put all data into a source queue
    2.     Build a queueOfSortedQueues by dividing and
           sorting the source queue
    3.     Merge sort the queueOfSortedQueues
    4.     The last one left in the queueOfSortedQueues
           is the final sorted queue
Source, Samples, Docs and Tutorials


    https://github.com/bulldog2011/bigqueue
Other Alternatives

   Apache ActiveMQ
       http://activemq.apache.org
   RabbitMQ
       http://www.rabbitmq.com/
   ZeroMQ
       http://www.zeromq.org
   Kestrel
       https://github.com/robey/kestrel
   Apache Kafka
       http://kafka.apache.org
Ad

More Related Content

Similar to A Big, Fast and Persistent Queue (20)

Java 7 Features and Enhancements
Java 7 Features and EnhancementsJava 7 Features and Enhancements
Java 7 Features and Enhancements
Gagan Agrawal
 
Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...
Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...
Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...
Alluxio, Inc.
 
Project Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalProject Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare Metal
Databricks
 
What’s new in Alluxio 2: from seamless operations to structured data management
What’s new in Alluxio 2: from seamless operations to structured data managementWhat’s new in Alluxio 2: from seamless operations to structured data management
What’s new in Alluxio 2: from seamless operations to structured data management
Alluxio, Inc.
 
Speeding Up Spark Performance using Alluxio at China Unicom
Speeding Up Spark Performance using Alluxio at China UnicomSpeeding Up Spark Performance using Alluxio at China Unicom
Speeding Up Spark Performance using Alluxio at China Unicom
Alluxio, Inc.
 
Fedora Overview
Fedora OverviewFedora Overview
Fedora Overview
eposthumus
 
PyCon Estonia 2019
PyCon Estonia 2019PyCon Estonia 2019
PyCon Estonia 2019
Travis Oliphant
 
dotMemory 4 - What's inside?
dotMemory 4 - What's inside?dotMemory 4 - What's inside?
dotMemory 4 - What's inside?
Maarten Balliauw
 
Hadoop Vectored IO
Hadoop Vectored IOHadoop Vectored IO
Hadoop Vectored IO
Steve Loughran
 
[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf
[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf
[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf
MukundThakur22
 
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio Use Cases at Strata+Hadoop World Beijing 2016Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio, Inc.
 
Reproducibility and automation of machine learning process
Reproducibility and automation of machine learning processReproducibility and automation of machine learning process
Reproducibility and automation of machine learning process
Denis Dus
 
Mutexes 2
Mutexes 2Mutexes 2
Mutexes 2
Narendranath Reddy T
 
Goroutine stack and local variable allocation in Go
Goroutine stack and local variable allocation in GoGoroutine stack and local variable allocation in Go
Goroutine stack and local variable allocation in Go
Yu-Shuan Hsieh
 
Hidden pearls for High-Performance-Persistence
Hidden pearls for High-Performance-PersistenceHidden pearls for High-Performance-Persistence
Hidden pearls for High-Performance-Persistence
Sven Ruppert
 
EUDAT Generic Execution Framework
EUDAT Generic Execution FrameworkEUDAT Generic Execution Framework
EUDAT Generic Execution Framework
EUDAT
 
Kqueue : Generic Event notification
Kqueue : Generic Event notificationKqueue : Generic Event notification
Kqueue : Generic Event notification
Mahendra M
 
XWiki Rendering @ FOSDEM 2014
XWiki Rendering @ FOSDEM 2014XWiki Rendering @ FOSDEM 2014
XWiki Rendering @ FOSDEM 2014
Vincent Massol
 
06 file processing
06 file processing06 file processing
06 file processing
Issay Meii
 
The Pillars Of Concurrency
The Pillars Of ConcurrencyThe Pillars Of Concurrency
The Pillars Of Concurrency
aviade
 
Java 7 Features and Enhancements
Java 7 Features and EnhancementsJava 7 Features and Enhancements
Java 7 Features and Enhancements
Gagan Agrawal
 
Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...
Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...
Alluxio Webinar | What’s New in Alluxio AI: 3X Faster Checkpoint File Creatio...
Alluxio, Inc.
 
Project Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalProject Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare Metal
Databricks
 
What’s new in Alluxio 2: from seamless operations to structured data management
What’s new in Alluxio 2: from seamless operations to structured data managementWhat’s new in Alluxio 2: from seamless operations to structured data management
What’s new in Alluxio 2: from seamless operations to structured data management
Alluxio, Inc.
 
Speeding Up Spark Performance using Alluxio at China Unicom
Speeding Up Spark Performance using Alluxio at China UnicomSpeeding Up Spark Performance using Alluxio at China Unicom
Speeding Up Spark Performance using Alluxio at China Unicom
Alluxio, Inc.
 
Fedora Overview
Fedora OverviewFedora Overview
Fedora Overview
eposthumus
 
dotMemory 4 - What's inside?
dotMemory 4 - What's inside?dotMemory 4 - What's inside?
dotMemory 4 - What's inside?
Maarten Balliauw
 
[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf
[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf
[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf
MukundThakur22
 
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio Use Cases at Strata+Hadoop World Beijing 2016Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio, Inc.
 
Reproducibility and automation of machine learning process
Reproducibility and automation of machine learning processReproducibility and automation of machine learning process
Reproducibility and automation of machine learning process
Denis Dus
 
Goroutine stack and local variable allocation in Go
Goroutine stack and local variable allocation in GoGoroutine stack and local variable allocation in Go
Goroutine stack and local variable allocation in Go
Yu-Shuan Hsieh
 
Hidden pearls for High-Performance-Persistence
Hidden pearls for High-Performance-PersistenceHidden pearls for High-Performance-Persistence
Hidden pearls for High-Performance-Persistence
Sven Ruppert
 
EUDAT Generic Execution Framework
EUDAT Generic Execution FrameworkEUDAT Generic Execution Framework
EUDAT Generic Execution Framework
EUDAT
 
Kqueue : Generic Event notification
Kqueue : Generic Event notificationKqueue : Generic Event notification
Kqueue : Generic Event notification
Mahendra M
 
XWiki Rendering @ FOSDEM 2014
XWiki Rendering @ FOSDEM 2014XWiki Rendering @ FOSDEM 2014
XWiki Rendering @ FOSDEM 2014
Vincent Massol
 
06 file processing
06 file processing06 file processing
06 file processing
Issay Meii
 
The Pillars Of Concurrency
The Pillars Of ConcurrencyThe Pillars Of Concurrency
The Pillars Of Concurrency
aviade
 

Recently uploaded (20)

How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?
Daniel Lehner
 
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityCyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of security
riccardosl1
 
TrsLabs - Fintech Product & Business Consulting
TrsLabs - Fintech Product & Business ConsultingTrsLabs - Fintech Product & Business Consulting
TrsLabs - Fintech Product & Business Consulting
Trs Labs
 
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 
Big Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur MorganBig Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur Morgan
Arthur Morgan
 
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc
 
HCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser EnvironmentsHCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser Environments
panagenda
 
Linux Support for SMARC: How Toradex Empowers Embedded Developers
Linux Support for SMARC: How Toradex Empowers Embedded DevelopersLinux Support for SMARC: How Toradex Empowers Embedded Developers
Linux Support for SMARC: How Toradex Empowers Embedded Developers
Toradex
 
Generative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in BusinessGenerative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in Business
Dr. Tathagat Varma
 
ThousandEyes Partner Innovation Updates for May 2025
ThousandEyes Partner Innovation Updates for May 2025ThousandEyes Partner Innovation Updates for May 2025
ThousandEyes Partner Innovation Updates for May 2025
ThousandEyes
 
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptx
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxDevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptx
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptx
Justin Reock
 
How analogue intelligence complements AI
How analogue intelligence complements AIHow analogue intelligence complements AI
How analogue intelligence complements AI
Paul Rowe
 
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond SparkRusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Spark
carlyakerly1
 
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxIncreasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Anoop Ashok
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
BookNet Canada
 
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfSAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
Precisely
 
Procurement Insights Cost To Value Guide.pptx
Procurement Insights Cost To Value Guide.pptxProcurement Insights Cost To Value Guide.pptx
Procurement Insights Cost To Value Guide.pptx
Jon Hansen
 
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
SOFTTECHHUB
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?
Daniel Lehner
 
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityCyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of security
riccardosl1
 
TrsLabs - Fintech Product & Business Consulting
TrsLabs - Fintech Product & Business ConsultingTrsLabs - Fintech Product & Business Consulting
TrsLabs - Fintech Product & Business Consulting
Trs Labs
 
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 
Big Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur MorganBig Data Analytics Quick Research Guide by Arthur Morgan
Big Data Analytics Quick Research Guide by Arthur Morgan
Arthur Morgan
 
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc
 
HCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser EnvironmentsHCL Nomad Web – Best Practices and Managing Multiuser Environments
HCL Nomad Web – Best Practices and Managing Multiuser Environments
panagenda
 
Linux Support for SMARC: How Toradex Empowers Embedded Developers
Linux Support for SMARC: How Toradex Empowers Embedded DevelopersLinux Support for SMARC: How Toradex Empowers Embedded Developers
Linux Support for SMARC: How Toradex Empowers Embedded Developers
Toradex
 
Generative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in BusinessGenerative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in Business
Dr. Tathagat Varma
 
ThousandEyes Partner Innovation Updates for May 2025
ThousandEyes Partner Innovation Updates for May 2025ThousandEyes Partner Innovation Updates for May 2025
ThousandEyes Partner Innovation Updates for May 2025
ThousandEyes
 
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptx
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxDevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptx
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptx
Justin Reock
 
How analogue intelligence complements AI
How analogue intelligence complements AIHow analogue intelligence complements AI
How analogue intelligence complements AI
Paul Rowe
 
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond SparkRusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Spark
carlyakerly1
 
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxIncreasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Anoop Ashok
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
BookNet Canada
 
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfSAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
Precisely
 
Procurement Insights Cost To Value Guide.pptx
Procurement Insights Cost To Value Guide.pptxProcurement Insights Cost To Value Guide.pptx
Procurement Insights Cost To Value Guide.pptx
Jon Hansen
 
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
SOFTTECHHUB
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
Ad

A Big, Fast and Persistent Queue

  • 1. BigQueue A Big, Fast and Persistent Queue - by William http://bulldog2011.github.com/
  • 2. Feature Highlights  Fast  enqueue and dequeue are close to O(1) direct memory access.  Big  Only limited by available disk space.  Persistent  Data is persisted on disk and is crash resistant.  Reliable  OS will be responsible for message persistence even your process crashes.  Realtime  Produced messages will be immediately visible to consumers  Flexible Queue Semantics  Consume once queue, fanout queue, can even consume by index  Memory-efficient  Automatic pagging & swapping algorithm, only most recently accessed data is kept in memory.  Thread-safe  Multiple threads can concurrently enqueue and dequeue without data corruption.  Simple & Light-weight  Current library jar is less than 40K.
  • 3. Performance Highlights  In concurrent producing and consuming case, average throughput is around 166MBps.  In sequential producing then consuming case, average throughput is around 333MBps.
  • 4. Design – Logical View  Looks just like a big array or a circular array
  • 5. Design – Consume Once and Fanout Semantics Support
  • 6. Design – Physical View : Paged Index + Data File
  • 8. Design – Dynamic View : Memory Mapped Sliding Window
  • 9. Concurrency  Produce(or append) is synchronized in the queue implementation  Consume(or read) is already thread-safe
  • 10. Simple Interface  Creation IBigQueue bigQueue = new BigQueueImpl("d:/bigqueue/tutorial", "demo");  Enqueue for(int i = 0; i < 10; i++) { String item = String.valueOf(i); bigQueue.enqueue(item.getBytes()); }  Dequeue for(int i = 0; i < 5; i++) { String item = new String(bigQueue.dequeue()); }  Peek byte[] data = bigQueue.peek();
  • 11. Fanout Queue  Creation IFanOutQueue foQueue = new FanOutQueueImpl("d:/tutorial/fanout-queue", "demo");  Enqueue for(int i = 0; i < 10; i++) { String log = "log-" + i; foQueue.enqueue(log.getBytes()); }  Fanout 1 Dequeue String fanoutId1 = "realtime"; while(!foQueue.isEmpty(fanoutId1)) { String item = new String(foQueue.dequeue(fanoutId1)); System.out.println(item); }  Fanout 2 Dequeue String fanoutId2 = "offline"; while(!foQueue.isEmpty(fanoutId2)) { String item = new String(foQueue.dequeue(fanoutId2)); System.out.println(item); } Fanout 1 and Fanout 2 consuming are independent
  • 13. Use Case 1 : Log Collecting & Consuming
  • 14. Use Case 2 : Big Data Sorting  Queue Only Algorithm: 1. Put all data into a source queue 2. Build a queueOfSortedQueues by dividing and sorting the source queue 3. Merge sort the queueOfSortedQueues 4. The last one left in the queueOfSortedQueues is the final sorted queue
  • 15. Source, Samples, Docs and Tutorials https://github.com/bulldog2011/bigqueue
  • 16. Other Alternatives  Apache ActiveMQ  http://activemq.apache.org  RabbitMQ  http://www.rabbitmq.com/  ZeroMQ  http://www.zeromq.org  Kestrel  https://github.com/robey/kestrel  Apache Kafka  http://kafka.apache.org