Introduction to SolrCloud
4 likes2,877 views
A brief introduction to Solr and where it fits in your application, followed by an introduction to SolrCloud.
Ad
More Related Content
What's hot (20)
Viewers also liked (12)
Ad
Similar to Introduction to SolrCloud (20)
![Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...](https://cdn.slidesharecdn.com/ss_thumbnails/2020-11-10influxdays-introducinginfluxdbiox-201110182839-thumbnail.jpg?width=560&fit=bounds)
Ad
Introduction to SolrCloud
- 1. Introduction To SolrCloud Varun Thacker
- 2. Apache Solr has a huge install base and tremendous momentum Solr is both established & growing 250,000+ most widely used search solution on the planet. 8M+ total downloads monthly downloads You use Solr everyday. Solr has tens of thousands of applications in production. 2500+ open Solr jobs. Activity Summary 30 Day summary Aug 18 - Sep 17 2014 • 128 Commits • 18 Contributors 12 Month Summary Sep 17, 2013 - Sep 17, 2014 • 1351 Commits • 29 Contributors via https://www.openhub.net/p/solr
- 3. Solr scalability is unmatched. • 10TB+ Index Size • 10 Billion+ Documents • 100 Million+ Daily Requests
- 4. Solr’s scalability is unmatched
- 5. What is Solr? • A system built to search text • A specialized type of database management system • A platform to build search applications on • Customizable, open source software
- 6. Where does Solr fit?
- 7. What is SolrCloud? Subset of optional features in Solr to enable and simplify horizontal scaling a search index using sharding and replication. Goals scalability, performance, high-availability, simplicity, and elasticity
- 8. Terminology • ZooKeeper: Distributed coordination service that provides centralised configuration, cluster state management, and leader election • Node: JVM process bound to a specific port on a machine • Collection: Search index distributed across multiple nodes with same configuration • Shard: Logical slice of a collection; each shard has a name, hash range, leader and replication factor. Documents are assigned to one and only one shard per collection using a hash-based document routing strategy • Replica: A copy of a shard in a collection • Overseer: A special node that executes cluster administration commands and writes updated state to ZooKeeper. Automatic failover and leader election.
- 10. Collection == Distributed Index • A collection is a distributed index defined by: • named configuration stored in ZooKeeper • number of shards: documents are distributed across N partitions of the index • document routing strategy: how documents get assigned to shards • replication factor: how many copies of each document in the collection • Collections API: • curl "http://localhost:8983/solr/admin/collections? action=CREATE&name=punemeetup&replicationFactor=2&numShards=2&coll ection.configName=myonf
- 11. DEMO
- 12. Document Routing • Each shard covers a hash-range • Default: Hash ID into 32-bit integer, map to range • leads to balanced (roughly) shards • Custom-hashing • Tri-level: app!user!doc • Implicit: no hash-range set for shards
- 13. Replication • Why replicate? • High-availability • Load balancing • How does it work in SolrCloud? • Near-real-time, not master-slave • Leader forwards to replicas in parallel, waits for response • Error handling during indexing is tricky
- 14. Distributed Indexing • Get cluster state from ZK • Route document directly to leader (hash on doc ID) • Persist document on durable storage (tlog) • Forward to healthy replicas • Acknowledge write succeed to client
- 15. Shard Leader • Additional responsibilities during indexing only! Not a master node • Leader is a replica (handles queries) • Accepts update requests for the shard • Increments the _version_ on the new or updated doc • Sends updates (in parallel) to all replicas
- 16. Distributed Queries • Query client can be ZK aware or just query via a load balancer • Client can send query to any node in the cluster • Controller node distributes the query to a replica for each shard to identify documents matching query • Controller node sorts the results from step 3 and issues a second query for all fields for a page of results
- 17. Scalability / Stability Highlights • All nodes in cluster perform indexing and execute queries; no master node • Distributed indexing: No SPoF, high throughput via direct updates to leaders, automated failover to new leader • Distributed queries: Add replicas to scale-out qps; parallelize complex query computations; fault-tolerance • Indexing / queries continue so long as there is 1 healthy replica per shard
- 18. Zookeeper • Is a very good thing ... clusters are a zoo! • Centralized configuration management • Cluster state management • Leader election (shard leader and overseer) • Overseer distributed work queue • Live Nodes • Ephemeral znodes used to signal a server is gone • Needs 3 nodes for quorum in production
- 19. Zookeeper: State Management • Keep track of live nodes /live_nodes znode • ephemeral nodes • ZooKeeper client timeout • Collection metadata and replica state in /clusterstate.json • Every core has watchers for /live_nodes and / clusterstate.json • Leader election • ZooKeeper sequence number on ephemeral znodes
- 20. Other Features/Highlights • Near-Real-Time Search: Documents are visible within a second or so after being indexed • Partial Document Update: Just update the fields you need to change on existing documents • Optimistic Locking: Ensure updates are applied to the correct version of a document • Transaction log: Better recoverability; peer-sync between nodes after hiccups • HTTPS • Use HDFS for storing indexes • Use MapReduce for building index (SOLR-1301)
- 21. Solr on YARN
- 22. Solr on YARN • Run the SolrClient application : • Allocate container to run SolrMaster • SolrMaster requests containers to run SolrCloud nodes • Solr containers allocated across cluster • SolrCloud node connects to ZooKeeper
- 23. More Information on Solr on YARN • https://lucidworks.com/blog/solr-yarn/ • https://github.com/LucidWorks/yarn-proto • https://issues.apache.org/jira/browse/SOLR-6743
- 24. Our users are also pushing the limits https://twitter.com/bretthoerner/status/476830302430437376
- 25. Up, up and away! https://twitter.com/bretthoerner/status/476838275106091008
- 26. Connect @ https://twitter.com/varunthacker http://in.linkedin.com/in/varunthacker [email protected]