How Open Source Embiggens Salesforce.com

How Open Source
Software Embiggens
Salesforce.com
Ian Varley
Principal Member of Technical Staff
@thefutureian

Safe harbor
Safe harbor statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the
assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make.
All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth,
earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements
concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new
products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our
Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature
market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our
service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further
information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year
and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC
Filings section of the Investor Information section of our Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on
time or at all. Customers who purchase our services should make the purchase decisions
based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.

Who am I?
• Ian Varley
• 4 years at Salesforce.com
• Core Database Team, Big Data Team
• @thefutureian

Show of hands...
• Use OSS?
• Contribute to OSS?
• Write their own OSS projects?

Developers at Salesforce spend all day in open source software…

Salesforce engineers work on an OSS stack ...
• Linux (Ubuntu, RHL)
• Java
• Eclipse (+ IntelliJ, vim, emacs)
• Guava, Apache Commons, more
• JUnit, Mockito, Selenium
• Git (+ p4)
• Memcached

What’s so great about open
source software anyway?

A rising tide ... … lifts all boats.

It’s a win-win situation.
• Everyone gets more out than they put in
• You have control over your own destiny
• You can attract the industry’s best minds
– The smartest devs seem to gravitate towards open source
– So if you raised your hand before, give yourself a pat on the back.

Servlet Container
• Servlet containers handle routing HTTP requests to code.
– Started w/ commercial product
– Feature: “steal” work from overloaded servers
– (Code name: Hamburglar)
• But! Show stopper bug, and no way to fix it ...

Solution: Jetty
• http://www.eclipse.org/jetty/
• Year-long migration process
• Tricky with 10+ years of legacy code!
• Now running Jetty (almost) everywhere.

Search Indexing
• Indexer takes text (e.g. chatter posts, etc.), makes it searchable.
– Original implementation: Lucene (forked)
– But, scale keeps increasing!
– Bottleneck: single-writer QFS on a SAN
– Needed solution to scale horizontally

Solution: Solr
• Horizontally scalable, REST interface
• Query / index on same host, no more SAN
• New features, core library is (latest) Lucene
• We’ve also contributed some small fixes, and contracted a big fix to allow handling
indexers with many cores (10K+!).

Message Queue
• Decouple calling code from its execution.
– Originally: 10-15 devs had rolled their own
– Centralized on a transactional queue (Vijay)
– Commercial product, deeply coupled to DB
– OK until a mysterious, unsolvable error.
– 3 years of back and forth!
– Eventually rewrote our layer to work around it.
– Scale challenges: 50 -> 500 queues
– CPU contention at head of queue

Solution: QPID
• Apache project, good reputation
• Separate tier from the DB
• Ran into bugs… and fixed them.
• 40% memory savings on client (QPID-4873; thanks Helen Kwong & Brian Toal)

Open source lets us bring our
experts to help everybody.

Build: Ant
• Build tools help get you from “code written” to “code running”.
– Used Apache Ant for years
– But, as the # of devs has grown …
– It’s become more difficult with a large, complex code base.

Solution: Maven
• Moving core build to Apache Maven
• Goal is a more modular and decoupled build structure
• Declarative dependencies FTW
• OSGi: Apache Felix

Plus: Jenkins
• Salesforce core uses home-grown “autobuilds”
• But for new projects outside core, set up builds as needed
• Additional automation on top of Jenkins for local builds

Deployment: Home Grown
• Deployment tools let you get code out to servers.
– Salesforce has always used home-grown tool, “ReleaseRunner”
– Required for Salesforce's extremely rigorous security model
– But as we scale out, manual methods aren’t cutting it

Solution: Puppet, Salt, Razor, Rundeck
• Get code out to lots of servers with little manual involvement.
– Razor: automated machine inventory
– Puppet: deployment of bits and configuration
– Salt, Rundeck: service orchestration for restarts
All of this still very much WIP… Salesforce.com is an industry leader in security, and is leading the way in
using tools like Puppet in an enterprise-class, multi-tenant environment.

Metric Collection: Home Grown
• Getting information about server health and performance
– Home-grown agent software
– Limited to only one app (Salesforce core)
– Metrics collected and pulled by central cluster

Metric Collection: Kafka
• Kafka is a high throughput distributed messaging system
– Written & open sourced by LinkedIn
– Used in a system (code-named Ajna)
– Pushes metrics out of prod, rather than pulling
– Intra-pod queue for local consumption
– Centralized pipeline to DMZ

Batch Processing
• Salesforce == RDBMS
• No great approach for batch processing
• Especially on sets that don’t fit in memory
• Working with data that doesn't fit into the standard
relational model is hard

Solution: Hadoop
• Map/Reduce: ship computations to your data instead vice versa
– Walter Macklem (Platform CTO); Codename: Gridforce
– +HDFS (distributed file storage)
– +Pig (a higher level language)
– Features: recommendations, search relevance, machine learning
– Log export pilot ...
(Ask your CSR/CSM/AM to
get nominated for the pilot!)

Big Data
• Relational databases are powerful … but …
– The model is so rich, it’s prohibitive for really large data
– RDBMS has strict scalability limits per object
– Hard to scale out because, runs on big iron
So we asked:
• What if we could store vast numbers of records, but with fewer capabilities and
assumptions? Scale horizontally, but with the same safety guarantees?

Big Data: HBase
• Horizontally scalable NoSQL database.
– Fewer capabilities (no joins, transactions)
– Scales by adding machines
– Fault tolerant (on HDFS)
– Features? Initially, audit & compliance, event tracking
– Eventually, a lot more: really big objects
– Got a lot of field history? Join the FHR retention pilot! (Talk to your CSM)
• This is my team, so I could talk for hours...

OK, that’s cool. But, does
salesforce.com contribute new
projects?

Historically: no, not many.But, this
is changing.

Aura: UI Framework
• Basis for new generation of Salesforce UI
– High performance client-server architecture
– Event-driven, MVC architecture
– https://github.com/forcedotcom/aura

Phoenix: a SQL Skin for HBase
• “We put the SQL back in NoSQL”
– A proper subset of SQL
– Familiar interface, scalable storage
– Unlike Hive, uses the HBase client API
– Blazing fast; queries in milliseconds
– Very broad contribution since we opened it
– Accepted in the Apache Incubator in 2013
– Included in Hortonworks Hadoop distribution in 2014

Mobile SDK
• All SDK dev for Salesforce done in open source
– https://github.com/forcedotcom/SalesforceMobileSDK-iOS
– https://github.com/forcedotcom/SalesforceMobileSDK-Android
– Also: heavy use of Apache Cordova, to blend web & native components

Lots more!
• So far, we’ve only been talking about Salesforce core.
– Many Salesforce companies use tons of Open Source:
– Heroku - https://github.com/heroku
– Radian6, Data.com, ExactTarget - you name it, we probably use it somewhere
• And lots of open source stuff on the platform, too!
– http://boards.developerforce.com/t5/Salesforce-Labs-Open-Source/bd-p/labs
• You can search github for Apex & Salesforce

Salesforce.com isn’t just an OSS
user.We’re an OSS pusher.

Committers on dozens of big projects
• Salesforce actively supports a lot of people who primarily contribute to open source
projects (not just a side thing).
– Postgres: Tom Lane (Project Lead)
– Ruby: Matz (Project Lead)
– Maven: Jason Van Zyl (Project Lead)
– HBase: Lars Hofhansl (PMC, release manager); Jesse Yates
– Phoenix: James Taylor (Project Lead)
– Aura: Doug Chasman (Project Lead)
– Pig: Prashant Kommereddi

Is Open Source right for
everything? No.

It’s great for …
• Core components
• Databases
• Common algorithms
• Reusable UI libraries & abstractions
• And any case where “the source isn’t the secret sauce”.

It’s not great for …
• Code entangled with your business model
• Code you didn’t write with a plan to open up
• Software that’s “all things to all people”
• Getting other people to do your work
• But, these are kind of anti-patterns anyway, right … ?

Most return on investment is from
open sourcing “the interesting
bits”, rather than the whole stack.

And embracing the Open Source
approach, particularly in the last 3
years, has been a sea change.

In contributing, we all gain.
Look for more OSS involvement from Salesforce in the future!

Follow us
• Ian Varley (@thefutureian)
• @salesforceeng
• @salesforcewit

How Open Source Embiggens Salesforce.com

How Open Source Embiggens Salesforce.com

Recommended

More Related Content

What's hot (20)

Viewers also liked (19)

Similar to How Open Source Embiggens Salesforce.com (20)

More from Salesforce Engineering (20)

Recently uploaded (20)

How Open Source Embiggens Salesforce.com