Effective SharePoint Scalability & Management. To BLOB or not to BLOB, that’s the question

Editor's Notes

• #5: 150k AUD with 100k still in pipe
• #7: Every company has a vision to use SharePoint as the sole ECM. There are many steps and challenges along the way. The greatest leaps are going from ‘collaboration’ to ‘development’ and customization, and then again from ‘development’ to full ‘enterprise content management’ (ECM). You must have tools to get you there, and DocAve will accelerate your growth.
• #8: Single farm to full, multi-platformSaaS
• #11: Considerations for scaling your SharePoint environment for growth
• #12: Best practice for the recommended architecture:Redundancy & HA for each tier in your production env.Maintain a test environment where you QA all changes and code before deploying in production…improve production stability
• #13: Scaling brings StabilityStability brings AvailabilityAvailability through deployment (Dev -> Prod)Deployment through Architecture (Virtual is common)One of the best ways to ensure production stability is to abide by the proven practice of maintaining a multiple farm environment- keeping all development and testing completely separate from production. This means that if for some reason, something like a looping workflow is created- only the dev or test farm will be brought down, and production will remain unaffected. If we were testing in a separate web-application, we would have been screwed. Maintaining this staged environment means all workflows, features, customizations, and metadata modifications, for instance, are all properly tested, and the affect they’ll have on production is known before anything’s even installed. But assuming we set up global SharePoint environments, and comply with MSFT best practices of maintaining separate SharePoint farms for Dev/Testing/Staging/Prod- how do we manage changes- change to design elements, solutions, features, workflow, site content- across these sharepoint deployments? Documentation is key here- if a step is missed when installing a feature, if we forget to reset IIS or forget that one file that was stored on the other web front end… repeatability is key. You’ll have to be diligent at documenting change, and who’s responsible for which components of change- To make your lives easier in this respect, there are 3rd party tools available to help you thru this process.All Inclusive Deployment – Design, Solutions, WFE
• #14: One of the great new features of service applications is that they can be published and shared between farms. This is achieved by the source server “publishing” one or more of its service applications. When a service application is published a URL is created that can be fed into the second farm, being the consumer of the service. The servers are required to be configured appropriately first, by doing things such as exchanging security certificates so the farms will trust each other, but once that is done the consumer will be able to interact with the service as if it was local, and it can associate it with its local web applications.
• #15: This theory of publishing a service application opens up some interesting possibilities around how you can architect environments that have multiple farms, and one such way to work with this is the model of having a services farm. The premise is that you can have one farm that will run service applications that would be common to your organisation, such as user profile import, or enterprise search, or managed metadata for an enterprise taxonomy. These could then be managed and run in only one location and then published out for other farms in the organisation to consume. This means that user profiles can be imported once and used globally, or content can be indexed in one location and the index published out to other farms, or a taxonomy can be built in one place and then shared to the entire organisation.
• #16: A fully distributed global architecture will provide quick access to local SharePoint content with good user experience;You want to replicate only the relevant/global content You also want to handle special remote location (like the Alaska oil ring in this slide) via local Infrastructure and replication; Requires 3rd party tool
• #18: Distribute admin tasks- site collection administration, permissions management, etc … It may suit you to break up into separate site collections for different business units in order to achieve the desired GovernanceCan only distribute tasks so muchWill require additional personnel or an admin tool
• #21: There are also significant cost savings to be realized by moving your data through different tiers of storage. I’ve provided an example for a 1TB content database stored on Tier 1 storage. In this example, the customer saved $11k for 1 single database by moving data to a cheaper tier of storage.Discussion:Are these numbers accurate? What are people paying for Tier 1 storage these days? Tier 2? Tier 3?In my example, I’ve referenced Cloud Storage in Tier 3. Is anyone storing data in the cloud? Are people comfortable storing data in the cloud?
• #22: Going back one step, what is actually stored in SharePoint and how is it stored? 95% of what’s stored are BLOBs = Binary Large Objects BLOB = anything I’m adding to SharePoint (> 256kb)
• #23: So, SharePoint content = BLOB + Metadata
• #24: So, content DB = database of … BLOBs + Metadata
• #25: This graph shows requests-per-second for varying number of users. Requests-per-second for a SQL DB is on the left, RPS with RBS is on the right.For SharePoint, the ability to scale is critical.We see with the first graph- regarding SharePoint’s Ability to Scale, that as we increase user threads, the number of requests SharePoint’s able to handle per second decreases- this correlates to the green line here- showing us that as user count increases, the time it takes for SharePoint to respond to a request drastically increases. Now lets look at a SharePoint environment with its BLOBs externalized… here we see right off the bat that SharePoint’s able to handle many more requests per second than before, and the time of response per request stays extremely low. This is definitely something that requires further investigation- to see what happens when we get up into the thousands. For the purpose of this test, we kept the user threads to 500 and under, as this is where we see drastic changes is how many requests per second SharePoint’s able to handle. Now, this test was also conducted on SharePoint 2007, so it’d also be interesting to see how this changes between SharePoint 2007 and 2010, natively, and then with EBS or RBS enabled.
• #27: For SharePoint 2010, Microsoft recommends you split content db when it hits 200GBFor 2007, split when it’s around 90GB
• #29: Here’s a look at a very basic SharePoint architecture. Here we have the front end with the object model, and the SharePoint content (blobs and metadata) are stored in SQL.
• #30: No native archiving toolsEBS extended to include RBS for BLOB removal Available only in SQL Server 2008 SP2Only accessible via APIBCS (BDC in 2007) extended to allow for easier connectivity with legacy data systems Not intended for controlling growth, only exposing additional data from other systemsIntegrated with SharePoint!Users can access contents by:Clicking and downloading directly through SharePointOpening the file using their Office clientReferencing the URLSearching for contents natively in SharePointUsers can interact with contents by:Modifying metadata and content typesModifying permissionsApplying alertsUsing workflows or publishing templatesUsing site Quotas and Locks
• #31: So now let’s take a look at RBS… As RBS is SQL specific, it can be used across applications that leverage SQL, not just SharePoint, so this gives you more of an enterprise-wide storage architecture versus EBS, and here’s how enabling an RBS provider would affect your SharePoint storage architecture. You can have an RBS provider per database. No context, no ability to manage the object
• #32: Free to help with your migration planning.
• #33: Anticipation starts when you plan your architecture
• #34: When you anticipate growth and architect your environment for scalability, you’d reap the benefits!HOWEVER…. We do need to realize that externalization, while great for performance and user experience and cost, does create complexities in how we manage our infrastructure. Backup and Recovery operations- for instance- we still will need a way to back up BLOBs SYNCHRONOUSLY. If we want to leverage content management tools, for restructuring, or replication technology, for keeping multiple sites or farms in synch, those tools need to be able to account for externalized content- whether these tools copy or move BLOBs as well, whether they just copy or move the stubs, or even whether replication technology can copy the stubs and redirect with DFSR.Backup Implications - Need a method to backup BLOBs synchronously or your DBs can get out of sync with your filesystem SharePoint 2007 – this isn’t very efficient SharePoint 2010 – this works very wellTo BLOB or not to BLOB (MIT Research for Microsoft)<256kb, SQL better256kb to 1mb, SQL and file system comparable>1mb, file system betterMicrosoft will provide a powershellsolution to migrate from EBS to RBS (check this fact!)AND LAST BUT NOT LEAST>>>> IT”S COMPLETELY SEAMLESS FOR ENDUSERS!!!! End users can still access, and INTERACT with content! Content still works with workflows, alerts, office applications, etc.
• #35: It is extremely critical to factor in BLOB externalization to your Data Protection Strategies- just as you should consider it in other content management strategies as well.And here’s why…
• #39: These are just a few options for planning for BLOB externalization- The thing to remember here- is that even if some of these methods will keep BLOBs protected- if they’re not allowing you to perform SYNCHRONOUS backups of your databases AND BLOB store- whether its in the cloud, etc- the BLOB backups will only be useful if, by chance, the timer jobs are in sync. Make sure and test whatever strategy you want to leverage.
• #40: To optimize user experience, and their reliance on SharePoint as a platform- a safe place to store their documents, etc- you’ll need to find away to provide for granular restore capabilities. Perfect example- I’ve set up countless SharePoint libraries internally- one, I made the mistake of not turning versioning on right away to keep track of document histories- critical to collaboration. While looking thru a SharePoint doc- I could have sworn I’d made other edits- so I thought, maybe the latest version is on my laptop. Instinctively, I uploaded the document and overwrote the file. Come to find out, I’d actually uploaded a previous version, overwritten my latest edits, and lost a week’s worth of work. This is a risk. Because I didn’t delete the file, it wasn’t in recycle bin- the document had to be restored for me to continue on.
• #41: Now lets look at Archiving- we mentioned this was another way to optimize storage to save costs, increase sharepoint’s scalability, and improve performance. So first I just want to do a re-cap on the types of Archiving… because depending on who you’re talking to- you could mean a couple different things. First, there’s archiving for compliance….Natively, SharePoint offers the records center. If you’re leveraging the Records center, be aware that it is essentially just another location, still in SQL, to store content. The best practice here would be to put the records center on its own database, and leverage RBS to offload content. Next, we have archiving for storage savings. Essentially, leveraging multiple tiers of storage- not just 0 or 1 or 1 and 2 like we do with BLOB externalization, but maybe leverage tiers 1-4, for example, to really achieve the greatest cost savings. Now, archiving… natively, I mentioned there were no tools, but in reality, you could essentially just create backup files of the content you’d want to “archive” and then delete them out of SharePoint. OR, you look at 3rd parties, like AvePoint’s DocAve Archiver to build business rules into your archival plans.
• #42: Things to look for in 3rd party tools- EBS/RBS is key- keeps content “in”- accessible, allows interaction