Leveraging Structured Data To Reduce Disk, IO & Network Bandwidth
Download as PPTX, PDF0 likes304 views
Most of the data that is pulled out of an SCM like Perforce Helix is common across multiple workspaces. Leveraging this fact means only fetching the data once from the repository. By creating cheap copies or clones of this data on demand, it is possible to dramatically reduce the load on the network, disks and Perforce servers, while making near-instant workspaces available to users.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Leveraging Structured Data To Reduce Disk, IO & Network Bandwidth (20)
Ad
More from Perforce (20)
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
Leveraging Structured Data To Reduce Disk, IO & Network Bandwidth
- 1. Leveraging Structured Data To Reduce Disk, IO & Network Bandwidth Vishal Moondhra VP Applications Methodics Inc.
- 2. 2 The Problem – We Are Drowning in Data! According to IBM, 90% of the world’s data has been created in the last 2 years! (www.ibm.com/smarterplanet) Semiconductors, Computational Fluid Dynamic simulations, Biotech simulations, game development… • Smartphone CPU development requires 100GB user workspaces
- 3. 3 Users Only Modify Small Subsets of Data Up to 95% of all data in a workspace is not modified • All the data is needed for reference – simulations, builds, rendering… • Majority of users only edit/modify their small portion USER1 WS USER2 WS ModificationsUnmodified
- 4. 4 CI Makes it worse User Commit(s) Same Data is ‘sync’ed to multiple workspaces
- 5. 5 Design Data is Lazy! The majority of your design data is duplicated, occupying expensive disks and choking your network!
- 6. 6 ‘Data De-duplication’ is not the solution De-duplication is done ‘after the fact’ – data is already created Filer CPU and Disk bandwidth hog • Might run for hours • 20-30% filer CPU utilization, bringing filer performance down It is a very broad technique • Looks at all available data – cannot be partitioned by project etc. Does not change the load on the Helix server
- 7. 7 Structured Data Each workspace has a ‘structure’ – provided by Perforce Helix • All workspaces of a project have similar (if not the same) structures • Workspaces are managed by a P4 client • The client determines which files are present in the workspace • P4 Protect determines which files are visible to a user • Files are versioned and have ‘changelists’ This structure can be leveraged to create smarter, smaller and faster workspaces
- 8. 8 A Smarter way to build workspaces Start with a ‘master’ workspace – fully populated with all the data Build ‘clones’ of this master workspace on demand Each clone gets its own P4 client Each clone can be independently sync’ed to any changelist • Sync the clone to the required (or latest) changelist when created • Clone only stores the ‘diff’ between master and the sync’ed changelist
- 9. 9 Permissions The Master workspace has all content P4 Protect is baked into the clone creation process • Clones will only get the files visible to the user creating the workspace • Transactional nature of this process ensures that it cannot be interrupted / hijacked Master Workspace Clone Files not available in clone due to P4 protections
- 10. 10 Bring Data into Clones when needed Clones use the Master workspace as reference • All common data is referred from the master, at the file block level When a file is written to, the system delivers the blocks to the clone – just-in-time • Happens ‘under the covers’ • Transparent to users The modified files are now contained in the clone • Allows each user to modify his/her workspace independently, as expected
- 11. 11 Clones are like any normal workspace P4 users will see no difference between a clone and a traditional ‘sync’ from P4 All P4 commands will work as before • Add/edit/delete/submit • Client commands Commits go back to P4 server from the clone ‘sync’ commands will bring in new data
- 12. 12 Standard P4 Workspace creation Project Reference Client P4 Client Speed: SLOW Byte count: HIGH TEMPLATE Create a new workspace Sync … Fetch files From depot
- 13. 13 Smarter Workspaces Project Reference Client Masters Create a new workspace Pick a master Clone ‘sync’ latest changes Speed: FAST Byte count: LOW Speed: INSTANT Byte count: FREE Speed: FAST Byte count: LOW
- 14. 14 Sample Disk Savings (1GB workspaces) Workspaces Disk Space Used (Traditionally) Disk Space Used (Initial) Disk Space Used (Steady State)++ Savings (Total) Master+ -- 1.00GB 1.5GB -- 1 WS 1.00GB 84KB 100.084MB 0% 10 Workspaces 10GB 840KB 260.092MB 78.33% 20 Workspaces 20GB 1.68MB 350.168MB 83.33% 100 Workspaces 100GB 8.4MB 11.15GB 88.61% ++ Estimated. The rule used here is that users will sync up to 10% of their workspace at steady state.
- 16. Never Pull The Same Data From Your Helix Server Twice!