C# development workflow @ criteo

Editor's Notes

• #4: We splitted our codebase in many components packaged as Nugets. Each team was responsible of a few Nugets. Each Nuget had a version, and new versions were periodically uploaded to a nuget server. This setup worked for some time, it allowed teams to work independently from each other. On this example, team B can work on next version without worrying about their clients. And Team A is safe, because it can control exactly when it wants to integrate changes from other teams. You need only your repo to work, you don’t care about the repos of your dependencies. So it worked, but we quickly faced major issues.
• #5: A breaking change in C can take a long time to be deployed in prod. C does is development and publishes its new major version. B has to upgrade its dependencies, but they may have no reason to do so. If it works fine with the version they have, why would they upgrade ? Upgrading a dependency is scary, you get many changes at one. It can be costly if breaking changes have been introduced. So the guys from team C may have to convince their clients to upgrade their dependencies. So if you have a complex dependency graph, we a lot of levels of depenency, it can become very hard to propagate changes quickly. Also, since a Nuget can be used by several components, at different versions, a team has to maintain several versions of it Nugets. This a waste of time. You cannot be sure your commit is OK until it’s deployed in production, and it happens weeks after, it’s not efficient.
• #6: There is a second issue with the Nuget system. It’s the Nuget Dependency Hell. It’s quite simple, it happends when you have diamon dependencies. If A depends on B and C. B and C depends on D, but with different versions. Then you have problems. B and C must be synchronized in terms of dependency which is hard to enforce.
• #7: Spending too much time in integration. Bad for agility and scaling. We decided that each component would always depend on the latest commits of its dependencies. There are still components but there are simple assemblies, not versionned nugets. All development is made is master branch. When you do a change in D, then next time A is built, this change will be integrated in A immediately. It solves our problem of late integration. You do a commit in the code base, it’s immediately visible to all. Baiscally, it’s like if we had a single repository with a giant solution.
• #8: Dev are done in a single shared trunk. It’s the model that is used by companies likes Google or Facebook. There is no feature branches, all changes are done in master. Every commit produces a production ready build. It allows to do true Continuous Delivery. If you have feature branches or bumping of internal versions, then you cannot do continous delivery, because of the late integration. A commit in any repository can break the build of everybody, so we need proper tooling to avoid this situation. This is the opposite of the previous model, where when you do a commit in your component, nodody is impacted, until they actually upgrade their dependencies. Also, since you cannot break anything, it’s harder to do breaking changes, because each commit must keep the codebase in a stable state. TBD cannot be done properly if no proper test coverage: you need to be sure at any time that your code is working fine,
• #9: Since there is no more Nugets and versions, it means everybody has to build using the sources. As a dev, since there are many repos, it’s hard to know where all the sources of your dependencies are. We have actually diferent use cases: On the continous integration servers, we want to build from trunk On the dev machines, we have to build from also, but we would like some optimization to avoid cloning or building the sources we don’t need We need strong tests before merge, to be able to be sure that a commit is not going to build the trunk. And for this we need to answer the questions like: what projects that depend on a given one. This is necessary, so that when you do a change on a project, you are sure you’re not going to break project that depend on you.
• #10: Our setup is based on Gerrit for Code Review and git hosting, Jenkins for running the build jobs, MSBuild for actual build, and a custom tool called CBS (Criteo Build System) We had to add some tooling on top of MSBuild and Visual Studio, because we needed something to manage the multiple repos. Also the list of projects that we need to be built depends on the situation, we cannot have static msbuild or visual studio solutions. Instead with CBS, with can manage a dynamic list of projects to build.
• #11: We have a job that we call the MOAB (stands for mother of all builds), that runs as fast as possible. When a run finished, a new one is launched. The goal of this job of simply to get all latest commits from C# repositories, and build the master branch. A build number is assigned at each build.
• #12: cbs checkout that clones and fetches all latest sources. The we build using cbs build. Cbs is command line tool built in C#. ‘Cbs build’ computes the dependency graph between projects and build them in the right order. Cbs calls MSBuild for building each project, but it manages the build queue itself. It gives us more control than generating a build file for example.
• #13: Then we run test uwing NUnit. we have a runner in CBS that allows us to run tests of different assemblies in parallel.
• #14: And finally we publish artifacts of the build: We publish a file, where we store the sha1 that was used for each repo. It makes hotfix possible. We also store a dependency graph of all projects. This graph allows cbs to answer the question: what is the clients of my projects ? We store all the deployable artifacts, ready to be deployable in test environments and production We also store individual assemblies, the DLL files, that will be useful for the developers.
• #15: Cbs checkout --with-dependencies gets the source code of the project I want to work on, and all the source code of its dependencies It generates a solution to build the project and the dependencies. It can also be built usng cbs build, To get the latest commits of your dependencies, just run git pull in all repos (or cbs refresh sources)
• #16: It’s also possible to build using only the source code of the project you want to work on. You can ask cbs to not get the source code of your dependencies. It will get the assemblies of the latest MOAB build on Jenkins instead. It’s laike using snaphot in Maven. To get the latest commits of your dependencies, run ‘cbs refresh moab’, which will download the latest built assemblies. In our projects we don’t have any version or hintpath, when referencing an internal component. A special targets file, cbs.targets, tells msbuild where if has to find the dependencies, depending on the fact we have the source code or not.
• #17: Pre-submit tests are executed when a commit is proposed for review. It verifies that the commit is not going to break any source code that is depending on it. We use cbs that knows the dependency graph of all projecs to compute the project list impacted by the change. If pre-submit tests fail, then the commit is not merged in the trunk.