Version control for games

Reasons why Game Development teams are changing their version control

  • A strong need for better branching and merging
  • A better way to manage distributed development
Download now!
Picture: ASHES OF MANKIND by Black Ice Studios

Specific version control challenges in game development

Handle huge projects

Handle huge projects (300k files is regular count for a workspace, 1.5 million is big).

Collaborate effectively on code & art

Collaborate effectively on code & art among different teams (and studios) in distant locations – potentially transferring gigabytes.

Handle huge repository sizes

Handle huge repository sizes including really big binaries (typically game art, both in raw and processed formats).

Artists work on files that can’t be merged

Artists work on files that can’t be merged – so they need exclusive checkouts (locking).

Need to work centralized

Need to work centralized (without local repositories) in order to save space and make many team members feel at home with the workflow. Since teams can be pretty big, server scalability and performance under heavy load is a must.

Flexible workspace configuration

Flexible workspace configuration: being able to work on sparse trees. It is important to avoid losing time downloading huge binaries that won’t be used all the time.

Direct access to the engineering team

Direct access to the version control engineering team: tools teams inside game studios face huge challenges to keep the pipeline up and running. Engineer to engineer communication saves them precious time.

A clear migration path

A clear migration path from the current version control into the new one, so that the move is as painless as possible.

Good ways to structure the codebase hierarchies

Good ways to structure the codebase hierarchies to simplify component oriented development: solid alternatives to monolithic repositories that can be hard to maintain and evolve.

In short: game dev teams need the distributed and branching/merging capabilities of the modern version controls, but can’t afford losing any of the strengths of their current systems: performance, scalability, exclusive locks and centralized workflow support.

GameDev Scenarios

Game engine scenarios

Collaborative and multi-site GameEngine Development
Description
  1. Engine development happens at the central New York studio.
  2. Each time the engine team publishes a new upgrade or major release, the different game teams will pull the changes from the central server to their team servers. They will then integrate the engine changes in the games they’re developing.
  3. Engine development typically follows a shared development model, so teams will make changes to the engine to make improvements to their game or fix issues. They’ll be able to push back the changes to the ‘engine team’ so that they benefit the rest of teams.
Benefits
  1. Distributed development using replica: no need for permanent connection between game teams and the ‘engine central server’. Each team will have its own server, so access is local for each of them. View more
  2. Partial replica: game teams don’t need to replicate the entire ‘engine repo’, they’ll just need to get the ‘main branch’ (release branch). They’ll still be able to push back. View more
  3. Optimized network performance: for data transfer through high latency networks.
  4. Branching and merging: to structure the development process. View more
  5. Distributed concurrent development: even when changes were made in the same branch on two different sites, Plastic is able to merge them back. View more
  6. All network data transfer can be secured through SSL – Plastic implements native SSL support.

Distributed teams inside different distributed studios

Description
  1. The company has different studios around the world.
  2. The engine studio is based in New York. Different studios will push and pull to it.
  3. One of the studios, in London, has different teams that are also distributed in different sites.
  4. The scenario shows a hierarchy of servers:
    • One ‘main engine’ server in New York.
    • One ‘studio’ central server in London.
    • Developers working in centralized mode in London (direct LAN connection to the server).
    • Two additional sites with additional servers. They’ll push and pull from the London server.
    • One developer working from home and having a local partial replica of the repository (or repositories) he needs to work.
Benefits
  1. Multiple servers that can use different underlying technology: suppose the New York studio is a ‘Windows shop’ while the London one prefers to run servers on Linux. Replication will work independently of the underlying operating system and database backend.
  2. Servers running with different database backends and hardware requirements:
    • The New York studio can be a big server (32Gb RAM, 16 cores) using a SQL Server and handling the load of 400 concurrent developers.
    • The London studio can use a big server (16Gb, 8 cores) using MySQL and handling the load of 100 concurrent developers.
    • The developer working from home will run a replicated server running on his laptop and using a SQLite backend (really small memory footprint and only handling single user load).
    • The 2 distributed sites inside the ‘UK Studio’ can use much smaller machines to handle the load of 10 developers each.
  3. Replication between environments ensure proper asset and code delivery.

Remote Access and network optimization using Proxies

Proxy vs distributed:

Under some specific circumstances teams will prefer proxies than using a replicated server.

The benefits of distributed/replicated over proxy based solutions are clear: if the connection goes down, the teams depending on proxies won’t be able to continue working with the main server.

That’s the reason why we always encourage the use of distributed servers instead of proxies for distributed scenarios.

Plastic is all about flexibility, though, so a proxy server is available.

Typical proxy-based multi-site scenario. Plastic recommends the use of distributed servers instead (proxies are supported too)
Using proxies for sites with really heavy load:

There are scenarios where we recommend the use of proxies:

  • Consider a centralized setup with several hundreds of developers working directly with the same central server.
  • Different developer groups are on different network segments
  • The high data traffic (more than 1000 developers overall) can potentially be too high to be directed to the central server.

Under these circumstances is where introducing a proxy server on each network segment can dramatically improve overall performance by reducing data requests (updates) to the central server.

Proxies on different network segments on a central server scenario

Centralized development

Description
  1. Inside a given location, programmers and artists specially, work directly connected to the version control server without any local clone.
  2. This direct connection to the server enables a simpler ‘lock-edit-checkin’ cycle: more natural for artists and developers. Disk space is also saved (considering in game dev projects it is measured in GB).
  3. The main server has to handle a big load from potentially hundreds of concurrent users, avoid unwanted locks, be responsive and fast.
Benefits
  1. Plastic SCM supports the centralized and distributed scenarios. A Plastic workspace can be directly connected to the central server, no need for intermediate repository clones (although it is a possibility). That’s why Plastic can work in *real* centralized mode.
  2. Plastic supports heavy load on a single server as it is explained in this test scenario. Any team interested on checking how Plastic behaves under heavy load will be granted access to the Amazon EC2 setup to see it in action and even alter the scenario to better reproduce their environment.
  3. Multiple Plastic servers can be configured to split the repository catalog among multiple machines if necessary.
  4. Speed is also remarkably fast as explained in the ‘performance results’ sections, consistently beating established competitors.

Performance, scalability and network optimizations

Optimized network channel for high latency

Optimized data transfer for high-latency & high bandwidth networks – intercontinental data transfer

While working with different studios we found out that transferring large amounts of data between distant sites is a key operation for many of them.

That’s why we included an improved ‘WAN network channel’ in Plastic SCM 5.4.

Data transfer between servers hosted on different continents faces different problems than LAN scenarios.

Typically when replicating data between distant servers through the Internet, applications have to deal with high latency, so even when the bandwidth is high the final transfer rate will be greatly reduced.

TCP is not ideal under these circumstances and that’s why the new WAN channel takes advantage of a UDP based protocol to improve the transfer rate. A 2.3 improvement is easy to achieve with a 100msec latency, and then as the latency increases, WAN channel outperforms the TCP one up to 5-10 times.

Scalability results of Plastic SCM

How does Plastic SCM scale under heavy load?

One of the key features for game dev teams is to be able to use highly scalable centralized servers

We run frequent scalability tests using automated bots that simulate actual users to tune Plastic under heavy load.

We have an Amazon EC2 based setup to enable teams to connect and check scalability by themselves. The typical scenario is described here including a screencast showing how a developer using the Plastic GUI is affected by the load created by hundreds of concurrent users.

This section explains how Plastic scales using different operating systems and database backends.

Test Description:

    Each “test bot” performs the following operations:

  1. Create a branch.
  2. Switch to it.
  3. Loop 100 times:

  4. Modifies 10 files.
  5. Checkins the 10 files.

So each “test bot” performs 100 checkins of 10 files each.

Code base size: 197,356 files, 15,179 directories – 2.59 GB

Linux server setup:

OpenSUSE 12.3 x64 14.0 GB RAM, 3.00 GHZ 8 processors

MySQL (7GB innodb) MySQL-server-5.6.15-1.linux_glibc2.5.x86_64.rpm

Client setup:

Windows 7 Pro SP1 64 bit, 8GB RAM 3.60 GHZ Intel Xeon E5-1620 0

Windows server setup:

Windows Server 2012 R2 x64 14.0 GB RAM, 3.00 GHZ 8 processors

SQLServer 2012 (Limit to 7GB max mem)

MySQL (7GB innodb) mysql-installer-community-5.6.15.0

Linux – std sgen means the test uses the standard sgen garbage collector with no extra params
Linux – boehm means the Plastic server runs on the standard setup, using the Boehm garbage collector
Linux – tweak sgen means the Plastic servers uses sgen with the following params (highly recommended for high load linux servers) - Mono 3.0.3 provided with PlasticSCM5, with sgen. Server launched with the following MONO ENV variable: MONO_GC_PARAMS=nursery-size=64m,major=marksweep-fixed,major-heap-size=2g

MySQL – int ids shows how the overall performance is greatly improved when the database stores object identifiers using ints instead of longs. This improvement is already present in the 5.4 release.

Conclusion

The results show how Plastic scales under a really heavy load scenario. The Windows server + SQL Server combination probes to be the fastest, closely followed by the Linux setup correctly configured to work under heavy load

It is important to highlight that the number of checkins performed during the test by each ‘test bot’ is extremely high compared to what a human developer will do. This test with 100 bots can be easily compared to a real scenario with several hundreds of concurrent developers.

Performance results of Plastic SCM

How does Plastic compare to other version control systems in pure speed?

We frequently benchmark Plastic against other key version control systems known for its high performance and continuously improve in key operations such us add, checkin and update.
We’ve compared Plastic, Git and a very well-known commercial version control system extensively used in the game industry. We’ve tested with 3 types of codebases: small, medium and large.

The tests benchmark a full add+checkin (adding the entire code base to the version control) and also a ‘clean update’ (downloading the entire codebase to a clean directory).
While certainly adding a big codebase of 140GB is not something a team will do on a daily basis, it shows how Plastic can handle the scenario. When the data to be checked in is really large, disk and network IO tend to be the bottleneck, so probably the ‘medium’ scenario is the most relevant since even in large repositories the team members will more often do ‘medium’ checkins (or even ‘small’) than large ones.

All the tests have been performed with Plastic SCM 5.4 using the ‘filesystem blob storage’ which saves big blobs (bigger than a certain configurable size) on the filesystem instead of the database. Plastic uses a SQL Server database backend.

When comparing to Git it is important to highlight that Plastic is configured in client/server mode, so all data transfer has to be sent through the network.

SMALL - 45,925 files 2,946 directories – 601 MB
The code for the SMALL repo is the Linux kernel

Plastic is 7.3 times faster than ‘commercial version control’ doing add + checkin

Plastic is 4 times faster than Gitconsider that Git is checking in locally while Plastic is sending data through the network from client to server


MEDIUM - 221,141 files 13,247 directories – 8.08 GB
The code for the MEDIUM repo is created as follows: linux + mono + openoffice + alienarena + war.dat

This scenario is specially relevant because even if game teams use LARGE repositories, most of the time they’ll be doing ‘medium’ checkins (it is not so common to checkin 142Gb as it is a few Gb). In these cases is where Plastic shines since it excels managing the metadata and IO speed is not yet the limit factor.

Plastic is 6 times faster than ‘commercial version control’ doing add + checkin

Plastic is 3.3 times faster than Gitconsider that Git is checking in locally while Plastic is sending data through the network from client to server.


LARGE - 360,399 files 22,572 directories – 142 GB
The code for the LARGE repo is created as follows: mediumWks + mono (again) + 4 x example data containing big binaries (from a real game)

Plastic is 3 times faster than the ‘other commercial version control’.

Plastic needs 35 minutes less to complete a add/ci than Git -> 1.7 times faster.


Note: the Git test was repeated several times because we found occasional issues (out of memory).

It is important to highlight that:
  1. With a checkin of 142 Gb most of the time is spent in network data transfer and writing data to disk. Disk and network IO set the limits in these scenarios.
  2. While a LARGE repo is common in the gaming industry, adding 142 Gb is not the most common scenario. It is more common to do updates or checkins of smaller sizes *inside* the LARGE repo. In these scenarios the way to handle the metadata is more relevant than the IO limits, and Plastic clearly beats competitors.
Test environment:
  • Both client and server:

    CPU: 4CPU, 14 ECU (x64) / Intel Xeon CPU E5-2670 v2 @ 2.50GHz
    RAM: 30.5 GB
    Network: 1Gbps
    OS: Windows Server 2012

  • Disk IO:

    Server Machine: 64.5MB/sec Read speed (HDTune)
    Client Machine: 145 MB/sec Read speed (HDTune)
    Write speed: 195Mb/s (cm iostats)

The challenges of game development

Meet Digital Legends Entertainment as they explain the challenges they faced delivering new multi-platform titles and how they transitioned to Plastic SCM to reduce their time to market.

Digital Legends moved from a centralized workflow to a fully parallel one, embracing branching and merging and connecting distributed teams through Plastic SCM.

  • The future of gaming
    Xavi explains his view on the future of games and the challenges for the industry of game development in the coming years.
  • Game development is changing
    With new downloadable content releases and changing business models, games today are dramatically different. Unai explains what it means for a dev team.
  • Digital Legends and Codice Software
    Digital decided to migrate away from their previous version control solution into Plastic, and it ended up being much more than a tool change, it enabled a new way of working.
  • How Plastic SCM improves game development
    From branch per task to game-content-aware distributed version control, Xavi explains the impact of a version control change.

Other testimonials

“The Telltale team adopted Plastic SCM and had contributed to design and test the new Plastic Gluon to improve our version control workflow”

Zac Litton, CTO at Telltale Games.

Would you like to develop a cool new Unity game and not worry about losing your file changes?

Go to Unity stealth demo

Get the latest news

View more... x