1 Overview
The current guide explains the Plastic SCM distributed
system capabilities: a general description of the different distributed and
replication scenarios supported, followed by detailed explanations.
Plastic SCM distributed capabilities enable
setting up different servers for multi-site development support, being able to
both replicate and reconcile changes made on replicated branches.
This document will give a general overview
of the system’s distributed capabilities.
Plastic SCM has the ability to create
different multi-site scenarios ranging from single-server to fully distributed
deployments.
Figure 1. Deployment scenarios supported
As Figure 1 shows,
Plastic SCM can be configured to work in a single server mode, which is the
default mode on installation and the conventional mode available on all SCM
products.
The next step has been called classic multisite in which several
servers exist, one for each development location, and contents are replicated
among them. The basic rule at this distribution
stage is that branch mastership is
kept by only one site at a time: if a branch is modified at one site, the other
ones won’t modify it until the branch is replicated again. In many systems this
behaviour is encouraged by the software itself,
preventing simultaneous changes in a master/slave
relationship. Plastic is not restricted to work on this mode. Several
deployment strategies will find this configuration possibility useful, but
Plastic won’t have any restriction to perform changes in the distributed copies
because it doesn’t implement any master/slave functionality, although it can be
simulated by the system administrator if necessary using both permissions and a
clear replication policy.
Full multisite support is almost identical
to the previous distribution stage with only one difference: all the SCM
servers can modify their branches at any time. Changes can be reconciled back
later on if the same branch is modified more than once at different locations.
Full distribution is exactly the same as
full multisite, but on this deployment scenario each developer has his own SCM
server. There’s only one restriction imposed to systems working on this mode:
servers must be light enough to run on non-dedicated workstations and even
laptops. Plastic SCM servers can be easily configured to work in this mode,
introducing full disconnected support. A developer can take his laptop home and
continue working as if he were at the office and reconcile his work when he’s
back at the office.
The main distributed operation is replication. By means of this operation
repositories can be distributed on several machines. The replication unit in
Plastic SCM is the branch. Users specify a branch to be replicated from a given
source repository to a destination one, and then the revisions on the branch
plus their labels, links, attributes, changesets and so on
will be replicated from the source to the destination repository.
Figure 2. Branch replication between two locations
Figure 2 shows
two repository servers at two different locations. The server at location 2 has replicated the branch main at location 1. Then it has created another two branches which have
been later on replicated into location 1.
As you can see from the figure, distributed
repositories don’t have to be exact clones. They share replicated branches and their contents but the entire repositories
don’t have to be identical, they can evolve separately sharing some common
branches.
There are several possible distributed
scenarios with Plastic SCM. They will be explained in detail in this topic.
In this scenario two or more servers are
used in replication. Servers will normally run at different locations to enable
geographically distributed teams to work together on the same project. A server
at each location will solve the problem of slow or unreliable internet
connections between sites.
Figure 3. Multi-site readonly
Figure 3 shows
both a deployment diagram and a detailed view of the branching strategy. This
set up resembles classic multi-site
replication as implemented by many master/slave
based products. In Plastic this scenario is just one possibility and it will be
used to explain replication.
The two sites, location 1 and location 2,
will have their own servers. Both sites will be working on the same code-base,
so developers will need to check-in changes at any time. The chosen strategy would
be the following one:
- Both servers will have an exact replicated copy of the main
branch, containing the latest baseline.
- The new baselines will be generated only in one server at a
time, so they will be implementing a sort of mastership behaviour. Let’s assume new
baselines will be created on server
01.
- Developers at both locations will follow the branch per task pattern. Branches
will be created using the latest baseline as starting point. The main
branch won’t be modified in parallel at the two sites.
- Periodically, for instance twice a week depending on the amount
of work finished, all the task-branches
created at site 2 will be
replicated into site 1. All
branches will be integrated on main, tested, and a new baseline will be
created. Alternative integration branches could exist at each site in
order to ease the integration process.
- Once the release is finished the main branch will be replicated
from site 1 to site 2 and a new
development iteration will begin.
Figure 3 shows
how branches are replicated from site 2 to
site 1 in its lower area.
Figure 4 and Figure 5 describe
the previous scenario step by step. They show how the main branch is first
replicated from location 1 into location 2, how the newly created
release 58 is then available to the
two development groups.
Then the groups start working on and
creating task branches independently
from each other, but starting at a well-known point: release 58.
Once the iteration is finished, branches task1012, task1013 and task1030 created
at location 2 are replicated to location 1 to be integrated.
Once the integration is finished, the
branch main will be replicated again
to location 2, so that the
development group there can continue working with the latest approved baseline.
Note that both repositories are not
identical after the development iteration finishes, but the content on the main
branches, considering they’re being modified at only one site, is exactly the
same.
The deployment required for this scenario
is exactly the same as in the mastership
case. The difference will be on the way in which the replicated branch evolves.
Now developers will make simultaneous changes to the replicated branch and
Plastic will have to help reconciling these changes together.
Figure 6 depicts
the situation: developers at two different sites are working against the same
branch which has been replicated from location
1 to location 2. Then both groups
perform changes directly on their replica
of the main branch.
Figure 6. Multi-site replication with changes performed in parallel
Changes are drawn as performed directly on
the main branch, but sites could also be using the branch per task pattern and directly integrating their own work on main.
When site
1 (or viceversa) requests changes made at site 2 using replication, the newly
created changesets on branch main at site 2 will be linked to the right
changesets on the site 1, keeping the correct changeset linking. If a subbranch is created (more than one head or “last
changeset” on a given branch, which can happen after pulling changes from a
remote repository) it will have to be merged in order to reconcile the changes
created remotely.
At a pure distributed scenario there isn’t
a central server, but each developer runs his own server containing his own
repositories.
This strategy can be fully implemented with
Plastic configuring a server on each developer’s workstation.
This fully distributed scenario can be
adopted by any company, although they would normally prefer to count on a
central copy. On distributed development there will always be a master server, not necessarily due to
software restrictitions but to some sort of meritocracy, as it happens at open
source projects. So it is usually better to explicity
decide which one will be the computer containing the well-known stable
releases. Obviously there will be more than one satisfying this requisite, but
it is better for simplicity to exactly determine which will be the master one at any time.
In corporate
scenarios this pure distributed ability can be tuned to support a mixed
scenario:
- Onsite developers continue using a regular client/server configuration when working against a central
server.
- The central server plays the role of master copy.
- Developers working at a different location (at home, for
instance) have their own repository server which they can keep in synch
regularly.
- Developers working on laptops can also run their own servers
and then implement fully
disconnected support.
Alternatively all developer’s workstations
could run Plastic SCM servers. This is totally supported but the system. Using
or not this capability will depend on the organization itself, developers’
skills and the amount of administrative burden required.
Figure 7
depicts the concepts described above.
Figure 7. Fully distributed scenario in detail
So far general distributed system’s behaviour has been introduced. This topic will explain in
detail how Plastic SCM replicates changesets between branches on different
repositories and how to reconcile conflicting changesets created in parallel in
the same replicated branch on two different repositories.
The diagrams and samples introduced above
focused on overall branch behaviour. Figure 8
details a replication sample studying what happens at the changeset level.
Figure 8. Replication: step by step explanation.
The sample focuses on a file named /src/main.cpp
at the branch main/fix. The branch is
replicated from repository A at Location A to repository B at Location B. Note
that the figure specifies the Plastic command needed in order to run a
replication.
At step
1 there is only one changeset on the two replicated repositories,
containing the first change on /src/main.cpp.
Step
2 shows how the file is modified at rep A: two new revisions are created.
At step
3 the developer at location B runs
once more the same replication command. The two new revisions created at rep A are now
copied into rep B.
During replication Plastic first pulls the changesets
at the branch specified by the user (starting at the last previously replicated
changeset if any). Then it will try to pull the changesets from the source
repository. To do so, Plastic finds the parent changeset of the new changesets
being pulled, and links them accordingly.
At step
4 the developer at rep B makes a
new change starting from the latest replicated changeset and modifying again main.cpp.
At step
5 the developer at rep A replicates /main/fix
at rep B. The newly created changeset
3 gets replicated and correctly
placed on his repository.
Note that the sample from Figure 8 shows
only one change at a time on the branch, so no conflicts can happen. While
following this strategy the two replicated branches will continue being exact
clones on replication.
Figure 9 shows
a more complex scenario. Both locations start with the same configuration:
three changesets at branch /main.
At step
2 the two repositories evolve in parallel when the developers introduce new
changes on main.cpp.
At step
3 the user at rep A tries to
replicate changes from rep B. Now
Plastic can’t directly link revisions
3 and 4, created at rep B to revision
2 because a new revision 3 has also been created at the branch.
Note that internally Plastic SCM identifies
each object by a GUID (Globally Unique Identifier. So
don’t get confused by the changeset numbers
shown in the sample.
If changeset 4 at repA
didn’t exist, then Plastic would have placed revision 5 and 6 from rep B just
hanging from the existing changeset 3. But now it can’t do that. So what
Plastic actually does is creating a subbranch to place the replicated changesets.
Figure 9. Replication of branches modified at two locations
There are two replication modes available:
- Direct server to server replication: a Plastic client will tell
the destination server to replicate a branch from a source server. Servers
will communicate to replicate data.
- Package based replication: Plastic client connects with the
source server and creates a replication package. The package will be
imported later on by the destination server.
Figure 10
depicts the two available replication modes.
Figure 10. Replication modes
The package
based replication introduces the ability to keep in synch servers which are
not allowed to connect directly due to security restrictions.
All the replication scenarios and
possibilities described can be set up with a single Plastic command: replicate.
cm replicate srcbranch
destinationrepos
Where src branch is a branch spec identifying the branch to be replicated
and its repository, and destination repos
is the repository where the branch is going to be replicated.
Suppose you want to replicate the branch main at repository code at server london:8084 to repository code_clone at bangalore:7070. The command would be:
cm replicate main@code@london:8084
rep:code_clone@bangalore:7070
To replicate branches using packages the first step will be creating
a replication package, and then importing the package into another server.
Suppose you have to create a replication
package for the main branch at repository code
at server box:8084.
cm replicate br:/main@code@box:8084
–package=box.pk
The previous command will generate a
package named box.pk with all the
content of the main branch.
Later on the package will be imported at the
repository server berlin:7070.
cm replicate rep:code@berlin:7070 –import=box.pk
During replication different servers have
to communicate with each other.This means that
servers running different authentication modes will have to exchange data.
To do so the replication system is able to
set up different authentication options.
Figure 11 shows
a tipycal scenario with a client and two servers. All
the involved Plastic components are configured to work in LDAP and they share
the same LDAP credential, so no translation is required.
Note that authentication happens at two
levels:
- The client needs to be authenticated in order to connect to the
destination server. In the
figure the destination server is berlin.
- Then berlin
will need to connect to server london to retrieve
information about the branch to be replicated (main in the sample).
If both servers were not using the same
authentication mechanism or not authenticating against the same LDAP authority,
step 2 would fail.
Figure 11. Distributed system authentication diagram
Figure 12 shows
a scenario in which the server at london is configured to
use user/password authentication. In
this case a command like the one specified at the top of the figure will fail
because authentication between servers won’t work at step 2.
Figure 12. Specifying credential to replicate between servers
To solve this problem the replication
system has the ability to specify authentication credentials to be used between
servers. In the sample the client can specify to the server berlin a user and password to
communicate with server london.
Figure 13 shows
two different ways to specify authentication credentials when using user/password at the source server.
Figure 13. Two different ways to specify authentication credentials
The first option is actually specifying the
mode plus the user and password (for UP) at the command line.
The second one uses an authentication file, which is useful when authentication
credentials are going to be used repeteadly. As the
figure shows, an authentication file is a simple text file containing two
lines:
- The authentication working mode: UPWorkingMode,
LDAPWorkingMode, NameWorkingMode,
ADWorkingMode or NameIDWorkingMode
- Specific authentication data for the authentication mode. The
data specified in the second column is exactly the same the client
generates at client.conf
configuration file under the key SecurityConfig. So a way to create an authfile is
manually editing the file, specifying the working mode on the first line
and copying the right data from a correctly configured (using the
configuration wizard, for instance) client.conf file.
Suppose now that replication must happen in
the opposite direction, from berlin to london as Figure 14 shows.
The parameters to connect to a LDAP server (in this case an Active Directory acessed through LDAP) are specified. Normally in LDAP an authentication
file will be used to ease the process.
Figure 14. UP server communicating with LDAP one
Note: If replication is performed through replication packages the client needs to
be able to connect to the source or destination servers, depending on whether
it is performing an export or import operation.
When replication is performed between
servers with different security modes, authentication is not the only issue.
User and group identifications have to be translated between the different
security modes.
The sample at Figure 15 tries
to replicate from a user/password
authentication mode to a LDAP based one. The user list at the UP node stores
plain names but the user list at the LDAP server stores SIDs. When the owner of
a certain revision being replicated needs to be copied from repA
to repB, a user or group will be taken from the user
list at repA and introduced into the list at repB. If a name coming from repA
is directly inserted on the list at repB, there will
be a problem later on when the server at berlin tries to resolve the LDAP
identifier because it will find an invalid one.
So in order to solve the problem
translation will be needed.
Figure 15. User and group translation on replication
The Plastic replication system supports
three different translation modes:
- Copy mode: it is the default behaviour.
The security IDs are just copied between repositories on replication. It
is only valid when the servers hosting the different repositories involved
work in the same authentication mode.
- Name mode: translation between security identifiers is done
based on name. In the sample at Figure 15
suppose user daniel has to be translated by name from repA to repB. At repB the Plastic server will try to locate a user with
name daniel and will introduce its LDAP SID into the
table if required.
- Translation table: it also performs a translation based on
name, but driven by a table. The table, specified by the user, tells the
destination server how to match names: it tells how a source user or group
name has to be converted into a destination name. Figure 16
explains how a translation table is built and how it can translate between
different authentication modes.
Figure 16. Translation table explained
Note: a
translation table is just a plain text
file with two names per line separated by a semi-colon “;”. The first name
indicates the user or group to be translated (source) and the one on the right
the destination one.
Replication can be used from both the command
line interface (CLI) and the Plastic Graphical User Interface (GUI) tool. All
the possible actions are located in a submenu under the branch options, because replication is primarily related to
branches. This topic will describe how to perform the most common replication
actions from the GUI.
From the GUI replication and distributed
collaboration has been organized in the following actions:
1. Branch actions:
a. Push the selected branch
b. Pull the selected branch
c. Pull a remote branch
2. Package actions:
a. Create a replication package from the current branch
b. Create a replication package from a branch
c. Import a replication package
The Figure 17
depicts the different available operations. From the command line all the
operations are issued from a single command, but the GUI makes a distinction
between push (move changes from your
server to a destination) and pull
(bring changes from a remote repository to yours) actions.
Figure 17. GUI replication actions
As it was mentioned before, all replication
actions can be accessed from the branch menu (check Figure 18).
The options push this branch, pull this
branch and create replication package
from this branch are related to the branch currently selected in the branch
view. The other options: pull remote
branch, create replication package
and import replication package are
generic replication actions which are not constrained to the current branch but
are located under the branch menu to keep all the replication options together.
Figure 18. Replication options in GUI branch menu
Whenever you want to push your changes to a
remote repository, select push this
branch on the branch menu. Pushing your
changes means sending the changes made on the selected branch to a remote
repository.
Figure 19. Pushing changes
If the branch already exists on the
destination repository the changes will be synchronized
(a warning message will show up if there are conflicting changesets on
destination so the developer will have to reconcile changes pulling first the
branch to the local server and pushing once the merge conflicts have been
resolved). If the branch doesn’t exist on the destination repository a new
branch will be created (identified by the same GUID used on the source
repository).
Check Figure 19 for a
detailed explanation.
Once you’ve pushed your branch to a
different repository, the branch can be modified remotely. At a certain point
in time you’ll be interested on retrieving the changes made remotely into your
branch. In order to do that you’ve to use the pull this branch action from the replication branch menu.
The dialog depicted on Figure 20 is
very similar to the one used to push changes but this time your server is
located on the right as destination
of the operation.
Figure 20. Pull changes from a remote branch
When you pull changes from a remote branch
a subbranch
can be potentially created if there are conflicting changesets on the two
locations.
Another common scenario during replication
is importing a branch from a remote repository into yours in order to start
making changes or create child branches from it.
In order to perform the import
use the pull remote branch
option. The dialog show at Figure 21 will
be displayed. Notice that this time you can choose both the
source server, repository and branch, and also the destination repository on
your server.
Figure 21. Pull a remote branch
As it was described on chapter 7,
different Plastic servers can be using different authentication modes. By
default when you try to connect to a remote server you’ll be using your current
profile (the configuration used to connect to your server). But sometimes the
default profile won’t be valid on the remote server.
In order to configure Plastic to be able to
connect with a remote server with different authentication mode, use the advanced options button on the
replication dialog. It will pull up a dialog like the one on Figure 22.
Figure 22. Advanced options dialog – I3 visual theme
The dialog shows the profile currently
selected (the default one on the
screenshot) and also the translation mode (refer to chapter 7 for
more information) and the optional translation table.
You can have different authentication
profiles created from previous replication operations, and you can list them or
create new ones pressing the browse
button located on the right of the remote
server configuration profile edit box.
It will display a dialog like the one on Figure 23 which
will allow you to select, edit, create or remove a profile.
Figure 23. Profile selection dialog – I3 visual theme
Note: the replication
dialog will try to choose a profile automatically each time you change the
server. It will look for the most suitable profile based on the server
information provided.
So far all the steps have been focused on
setting up the replication process. Once the operation is correctly configured,
press the replicate button and you’ll
enter the replication progress dialog as explained on Figure 24.
The replication operation is divided into
three main states: fetch metadata, push metadata and transfer revision data.
The first one happens on the source server, the second one on the destination
one and the third one involves the two servers as data is transferred from the
source to the destination.
At any point in time the operation can be
cancelled pressing the cancel button.
When the replication operation finishes a
summary is displayed containing detailed information about the number of
objects created.
Figure 24. Replication operation progress – I3 visual theme
A replication package can be created from a
branch on your repository or from any branch on any server you can connect to.
In order to create a package from the selected branch on the branch view, click
on create replication package from this
branch. If you want to create a package from any remote branch, click on create
replication package on the replication menu.
Figure 25. Create replication package from the GUI
Figure 25 shows
the package creation dialog. It will generate a replication package from the
selected branch which will contain all data and metadata from the branch. It
can be used to replicate between servers when no direct connection is
available.
From the replication menu select import replication package and select a
package file to be imported. The dialog is shown at Figure 26.
Figure 26. Import a replication package
9
Distributed Branch Explorer
The “Branch Explorer” is one of the core
features in the Plastic SCM GUI and it has been deeply improved in 4.0 release to be able to deal with distributed scenarios.
That’s the reason why it now receives the “distributed” branch explorer name.
The short name it get is DBrEx.
9.1 How the
DBrEx works
Consider two replicated servers. The second
one has first replicated the “main” branch from the first, and later a second
branch was created and developers worked on it. At a certain point in time it
will look like Figure 27.
Figure 27. Distributed
scenario with two servers
At this point each of the servers contains
part of the development, but prior to 4.0 there wasn’t a good way to understand
the “whole branching schema” other than connecting and browsing each repository
branch explorer separately.
The DBrEx is able
to render a “distributed” diagram collecting data from different sources and
then render the changesets and branches on a single diagram as the Figure 28
shows.
Figure 28. DBrEx draft
The DBrEx will
combine the different sources and create an interactive diagram with the
information gathered from the different sources.
There are several options in order to
combine more than one replicated repository into the same DBrEx
diagram. The first one is used to create a combined render including all the
changesets and branches coming from the selected “replication sources”.
Consider Figure 29 to
start configuring the diagram.
Figure 29. Options to combine remote repositories on the branch explorer
Once you click on one of more “replication
sources” the distributed diagram will be rendered as depicted by Figure 30.
This way the Distributed Branch Explorer
introduces a new way to understand how the project and branches evolve across
different replicas.
It is also possible to run the replication
operations from the DBrEx, so pulling a remote branch
is now as easy as selecting the remote branch rendered in the DBrEx and clicking on “pull this branch”. Remote branches
and changesets are available for “diffing” too, which
greatly enhance the way to work with distributed changes.
Figure 30. Remote changesets rendered on the branch explorer
9.3
Diffing remote branches and changesets
It is possible to right click a remote
branch or changeset on the DBrEx to explore and
understand what was modified remotely. This way developers or integrators can
better understand what changes are going to be pulled from the remote sources
prior to complete the operation. The shows the options enabled on a remote
changeset.
Figure 31.
Menu options (diffing) on remote changesets
Sometimes it is not necessary to render the
whole distributed diagram because the SCM manager or developer needs to focus
on a specific branch only.
Figure 32. Single branch extended with remote information
Figure 32 shows
the Branch Explorer / Show remote changesets menu
option which allows to select a remote source to “decorate” a branch with
remote data to understand what needs to be pulled, explorer differences and
trigger replication commands.
10
Synchronization View
Plastic SCM 4.0 is all about helping teams
to embrace distributed development. To do so, 4.0 enhances the DBrEx but in order to deal with hundreds of distributed
changesets a new “perspective” has been created: the distributed view.
The sync view enables to synchronize any
pair of repositories easily, browsing and diffing the
pending changes to push or pull.
In order to get advantage of the new “sync
view” it is necessary to configure one. The entry point to the “sync view” has
been placed on the left menu bar as depicted by Figure 33.
The entry point will let you browse all the
defined sync views once you’ve created them.
Figure 33. Sync view entry point on the menu
A synchronization configuration is about
selecting a source repository and a destination repository and then run to find
the outgoing and incoming changesets.
The Figure 34 guides
you through the process of setting up a sync view.
Figure 34. Creating a new sync view
10.2Source and destination
explained
Whenever you define a new sync view
configuration you’ll have to define a repository source and a destination. It
basically means which are the repositories (and servers) that will be source
and destination of the replication operations that will be run from the sync
view.
·
You can set up a sync view with
your code repo at your laptop as source and your project repo at the central
server as destination to use it to push and pull changesets from your central
server to your laptop and viceversa.
·
You can set up a sync view
between your main server and a mirror, to use it to keep the mirror in sync.
·
You can set up a sync view to
keep two servers, at two distant locations, in sync.
10.3Sync view
details
The “details” of the sync view shows the
branches to be synchronized grouped in “outgoing” and “incoming” and then
containing the related changesets. Figure 35 shows
a sync view containing the mentioned details.
Figure 35. Sync view details
The menus available for the destination
repository, branches are changesets are depicted on Figure 36.
Figure 36. Available menus for sync view details
As Figure 37
shows, it is possible to define more than one source-destination pair on a
single view. It is useful to replicate more than one repository between the
same two servers.
Figure 37. More than one source/destination pair on a sync view
