How to use Git submodules to facilitate your development routine

I have been using git submodules for a while now and I was sometimes lost on how to use it correctly and efficiently. This post will guide you through the needs and usage of a Git submodule.

What is Git submodule?

Git submodule is a Git command that essentially allows external repositories to be embedded within a dedicated subdirectory of the source tree.

Why using Git submodule?

It often happens that while working on one project, you need to use another project from within it. Perhaps it’s a library that a third party developed or that you’re developing separately and using in multiple parent projects.

In my opinion there are several situations where using submodules is pretty useful:

  • When you want to separate a part of your code into another repository for other projects to use it and still be able tu use within your project as a dependency.
  • When you do NOT want to use a dependency manager (such as Composer for PHP). Using Composer for personal repositories can be overkill, submodules in that case is a fair choice.
  • When you do use a front end dependency manager type Bower. Bower is good but only get distribution files of each dependencies. It will never clone Git repositories (Therefor your dependencies are read-only).

Starting a repository with submodules

From scratch

Using a Git repository that do not have submodules yet.

Add a submodule

$ git submodule add git@github.com:shprink/BttrLazyLoading.git

From existing submodules

Using a Git repository that already have submodules registered.

Register submodules

$ git submodule init
Submodule 'lib/BttrLazyLoading' (git@github.com:shprink/BttrLazyLoading.git) registered for path 'lib/BttrLazyLoading'
Submodule 'lib/ShprinkOne' (git@github.com:shprink/Shprink-One.git) registered for path 'lib/ShprinkOne'

Checkout submodules

$ git submodule update
Cloning into 'lib/BttrLazyLoading'...
Submodule path 'lib/BttrLazyLoading': checked out '270b55e177ca555bb4fa559d0e663178ac5006a3'
Cloning into 'lib/ShprinkOne'...
Submodule path 'lib/ShprinkOne': checked out '0dddd0f3e24a473675022c7f79c6b1a27a095914'

From .gitmodule file

Using the .gitmodule file of another repository you can build your own by just running a shell script.

.gitmodule file

[submodule "lib/native"]
path = lib/native
url = git@github.com:shprink/BttrLazyLoading.git
[submodule "lib/mobile"]
path = lib/mobile
url = git@github.com:shprink/Shprink-One.git

Shell script

Create an empty file yourfilename.sh and paste the script below:


#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule..*.path$' |
    while read path_key path
    do
        url_key=$(echo $path_key | sed 's/.path/.url/')
        url=$(git config -f .gitmodules --get "$url_key")
        git submodule add $url $path
    done

Run the script by running the command:

$ sh yourfilename.sh

Daily routine

Now that your repository is ready, it is time to start working! Using submodules can increase the number of commands you will need to run, but not necessarily. I personally gained productivity using them.

Status

The git submodule status command let you see the latest commit hash and the branch currently checked out.

$ git submodule status
270b55e177ca555bb4fa559d0e663178ac5006a3 lib/BttrLazyLoading (heads/master)
0dddd0f3e24a473675022c7f79c6b1a27a095914 lib/ShprinkOne (heads/master)

Update

The git submodule update will update your submodules according to the latest source tree (Root repository) known. It means that if your source tree is not up to date, you can end up using old versions of your submodules. This is pretty annoying, especially when working with a team.

If you want to be always up to date it means that your team needs to be rigorous on updating the source tree on every submodule’s commit…

After experiencing this problematic I decided to use the git submodule foreach command instead. It simply loop over your submodules and execute something.

update master on all submodules

$ git submodule foreach 'git pull origin master'

Commit

Now that you have worked on one or several submodules, it is time to commit your changes. to make sure you are ready to share your work run git status on all submodule:

$ git submodule foreach 'git status'
Entering 'lib/BttrLazyLoading'
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#    modified:   newFile.js
#
Entering 'lib/ShprinkOne'
# On branch master
nothing to commit (working directory clean)

Add your new files

$ git submodule foreach 'git add --all'

Commit your changes

$ git submodule foreach 'git commit -m "your commit message" || true'

Using || true makes sure that the loop will not stop when a repository with nothing to commit is reached. Otherwise you will get the following error message:

Entering 'lib/ShprinkOne'
# On branch master
nothing to commit (working directory clean)
Stopping at 'lib/ShprinkOne'; script returned non-zero status.

Push to your remote

$ git submodule foreach 'git push origin master'

Sources

Leading the SVN to Git migration

Once upon a time I started a new challenge within a new company. This company grew up so fast that the development processes and tools were not suitable anymore. Most projects started with only one developer and, at the time the company chose Subversion (SVN) as the main Concurrent Versions System (CVS) and a classic Waterfall process (linear) to manage projects. This choices froze things for years..

Coming from a large team (10 members) in a totally Agile environment the gap I faced was kind of huge. The lack of productivity that the company faced resulted from those ancient choices. A colleague and I decided to change things by introducing several development tools such as Git, Gitlab (GitHub alike), Jenkins (Continuous Integration), Composer (Dependency Manager for PHP) and Scrum (Agile development framework).

The SVN to GIT transition can be pretty hard to handle, depending on the size of your company, the number of repositories at stake and the number branches, tags created…

To avoid losing time and money, things have to be well prepared. In this post I will share my experience and some tricks to lead successfully the SVN to GIT transition.

Why companies still use Subversion?

Despite the popularity of GIT nowadays there is still a tremendous amount of company that still use Subversion

Git is not better than Subversion. But is also not worse. It’s different. There are many reasons possible why companies still use Subversion:

  1. Subversion’s initial release was in 2000 while Git was in 2005. Any Software Company created before 2005 might therefore have used Subversion (It is the case of eBay, inc. for example)
  2. Subversion is less complex and suit better developers working alone. Indeed Git adds complexity. Two modes of creating repositories, checkout vs. clone, commit vs. push… You have to know which commands work locally and which work with “the remote”
  3. The company grew up too quickly: The lack of time and money that most start-ups face can lead to underestimating developers needs and making unreasonable architecture choices.
  4. The fear of changing well explained by Kurt Lewin being a three-stage process:
    1. Unfreezing: dismantling the existing “mind set”, defense mechanisms have to be bypassed
    2. Movement: When the change occurs, this is typically a period of confusion and transition
    3. Freezing: The new mindset is crystallizing and one’s comfort level is returning to previous levels

What does Git do better than SVN?

I invite you to watch this video about Linus Torvalds (Git creator and Linux project’s coordinator) tech talk at Google.

If you like using CVS, you should be in some kind of mental institution or somewhere else. Linus Torvalds

Git is Scalable

Git is perfect to handle medium to large projects because it is scalable, the more files and developers involved in your project the more you can leverage from it.

Git is Distributed

Git is a DVCS (Distributed Version Control System), meaning that every user has a complete copy of the repository data (including history) stored locally (While SVN has one Central repository). Developers can therefore commit off-line, and synchronize their work to distant repositories when back online.

Git branching and merging support is a lot better

SVN isn’t branch-centric while Git is designed around the idea of branching. Making branches, using branches, merging two branches together, merging three branches together, merging branches from local and remote repositories together – Git is good at branches.

Git Flow

Git-Flow is a set of git extensions that handle most high-level repository operations (feature, hotfix and release) based on Vincent Driessen’s branching model. I recommend using this tool but only if you understand Git basic commands (git branch, git checkout, git pull etc.) otherwise in some cases you will get lost.

OSX Installation
$ brew install git-flow
Linux Installation
$ apt-get install git-flow
Windows (Cygwin) Installation
$ wget -q -O - --no-check-certificate https://github.com/nvie/gitflow/raw/develop/contrib/gitflow-installer.sh | bash

Git is Speed

Git operations are much faster than on SVN. All operations (except for push and pull) are done locally, there is therefore no network latency involved for most of the daily routine commands (git diff, git log, git commit, git branch, git merge etc.). A Git repository is also around 30x smaller than a SVN repository which is not negligible when cloning (backup).

Preparation

Teaching

Being on GitHub does not mean being competent at GIT!

Teaching Git to developers is essential. Git can be hard to learn, especially if you are used to SVN. You need to insist on what’s better than SVN and the things that you now can do with Git that you could not with SVN. I have seen developers on Github that were lost using Git in a work environment, being on GitHub does not mean being competent at GIT!

Here is some of the slides I presented to developers:

[iframely]http://www.slideshare.net/julienrenaux/git-presentation-31666204[/iframely]

Creating a authors.txt file to map SVN users to Git users.

SVN and Git do not store the same way the developer identity when committing. SVN stores the username while Git stores the full name and the email address.

Therefore prior to migrating to Git, you need to create an author mapping file that has the following format:

fdeveloper  = First Developer <first.developer@company.com>
sdeveloper = Second Developer <second.developer@company.com>
tdeveloper  = Third Developer <third.developer@company.com>
etc..

If you have missed this step and already migrated to Git, don’t worry, we still can rewrite history using the command git filter-branch!

git filter-branch --commit-filter '
        if [ "$GIT_COMMITTER_NAME" = "fdeveloper" ]; // SVN username
        then
                GIT_COMMITTER_NAME="First Developer";
                GIT_AUTHOR_NAME="First Developer";
                GIT_COMMITTER_EMAIL="first.developer@company.com";
                GIT_AUTHOR_EMAIL="first.developer@company.com";
                git commit-tree "$@";
        else
                git commit-tree "$@";
        fi' HEAD

This command will go through every single commit and if necessary change the developer information.

Using this command could be pretty slow depending of the number of commit involved.

Migrating

git svn is a simple conduit for changesets between Subversion and git. It provides a bidirectional flow of changes between a Subversion and a git repository.

Clone

For a complete transparent transition (importing commit history) you will need to use GitSvn

$ git svn clone -A ~/Desktop/authors.txt svn://IP@/Project/trunk .

You can also tell git svn not to include the metadata that Subversion normally imports, by passing –no-metadata

$ git svn clone -A ~/Desktop/authors.txt svn://IP@/Project/trunk . --no-metadata

Migrate tags

This takes the references that were remote branches that started with tag/ and makes them real (lightweight) tags.

$ git for-each-ref refs/remotes/tags | cut -d / -f 4- | grep -v @ | while read tagname; do git tag "$tagname" "tags/$tagname"; git branch -r -d "tags/$tagname"; done

Conclusion

Do not underestimate the unwillingness of your coworkers!

If you want to succeed your migration I suggest you spend a lot of time teaching Git to developers, confront them with real cases and things that Git do better than SVN, do not underestimate the unwillingness of your coworkers!

Once everybody is up to date with Git, then you can start working on your migration. If you use GitSvn things should be pretty smooth but do not forget to make backups, just in case 😉

Sources

How to automatically checkout the latest tag of a Git repository

Now that you know How to automatically tag a Git repository an interesting command to know about is “How to automatically checkout the latest tag of a Git repository”. Let’s imagine that you want to automatically release a stable version of your product (that have dependencies or not) based on the latest version available on your repository.

An easy way to do it is to follow the following steps:

# Get new tags from the remote
$ git fetch --tags

# Get the latest tag name
$ latestTag=$(git describe --tags `git rev-list --tags --max-count=1`)

# Checkout the latest tag
$ git checkout $latestTag

How to automatically tag a Git repository using NodeJs, CoffeeScript and Cake

If you have worked in a project that have several developers and several repositories, then you know that tagging can be pretty annoying.

Indeed to tag a repository you need to know two things: The latest version of your application and whether the new version contains bug fixes, new features or things that would break the backward compatibility (If following a Semantic Versioning based on MAJOR.MINOR.PATCH).

The purpose of this tutorial is to be able via a Cakefile to tag a repository using a simple task, without knowing the number of the latest version of your application.

Get this tutorial source code from GitHub

What is a Cakefile?

CoffeeScript includes a (very) simple build system similar to Make and Rake. Naturally, it’s called Cake, and is used for the tasks that build and test the CoffeeScript language itself. Tasks are defined in a file named Cakefile, and can be invoked by running cake [task] from within the directory. To print a list of all the tasks and options, just type cake.

What is Semantic Versioning?

Given a version number MAJOR.MINOR.PATCH, increment the:

  • MAJOR version when you make incompatible API changes,
  • MINOR version when you add functionality in a backwards-compatible manner, and
  • PATCH version when you make backwards-compatible bug fixes.

This is not a new or revolutionary idea, but by giving a name and clear definition of the above concept we ensure that the Software user is aware of your intentions.

Prerequisite

This tutorial requires several prerequisite in order to be completed successfully:

  • NodeJs must be installed
  • CoffeeScript must be installed

If those requirements are not fulfilled yet, I invite you to read a post I wrote: How to install Node.js, CoffeeScript, LESS and Uglify-js on Ubuntu

Let’s get started!

Create the needed files

For this tutorial purpose we need only two files:

Cakefile
Contains the tasks to tag the Git repository and all the logic behind it.
version
Contains the latest version of your application
If you are using Unix a simple mkdir Cakefile version would do. If you are on Windows please note that those files do not have any extension (.something) and therefore you are on our own…

Leave the Cakefile empty for now and add a line containing your application latest version (or 1.0.0) within the version file like this.

Add NodeJs packages

We will only need two NodeJs packages, the File System (fs) to read and write the version file and the Child Process (child_process) to execute command lines for us (e.g: git tag -a v2.0.1). Add the below lines to your Cakefile:

# Required packages
fs      = require 'fs'
{spawn}	= require 'child_process'

Create empty tasks

We want to be able to increase the Major, the Minor or the Patch

###
TASKS
###
task 'tag.major', 'Major tag incrementation', -&gt;

task 'tag.minor', 'Minor tag incrementation', -&gt;

task 'tag.patch', 'Patch tag incrementation', -&gt;

To make sure your new tasks are referenced try to run the cake command:

$ cake
Cakefile defines the following tasks:
cake tag.major            # Major tag incrementation
cake tag.minor            # Minor tag incrementation
cake tag.patch            # Patch tag incrementation

Create the methods

Our Cakefile will need four methods to be able to tag a Git repository:

tag()
The entry point method to tag repository.
run()
Execute commands.
getVersion()
Reads the version saved within the version file.
getIncreasedVersion()
Get the current version via the getVersion method and increase the major, minor or the patch.

run(cmd, args, successCallback)

The run method is pretty generic and allow us to execute wathever command needed via the spawn object. It has three parameters:

cmd
[string] The command, in our case “Git”
args
[array] The arguments, options used with the command.
successCallback
[function] Ran once the command is successfully executed.
run = (cmd, args, successCallback) -&gt;
	# Execute the command
	child = spawn cmd, args

	# Listen to exit event
	child.on 'exit', (code) -&gt;
		# Success
		if !code
			successCallback()

	# Listen to errors
	child.stderr.on 'data', (data) -&gt;
		console.log 'Oups something wrong happened: ' + data

getVersion()

Reads the version saved within the version file via the NodeJs File System. It has no parameter.

# define the version file name
versionFile = 'version'
getVersion = -&gt;
	&quot;#{fs.readFileSync versionFile}&quot;

getIncreasedVersion(label)

Get the current version and increase it. It has only one parameter:

label
[string] Could be: major | minor | patch
getIncreasedVersion = (label) -&gt;
	v = getVersion()
	vSplitted = v.split('.');
	switch label
		when &quot;major&quot;
			vSplitted[0] = parseInt(vSplitted[0]) + 1
			vSplitted[1] = 0
			vSplitted[2] = 0
		when &quot;minor&quot;
			vSplitted[1] = parseInt(vSplitted[1]) + 1
			vSplitted[2] = 0
		when &quot;patch&quot;
			vSplitted[2] = parseInt(vSplitted[2]) + 1
	vSplitted.join('.');

tag(version)

The tag method calls the run() method with the appropriate parameters. It has only one parameter:

version
[string] The version in which you want your Git repository to be tagged.
tag = (version) -&gt;
	run 'git', ['tag', '-a', '-m', &quot;&quot;Version #{version}&quot;&quot;, version], () -&gt;
		# Save the new version within the version file if success
		fs.writeFileSync versionFile, version

Populate tasks

Update the tasks by adding a call to the tag method.

###
TASKS
###
task 'tag.major', 'Major tag incrementation', -&gt;
	tag getIncreasedVersion 'major'

task 'tag.minor', 'Minor tag incrementation', -&gt;
	tag getIncreasedVersion 'minor'

task 'tag.patch', 'Patch tag incrementation', -&gt;
	tag getIncreasedVersion 'patch'

Make a release!

Everything is now ready to execute our tasks. Let’s try shall we!

Release a Patch

$ cake tag.patch
Increasing from 1.0.0 to 1.0.1...
command used: git tag -a -m &quot;Version 1.0.1&quot; 1.0.1

Release a Minor

$ cake tag.minor
Increasing from 1.0.1 to 1.1.0...
command used: git tag -a -m &quot;Version 1.1.0&quot; 1.1.0

Release a Major

$ cake tag.major
Increasing from 1.1.0 to 2.0.0...
command used: git tag -a -m &quot;Version 2.0.0&quot; 2.0.0

In a next post I will continue working with this Cakefile and explain how to make a complete release of code (Minification, Packaging and Tagging). This tutorial is now finished, let me know if you liked it or not by commenting 🙂

Get this tutorial source code from GitHub

Sources