GIT and Puppet¶
Puppet as a tool has become quite useful. Since the major upgrade from version 2.7 to 3.x, many things have changed.
But not only in Puppet, also the environment around it has changed and adopted to new challenges.
With Puppet supporting different environments and configurations connected to it, deploying changes to them can be quite challenging. The marriage of GIT with Puppet comes immediately into some ones mind and it makes actually sense. The codebase of Puppet is plain text, putting this into a versioning system is a huge benefit.
The Puppetlabs documentation describes an interesting and quite useful way to automatically deploy branches of a GIT repository into the file-system and Puppet environment setup. It uses the GIT hooks and a ruby script to trigger a repository export into the file-system.
Due to the nature of GIT, a repository always contains a sub-directory called .git. GIT keeps a snapshot of all files under it’s control, so we can go back whenever we want.
One way of checking out the content of a GIT branch is to use a GIT hook and fetch the just checked in version to the file-system. That works very reliable and stable and makes it easy to turn changes in the repository into actual changes on a system.
Until it doesn’t.
The drawback of this method becomes clear only after a while.
With each update, a complete copy of all files get deployed into the file-system. You’re basically creating zillions of files. This wouldn’t hurt really much, but it’s unnecessary. When this constellation meets a rather small partition size as well, the number of i-nodes on a disk get so low, that you can’t create new files and can’t deploy changes any more.
This is the number of i-nodes I had available, when I hit the problem. The Puppet configuration directory is under /etc/puppet. On that partition, though rather small of size, 63103 i-nodes are used. I could create single files, but I couldn’t roll out major changes into the Puppet environments anymore.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 65536 63103 2433 97% /
tmpfs 490610 1 490609 1% /dev/shm
/dev/sda1 65536 39 65497 1% /boot
/dev/mapper/vg0-home 65536 626 64910 1% /home
/dev/mapper/vg0-opt 65536 10631 54905 17% /opt
/dev/mapper/vg0-tmp 32768 12 32756 1% /tmp
/dev/mapper/vg0-usr 192768 42689 150079 23% /usr
/dev/mapper/vg0-var 642560 168830 473730 27% /var
/dev/mapper/vg0-data 198800 2008 196792 2% /data
There are many ways of solving this issue.
Delete branches¶
As first action I listed the setup branches and removed them from the GIT repository. Especially older branches that have been created in order to develop modules,to investigate an incident or to test a feature. Usually they are created with the full history of the parent branch. Not, because it would be necessary, but because it’s easy to forget to put the parameter ‘–orphan’ in the checkout command.
$ git checkout --orphan new-branch
Each of those branches deployed to the puppet server starts eating up the i-nodes. Just deleting them when they aren’t needed any more, will free up space
$ git -r -d|-D branch-name
# or
$ git push origin :branch-name
Deleting history¶
As a simpler version of the previous workaround, deleting the history is also an option. What you basically do is you spawn a new branch of the current one with the history and rename the new one to the old one. That way you drop all the history of the current branch and reduce that way the number of i-node you use.
$ git push origin :old-branch-name
$ git checkout -b --orphan new-branch-name
$ git checkout new-branch-name
$ git -D old-branch-name
$ git branch -m new-branch-name old-branch-name
$ git push origin new-branch-name
Separate Partition¶
While the first two solutions address more the source of the problem (GIT) and temper with the configuration there, this workaround moves the Puppet configuration onto a separate partition and gives it therefore its own set of i-<nodes.
I’ve created a 5GB partition in the LVM setup (also using Puppet) and mounted it into /etc/puppet.new. This was the partition layout when still running low on i-nodes:
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 65536 63103 2433 97% /
tmpfs 490610 1 490609 1% /dev/shm
/dev/sda1 65536 39 65497 1% /boot
/dev/mapper/vg0-home 65536 626 64910 1% /home
/dev/mapper/vg0-opt 65536 10631 54905 17% /opt
/dev/mapper/vg0-tmp 32768 12 32756 1% /tmp
/dev/mapper/vg0-usr 192768 42689 150079 23% /usr
/dev/mapper/vg0-var 642560 168830 473730 27% /var
/dev/mapper/vg0-data 198800 2008 196792 2% /data
/dev/mapper/vg0-puppet
131072 0 131072 0% /etc/puppet.new
The root partition containing /etc/puppet is down to ~2400 i-nodes (that means ~2400 files can be created additionally). The new partition with 5GB space has the double amount of i-nodes available and will be only used for the Puppet configuration in the GIT repository. This one is currently mounted on /etc/puppet.new.
The next step was simply to sync all files and attributes to the new location
$ sudo rysnc -rvt /etc/puppet /etc/puppet.new
Update: (2014-06-26)
It seems that even that was not enough. The new partition ran into the same problems just after a couple of weeks. Just increasing the disc space does not seem to be right, since only so little is used. Instead I have decreased the size of the inodes/number of inodes per group.
Before the change is was like this:
$ sudo tune2fs -l /dev/mapper/vg0-puppet | grep Inode
Inode count: 131072
Inodes per group: 8192
Inode blocks per group: 512
Inode size: 256
By taking a backup of the partition, unmounting it and re-creating the file-system on it, I was able to increase the available numbers of inodes
# Backup already taken
$ sudo umount /etc/puppet
$ sudo mkfs.ext4 -I 2048 /dev/mapper/vg0-puppet
$ sudo mount /etc/puppet
# Restore Backup from here
The result will (hopefully) last a bit longer then just moving the files to a separate partition.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
...
/dev/mapper/vg0-puppet
2621440 119223 2502217 5% /etc/puppet
Changing the partition layout in /etc/fstab will help us doing the re-mount faster and permanent
- /dev/mapper/vg0-puppet /etc/puppet.new ext3 defaults 0 0
+ /dev/mapper/vg0-puppet /etc/puppet ext3 defaults 0 0
Now just stopping the Puppetmaster and the Puppet agent, so nothing breaks while doing the magic.
$ sudo service puppetmaster stop
$ sudo service puppet stop
Finally, the moment of truth has come. The old configuration folders moves out of the way and the new partition will take it’s place.
$ sudo mv /etc/puppet /etc/puppet.old
$ sudo umount /dev/mapper/vg0-puppet
$ sudo mount -a
There’s now two folders with Puppet configuration files under /etc/:
/etc/puppet.old #( The old configuration files on the same partition as /,
/etc/puppet #( The copy of the configuration files
After starting the Puppetmaster and the Puppet agent again, everything should work as usual and Puppet will continue to distribute the configuration.
The GIT deployment will also still work, because none of the files or paths have actually changed. The only thing that changed was the location, where all data is stored.
Having final look at the i-nodes over the partitions, we’ll see, that this solved our problem for now.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 65536 6123 59413 10% /
tmpfs 490610 1 490609 1% /dev/shm
/dev/sda1 65536 39 65497 1% /boot
/dev/mapper/vg0-home 65536 627 64909 1% /home
/dev/mapper/vg0-opt 65536 10635 54901 17% /opt
/dev/mapper/vg0-tmp 32768 12 32756 1% /tmp
/dev/mapper/vg0-usr 192768 42689 150079 23% /usr
/dev/mapper/vg0-var 642560 168853 473707 27% /var
/dev/mapper/vg0-data 198800 2008 196792 2% /data
/dev/mapper/vg0-puppet
131072 56993 74079 44% /etc/puppet
Final thoughts¶
Personally I think think that extending the Ruby-Script with some functions to clean up the partitions or to reduce the number of Inodes could have been a better solution. Even a pull of only the last version (git pull –depth=1 origin) could be a solution here. However, since I’m not that fluent in Ruby and I was confident in what I had to do, switching over to the other partition, was a fast and easy way to do it.