Commit fbddc25e authored by Antoine Beaupré's avatar Antoine Beaupré

Merge branch 'march2015-refactor' into 'master'

March2015 refactor

This branch is an attempt to bring the backupninja module in the 21st century. See the upgrade notice in the README for details.

See merge request !4
parents 74ce68bc b24afc7f
......@@ -4,28 +4,37 @@ Backupninja Module
This module helps you configure all of your backups with puppet, using
backupninja!
! Upgrade notice !
!! UPGRADE NOTICE !!
If you were previously using this module, some pieces have changed,
and you need to carefully change your use of them, or you will find
your backups to be duplicated on your backup server. The important
part that changed has to do with the rdiff-backup handler, if you
weren't using that, you don't need to worry.
If you were, you will need to make sure you change all of your
"$directory" parameters to be "$home" instead, and on your
backupserver you will need to move all of your backups into
"$home"/rdiff-backup. Previously, they were put in "$directory", which
doubled as the home for the user that was created. This caused
problems with rdiff-backup because of dot files and other things which
were not part of any rdiff-backup.
Getting started
your backups could stop working.
The backupninja::client class has been renamed to backupninja, and is
now *required* in all node manifests. Make sure the backupninja class
is now declared in all your node manifests! This new class now defines
defaults which were previously provided by backupninja::client::defaults,
and can now be overridden thanks to the brand new technology of class
parameters. This class also manages the backupninja configuration file,
replacing the backupninja::config ressource.
The backupninja::server class now takes parameters, replacing several
global variables such as $backupdir, $backupserver_tag and
$nagios_server. The $manage_nagios parameter also replaces the
$use_nagios global.
As for handlers, they don't include the backupninja::client anymore and
now read several default values from the backupninja base class. Most
handler declarations shouldn't need any changes.
See below for dependencies which have been introduced in this version.
Dependencies
---------------
First you will need to import the module:
This module requires Puppet versions 2.7 and up.
import "backupninja"
An up-to-date version of the puppet-stdlib module is also required.
Configure your backup server
----------------------------
......@@ -35,41 +44,41 @@ to your node definition for that server:
include backupninja::server
The default configuration will store backup data in the "/backup"
directory. To change this you may declare the class with a "backupdir"
parameter:
class { 'backupninja::server':
backupdir => '/mnt/backupdata'
}
By configuring a backupninja::server, this module will automatically
create sandboxed users on the server for each client for their
backups.
You may also want to set some variables on your backup server, such as:
$backupdir = "/backups"
Configure your backup clients
-----------------------------
The backupninja package and the necessary backup software will be
installed automatically when you include any of the different handlers
(as long as you are not handling it elsewhere in your manifests), for
example:
First, you need to include the backupninja class or declare it with
custom parameters:
include backupninja::client::rdiff_backup
class { 'backupninja':
loglvl => 3,
usecolors => false,
reportsuccess => false,
reportwarning => true,
ensure_backupninja_version => '1.0.1-1',
ensure_rdiffbackup_version => '1.2.8-7'
}
In this case, the module will make sure that the backupninja package
and the required rdiff-backup package are 'installed'/'present' (using
puppet's ensure parameter language). If you need to specify a specific
version of either backupninja itself, or the specific programs that
the handler class installs, you can specify the version you need
installed by providing a variable, for example:
$backupninja_ensure_version = "0.9.7~bpo50+1"
$rdiff_backup_ensure_version = "1.2.5-1~bpo40+1"
$rsync_ensure_version = "3.0.6-1~bpo50+1"
$duplicity_ensure_version = "0.6.04-1~bpo50+1"
$debconf_utils_ensure_version = "1.5.28"
$hwinfo_ensure_version = "16.0-2"
If you do not specify these variables the default 'installed/present'
version will be installed when you include this class.
is installed (using puppet's ensure parameter language) and create the
/etc/backupninja.conf configuration file.
If you need to specify a specific version of either backupninja itself,
or the specific programs that the handler class installs, you can
specify the version you need installed by providing a class parameter,
as shown in the example.
Configuring handlers
--------------------
......@@ -87,7 +96,7 @@ Included below are some configuration examples for different handlers.
* An example mysql handler configuration:
backupninja::mysql { all_databases:
backupninja::mysql { 'all_databases':
user => root,
backupdir => '/var/backups',
compress => true,
......@@ -96,7 +105,7 @@ backupninja::mysql { all_databases:
* An example rdiff-backup handler configuration:
backupninja::rdiff { backup_all:
backupninja::rdiff { 'backup_all':
directory => '/media/backupdisk',
include => ['/var/backups', '/home', '/var/lib/dpkg/status'],
exclude => '/home/*/.gnupg'
......@@ -104,32 +113,13 @@ backupninja::rdiff { backup_all:
* A remote rdiff-backup handler:
backupninja::rdiff { "main":
host => "backup.example.com",
type => "remote",
directory => "/backup/$fqdn",
user => "backup-$hostname",
}
Configuring backupninja itself
------------------------------
You may wish to configure backupninja itself. You can do that by doing
the following, and the /etc/backupninja.conf will be managed by
puppet, all the backupninja configuration options are available, you
can find them inside this module as well.
For example:
backupninja::config { conf:
loglvl => 3,
usecolors => false,
reportsuccess => false,
reportwarning => true;
backupninja::rdiff { 'main':
host => 'backup.example.com',
type => 'remote',
directory => "/backup/${::fqdn}",
user => "backup-${::hostname}",
}
Automatic creation of ssh-keys for duplicity
--------------------------------------------
......@@ -160,9 +150,9 @@ i.e.:
Nagios alerts about backup freshness
------------------------------------
If you set the $nagios_server variable to be the name of your nagios
server, then a passive nagios service gets setup so that the backup
server pushes checks, via a cronjob that calls
If you set the $backupninja::server::nagios_server variable to be the
name of your nagios server, then a passive nagios service gets setup so
that the backup server pushes checks, via a cronjob that calls
/usr/local/bin/checkbackups.pl, to the nagios server to alert about
relative backup freshness.
......@@ -171,9 +161,4 @@ To use this feature a few pre-requisites are necessary:
. configure nsca on your backup server (not done via puppet yet)
. configure nsca on your nagios server (not done via puppet yet)
. server backup directories are named after their $fqdn
. using nagios2 module, nagios/nagios3 modules/nativetypes not supported yet
. using a nagios puppet module that can create passive service checks
. backups must be under $home/dup, $home/rdiff-backup depending on method
. $nagios_server must be set before the class is included
class backupninja::client::defaults {
$configdir = $cfg_override ? {
'' => "/etc/backup.d",
default => $cfg_override,
}
$real_keystore = $backupkeystore ? {
'' => "$fileserver/keys/backupkeys",
default => $backupkeystore,
}
$real_keytype = $backupkeytype ? {
'' => 'rsa',
false => 'rsa',
default => $backupkeytype,
}
$real_keydestination = $keydestination ? {
'' => '/root/.ssh',
default => $keydestination,
}
$real_keyowner = $keyowner ? {
'' => 0,
default => $keyowner,
}
$real_keygroup = $keygroup ? {
'' => 0,
default => $keygroup,
}
$real_keymanage = $keymanage ? {
'' => true,
default => $keymanage
}
$real_ssh_dir_manage = $ssh_dir_manage ? {
'' => true,
default => $ssh_dir_manage
}
if !defined(Package["backupninja"]) {
if $backupninja_ensure_version == '' { $backupninja_ensure_version = 'installed' }
package { 'backupninja':
ensure => $backupninja_ensure_version
}
}
file { $configdir:
ensure => directory,
mode => 750, owner => 0, group => 0;
}
}
class backupninja::client inherits backupninja::client::defaults {
define key(
$user = false, $host = false, $createkey=false, $installkey=false,
$keyowner=false, $keygroup=false, $keystore=false, $keystorefspath='',
$keytype=false,
$keydest=false, $keydestname=false )
{
$real_user = $user ? {
false => $name,
default => $user
}
$real_host = $host ? {
false => $user,
default => $host
}
$install_key = $installkey ? {
false => "${backupninja::client::defaults::real_keymanage}",
default => $installkey,
}
$key_owner = $keyowner ? {
false => "${backupninja::client::defaults::real_keyowner}",
default => $keyowner,
}
$key_group = $keygroup ? {
false => "${backupninja::client::defaults::real_keygroup}",
default => $keygroup,
}
$key_store = $keystore ? {
false => "${backupninja::client::defaults::real_keystore}",
default => $keystore,
}
$key_type = $keytype ? {
'' => "${backupninja::client::defaults::real_keytype}",
false => "${backupninja::client::defaults::real_keytype}",
default => $keytype,
}
$key_dest = $keydest ? {
false => "${backupninja::client::defaults::real_keydestination}",
default => $keydest,
}
$key_dest_name = $keydestname ? {
false => "id_$key_type",
default => $keydestname,
}
$key_dest_file = "${key_dest}/${key_dest_name}"
if $createkey == true {
if $keystorefspath == false {
err("need to define a destination directory for sshkey creation!")
}
$ssh_keys = ssh_keygen("${keystorefspath}/${key_dest_name}")
}
case $install_key {
true: {
if !defined(File["$key_dest"]) {
file { "$key_dest":
ensure => directory,
mode => 0700, owner => $key_owner, group => $key_group,
}
}
if !defined(File["$key_dest_file"]) {
file { "$key_dest_file":
source => "${key_store}/${key_dest_name}",
mode => 0400, owner => $key_owner, group => $key_group,
require => File["$key_dest"],
}
}
}
}
}
}
class backupninja::client::maildir inherits backupninja::client::defaults {
if !defined(Package["rsync"]) {
if $rsync_ensure_version == '' { $rsync_ensure_version = 'installed' }
package { 'rsync':
ensure => $rsync_ensure_version,
}
}
}
class backupninja::client::rdiff_backup inherits backupninja::client::defaults {
if !defined(Package["rdiff-backup"]) {
if $rdiff_backup_ensure_version == '' { $rdiff_backup_ensure_version = 'installed' }
package { 'rdiff-backup':
ensure => $rdiff_backup_ensure_version,
}
}
}
class backupninja::client::duplicity inherits backupninja::client::defaults {
if !defined(Package["duplicity"]) {
if $duplicity_ensure_version == '' { $duplicity_ensure_version = 'installed' }
package { 'duplicity':
ensure => $duplicity_ensure_version,
}
}
}
class backupninja::client::sys inherits backupninja::client::defaults {
case $operatingsystem {
debian,ubuntu: {
if !defined(Package["debconf-utils"]) {
if $debconf_utils_ensure_version == '' { $debconf_utils_ensure_version = 'installed' }
package { 'debconf-utils':
ensure => $debconf_utils_ensure_version,
}
}
if !defined(Package["hwinfo"]) {
if $hwinfo_ensure_version == '' { $hwinfo_ensure_version = 'installed' }
package { 'hwinfo':
ensure => $hwinfo_ensure_version,
}
}
}
default: {}
}
}
class backupninja::client::rsync inherits backupninja::client::defaults {
if !defined(Package["rsync"]) {
if $rsync_ensure_version == '' { $rsync_ensure_version = 'installed' }
package { 'rsync':
ensure => $rsync_ensure_version,
}
}
}
# Write a "main" configuration file for backupninja. Effectively, it does
# little more than just take the config options you specify in the define and
# write them to the config file as-is.
#
define backupninja::config(
$configfile = '/etc/backupninja.conf', $loglvl = 4, $when = 'everyday at 01:00',
$reportemail = 'root', $reportsuccess = false, $reportwarning = true,
$reporthost = $reporthost, $reportuser = $reportuser,
$reportdirectory = $reportdirectory,
$logfile = '/var/log/backupninja.log', $configdir = '/etc/backup.d',
$scriptdir = '/usr/share/backupninja', $libdir = '/usr/lib/backupninja',
$usecolors = true, $vservers = false)
{
file { $configfile:
content => template('backupninja/backupninja.conf.erb'),
owner => root,
group => root,
mode => 0644
}
}
# Write the backupninja cron job, allowing you to specify an alternate backupninja
# command (if you want to wrap it in any other commands, e.g. to allow it to use
# the monkeysphere for authentication), or a different schedule to run it on.
......
......@@ -81,11 +81,11 @@ define backupninja::duplicity( $order = 90,
$destuser = false,
$desturl = false,
# configs to backupninja client
$backupkeystore = false,
$backupkeystorefspath = '',
$backupkeytype = "rsa",
$backupkeydest = false,
$backupkeydestname = false,
$backupkeystore = $backupninja::keystore,
$backupkeystorefspath = $backupninja::keystorefspath,
$backupkeytype = $backupninja::keytype,
$backupkeydest = $backupninja::keydest,
$backupkeydestname = $backupninja::keydestname,
# options to backupninja server sandbox
$ssh_dir_manage = true,
$ssh_dir = false,
......@@ -96,8 +96,8 @@ define backupninja::duplicity( $order = 90,
$createkey = false,
$installkey = true ) {
# the client with configs for this machine
include backupninja::client::duplicity
# install client dependencies
ensure_resource('package', 'duplicity', {'ensure' => $backupninja::ensure_duplicity_version})
case $desthost { false: { err("need to define a destination host for remote backups!") } }
case $destdir { false: { err("need to define a destination directory for remote backups!") } }
......@@ -118,7 +118,7 @@ define backupninja::duplicity( $order = 90,
}
# the client's ssh key
backupninja::client::key { "${destuser}-${name}":
backupninja::key { "${destuser}-${name}":
user => $destuser,
host => $desthost,
createkey => $createkey,
......@@ -131,13 +131,13 @@ define backupninja::duplicity( $order = 90,
}
# the backupninja rule for this duplicity backup
file { "${backupninja::client::defaults::configdir}/${order}_${name}.dup":
file { "${backupninja::configdir}/${order}_${name}.dup":
ensure => $ensure,
content => template('backupninja/dup.conf.erb'),
owner => root,
group => root,
mode => 0600,
require => File["${backupninja::client::defaults::configdir}"]
require => File["${backupninja::configdir}"]
}
}
import "client.pp"
import "config.pp"
import "dup.pp"
import "labelmount.pp"
import "maildir.pp"
import "mysql.pp"
import "pgsql.pp"
import "rdiff.pp"
import "server.pp"
import "sh.pp"
import "svn.pp"
import "sys.pp"
class backupninja (
$ensure_backupninja_version = 'installed',
$ensure_rsync_version = 'installed',
$ensure_rdiffbackup_version = 'installed',
$ensure_debconfutils_version = 'installed',
$ensure_hwinfo_version = 'installed',
$ensure_duplicity_version = 'installed',
$configdir = '/etc/backup.d',
$keystore = "${::fileserver}/keys/backupkeys",
$keystorefspath = false,
$keytype = 'rsa',
$keydest = '/root/.ssh',
$keyowner = 0,
$keygroup = 0,
$keymanage = true,
$configfile = '/etc/backupninja.conf',
$loglvl = 4,
$when = 'everyday at 01:00',
$reportemail = 'root',
$reportsuccess = false,
$reportwarning = true,
$reporthost = undef,
$reportuser = undef,
$reportdirectory = undef,
$logfile = '/var/log/backupninja.log',
$configdir = '/etc/backup.d',
$scriptdir = '/usr/share/backupninja',
$libdir = '/usr/lib/backupninja',
$usecolors = true,
$vservers = false
) {
class backupninja {
# install client dependencies
ensure_resource('package', 'backupninja', {'ensure' => $ensure_backupninja_version})
}
# set up backupninja config directory
file { $configdir:
ensure => directory,
mode => 750, owner => 0, group => 0;
}
define key(
$user = $name,
$createkey = false,
$keymanage = $backupninja::keymanage,
$keyowner = $backupninja::keyowner,
$keygroup = $backupninja::keygroup,
$keystore= $backupninja::keystore,
$keystorefspath = $backupninja::keystorefspath,
$keytype = $backupninja::keytype,
$keydest = $backupninja::keydest,
$keydestname = "id_${backupninja::keytpe}" )
{
# generate the key
if $createkey == true {
if $keystorefspath == false {
err("need to define a destination directory for sshkey creation!")
}
$ssh_keys = ssh_keygen("${keystorefspath}/${keydestname}")
}
# deploy/manage the key
if $keymanage == true {
$keydestfile = "${keydest}/${keydestname}"
ensure_resource('file', $keydest, {
'ensure' => 'directory',
'mode' => '0700',
'owner' => $keyowner,
'group' => $keygroup
})
ensure_resource('file', $keydestfile, {
'ensure' => 'present',
'source' => "${keystore}/${keydestname}",
'mode' => '0700',
'owner' => $keyowner,
'group' => $keygroup,
'require' => 'File["$key_dest"]'
})
}
}
file { $configfile:
content => template('backupninja/backupninja.conf.erb'),
owner => root,
group => 0,
mode => '0644'
}
}
......@@ -23,23 +23,22 @@ define backupninja::labelmount($order = 10,
$label,
$dest
) {
include backupninja::client::defaults
file { "${backupninja::client::defaults::configdir}/${order}_${name}.labelmount":
file { "${backupninja::configdir}/${order}_${name}.labelmount":
ensure => $ensure,
content => template('backupninja/labelmount.conf.erb'),
owner => root,
group => root,
mode => 0600,
require => File["${backupninja::client::defaults::configdir}"]
require => File["${backupninja::configdir}"]
}
file { "${backupninja::client::defaults::configdir}/99_${name}.umount":
file { "${backupninja::configdir}/99_${name}.umount":
ensure => $ensure,
content => template('backupninja/umount.conf.erb'),
owner => root,
group => root,
mode => 0600,
require => File["${backupninja::client::defaults::configdir}"]
require => File["${backupninja::configdir}"]
}
# Copy over the handler scripts themselves, since they're not in the
......
......@@ -17,25 +17,27 @@
#
#
define backupninja::maildir(
$order = 99, $ensure = present, $when = 'everyday at 21:00', $srcdir = false,
$order = 99, $ensure = present,
$when = 'everyday at 21:00', $srcdir = false,
$destdir = false, $desthost = false, $destuser = false, $destid_file = false,
$remove = false, $multiconnection = yes, $keepdaily='4', $keepweekly='2',
$keepmonthly='2')
{
include backupninja::client::maildir
# install client dependencies
ensure_resource('package', 'rsync', {'ensure' => $backupninja::ensure_rsync_version})
case $srcdir { false: { err("need to define a source directory to backup!") } }
case $destdir { false: { err("need to define a destination directory to backup!") } }
case $desthost { false: { err("need to define a destination host for backups!") } }
case $destuser { false: { err("need to define a destination user for backups!") } }
case $destid_file { false: { err("need to define a ssh key id file to use!") } }
file { "${backupninja::client::defaults::configdir}/${order}_${name}.maildir":
file { "${backupninja::configdir}/${order}_${name}.maildir":
ensure => $ensure,
content => template('backupninja/maildir.conf.erb'),
owner => root,
group => root,
mode => 0600,
require => File["${backupninja::client::defaults::configdir}"]
require => File["${backupninja::configdir}"]
}
}
......@@ -27,13 +27,12 @@ define backupninja::mysql(
default => $configfile,
}
include backupninja::client::defaults
file { "${backupninja::client::defaults::configdir}/${order}_${name}.mysql":
file { "${backupninja::configdir}/${order}_${name}.mysql":
ensure => $ensure,
content => template('backupninja/mysql.conf.erb'),
owner => root,
group => root,
mode => 0600,
require => File["${backupninja::client::defaults::configdir}"]
require => File["${backupninja::configdir}"]
}
}
......@@ -16,14 +16,12 @@
define backupninja::pgsql(
$order = 10, $ensure = present, $databases = 'all', $backupdir = "/var/backups/postgres", $compress = true, $vsname = false)
{
include backupninja::client::defaults
file { "${backupninja::client::defaults::configdir}/${order}_${name}.pgsql":
file { "${backupninja::configdir}/${order}_${name}.pgsql":
ensure => $ensure,
content => template('backupninja/pgsql.conf.erb'),
owner => root,
group => root,
mode => 0600,
require => File["${backupninja::client::defaults::configdir}"]
require => File["${backupninja::configdir}"]
}
}
......@@ -16,8 +16,8 @@
# directories.
#
define backupninja::rdiff(
$order = 90, $ensure = present, $user = false,
$home = "/home/${user}-${name}", $host = false,
$order = 90, $ensure = present,
$user = false, $home = "/home/${user}-${name}", $host = false,
$type = 'local',
$exclude = [ "/home/*/.gnupg", "/home/*/.local/share/Trash", "/home/*/.Trash",
"/home/*/.thumbnails", "/home/*/.beagle", "/home/*/.aMule",
......@@ -26,9 +26,11 @@ define backupninja::rdiff(
"/home", "/usr/local/*bin", "/var/lib/dpkg/status*" ],
$vsinclude = false, $keep = 30, $sshoptions = false, $options = '--force', $ssh_dir_manage = true,
$ssh_dir = false, $authorized_keys_file = false, $installuser = true, $installkey = true, $key = false,
$backuptag = false, $backupkeytype = "rsa", $backupkeystore = false, $extras = false, $nagios2_description = 'backups')
$backuptag = false, $backupkeytype = $backupninja::keytype, $backupkeystore = $backupninja::keystore,
$extras = false, $nagios_description = 'backups')
{
include backupninja::client::rdiff_backup
# install client dependencies
ensure_resource('package', 'rdiff-backup', {'ensure' => $backupninja::ensure_rdiffbackup_version})
$directory = "$home/rdiff-backup/"
......@@ -46,10 +48,10 @@ define backupninja::rdiff(
manage_ssh_dir => $ssh_dir_manage, ssh_dir => $ssh_dir, key => $key,
authorized_keys_file => $authorized_keys_file, installuser => $installuser,
backuptag => $real_backuptag, keytype => $backupkeytype, backupkeys => $backupkeystore,
nagios2_description => $nagios2_description
nagios_description => $nagios_description
}
backupninja::client::key
backupninja::key
{
"${user}-${name}": user => $user, host => $host,
installkey => $installkey,
......@@ -60,13 +62,13 @@ define backupninja::rdiff(
}
file { "${backupninja::client::defaults::configdir}/${order}_${name}.rdiff":
file { "${backupninja::configdir}/${order}_${name}.rdiff":
ensure => $ensure,
content => template('backupninja/rdiff.conf.erb'),
owner => root,
group => root,
mode => 0600,
require => File["${backupninja::client::defaults::configdir}"]
require => File["${backupninja::configdir}"]
}
}
......@@ -2,11 +2,12 @@
# Based on backupninja::rdiff
define backupninja::rsync(
$order = 90, $ensure = present, $user = false, $home = false, $host = false,
$order = 90, $ensure = present,
$user = false, $home = false, $host = false,
$ssh_dir_manage = true, $ssh_dir = false, $authorized_keys_file = false,
$installuser = true, $installkey = true, $key = false, $backuptag = false,
$home = false, $backupkeytype = "rsa", $backupkeystore = false, $extras = false,
$nagios2_description = 'backups', $subfolder = 'rsync',
$home = false, $backupkeytype = $backupninja::keytype, $backupkeystore = $backupninja::keystore, $extras = false,
$nagios_description = 'backups', $subfolder = 'rsync',
$log = false, $partition = false, $fscheck = false, $read_only = false,
$mountpoint = false, $backupdir = false, $format = false, $days = false,
......@@ -29,7 +30,8 @@ define backupninja::rsync(
$rm = false, $cp = false, $touch = false, $mv = false, $fsck = false)
{
include backupninja::client::rsync
# install client dependencies
ensure_resource('package', 'rsync', {'ensure' => $backupninja::ensure_rsync_version})
# Right now just local origin with remote destination is supported.
$from = 'local'
......@@ -63,10 +65,10 @@ define backupninja::rsync(
backuptag => $real_backuptag,
keytype => $backupkeytype,
backupkeys => $backupkeystore,
nagios2_description => $nagios2_description
nagios_description => $nagios_description
}
backupninja::client::key { "${user}-${name}":
backupninja::key { "${user}-${name}":
user => $user,
host => $host,
installkey => $installkey,
......@@ -76,12 +78,12 @@ define backupninja::rsync(
}
}
file { "${backupninja::client::defaults::configdir}/${order}_${name}.rsync":
file { "${backupninja::configdir}/${order}_${name}.rsync":
ensure => $ensure,