Dumping MongoDB database

Last couple of months I was wandering through the world of MongoDb, both as a developer and as a sysadmin. I won’t say how happy I’m with it or how much I got disappointed , I will just note that you don’t know how deep or serious you dived into a software product till the moment when you feel the need for a backup. And that day came, and I did something about it.

I’m not so much into installing backup software that will just do the things for me, I always start with the idea that I can do it with my own simple implementation that will fill my needs for saving data or doing something else. Mongo has become a huge software/database monster and there are a lot of different approaches regarding this question. Off course the right way for you depends of several factors. Some of them are: the infrastructure of the Mongo deployment, the importance of the data, the quantity of the data, the performance factor etc.

For my needs that are associated with a single cluster instance of Mongo database without replication or sharding I realized that using the “classical” dumping method will fulfil .
One of the things I like about Mongo is its nicely done documentation. A lot of information about backuping MongoDb can be found on its official site:

http://docs.mongodb.org/manual/core/backups/

The documentation is like a guide showing you the different ways of backuping. Generally there are 3 different approaches: doing file system snaphosts, using mongodump command and using MongoDB Management Service.

For my dumping approach I wrote simple and primitive bash shell script that you can use for local backups or as a push towards the idea how to backup mongo data. It is oriented around dumping databases. Mongo data can be dumped as whole instance, whole collection or part collection and as a database. Here is the scirpt, the product of it are BSON and metadata JSON files at the designated directory.

#!/bin/bash

#
# GNRP LICENSE: Script licensed by GOD
#
# @author: Igor Ivanovski <igor at genrepsoft.com>
#
# March, 2015
#

#
# Filesystem directory variables
#

# Now in format: YYYY-MM
DATEYYMM=`date +"%Y-%m"`

# Now in format: DD
DATEDD=`date +%d`

# Backup directory 
BACKUPDIR="/opt/backup/mongodumps"

# Daily backup directory
BACKUPDIRDAILY="$BACKUPDIR/$DATEYYMM/$DATEDD/"

#
# List of databases to backup
#
DBs="
admin
someDb
";

#
### Mongo Server Setup ###
#

# Don't forget to add adequate roles if you are using authenticatio
#use someDb db.createUser({user:"backup",pwd:"pwd",roles:["readWrite"]})
#use admin db.createUser({user:"backup",pwd:"pwd",roles:["backup"]})

# Mongo backup username  
MUSER="backup"

# Mongo backup password
MPASS="pwd"

# Mongo HOST  name
MHOST="localhost"

# Mongo PORT number 
MPORT="27017"

# Mongo dump binary 

# Check if mongodump is installed
STATUS=0
[ "Y`which mongodump`" != "Y" ] && STATUS=$? || STATUS=$?
[ $STATUS == 1 ] && echo "No mongodump found. Exiting"; exit 1 
MONGODUMP=`which mongodump`;

# Starting to dump databases one by one
if [ "X$DBs" != "X" ]; then
    for db in $DBs
        do
            echo "Backing up database $db"
            $MONGODUMP --host $MHOST --port $MPORT --username $MUSER --password $MPASS --out $BACKUPDIRDAILY --db $db
        done

else
        echo "All listed mongo databases dumped. Bye"
fi

Thank you for reading me, here is a bonus from me for swinging 😉

Remove old backups with Bash Shell

Hello folks, I present you a bash script I wrote recently. It’s purpose  is deleting ‘old’ directories  marked by time period expressed in days. The script is simple and at its bases uses the command find to locate directories that we want to remove.

The general thinking on which this script is based are directories with backups categorized by some logical distinction and retention period for the second level of directories that are sorted by some other means like let’s say date. The script run by cron job searches the second level of directories and deletes the one older than some number of days n.

Example structure:

/mnt/backup/apps/2014-06-20/*

/mnt/backup/dbs/2014-06-1/*

/mnt/backup/logs/2014-06-1/*

 

If we want to exclude some of the directories in /mnt/backup , for example /mnt/backup/logs we can do that by putting the path into the variable EXCLUDED_DIRS. Note that only the directories from depth 1 can be excluded with this implementation (/mnt/backup/<> = OK , /mnt/backup/logs/<> != OK ).

EXCLUDED_DIRS=”/mnt/backup/logs /some/other/etc”

 

USAGE of the script:

<script_name> {arg1:/path/to/folder} {arg2:n days}  

or

./deleteOldDirs.sh         /mnt/backup           60

 

Note on find. For checking the timestamp of the directory I use ctime like option from find.  When this argument is used ,  find checks for the iNode pointing to the file/folder and collects information (like permissions, owner, etc) from which can be determinated when it was modified. In other hand if we want to check modification of the data in the backup itself we can use the argument mtime instead.

 

The script , beware of removing yourself.

May the force be with you !

#!/bin/bash

#
# GNRP LICENSE: Script licensed by GOD
#
# @author: Igor Ivanovski <igor at genrepsoft.net>
#
# July, 2014
#

#
# *** VARIABLES
#

PATH=$1 # First argument should be PATH of the backup folder
DAYS=$2 # Secound argument should be the max number of days we want to keep backups

GREP="/bin/grep"
MOUNT="/bin/mount"
FIND="/usr/bin/find"
RM="/bin/rm"

PATH_STATUS=0
MOUNT_STATUS=0

EXCLUDED_DIRS="/home/igor/Documents /home/igor/Programs"

#
# *** CHECKERS
#

if [ $# -eq 0 ]; then
    echo "No arguments provided"
    exit 1
fi

if [ -z $PATH ] || [ -z $DAYS ]; then
    echo "Not all arguments provided"
    exit 1
fi

#
# *** FUNCTIONS
#

function check_if_path_exists {

	[ ! -d $1 ] && echo "Path is invalild" || PATH_STATUS=1
}

function check_mount {
	 if [ -n "$($GREP $1 /etc/fstab)" ]; then
                echo "Mounting point exists..."
		if [ ! -n "$($MOUNT -l | $GREP $1)" ]; then
		    echo "...but not mounted."
		    echo "Mounting now."
                    $MOUNT $1
                    [ $? -eq 0 ] && { echo "Mounting was successfull!"; MOUNT_STATUS=1; } || echo "Can't mount";
		fi
         else
                echo "Mounting point doesn't exist"
         fi
}

# Excludes only if parent directory fits
function check_if_parent_dir_excluded { 
            for dir in $EXCLUDED_DIRS; do
		size=${#dir}
	        check=${1:0:size}
		#echo "$dir : ${#dir} == $check : ${#check}"
		if [ $dir ==  $check ]; then 
			echo "1"
		fi
	    done
}

function find_dirs {
	$FIND $PATH -mindepth 2 -maxdepth 2 -type d -ctime +$DAYS -print0
}

function print_dirs {
	while read -r n; do
                 result=$(check_if_parent_dir_excluded $n)
		 [ -z $result ] && printf '%q\n' "$n" || echo "Found EXCLUDE_DIR: $n" 
	done < <($FIND $PATH -mindepth 2 -maxdepth 2 -type d -ctime +$DAYS -print) # Use NL delimiter
}

function del_dirs {
        while read -r -d '' n; do
                 printf '%q\n' "$n"
                 result=$(check_if_parent_dir_excluded $n)
                 if [ -z $result ]; then
			 echo "Directory will be deleted now..."
			 $RM -rf $n
			 [ $? -eq 0 ] && echo "Gone!" || echo "Some error occured!"
		 else	
	 		echo "Found EXCLUDE_DIR: $n continue..." 
		 fi
        done < <($FIND $PATH -mindepth 2 -maxdepth 2 -type d -ctime +$DAYS -print0) # Use NUL delimiter
}

#
# *** MAIN
#

check_if_path_exists $PATH
[ $PATH_STATUS -eq 0 ] && check_mount $PATH
[ $MOUNT_STATUS -eq 1 ] && check_if_path_exists $PATH
[ $PATH_STATUS -eq 1 ] &&  del_dirs || { echo "Bye."; exit 0; }

#
# *** END
#