Remove old backups with Bash Shell

Hello folks, I present you a bash script I wrote recently. It’s purpose  is deleting ‘old’ directories  marked by time period expressed in days. The script is simple and at its bases uses the command find to locate directories that we want to remove.

The general thinking on which this script is based are directories with backups categorized by some logical distinction and retention period for the second level of directories that are sorted by some other means like let’s say date. The script run by cron job searches the second level of directories and deletes the one older than some number of days n.

Example structure:

/mnt/backup/apps/2014-06-20/*

/mnt/backup/dbs/2014-06-1/*

/mnt/backup/logs/2014-06-1/*

 

If we want to exclude some of the directories in /mnt/backup , for example /mnt/backup/logs we can do that by putting the path into the variable EXCLUDED_DIRS. Note that only the directories from depth 1 can be excluded with this implementation (/mnt/backup/<> = OK , /mnt/backup/logs/<> != OK ).

EXCLUDED_DIRS=”/mnt/backup/logs /some/other/etc”

 

USAGE of the script:

<script_name> {arg1:/path/to/folder} {arg2:n days}  

or

./deleteOldDirs.sh         /mnt/backup           60

 

Note on find. For checking the timestamp of the directory I use ctime like option from find.  When this argument is used ,  find checks for the iNode pointing to the file/folder and collects information (like permissions, owner, etc) from which can be determinated when it was modified. In other hand if we want to check modification of the data in the backup itself we can use the argument mtime instead.

 

The script , beware of removing yourself.

May the force be with you !

#!/bin/bash

#
# GNRP LICENSE: Script licensed by GOD
#
# @author: Igor Ivanovski <igor at genrepsoft.net>
#
# July, 2014
#

#
# *** VARIABLES
#

PATH=$1 # First argument should be PATH of the backup folder
DAYS=$2 # Secound argument should be the max number of days we want to keep backups

GREP="/bin/grep"
MOUNT="/bin/mount"
FIND="/usr/bin/find"
RM="/bin/rm"

PATH_STATUS=0
MOUNT_STATUS=0

EXCLUDED_DIRS="/home/igor/Documents /home/igor/Programs"

#
# *** CHECKERS
#

if [ $# -eq 0 ]; then
    echo "No arguments provided"
    exit 1
fi

if [ -z $PATH ] || [ -z $DAYS ]; then
    echo "Not all arguments provided"
    exit 1
fi

#
# *** FUNCTIONS
#

function check_if_path_exists {

	[ ! -d $1 ] && echo "Path is invalild" || PATH_STATUS=1
}

function check_mount {
	 if [ -n "$($GREP $1 /etc/fstab)" ]; then
                echo "Mounting point exists..."
		if [ ! -n "$($MOUNT -l | $GREP $1)" ]; then
		    echo "...but not mounted."
		    echo "Mounting now."
                    $MOUNT $1
                    [ $? -eq 0 ] && { echo "Mounting was successfull!"; MOUNT_STATUS=1; } || echo "Can't mount";
		fi
         else
                echo "Mounting point doesn't exist"
         fi
}

# Excludes only if parent directory fits
function check_if_parent_dir_excluded { 
            for dir in $EXCLUDED_DIRS; do
		size=${#dir}
	        check=${1:0:size}
		#echo "$dir : ${#dir} == $check : ${#check}"
		if [ $dir ==  $check ]; then 
			echo "1"
		fi
	    done
}

function find_dirs {
	$FIND $PATH -mindepth 2 -maxdepth 2 -type d -ctime +$DAYS -print0
}

function print_dirs {
	while read -r n; do
                 result=$(check_if_parent_dir_excluded $n)
		 [ -z $result ] && printf '%q\n' "$n" || echo "Found EXCLUDE_DIR: $n" 
	done < <($FIND $PATH -mindepth 2 -maxdepth 2 -type d -ctime +$DAYS -print) # Use NL delimiter
}

function del_dirs {
        while read -r -d '' n; do
                 printf '%q\n' "$n"
                 result=$(check_if_parent_dir_excluded $n)
                 if [ -z $result ]; then
			 echo "Directory will be deleted now..."
			 $RM -rf $n
			 [ $? -eq 0 ] && echo "Gone!" || echo "Some error occured!"
		 else	
	 		echo "Found EXCLUDE_DIR: $n continue..." 
		 fi
        done < <($FIND $PATH -mindepth 2 -maxdepth 2 -type d -ctime +$DAYS -print0) # Use NUL delimiter
}

#
# *** MAIN
#

check_if_path_exists $PATH
[ $PATH_STATUS -eq 0 ] && check_mount $PATH
[ $MOUNT_STATUS -eq 1 ] && check_if_path_exists $PATH
[ $PATH_STATUS -eq 1 ] &&  del_dirs || { echo "Bye."; exit 0; }

#
# *** END
#

Resolving error 413 (Request Entity Too Large/Not Allowed) in IIS 7.5

Working with IIS lately have brought me a lot of trouble, however it also increased my in-depth knowledge of its working abilities and adaptability.

One loving situation (between the minor ones) appeared after we did transition from http to https. After fixing the minor ones everything was working smooth and groovy except that sometimes the upload of files was broken. Then we realized it was not someTIME, but someTHING or the concrete size of the uploaded file that was causing the problem. Haa, so common problem when setting up Web Server you say, me said also, but this time the workaround was a little more pain in the *ss if you know what I mean.

The response given when uploading was an intelligent block by the Web Server resulting with error 413 – Request Entity Too Large. That doesn’t make a sense, I’m uploading  file that is 100KB but the settings allow max to 100MB file…

So with the help of rogue googling enabled by http://www.startpage.com I set my self digging into the problem overcome. One thing was clear, the trigger for this dysfunctionality certainly was https, since the uploading worked fine on http not secure. That gave me the rough but brilliant idea what https do. It encrypts, keeps extra request data, it certainly enlarges the request payload.

Hmm, OK first lets check the standard file limits in IIS.

Normal setting for max upload file size:

Setting the request limits in the root web.config of the site (default is 30 MB). This can be set in Internet Information Services Manager Program also (MACHINE->Site->IIS->Request Filtering->Edit Feature Settings)

<!– 100 MB . Format uses Bytes –>

<security>

    <requestFiltering>

        <requestLimits maxAllowedContentLength=”102400000″ />

    </requestFiltering>

   </security>

For ASP.NET you have more specific configuration:

http://msdn.microsoft.com/en-us/library/e1f13641%28v=vs.85%29.aspx

Example:

 <system.web>

   <httpRuntime maxRequestLength=”102400000″ executionTimeout=”3600″ />

 </system.web>

For legacy ASP:

<!– This goes under ASP – can be set with IIS Manager also 😉 –>

<limits maxRequestEntityAllowed=”102400000″/>

And here comes our solution:

http://www.iis.net/configreference/system.webserver/serverruntime

What we faced here is a buffer related problem. It’s not about the maxRequestEntityAllowed since the default size is Unlimited , but how IIS handles the request. After some empirics we noticed that the problem was occurring only with files larger or equal to 42 KB. And what is the default value of uploadReadAheadSize? 42 KB.

Then I charged myself to change this property. So I did the following:

Since we need to do section overriding through the means of <location> to change the default values (this configuration is not possible through IIS Manager)

<location path=”SiteName” overrideMode=”Allow”>

  <system.webServer>

   <asp>

    <session />

    <comPlus />

    <cache />

    <limits maxRequestEntityAllowed=”102400000″/>

   </asp>

    <serverRuntime enabled=”true” uploadReadAheadSize=”102400″ />

   </system.webServer>

</location>

However for some God sake reason, this configuration didn’t change the default values, even with overriding enabled for the serverRuntime section in the applicationHost.config . For easy check there is one good friend – the appcmd.

C:\Windows\System32\inetsrv\appcmd.exe list config “SiteName” -section:system.webServer/serverRuntime

And this bring us to the solution. To cut the story:

* Enable the serverRuntime section

C:\Windows\System32\inetsrv\appcmd.exe set config “SiteName” -section:system.webServer/ServerRuntime /enabled:”True” /commit:apphost

* Set the uploadReadAheadSize to 10MB

C:\Windows\System32\inetsrv\appcmd.exe set config “SiteName” -section:system.webServer/ServerRuntime /uploadReadAheadSize:”1024000” /commit:apphost

Restart if required, and that’s it.

Just aside note don’t forget to change the uploadReadAheadSize to something smaller and more realistic since 10MB is huge for a buffer, cause you don’t want to be hit by the nasty bad boyz with their huge payload packets.

Setting Up Context in Apache Tomcat for Serving Static Files

The intro:

So I’ve heard you want to serve static files from your Tomcat Web App in a way that they won’t be deleted on WAR redeploy or Tomcat restarted?

You have a solution, and that is mapping a custom Context in your Apache Tomcat server.xml .

The scenario :

You have a site that allows users to upload images that are public,shared and not under a hood of some security filter. The most intuitive solution is to put them in some directory i.e. ‘uploads’ , but then you realize that the things in the exploded WAR rewrite on redeploy or if the war is in the webapps directory on Tomcat restart. (you can change this behaviour)

The solution is simple: save the files to some directory outside of the war (something like ‘/usr/share/tomcat/uploads’) and map that directory on the server context of your Tomcat AS (something like http://lesite:8080/uploads).

With workaround like this you will see your uploaded cute kitty picture like this: http://lesite:8080/uploads/kitty.jpg

The implementation:

Let’s use the same examples. The mapping is done in <CATALINA_HOME>/conf/server.xml (hopes you know what and where catalina_home is )

This is default situation on new Tomcat install (a snippet from sever.xml):

<Host name=”localhost” appBase=”webapps” unpackWARs=”true” autoDeploy=”true”>

<Valve className=”org.apache.catalina.valves.AccessLogValve” directory=”logs”
prefix=”localhost_access_log.” suffix=”.txt”
pattern=”%h %l %u %t &quot;%r&quot; %s %b” />

</Host>

But we want to change that in this:

<Host name=”localhost” appBase=”webapps” unpackWARs=”true” autoDeploy=”true”>

<Context docBase="/usr/share/tomcat/uploads" path="/uploads" />

<Valve className=”org.apache.catalina.valves.AccessLogValve” directory=”logs”
prefix=”localhost_access_log.” suffix=”.txt”
pattern=”%h %l %u %t &quot;%r&quot; %s %b” />

</Host>

And that’s it, end of setting. Restart , code and redeploy.

The cookie:

Java snippet of simple utilization:

public class UploadsServlet extends HttpServlet {

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletExcpetion, IOException {
        File file = new File("/usr/share/tomcat/uploads", request.getPathInfo());
        response.setHeader("Content-Type", Files.probeContentType(file.toPath()));
        response.setHeader("Content-Length", String.valueOf(file.length()));
        Files.copy(file.toPath(), response.getOutputStream());
    }

}

The conclusion:
In exact things the need for conclusion is deprecated. Everything should be concluded the one way :

return goToTopAndReadAgain();

The hint:

Maybe you won’t be impressed, and probably you have a better solution/implementation for/of the scenario. However let me give you a clue how this can be found useful in different situation. Proxying and load balancing, possibly with Nginx on front and couple of Tomcats behind. Defining new server contextses and getting a feel of that damn Superman speed.

Fiuuuuuuuuuuuuuuuuu…

(salutations and thanks to a friend of mine for collaboration)