Remove old backups with Bash Shell

Hello folks, I present you a bash script I wrote recently. It’s purpose  is deleting ‘old’ directories  marked by time period expressed in days. The script is simple and at its bases uses the command find to locate directories that we want to remove.

The general thinking on which this script is based are directories with backups categorized by some logical distinction and retention period for the second level of directories that are sorted by some other means like let’s say date. The script run by cron job searches the second level of directories and deletes the one older than some number of days n.

Example structure:

/mnt/backup/apps/2014-06-20/*

/mnt/backup/dbs/2014-06-1/*

/mnt/backup/logs/2014-06-1/*

 

If we want to exclude some of the directories in /mnt/backup , for example /mnt/backup/logs we can do that by putting the path into the variable EXCLUDED_DIRS. Note that only the directories from depth 1 can be excluded with this implementation (/mnt/backup/<> = OK , /mnt/backup/logs/<> != OK ).

EXCLUDED_DIRS=”/mnt/backup/logs /some/other/etc”

 

USAGE of the script:

<script_name> {arg1:/path/to/folder} {arg2:n days}  

or

./deleteOldDirs.sh         /mnt/backup           60

 

Note on find. For checking the timestamp of the directory I use ctime like option from find.  When this argument is used ,  find checks for the iNode pointing to the file/folder and collects information (like permissions, owner, etc) from which can be determinated when it was modified. In other hand if we want to check modification of the data in the backup itself we can use the argument mtime instead.

 

The script , beware of removing yourself.

May the force be with you !

#!/bin/bash

#
# GNRP LICENSE: Script licensed by GOD
#
# @author: Igor Ivanovski <igor at genrepsoft.net>
#
# July, 2014
#

#
# *** VARIABLES
#

PATH=$1 # First argument should be PATH of the backup folder
DAYS=$2 # Secound argument should be the max number of days we want to keep backups

GREP="/bin/grep"
MOUNT="/bin/mount"
FIND="/usr/bin/find"
RM="/bin/rm"

PATH_STATUS=0
MOUNT_STATUS=0

EXCLUDED_DIRS="/home/igor/Documents /home/igor/Programs"

#
# *** CHECKERS
#

if [ $# -eq 0 ]; then
    echo "No arguments provided"
    exit 1
fi

if [ -z $PATH ] || [ -z $DAYS ]; then
    echo "Not all arguments provided"
    exit 1
fi

#
# *** FUNCTIONS
#

function check_if_path_exists {

	[ ! -d $1 ] && echo "Path is invalild" || PATH_STATUS=1
}

function check_mount {
	 if [ -n "$($GREP $1 /etc/fstab)" ]; then
                echo "Mounting point exists..."
		if [ ! -n "$($MOUNT -l | $GREP $1)" ]; then
		    echo "...but not mounted."
		    echo "Mounting now."
                    $MOUNT $1
                    [ $? -eq 0 ] && { echo "Mounting was successfull!"; MOUNT_STATUS=1; } || echo "Can't mount";
		fi
         else
                echo "Mounting point doesn't exist"
         fi
}

# Excludes only if parent directory fits
function check_if_parent_dir_excluded { 
            for dir in $EXCLUDED_DIRS; do
		size=${#dir}
	        check=${1:0:size}
		#echo "$dir : ${#dir} == $check : ${#check}"
		if [ $dir ==  $check ]; then 
			echo "1"
		fi
	    done
}

function find_dirs {
	$FIND $PATH -mindepth 2 -maxdepth 2 -type d -ctime +$DAYS -print0
}

function print_dirs {
	while read -r n; do
                 result=$(check_if_parent_dir_excluded $n)
		 [ -z $result ] && printf '%q\n' "$n" || echo "Found EXCLUDE_DIR: $n" 
	done < <($FIND $PATH -mindepth 2 -maxdepth 2 -type d -ctime +$DAYS -print) # Use NL delimiter
}

function del_dirs {
        while read -r -d '' n; do
                 printf '%q\n' "$n"
                 result=$(check_if_parent_dir_excluded $n)
                 if [ -z $result ]; then
			 echo "Directory will be deleted now..."
			 $RM -rf $n
			 [ $? -eq 0 ] && echo "Gone!" || echo "Some error occured!"
		 else	
	 		echo "Found EXCLUDE_DIR: $n continue..." 
		 fi
        done < <($FIND $PATH -mindepth 2 -maxdepth 2 -type d -ctime +$DAYS -print0) # Use NUL delimiter
}

#
# *** MAIN
#

check_if_path_exists $PATH
[ $PATH_STATUS -eq 0 ] && check_mount $PATH
[ $MOUNT_STATUS -eq 1 ] && check_if_path_exists $PATH
[ $PATH_STATUS -eq 1 ] &&  del_dirs || { echo "Bye."; exit 0; }

#
# *** END
#

Creating Java keystore from existing private key and certificate

With the bunch of programming codes and programs found on the Web this days, Code Signing Certificates are fact and necessity. But the people who are end-users or developers are still in process of adjusting the awareness what programs/scripts/code are safe and junk free and should be trusted before running them on local machine. With this little guide I want to help new people which are diving into this area of problems.

Different platforms offer different way for code signing their apps , and in this post I will focus just on Java based systems.

Java web and desktop apps are bound with keystore files that keep the certificate chains signed by Internet authorities. With this technique it is easy to make distinction from trusted to untrusted programs with investment of some time and money.

Generally the process of creating Java keystore that can sign applications(source codes) can be covered in couple of steps that include the client and the certificate issuer:

  1. The client creates keystore file and generates private and public key pair
  2. The client exports Code Signing Request from the keys with personal and trustworthy data
  3. The client sends the CSR to Certificate issuer and waits for approval. Normally it is contacted during pending time.
  4. Certificate Issuer sends to the client the signed certificate and probably additional intermediate/root chain certificates that need to be included into the keystore.
  5. The client imports the certificate (probably in pkcs7 format) into the original keystore that was used to generate the keys and CSR with the appropriate alias that was used during the creation of the keystore.
  6. The keystore is included in Java applications and referenced with the alias so to sign the JARs used in the apps.

However it can happen the client to receive private key that ought to be used, without previously creating a valid keystore and generating key pair within it. This received key was used for generating CSR and certificate request was already sent to authority.

Well at this point it gets confusing what is the next step that should be taken and is it possible this key to be used for creating a new keystore? Some will say it is not possible (and seems logical because keytool doesn’t allow it), you will need to create new keystore and generate key pair  and issue a new certificate request with the CSR exported from this keystore key pair. That’s not true though, there is always a way.

Let’s suppose the original request has been approved and you received valid certificate cert.crt. At this point you have private.key and trusted cert.crt.

This files need to be merged and exported into pkcs12 format with the help of libssl library.

openssl pkcs12 -export -in cert.crt -inkey private.key -certfile cert.crt -name <certificate(alias)_name> -out keystore.p12

Next this new generated keystore.p12 should be used to create new keystore in JKS format with the help of keytool from the JDK.

keytool -importkeystore -srckeystore keystore.p12 -srcstoretype pkcs12 -destkeystore keystore.jks -deststoretype JKS
And that’s it voila! We have created keystore in jks format from existing private key.

After we have the keystore needed it is easy to import new certificates if required. Example:

keytool -import -trustcacerts -alias <alias(certificate)_name> -keystore keystore.jks -file <certificate_filename>

That’s it, three commands that will make your life easier.

SkeletorLead

Cheers,

Igor

Resolving error 413 (Request Entity Too Large/Not Allowed) in IIS 7.5

Working with IIS lately have brought me a lot of trouble, however it also increased my in-depth knowledge of its working abilities and adaptability.

One loving situation (between the minor ones) appeared after we did transition from http to https. After fixing the minor ones everything was working smooth and groovy except that sometimes the upload of files was broken. Then we realized it was not someTIME, but someTHING or the concrete size of the uploaded file that was causing the problem. Haa, so common problem when setting up Web Server you say, me said also, but this time the workaround was a little more pain in the *ss if you know what I mean.

The response given when uploading was an intelligent block by the Web Server resulting with error 413 – Request Entity Too Large. That doesn’t make a sense, I’m uploading  file that is 100KB but the settings allow max to 100MB file…

So with the help of rogue googling enabled by http://www.startpage.com I set my self digging into the problem overcome. One thing was clear, the trigger for this dysfunctionality certainly was https, since the uploading worked fine on http not secure. That gave me the rough but brilliant idea what https do. It encrypts, keeps extra request data, it certainly enlarges the request payload.

Hmm, OK first lets check the standard file limits in IIS.

Normal setting for max upload file size:

Setting the request limits in the root web.config of the site (default is 30 MB). This can be set in Internet Information Services Manager Program also (MACHINE->Site->IIS->Request Filtering->Edit Feature Settings)

<!– 100 MB . Format uses Bytes –>

<security>

    <requestFiltering>

        <requestLimits maxAllowedContentLength=”102400000″ />

    </requestFiltering>

   </security>

For ASP.NET you have more specific configuration:

http://msdn.microsoft.com/en-us/library/e1f13641%28v=vs.85%29.aspx

Example:

 <system.web>

   <httpRuntime maxRequestLength=”102400000″ executionTimeout=”3600″ />

 </system.web>

For legacy ASP:

<!– This goes under ASP – can be set with IIS Manager also 😉 –>

<limits maxRequestEntityAllowed=”102400000″/>

And here comes our solution:

http://www.iis.net/configreference/system.webserver/serverruntime

What we faced here is a buffer related problem. It’s not about the maxRequestEntityAllowed since the default size is Unlimited , but how IIS handles the request. After some empirics we noticed that the problem was occurring only with files larger or equal to 42 KB. And what is the default value of uploadReadAheadSize? 42 KB.

Then I charged myself to change this property. So I did the following:

Since we need to do section overriding through the means of <location> to change the default values (this configuration is not possible through IIS Manager)

<location path=”SiteName” overrideMode=”Allow”>

  <system.webServer>

   <asp>

    <session />

    <comPlus />

    <cache />

    <limits maxRequestEntityAllowed=”102400000″/>

   </asp>

    <serverRuntime enabled=”true” uploadReadAheadSize=”102400″ />

   </system.webServer>

</location>

However for some God sake reason, this configuration didn’t change the default values, even with overriding enabled for the serverRuntime section in the applicationHost.config . For easy check there is one good friend – the appcmd.

C:\Windows\System32\inetsrv\appcmd.exe list config “SiteName” -section:system.webServer/serverRuntime

And this bring us to the solution. To cut the story:

* Enable the serverRuntime section

C:\Windows\System32\inetsrv\appcmd.exe set config “SiteName” -section:system.webServer/ServerRuntime /enabled:”True” /commit:apphost

* Set the uploadReadAheadSize to 10MB

C:\Windows\System32\inetsrv\appcmd.exe set config “SiteName” -section:system.webServer/ServerRuntime /uploadReadAheadSize:”1024000” /commit:apphost

Restart if required, and that’s it.

Just aside note don’t forget to change the uploadReadAheadSize to something smaller and more realistic since 10MB is huge for a buffer, cause you don’t want to be hit by the nasty bad boyz with their huge payload packets.

Setting Up Context in Apache Tomcat for Serving Static Files

The intro:

So I’ve heard you want to serve static files from your Tomcat Web App in a way that they won’t be deleted on WAR redeploy or Tomcat restarted?

You have a solution, and that is mapping a custom Context in your Apache Tomcat server.xml .

The scenario :

You have a site that allows users to upload images that are public,shared and not under a hood of some security filter. The most intuitive solution is to put them in some directory i.e. ‘uploads’ , but then you realize that the things in the exploded WAR rewrite on redeploy or if the war is in the webapps directory on Tomcat restart. (you can change this behaviour)

The solution is simple: save the files to some directory outside of the war (something like ‘/usr/share/tomcat/uploads’) and map that directory on the server context of your Tomcat AS (something like http://lesite:8080/uploads).

With workaround like this you will see your uploaded cute kitty picture like this: http://lesite:8080/uploads/kitty.jpg

The implementation:

Let’s use the same examples. The mapping is done in <CATALINA_HOME>/conf/server.xml (hopes you know what and where catalina_home is )

This is default situation on new Tomcat install (a snippet from sever.xml):

<Host name=”localhost” appBase=”webapps” unpackWARs=”true” autoDeploy=”true”>

<Valve className=”org.apache.catalina.valves.AccessLogValve” directory=”logs”
prefix=”localhost_access_log.” suffix=”.txt”
pattern=”%h %l %u %t &quot;%r&quot; %s %b” />

</Host>

But we want to change that in this:

<Host name=”localhost” appBase=”webapps” unpackWARs=”true” autoDeploy=”true”>

<Context docBase="/usr/share/tomcat/uploads" path="/uploads" />

<Valve className=”org.apache.catalina.valves.AccessLogValve” directory=”logs”
prefix=”localhost_access_log.” suffix=”.txt”
pattern=”%h %l %u %t &quot;%r&quot; %s %b” />

</Host>

And that’s it, end of setting. Restart , code and redeploy.

The cookie:

Java snippet of simple utilization:

public class UploadsServlet extends HttpServlet {

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletExcpetion, IOException {
        File file = new File("/usr/share/tomcat/uploads", request.getPathInfo());
        response.setHeader("Content-Type", Files.probeContentType(file.toPath()));
        response.setHeader("Content-Length", String.valueOf(file.length()));
        Files.copy(file.toPath(), response.getOutputStream());
    }

}

The conclusion:
In exact things the need for conclusion is deprecated. Everything should be concluded the one way :

return goToTopAndReadAgain();

The hint:

Maybe you won’t be impressed, and probably you have a better solution/implementation for/of the scenario. However let me give you a clue how this can be found useful in different situation. Proxying and load balancing, possibly with Nginx on front and couple of Tomcats behind. Defining new server contextses and getting a feel of that damn Superman speed.

Fiuuuuuuuuuuuuuuuuu…

(salutations and thanks to a friend of mine for collaboration)

Setting up MySql server with utf-8 charset(s)

I can’t remember how many times I installed mysql server or mysql client on some Linux machines (occasionaly on Windows also) and forgot to change the charset of both client, connection, server etc.
For me being from non-latin culture and also working in such an enviroment it is always a step plus and torment while configuring the mysql servers to make them workable with utf8.
With that tought in mind, the purpose of this post is to give you fast solution on setting up utf8 or any other charset in your servers or development machines.
I also suggest making a skeleton my.cnf file and copy-pasting it on new instances of MySql.

Log in on mysql console and execute the query: mysql> show variables like ‘%char%’;

This will give you probably an output like this:

mysql> show variables like ‘%char%’;
+————————–+—————————-+
| Variable_name | Value |
+————————–+—————————-+
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+————————–+—————————-+
8 rows in set (0.00 sec)

(sorry for bad formatting)

Well it’s latin1 for the client, connection, current selected database, results, and whole server. Filesystem and system are fine . Let’s do the changes that are needed and set up utf-8.

Open up /etc/my.cnf or /etc/mysql/my.cnf (depending od distribution) with some editor (Vim, nano …). As we know MySql configuration is set in blocks that are categorized by function or logical part of the full system where we put different MySql parameters that are appropriate for that part of mysql. The beginning of this blocks is marked with square brackets, like [mysqld].

That are some basic stuff, the changes that are needed are following.

For the character_set_client in the block [client] we should set default-character-set=utf8 .

[client]
default-character-set=utf8

The encoding  and collation type of whole server and all new created databases will be set by the following parameters:

[mysqld]
character_set_server=utf8
collation_server=utf8_general_ci
init-connect=’SET NAMES utf8′      # A string to be executed by the server for each client that connects

(be aware of the group [mysqld])

A change is also needed in [mysql] for the mysql console client.

[mysql]

default-character-set=utf8

Remember you can always change this parameters by database scope, this is just the default behaviour of the server. You can also do system variables changes on runtime but not everything evaluates immediately. Of course the system variables have scopes of their own, some are global some or not.

See the following documentation for diving in : http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

That are the changes we do in the default option file. Remember, the existance of this file is recommended not needed for mysql to work.

Next we should restart the server and print the char variables.

This is the output:

mysql> show variables like ‘%char%’;
+————————–+—————————-+
| Variable_name            | Value                      |
+————————–+—————————-+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+————————–+—————————-+
8 rows in set (0.00 sec)

On Windows I use the Mysql Workbench program for managing , its operable on Linux too.

Have a nice day ! o/

Deploying Java Applications: Apache Tomcat + IIS 7.5

I’m on vacation and I’m writing my first technical post. That’s unfortunate 🙂

So this will be for all you lads and gentleman (and ladies) struggling in the world of Java and its ecosystem. Have you ever deployed Java apps on IIS? Well If I am the man to be consulted I’ll never choose that solution. But of course here comes the project managers, the Active Directory stuff and so on, when you have such climate you must bend yourself and do your best.

I have the following scenario: couple of different Java apps deployed on Tomcats. Well that doesn’t sound so bad there are easy service wizard installs of Apache Tomcat on Windows, easier deployment of the same so I will not really bother with such a trivial things, I hope you can do that by yourself.

Where is the problem? Well there a plural applications and one open port for the outside world. That’s right, the standard HTTP 80 port.

Simple thing we should do is to port forward and url rewrite the addressing applications.

Before finding this solution I was aware of one way on connecting Java container with IIS, through ISAPI and JK. But that requires changing Windows registry, defining couple of configuration files, using isapi_redirect , in other words messy and hardly-working solution. There are couple of guides on the net, just google it if you want to try them. The next solution is clean, simple and productive.

The IIS I’m working on is version 7.5. That is nice because the fellow Microsoft guys introduce a new solution to us, ARR. The use of this module here is to be set up to work as proxy server and that is exactly what we need in this situation.

After installing this extension on the server machine (I’m working on Windows Server 2008) next thing is to enable proxy server. Follow this steps:

  1. Open the IIS Manager and in the IIS Group there should be Application Request Routing Cache module showing up. Select it.
  2. So here we are. In the Action menu choose – Server Proxy Settings .
  3. For the following scenario this set up will work : Enable proxy = true, checked keep-alive, time-out by will, reverse rewrite host headers in response checked. Apply the changes and proceed.

Half work done. Next thing that we need is the URL rewrite feature. Install it. (Here I don’t mention  anything about the features and roles that need to be installed to make IIS working. I think that is just not-needed information here. You have the wizards in Computer Management, use them).

Well I hope you got URL rewrite working. That’s where we will define the rewrites.

Lets imagine. Two  Tomcat installation with just one application per server.

Installation folders: D:\Tomcat1 , D:\Tomcat2 . We need only the HTTP connectors, so in $tomcat_home\conf\server.xml you can comment out the AJP connectors.

Don’t forget while installing the Tomcats, to configure them on different ports. Example :

  <Connector port=”8080″ protocol=”HTTP/1.1″ proxyName=”www.example.com” proxyPort=”80″
               connectionTimeout=”20000″
               redirectPort=”8443″ />

  <Connector port=”8081″ protocol=”HTTP/1.1″ proxyName=”www.example.com” proxyPort=”80″
               connectionTimeout=”20000″
               redirectPort=”8443″ />

You ask yourself why I have put proxyName and proxyPort here. Well they are not required but since we want this application to work for outside world as I mentioned , you don’t want Java calls like getServerName() to return localhost  and local port. We want ajax and applets to work. This way we ensure the right host name and port are returned.

Installation ports: Tomcat1 on port 8080, Tomcat2 on port 8081. Installed apps: on Tomcat1 app1, on Tomcat2 app2.

So we can open the deployed applications on following urls : http://localhost:8080/app1 and http://localhost:8081/app2 .

It’s time to finalize the installations. We want the apps to be accessible on the following urls http://www.example.com/app1 and http://www.example.com/app2 .

This is where URL rewrite comes in play and that’s where we do the things explicitly. You can define URL rewrite rules in the main configuration but I prefer creating new site bound to port 80 and defining it there. Add new site with name exampe.com and for physical path put D:\ or whatever you want for your local situation, this is just example. That’s the place where web.config file will be created and everything that we do in the IIS manager will be mapped in configuration file, which is an xml configuration file the IIS use.

  1. In the new site, IIS group, open URL rewrite.
  2. Add new rule from the actions menu. Choose blank rule.
  3. Configurations : Name – Tomcat1 rule,  requested url -> matches the pattern , simple pattern: (app1.+) , ignore cases true,  action type – rewrite , rewrite url: http://localhost:8080/{R:0}
  4. Apply , make another rule for the other tomcat and  just change the name, pattern and the port.
  5. Restart IIS and you got it.

Just to get things straight. URL rewrite allow us to filter by some rule patterns, conditions and replace server variables. We just defined simple rewrite rule. That’s why we use {R:0} . It simple variable for the first rule pattern. There can be more of course , the format is {R:N} , where N is from 0 to 9.

This was very simple. I hope this will give you some clues how can you manipulate this modules to do more complex stuff. And you are not limited to use only Java servers of course. Any socket listening on any port can work.

I wish you happy days, for the end a reward from me

It is first, so let it be groggy

Welcome to you , welcome to me, what this blog will be?

Well it won’t be just groggy I hope. Primarily it will be written after work is done, when I’m tired, little restless and maybe a little more creative.

I will bother with technician stuff, sysadmin daily ad-hoc solutions, programming , things that interest me, maybe I will write something about you. Personal opinions will be included also.

So cheers.