2011-04-06

TAMING RSYNC

When it comes to keeping directories in sync there are many tools out there but I feel that the good old rsync still does a very good job. Moreover, rsync jobs are easy to automate with standard unix tools (cron).


For this article we will use a user's home and sync its content to another directory.
So /home/qatqat  (my home dir, you will obviously use yours) will be mirrored to /storage/qatqat_home_bak

Now the basics:
a quick look at rsyn's man page shows:

 rsync [OPTION...] SRC... [USER@]HOST:DEST


mmmh....cryptic! Well, not actually.
let's apply it to our test scenario

execute all in one line
rsync -vraWogp --delete /home/qatqat /storage/qatqat_home_bak


Almost clear. let's go through the options:

-v verbose, detailed output
-r recursive, go and sync recursively into subdirectories (fundamental for a good backup)
-a archive, it means many of the other options together so it is probably redunant
-W whole files, no incremental check, writes more, but compares less
-o preserve owner
-g preserve group
-p preserve permissions
--delete , deletes files in destination directory that are no longer present in source directory

Now it all starts to make sense. the command will copy the content of /home/qatqat to /storage/qatqat_home_bak keeping all files' settings the same (which is good) and, if I delete something in /home/qatqat, it will do some cleaning into /storage/qatqat_home_bak (which is also good) when run again. The command outputs many lines to screen, that's why we want to catch them and save them to a synchrony log.

execute all in one line
rsync -vraWogp --delete /home/qatqat /storage/qatqat_home_bak > /storage/qatqat_backup_results.txt


Now we have a (very) detailed log of all files that have been synchronised.
Now if you use a browser (I am pretty sure you do) you will notice in the qatqat_backup_results.txt file that many thousands of lines refer to your internet cache being synchronised too. Personally I don't like that so here is a way to exclude some directories from the synchony process.

Create a new file:

touch /storage/rsync_excluded_dirs
add all dirs to exclude to it
echo "/home/qatqat/.mozilla/firefox/" >> /storage/rsync_excluded_dirs
echo "home/qatqat/any_other_dir_to_esclude" >> /storage/rsync_excluded_dirs

Now we tell rsync to read the file when synchronising:

 execute all in one line


rsync -vraWogp --delete --exclude-from=/storage/rsync_excluded_dirs /home/qatqat /storage/qatqat_home_bak > /storage/qatqat_backup_results.txt

Pretty nifty now.
Now, what's left is to execute the synchrony periodically, so we need to create a bash script.
A sample script is available here, edit it as you please.

qatqat_home_backup.sh.gz


now add the relevant line to /etc/crontab

execute all in one line
echo "30 20 * * 1,3,5 root /storage/qatqat_home_backup.sh > /dev/null 2>&1" >> /etc/crontab

The line above executes the script every mon, wed, fri at 20:30 PM. Google CRONTAB SYNTAX if you are not familiar with cron.

That's it for now.

Comments are welcome,

Ciao
QatQat

















QatQat

2010-11-14

A BASH SCRIPT TO MANAGE DYNAMIC DNS

Have you ever wanted to host a site on your home linux box or simply reach your home PC from outside? If yes, you have come to the right place.

For this you will need an account and a DNS from one of the many services that offer dynamic DNS update. The most popular is DynDNS where you can get a free sub-domain on one of their registered domains. If you instead want to have a more professional solution I normally recommend eNom, where you will have to register your own domain.
In this article I will cover both the above mentioned services (and others) but nothing stops you from using the same technique on virtually every dynamic DNS service.

Before we start make sure that you have "curl" installed on your linux box; if not:

apt-get install curl (for all debian based linux distributions)
yum install curl (for all redhat based linux distributions)

For all other distributions you can find it at http://curl.haxx.se/download/curl-7.21.2.tar.gz

Let's prepare all files we will need. As root create a directory called /etc/dynip_scripts, move into it and create the following files:
touch cur_addr (it will be used to store our current address)
touch update_dns (our bash script)
touch log_new_dns (a log for all new addresses)
touch log_unchanged_dns (a log for all unchanged DNS)

Now we get to the scripting side of our solution.
Updating your DNS takes 2 steps: finding your current address and pushing it to a dynamic DNS service.

Step 1) we need to find out our IP address. we have several ways to accomplish this task but I will focus on two most generic cases: linux handles your DSL connection or you have a router to connect to the Internet.
If linux is handling the connection, there should be an interface created to access the Internet (something like ppp0).
In this case "ifconfig" command will return your current address. Unfortunately ifconfig returns a plethora of useful information, totally unnecessary though for us at present.
We need to filter the output in order to get the current ip address.
Bash has all the instruments need:

ifconfig ppp0 | grep -i "addr:" | cut -f2 -d: | cut -f1 -d " "

The above command returns your IP address stripped of all other information

If instead a router is handling your connection you need to rely on external services to get the IP address. DynDNS hosts a basic public page to identify your address.

curl --silent http://checkip.dyndns.org | cut -f6 -d " " | cut -f1 -d "<"

If that doesn't work you can use a service I am hosting on my web server.
curl --silent http://www.intellisan.eu/get_ip.php?auth=210873 | grep IPAddress | cut -f2 -d:


Step 2) communicating your address to a DynDNS or eNom. Both services support updating your address through the querystring.
Once obtained a valid account and a DNS, whether a subdomain from DynDNS or your own registered domain from eNom, here are examples of working querystrings.

- DynDNS (I registered a subdomain of their homelinux.org domain called "qatqat", but you obviously will register your own). I will be therefore updating qatqat.homelinux.org, which you will replace with yours:

curl -u your_username:your_password "http://members.dyndns.org/nic/update?hostname=qatqat.homelinux.org&wildcard=NOCHG&mx=NOCHG&backmx=NOCHG&myip=your_ip_address"

If you are using eNom instead here is the querystring:
curl "https://reseller.enom.com/interface.asp?command=setdnshost&hostname=your_hostname&zone=your_domain&domainpassword=your_password&address=your_ip_address"


Now let's put all of this in our bash script update_dns. Edit it with your favorite text editor and copy the text below in it. In the script I have also included some more nifty features such as obtaining your address from a NETGEAR router, updating a no-ip or everyDNS dynamic domain address, or sending an alert email after the DNS is updated.


#!/bin/bash

#get the address stored in the cur_addr file
export CUR_IPADDR=$(cat /home/antonio/cur_addr)

#obtain IP address, uncomment all options that you will be using
export DYNIP_ADDR=$(curl --silent http://checkip.dyndns.org | cut -f6 -d " " | cut -f1 -d "<") #intellisan.eu server
#export DYNIP_ADDR=$(curl --silent http://www.intellisan.eu/get_ip.php?auth=210873 | grep IPAddress | cut -f2 -d:)

#GET ADDRESS FROM NETGEAR ROUTERS (tested with DG834 or other similar firmware)
#export DYNIP_ADDR=$(curl --silent -u admin:your_router_password http://192.168.1.1/setup.cgi?next_file=s_status.htm | grep -A1 -m1 "IP Address" | grep -v "IP Address" | cut -d ">" -f2 | cut -d "<" -f1) #obtain your IP address from ifconfig command - it only works if linux is handling your internet connection directly
#export DYNIP_ADDR=$(ifconfig ppp0 | grep -i "addr:" | cut -f2 -d: | cut -f1 -d " ")


#now we compare the current address with the one stored in the cur_addr file
if [ "$CUR_IPADDR" = "$DYNIP_ADDR" ]; then

#if the address has not changed we simply log it
echo "$(date) - unchanged address " >> /home/antonio/log_unchanged_dns

else

#if the address has changed instead we log it and carry on with the rest

we update the cur_addr file
echo "$DYNIP_ADDR" > /home/antonio/cur_addr

#we log an entry in the log_new_dns file
echo "$(date) - New IP $DYNIP_ADDR" >> /home/antonio/log_new_dns


#now we update our dynamic address on DynDNS
curl -u your_username:your_password "http://members.dyndns.org/nic/update?hostname=your_subdomain.homelinux.org&wildcard=NOCHG&mx=NOCHG&backmx=NOCHG&myip="$DYNIP_ADDR

#uncomment the line below if instead you are using eNom
#curl "https://reseller.enom.com/interface.asp?command=setdnshost&hostname=your_hostname&zone=your_domain&domainpassword=your_password&address="$DYNIP_ADDR


#here are strings for more services
#everydns.net

#curl --silent -u your_username:your_password "http://dyn.everydns.net?ver=0.1&domain=your_domain&ip="$DYNIP_ADDR

#no-ip
#curl --silent "http://your_username:your_password@dynupdate.no-ip.com/nic/update?hostname=yourhostname.domain&myip="$DYNIP_ADDR


#now our final touch. we send an email to inform you of the new address
echo "YOUR PUBLIC IP ADDRESS HAS CHANGED TO $DYNIP_ADDR ON $(date)" | mail -s "a service message from my server" your@yourdomain.com





Now, all is left to do is to make the update_dns script executable and hidden to normal users and then create an entry in /etc/crontavb that will execute it every 20 minutes.

Always as root execute the commands below:

chmod 700 /etc/dynip_scripts/update_dns

echo "10,30,50 * * * * root /etc/dynip_scripts/update_dns > /dev/null 2>&1" >> /etc/crontab


This is it really.


Feel free to use and modify this script as you please, comments and suggestions are anyway appreciated.
This script can be adjusted to work with basically all dynamic DNS services by simply adding lines with the curl command for the appropriate querystring.
I just did not have the time to discover many other services' querystrings so if you happen to find them out kindly post a comment here.

Cheers,

QatQat

2010-10-16

HOW TO INSTALL VMWARE SERVER 2.0.2 ON FEDORA 13

(UPDATED ON on 2010-11-14: This procedure also works with kernel 2.6.34.7-61)



Hi there,

Having upgraded my F12 box to the latest release I soon found out, like you if you are reading this, that VMWare Server couldn't compile the necessary kernel modules as expected when running /usr/bin/vmware-config.pl

After reading a few posts on the net, none of which was a direct solution to my problem, I solved the issue by putting together hints from different blogs.

Here is what I have done, but first of all my setup:

Fedora 13
Kernel 2.6.34.7-56.fc13.x86_64 (64Bit kernel but this should also work on a 32bit)
VMware-server-2.0.2-203138.x86_64



In order for everything to work well you will need the ".tar.gz" version of vmware-server from www.vmware.com (the patch will not work with the .rpm version)

The correct file to download is:

VMware-server-2.0.2-203138.x86_64.tar.gz

You now need to get the patch written by Radu Cotescu (special thanks to Radu for this nice piece of work)
http://codebin.cotescu.com/vmware/vmware-server-2.0.x-kernel-2.6.3x-install.sh

unzip+untar the script, if needed, than make it executable and run it as root


chmod +x vmware-server-2.0.x-kernel-2.6.3x-install.sh

./vmware-server-2.0.x-kernel-2.6.3x-install.sh [PATH_TO_VMWARE_ARCHIVE.tar.gz]


Follow the instructions on screen and you will get stuck when the script invokes vmware-config.pl. for you.

To solve this issue you will need to manually edit the file vmware-config.pl.

Open the vmware-config.pl file in any editor (as long as you do this as root) then search for :

if (-e $answer . ‘/linux/utsrelease.h’) {
$uts_headers .= “#include \n”;

and replace with:

if (-e $answer . ‘/generated/utsrelease.h’) {
$uts_headers .= “#include <./generated/utsrelease.h> \n”;


At this point, run the /usr/bin/vmware-config.pl file and the kernel modules will compile without issues.

I hope this helps you.

QatQat

2010-01-19

LA UBUNTIZZAZIONE DI LINUX

la ubuntizzazione di linux ci sta rendendo sempre piu' simili agli utenti Microsoft. Canonical diventa sempre piu' simile a Microsoft. Tra le varie dispute apprendo che la nuova release di Ubuntu non include' "The Gimp" come software di manipolazione immagini perche' ritenuto troppo difficile e perche' (testuali parole del comunicato stampa) "la schermata iniziale non e' accattivante abbastanza".

Di questo passo, invece di elevare il livello medio di conoscenze IT si raggiungerĂ  un triste livellamento verso il basso, proprio dove Microsoft regna sovrana.

Sotto un video di Fedora 13, una distribuzione molto piu' in sintonia con lo spirito iniziale di Linux e, sopratutto, che non usa alcun software proprietario.