Global Policies - Squid Proxy

Hey There,

Today I was thrown with this task of making sure that all our proxies in my company have the same White and the Black List. The company already has the List of category that they block, with a standard squid.conf and sfagent, Now they want to go ahead and block a website or allow it in less than 30 minutes all over the company. If it is small number of proxies then I can guess this can be manual, but in our company, we have greater than 150 proxies which are a mix of Bluecoat and squid. Bluecoats, it was very easy by using the central policy concept, for the squids, I decided that we should create a script that goes ahead and makes changes the squid.conf file adds a crontab, so on and so forth.

So I used the same Webserver, which the company was  using for the Bluecoat central policies and made a blocked list and allowed list text files. Then I created a shell script installer, which when run in our company environment, will modify the squid.conf file adding references of the allowed and the blocked list and also insert a crontab

I wrote the script in haste, so there might be a lot of bugs, but for the people it might be interesting, I will develop this as a full squid addon and put this on GNU, for the people who know shell, it might not be a great deal.

Also, to use the script, go ahead and copy this in you squid proxies (change the 1.2.3.4 to  your webserver ip and set the paths) hope this atleast gives an idea to the people

#!/bin/sh
#
# Author: Alok. A. S
# Global Policy Poller for Squid Proxies on Linux
# This will use the WGET installed on the system
# You will also need to run the ./scriptname.sh install to get instructions
#


if [ "$1" == "install" ]
then
        clear
        echo ""
        echo "********************************************************************"
        echo " "
        echo "      Welcome to the Global Policy Script Installer Function        "
        echo "   This script function will install itself into the configuration  "
        echo "       folder, you will have to put the script in crontab           "
        echo "  "
        echo "         Author : Alok. A. S (alokshrivastwa@gmail.com)                "
        echo "            Please direct any bug reports to the Author             "
        echo ""
        echo "********************************************************************"
        echo "Starting ..."
        echo ""

        os=`uname -a | awk '{print $1}'`
        wget_check=`which wget`
        squid_check=`which squid`
        config_file=`find / -name squid.conf | grep etc | head -1`
        scriptname=`basename $0`
        scriptdirname=`dirname $0`


        check=`echo $os | grep x`
        if [ $? -ne 0 ]
        then
                echo "Sorry, Not a *nix system, Will Not install"
                exit 127
        fi
        echo "Operating System : $os ... (Ok)"
        if [ ! -f "$wget_check" ]
        then
                echo "WGET Not found, Will Not Install ..."
                exit 127
        fi
        echo "WGET Location: $wget_check ... (Ok)"
        if [ ! -f "$squid_check" ]
        then
                echo "Squid executable not found, Will Not Install ..."
                exit 127
        fi
        echo "Squid Executable: $squid_check ... (Ok)"
        echo ""
        echo "Prerequisites checked fine, Continuing with Installation ..."
        echo ""
        echo ""
        echo "Squid Config File : $config_file"
        read -p  'Is this Correct? (y/n) ' option
        if [ $option != 'y' ]
        then
                echo ""
                read -p "Sorry, my bad, can you please tell me the path and file name of the config file ? " config_file
                echo ""
                echo "Thanks, Proceeding ..."
        fi
        config_directory=`dirname $config_file`
        isinstalled=`cat $config_file | grep -v "#" | grep global_blocklist`
        if [ $? == 0 ]
        then
                echo "Looks like it is already installed, Will exit"
                echo "The following line(s) was/were found : "
                echo ""
                echo "$isinstalled"
                echo ""
                exit 127
        fi
        lastline=`cat -n $config_file | tail -1 | awk '{print $1}'`
        firstacl=`cat -n $config_file | grep acl | head -1 | awk '{print $1}'`
        echo ""
        echo "Taking backup of original squid.conf (It will be appended with beforeglobal extension)"
        cp $config_file $config_file.`date +"%Y%m%d"`.beforeglobal
        echo ""
        echo "Creating a New Config file ..."
        newfile=$config_file.`date +"%Y%m%d"`.newconfig
        rm -rf $newfile
        touch $newfile
        chmod 666 $newfile

        lineno=`expr "$firstacl" - 1`
        remain=`expr "$lastline" - "$lineno"`

        blocklist="$config_directory/global_blockedlist.txt"
        allowedlist="$config_directory/global_allowedlist.txt"

        `head -$lineno $config_file >> $newfile`
        echo "acl global_blocklist url_regex \"$blocklist\"" >> $newfile
        echo "acl global_allowedlist url_regex \"$allowedlist\"" >> $newfile
        `tail -$remain $config_file >> $newfile`

        newfile1=$config_file.`date +"%Y%m%d"`.newconfig1
        rm -rf $newfile1
        touch $newfile1
        chmod 666 $newfile1
        lastline=`cat -n $newfile | tail -1 | awk '{print $1}'`
        firstacl=`cat -n $newfile | grep http_access | head -1 | awk '{print $1}'`

        lineno=`expr "$firstacl" - 1`
        remain=`expr "$lastline" - "$lineno"`

        `head -$lineno $newfile >> $newfile1`
        echo "http_access deny global_blocklist all" >> $newfile1
        echo "http_access allow global_allowedlist all" >> $newfile1
        `tail -$remain $newfile >> $newfile1`

        rm -rf $newfile
        read -p 'Can i replace the squid.conf file with the new one ? (y/n)  ' option
        if [ $option != 'y' ]
        then
                echo ""
                echo "Ok, Please do it manually, Move the script to the $config_directory and dont forget to add the crontab entry ..."
                echo "Execute squid -k reconfigure for this to take effect"
                echo ""
                echo "To Add a crontab, type the command crontab -e"
                echo "Then add the following line (in the last line) "
                echo "0,20,40 * * * * $config_directory/$scriptname"
                echo ""
                echo "and exit with wq! (like VI)"
                echo "Ciao ... "
                echo ""
                exit 127
        fi
        rm -rf $config_file
        mv $newfile1 $config_file
        echo ""
        echo "The new config file is in place ... Proceeding ..."
        myfilename="$scriptdirname/$scriptname"
        myconfigdir="$config_directory/"
        cp -f $myfilename $myconfigdir
        rm -rf "$scriptdirname/temcrontabfile.txt"
        touch "$scriptdirname/temcrontabfile.txt"
        `crontab -l | grep -v "#" >> "$scriptdirname/temcrontabfile.txt"`
        echo "0,20,40 * * * * $config_directory/$scriptname" >> "$scriptdirname/temcrontabfile.txt"
        `crontab "$scriptdirname/temcrontabfile.txt"`
        rm -rf "$scriptdirname/temcrontabfile.txt"
        echo "Crontab Installed ..."
        echo "Script copied to the squid folder ... "
        echo "Reconfiguring Squid ... "
        `$squid_check -k reconfigure`
        echo "The installation completed, the global list will be downloaded after 20 mins (or) run the script without the install option in $config_directory/"
        echo "Thanks ... "
        exit 127
fi

scriptname=`basename $0`
scriptdirname=`dirname $0`

blockedlist="$scriptdirname/global_blockedlist.txt"
allowedlist="$scriptdirname/global_allowedlist.txt"
logfile="$scriptdirname/GLOBAL_POLICY.log"

if [ ! -f "$logfile" ]
then
    touch $logfile
    chmod 666 $logfile
    # echo "File created"
fi

if [ ! -f "$blockedlist" ]
then
        echo "$blockedlist not found, Creating it ..."
        touch $blockedlist
        chmod 666 $blockedlist
fi

if [ ! -f "$allowedlist" ]
then
        echo "$allowedlist not found, Creating it ..."
        touch $allowedlist
        chmod 666 $allowedlist
fi


blockmd5=`md5sum $blockedlist | awk '{print $1}'`
allowmd5=`md5sum $allowedlist | awk '{print $1}'`

wget_check=`which wget`

newblockedlist="$scriptdirname/newblockedlist".`date +"%Y%m%d"`
newallowedlist="$scriptdirname/newallowedlist".`date +"%Y%m%d"`

urlblock='http://1.2.3.4/squid_blocked.txt'
urlallow='http://1.2.3.4/squid_allowed.txt'

$wget_check -q $urlblock -O $newblockedlist -T 5
$wget_check -q $urlallow -O $newallowedlist -T 5

if [ ! -f "$newblockedlist" ]
then
        echo "$newblockedlist not found, not downloaded, so exiting ..."
        exit 127
fi

if [ ! -f "$newallowedlist" ]
then
        echo "$newallowedlist not found, not downloaded, so exiting ..."
        exit 127
fi

blockmd5new=`md5sum $newblockedlist | awk '{print $1}'`
allowmd5new=`md5sum $newallowedlist | awk '{print $1}'`

if [ "$blockmd5new" != "$blockmd5" ]
then
        mv "$blockedlist" "$blockedlist.backup".`date +"%Y%m%d"`
        mv "$newblockedlist" "$blockedlist"
        echo `date` "Blocked List Updated" >> $logfile
fi

if [ "$allowmd5new" != "$allowmd5" ]
then
        mv "$allowedlist" "$allowedlist.backup".`date +"%Y%m%d"`
        mv "$newallowedlist" "$allowedlist"
        echo `date` "Allowed List Updated" >> $logfile
fi

rm -rf "$newallowedlist"
rm -rf "$newblockedlist"
exit

You need to copy it run the script with ./scriptname.sh install, and then it will do its thing. Hope this helps ....