squidclamav http+https web filtering also utilizing squidblacklist.org acls

Installation of Squid, C-icap, Squidclamav, Clamav, Squidblacklist’s and Squid for web filtering on a GNU/Linux router.


 

This is not one big file as the continuous commenting would imply. you may need other -devel packages from yum.

Prerequisites:

Squid source (if using Fedora Beta 21), c-icap 3.2 source, squidclamav #source, clamav and clamav-scanner rpm from yum (fedora), #squidblacklists from squidblacklists.org, squid rpm from yum (fedora)

Firewall rules to be placed in that nat section of /etc/sysconfig/iptables remember to disable firewalld and enable iptables:

 

-A PREROUTING -s $INTERNAL_NETWORK -p tcp -m tcp –dport 443 -m conntrack –ctstate NEW,RELATED,ESTABLISHED -j REDIRECT –to-ports 3127
-A PREROUTING -s $INTERNAL_NETWORK -p tcp -m tcp –dport 80 -m conntrack –ctstate NEW,RELATED,ESTABLISHED -j REDIRECT –to-ports 3129
-A OUTPUT -p tcp -m tcp –dport 443 -m conntrack –ctstate NEW,RELATED,ESTABLISHED -m owner –gid-owner 23 -j ACCEPT
-A OUTPUT -p tcp -m tcp –dport 80 -m conntrack –ctstate NEW,RELATED,ESTABLISHED -m owner –gid-owner 23 -j ACCEPT

#where the https transparent proxy is run on port 3129. where the group owner id for squid is 23. some exceptions to bypass the squid proxy may be necessary. for instance my wireless thermostat doesn’t work with the squid https_port filtering.

#also to be placed in the nat section of /etc/sysconfig/iptables:

-A PREROUTING -s $THERM_IP -p tcp -m tcp –dport 443 -m conntrack –ctstate NEW,ESTABLISHED,RELATED -j ACCEPT

#install squid

yum install squid

#for Fedora Beta 21, install from source, i simply placed the binary “squid” compiled from source in /sbin/ this is a dirty hack but until Fedora fixes the problem, it works. the key is to compile squid with –disable-optimizations please see bug reports https://bugzilla.redhat.com/show_bug.cgi?id=1163874 and http://bugs.squid-cache.org/show_bug.cgi?id=4057

./configure –enable-ssl –enable-ssl-crtd –enable-icap-client –prefix=/usr –disable-optimizations
make
make install

#squid config relevent to web filtering (NOTE: this is not a complete squid config file, just parts relevant to this document):

#c-icap settings:

icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Authenticated-User
icap_preview_enable on
icap_preview_size 1024
icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/
squidclamav
adaptation_access service_req allow all
icap_service service_resp respmod_precache bypass=1 icap://127.0.0.1:134
4/squidclamav
adaptation_access service_resp allow all

#c-icap init script to be placed in /etc/init.d and made executable:

#!/bin/bash
#
# /etc/rc.d/init.d/c-icap
#
# Starts the c-icap daemon
#
# chkconfig: – 90 25
# description: c-icap Server
# processname: c-icap
# pidfile: /var/run/c-icap/c-icap.pid
# config: /usr/local/c-icap/etc/c-icap.conf
#
### BEGIN INIT INFO
# Provides: c-icap
# Short-Description: starting and stopping c-icap server
# Description: c-icap is an implementation of an ICAP server. It can be used \
# with HTTP proxies that support the ICAP protocol to implement content \
# adaptation and filtering services.
# Most of the commercial HTTP proxies must support the ICAP protocol. \
# The open source Squid 3.x proxy server supports it.
### END INIT INFO

PATH=/usr/bin:/sbin:/bin:/usr/sbin
export PATH

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

if [ -f /etc/sysconfig/c-icap ]; then
. /etc/sysconfig/c-icap
fi

ICAP_SHUTDOWN_TIMEOUT=20
exec=”/usr/local/c-icap/bin/c-icap”
config=”/usr/local/c-icap/etc/c-icap.conf”
pidfile=”/var/run/c-icap/c-icap.pid”
lockfile=”/var/lock/subsys/c-icap”

# determine the name of the c-icap binary
#[ -f ‘$exec’ ] && CICAP=c-icap

#prog=”$CICAP”
prog=”c-icap”

RETVAL=0

depend() {
after clamd@scan
}

probe() {
# Check that networking is up.
[ ${NETWORKING} = “no” ] && exit 1

[ `id -u` -ne 0 ] && exit 4

# check if the squid conf file is present
[ -f $config ] || exit 6
}

start() {
chown squid:squid /var/run/c-icap
probe
echo -n $”Starting $prog: ”
$exec -f $config
RETVAL=$?
[ $RETVAL -eq 0 ] && touch $lockfile
[ $RETVAL -eq 0 ] && echo_success
[ $RETVAL -ne 0 ] && echo_failure
return $RETVAL
}

stop() {
echo -n $”Stopping $prog: ”
if [ -f “$pidfile” ]; then
/usr/bin/killall $exec
RETVAL=$?
[ $RETVAL -eq 0 ] && rm -f $lockfile
[ $RETVAL -ne 0 ] && rm -f $pidfile
timeout=0
while : ; do
[ -f “$pidfile” ] || break
if [ $timeout -ge $ICAP_SHUTDOWN_TIMEOUT ]; then
return 1
fi
sleep 2 && echo -n “.”
timeout=$((timeout+2))
done
echo_success
else
/usr/bin/killall $exec 2> /dev/null
RETVAL=$?
[ $RETVAL -ne 0 ] && rm -f $pidfile
echo_success
echo c-icap is NOT running
return $RETVAL
fi
}

reload() {
echo -n $”Reloading $prog: ”
echo -n “reconfigure” > /var/run/c-icap/c-icap.ctl
}

restart() {
stop
start
}

condrestart() {
stop
start
}

rhstatus() {

# Short
#/usr/local/c-icap/bin/c-icap-client
# Long
pidof c-icap > /dev/null

RETVAL=$?
if [ $RETVAL -eq 0 ] ; then
/usr/local/c-icap/bin/c-icap-client -s “info?view=text” -req use-any-url
echo c-icap is running
fi
if [ $RETVAL -eq 1 ] ; then
echo c-icap is NOT running
fi

}

case “$1″ in
start)
start
;;

stop)
stop
;;

reload|force-reload)
reload
;;

restart)
restart
;;

condrestart|try-restart)
condrestart
;;

status)
rhstatus
;;

probe)
probe
;;

*)
echo $”Usage: $0 {start|stop|status|reload|force-reload|restart|try-restart|probe}”
exit 2
esac

exit $?

#squidblacklist settings (please see headers of each squidblacklist.org acl file for setup information):

acl blacklist dstdomain -i “/etc/squid/squid-malicious.acl”
acl blacklist2 dstdomain -i “/etc/squid/squid-usg.acl”
acl blacklist3 dstdomain -i “/etc/squid/squid-porn.acl”
acl blacklist4 dstdomain -i “/etc/squid/squid-proxies.acl”
acl blacklist5 dstdomain -i “/etc/squid/squid-ads.acl”
acl pornreg dstdom_regex “/etc/squid/squid-porn-regex.acl”

http_access deny blacklist
http_access deny blacklist2
http_access deny blacklist3
http_access deny blacklist4
http_access deny blacklist5
http_access deny pornreg

#http_port configuration directive:

http_port $SERVER_IP:3129 intercept

#https_port configuration directive:

https_port $SERVER_IP:3127 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl2/myCA.pem cafile=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem cipher=HIGH

#additional ssl proxy options:

always_direct allow all
ssl_bump server-first all
sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
sslcrtd_children 5
sslproxy_cert_error allow all
sslproxy_cert_adapt setCommonName ssl::certDomainMismatch
sslproxy_options ALL,SINGLE_DH_USE,NO_SSLv3,NO_SSLv2

#generate certificates as per squid-cache.org:

openssl req -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout myCA.pem -out myCA.pem
openssl x509 -in myCA.pem -outform DER -out myCA.der

#create ssl_db directory:

/usr/lib64/ssl_crtd -c -s /var/lib/ssl_db

#change ssl_db directory ownership:

#chown -R squid:squid /var/lib/ssl_db

#install certificate into desktop firefox browser or android os. firefox takes certificate in der format. android os: take the pem file and rename the extension to crt and import it for “VPN and apps” under security settings of your android os.

#warning: will still receive some ssl errors just inspect cert at that time and make sure the issuer is from the cert you created.

#clamav installation and configuration:

yum install clamav clamav-scanner

#edit /etc/clamd.d/scan.conf:

PidFile /var/run/clamd.scan/clamd.pid
LocalSocket /var/run/clamd.scan/clamd.sock
LocalSocketGroup squid
FixStaleSocket yes
c-icap and c-icap modules installation:
User squid
AllowSupplementaryGroups yes
DetectPUA yes
IncludePUA Spy
IncludePUA Scanner
IncludePUA RAT
IncludePUA Phishing
IncludePUA NetTool
IncludePUA PWTool
IncludePUA Tool

———-

usermod -G squid clamav
usermod -G clamav squid

#clamav unofficial sigs:

download clamav unofficial sigs from sourceforge

tar zxvf clamav-unofficial-sigs-3.7.1.tar.gz #(or whatever the current version is)
cd clamav-unofficial-sigs-3.7.1/
ls
#you’ll find all the files you need for operation:
clamav-unofficial-sigs.sh
clamav-unofficial-sigs.conf
clamav-unofficial-sigs-cron
clamav-unofficial-sigs-logrotate

#place the executable in a binary path, like /usr/local/bin
#place the configuration file in /etc
#place the cron file in /etc/cron.d
#place the logrotate file in /etc/logrotate.d

#relevant configuratuon file clamav-unofficial-sigs.conf

clam_user=”squid”
clam_group=”squid”
clamd_pid=”/var/run/clamd.scan/clamd.pid”
reload_dbs=”yes”
clamd_socket=”/var/run/clamd.scan/clamd.sock”
clamd_lock=”/var/lock/subsys/clamd.scan”
start_clamd=”service clamd.scan start”
user_configuration_complete=”yes”

#download c-icap from sourceforge.

tar zxvf c_icap-0.3.2.tar.gz
cd c_icap-0.3.2
./configure –enable-large-files
make
make install

#edit /usr/local/c-icap/etc/c-icap.conf
#commemt out service echo, set service to squidclamav and change user and group to squid
Service squidclamav squidclamav.so
#also set logfiles and directory:
ServerLog /var/log/c-icap/server.log
AccessLog /var/log/c-icap/access.log
#set up acls to deny connect from external:
acl localsquid_respmod src 127.0.0.1 type respmod
acl localsquid src 127.0.0.1
acl externalnet src 0.0.0.0/0.0.0.0
icap_access allow localsquid_respmod
icap_access allow localsquid
icap_access deny externalnet
MaxMemObject (you may want to increase this from the default)

#set squid ownership logfiles and directory for c-icap:
chown -R squid:squid /var/log/c-icap

#install squidclamav
./configure –with-c-icap=/usr/local/c-icap/ –sysconfdir=/etc/
make
make install

#edit /etc/squidclamav.conf
redirect http://YOUR_DOMAIN_HERE/cgi-bin/clwarn.cgi
#(you will need apache running and the clwarn.cgi file from the squidclamav site placed in /var/www/cgi-scripts
clamd_local /var/run/clamd.scan/clamd.sock
logredir 1
#place your whitelist domains on the “whitelist” line using squidclamav syntax. whitelisting domains with squidclamav is very easy.

#Here is your clamd.scan init script to be placed in /etc/init.d and made executable:

#!/bin/bash
#
# chkconfig: – 75 35
# description: The clamd server running for
# description: The clamd server running for Squid

CLAMD_SERVICE=scan
chown squid:squid /var/run/clamd.scan
. /usr/share/clamav/clamd-wrapper

Here is your c-icap init script:

#!/bin/bash
#
# /etc/rc.d/init.d/c-icap
#
# Starts the c-icap daemon
#
# chkconfig: – 90 25
# description: c-icap Server
# processname: c-icap
# pidfile: /var/run/c-icap/c-icap.pid
# config: /usr/local/c-icap/etc/c-icap.conf
#
### BEGIN INIT INFO
# Provides: c-icap
# Short-Description: starting and stopping c-icap server
# Description: c-icap is an implementation of an ICAP server. It can be used \
# with HTTP proxies that support the ICAP protocol to implement content \
# adaptation and filtering services.
# Most of the commercial HTTP proxies must support the ICAP protocol. \
# The open source Squid 3.x proxy server supports it.
### END INIT INFO

PATH=/usr/bin:/sbin:/bin:/usr/sbin
export PATH

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

if [ -f /etc/sysconfig/c-icap ]; then
. /etc/sysconfig/c-icap
fi

ICAP_SHUTDOWN_TIMEOUT=20
exec=”/usr/local/c-icap/bin/c-icap”
config=”/usr/local/c-icap/etc/c-icap.conf”
pidfile=”/var/run/c-icap/c-icap.pid”
lockfile=”/var/lock/subsys/c-icap”

# determine the name of the c-icap binary
#[ -f ‘$exec’ ] && CICAP=c-icap

#prog=”$CICAP”
prog=”c-icap”

RETVAL=0

depend() {
after clamd@scan
}

probe() {
# Check that networking is up.
[ ${NETWORKING} = “no” ] && exit 1

[ `id -u` -ne 0 ] && exit 4

# check if the squid conf file is present
[ -f $config ] || exit 6
}

start() {
chown squid:squid /var/run/c-icap
probe
echo -n $”Starting $prog: ”
$exec -f $config
RETVAL=$?
[ $RETVAL -eq 0 ] && touch $lockfile
[ $RETVAL -eq 0 ] && echo_success
[ $RETVAL -ne 0 ] && echo_failure
return $RETVAL
}

stop() {
echo -n $”Stopping $prog: ”
if [ -f “$pidfile” ]; then
/usr/bin/killall $exec
RETVAL=$?
[ $RETVAL -eq 0 ] && rm -f $lockfile
[ $RETVAL -ne 0 ] && rm -f $pidfile
timeout=0
while : ; do
[ -f “$pidfile” ] || break
if [ $timeout -ge $ICAP_SHUTDOWN_TIMEOUT ]; then
return 1
fi
sleep 2 && echo -n “.”
timeout=$((timeout+2))
done
echo_success
else
/usr/bin/killall $exec 2> /dev/null
RETVAL=$?
[ $RETVAL -ne 0 ] && rm -f $pidfile
echo_success
echo c-icap is NOT running
return $RETVAL
fi
}

reload() {
echo -n $”Reloading $prog: ”
echo -n “reconfigure” > /var/run/c-icap/c-icap.ctl
}

restart() {
stop
start
}

condrestart() {
stop
start
}

rhstatus() {

# Short
#/usr/local/c-icap/bin/c-icap-client
# Long
pidof c-icap > /dev/null

RETVAL=$?
if [ $RETVAL -eq 0 ] ; then
/usr/local/c-icap/bin/c-icap-client -s “info?view=text” -req use-any-url
echo c-icap is running
fi
if [ $RETVAL -eq 1 ] ; then
echo c-icap is NOT running
fi

}

case “$1″ in
start)
start
;;

stop)
stop
;;

reload|force-reload)
reload
;;

restart)
restart
;;

condrestart|try-restart)
condrestart
;;

status)
rhstatus
;;

probe)
probe
;;

*)
echo $”Usage: $0 {start|stop|status|reload|force-reload|restart|try-restart|probe}”
exit 2
esac

exit $?

——————-
#enable, start, and debug services:

systemctl enable squid.service
systemctl start squid.service
systemctl status squid.service -l

systemctl daemon-reload
systemctl enable clamd.scan.service
systemctl start clamd.scan.service
systemctl status clamd.scan.service -l

systemctl enable c-icap.service
systemctl start c-icap.service
systemctl status c-icap.service -l

#if any of the services don’t start it is most likely a permissions issue on a directory needed by a service and check your init.d scripts syntax.

#you can use swatch to email alerts once a virus is found on your network on http or https protocol:

yum install swatch

#create swatch directory, files, and init script, assuming you have an MTA set up that can mail to root@localhost. you can alias user root to your local account so email is sent to your user or set swatch to mail to another email account. make a symlink from /var/log/c-icap/server.log to /var/log/server.log:

mkdir /etc/swatch
create file server.log.conf in /etc/swatch with contents:

watchfor /Virus/
echo bold
mail=root@localhost,subject=VIRUS_DETECTED

swatchd init script:

#!/bin/sh
LOGS=”server.log”

start()
{
for i in `echo $LOGS` ; do
/usr/bin/swatch –config-file=/etc/swatch/$i.conf –tail-file=/var/log/$i –pid-file=/var/run/swatch-$i.pid –daemon > /dev/null >&1
done
}

stop()
{
for i in `echo $LOGS` ; do
PID=`cat /var/run/swatch-$i.pid`
kill $PID
done
}

case $1 in
start)
start
exit 0
;;
stop)
stop
exit 0
;;
restart)
stop
start
exit 0
;;
*)
echo “Usage: $0 { start | stop | restart }”
exit 1
;;
esac

—————-

systemctl daemon reload
systemctl enable swatchd.service
systemctl start swatchd.service
systemctl status swatchd.service -l

 

Make sure all services and processes are running and working together

Test iptables traffic with iptables -t nat -vnL

 

Conclusion:


 

Overall for android os browsers that accept android os root certificates works well. firefox on the desktop works well. have not tested in internet explorer or safari on windows or os x. would appreciate further testing on these platforms. firefox mobile doesn’t really work as it uses it’s own root certificate datastore and currently in firefox mobile there is no way to import the certificate where you don’t constantly get browser warnings about going to an untrusted site. every now and then your browser may ask you to verify the ssl cert for https. check it and make sure it’s valid before proceeding. https web virus filtering and squid on top of your https traffic may be overkill but if your paranoid about security this may be for you.

 

Browsers tested on:
Android Cynogenmod Native Browser (works)
Android Chrome (works)

GNU/Linux Firefox (works)
Android Firefox (doesn’t work)

 

Disable the blacklists and test your clamav https web filtering at:

https://www.etes.de/downloads/eicar-testvirus/

Also check your http web filtering(clamav) at:

http://www.f-secure.com/v-descs/eicar.shtml

Make sure to clear your browser cache before you retest on the same eicar file.

 

Please feel free to contact me with further testing or questions at lancelassetter (at) gmail (dot) com


Subscribe Today – Paypal or Credit Card Accepted.

Flat rate subscription. Select a membership option & subscribe.



Select Payment Option



  • You will be issued a username and password.
  • You will be granted access to our member area.
  • 5 Year Membership Option now vvailable.
  • For lifetime membership options click here.clipart
  • Contact us if you would like a pre order invoice.

Disclaimer: All sales are final, we do not issue refunds. Cancel your subscription anytime.