Thursday, 29 May 2008

Apache memory leak Fix

I ran into an apache memory leak today... the symptoms where:
[root@www /]# service httpd restart
Stopping httpd: [OK]
Starting httpd: [FAILED]
[root@www /]# service httpd status
httpd dead but subsys locked

Ok that was weird... so I check the logs in /var/log/httpd/error_log

[emerg] (28)No space left on device: mod_fcgid: Can't create global pipe mutex
[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed

so... memory kinda full of it... solution:
[root@www /]# ipcs -s | grep apache
0x00000000 7897096 apache 600 1
0x00000000 7929865 apache 600 1
0x00000000 7962634 apache 600 1
0x00000000 7995403 apache 600 1
0x00000000 8028172 apache 600 1
0x00000000 8060941 apache 600 1
0x00000000 8093710 apache 600 1
0x00000000 8126479 apache 600 1
0x00000000 8159248 apache 600 1
0x00000000 8192017 apache 600 1
 ok... right on the money! so continue...

[root@www /]# ipcs -s | grep apache | awk ' { print $2 } ' | xargs ipcrm sem
resource(s) deleted
 and all is ok with the world again :)

Carlos Rivero on the same issue states "When dealing with mem-leaks in my mod_perl-apps I ran into a curious apache-problem" and since it is 4 in the morning I will just agree with him :) Damn that mod_perl then!

Anyway.. problem solved apache back up and running

Thank you for playing :)

Wednesday, 28 May 2008

Software RAID 1 and GRUB issues

A small problem I ran into the other day while I was trying to install a new kernel on a CentOS box... After making the changes I wanted on the grub.conf file, I run "grub-install /dev/md0" and I got this error:
"md0 does not have a corresponding BIOS drive"

(a small middle finger) no worries though..
after looking it up turns out the middle finger was justified.. grub does not look at /etc/fstab to find the corresponding drives instead it uses /etc/mtab 

In this case when having a /dev/md0 as / which corresponds to /dev/hda1 all that needs to be done is:
  • edit /etc/mtab and switch /dev/md0 with /dev/hda
  • run "grub-install /dev/hda" to install GRUB to the MRB of the first drive
  • edit /etc/mtab and switch /dev/md0 with /dev/hdb
  • run "grub-install /dev/hda" to install GRUB to the MRB of the second drive (just in case of failure)
  • edit /etc/mtab again and put it back as you found it with devices set to md instead of hd

Changes might need to be made using the same logic to the "/boot/grub/device.map" file which according to the Romanian guy that wrote the solution I used "maps a logical name such as /dev/hda to a physical device such as (hd0) which means (to grub) the first hard drive detected by the BIOS"

The actual link if you want the full article can be found here

Special thanks to "that guy" his solution did work and helped a lot.

Thanks for playing :)

Tuesday, 27 May 2008

Server up script - ping check

This silly little script checks if a host responds to initially 3 and then 5 icmp packets, if all fail then it sends an SMS using clickatell once again...
#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin

if ifconfig | grep ppp0 > /dev/null
then
PKT_LOSS=`ping -c 3 xxx.xxx.xxx.xxx | grep transmitted | awk '{print $6}'`
PERCENT=`echo ${PKT_LOSS%%%}`
if [ $PERCENT = "100" ]
then
PKT_LOSS=`ping -c 5 xxx.xxx.xxx.xxx | grep transmitted | awk '{print $6}'`
PERCENT=`echo ${PKT_LOSS%%%}`
if [ $PERCENT = "100" ]
then
wget "http://api.clickatell.com/http/auth?api_id=yourapi_id&user=usrname&password=passwd" -O /tmp/sessionid -q
# Create session and save to /tmp/sessionid
sessionid=`cat /tmp/sessionid | awk '{print $2}'`
# use only the session id for the variable
smssend clickatell.sms $sessionid usrname passwd yourapi_id "SystemAlert" phone_No "Server not responding"
# send sms
else
echo "System Responding"
else
echo "System Responding"
fi
else
#yes I have a speedtouch STILL!
pppd call speedtch > /dev/null
#bandwith QoS thingy nothing important...
bandwith
fi

For the smssend part to work you will need the info from this post.

Thank you for playing :)

Linux - Router NAT with some restrictions

A simple IPtables script that allows forwarding from one interface to the other connecting 2 subnets while restricting access to the second.

eth0:192.168.0.1 (internet)

eth1:192.168.1.1 (lan)

The /bin/nat-up script:
iptables -t nat -F POSTROUTING
iptables -F FORWARD
iptables --policy FORWARD DROP
iptables -A POSTROUTING -t nat -s 192.168.1.0/24 -o eth0 -j SNAT --to-source 192.168.0.1
# Full forwarding access to one host
iptables -A FORWARD -s 192.168.1.2 -j ACCEPT
iptables -A FORWARD -d 192.168.1.2 -j ACCEPT
# Access to one host only for the rest of the subnet
iptables -A FORWARD -s 192.168.1.0/24 -d host1 -j ACCEPT
iptables -A FORWARD -s host1 -d 192.168.1.0/24 -j ACCEPT
# Enable Forwarding...
echo 1 > /proc/sys/net/ipv4/ip_forward
echo "IPtables - NAT enabled ... "

Another script has been created to disable the NAT on demand (/bin/nat-down) containing:
iptables -t nat -F POSTROUTING
iptables -F FORWARD
echo 0 > /proc/sys/net/ipv4/ip_forward
echo "IPtables - NAT disabled ... "

Thank you for playing :)

Saturday, 24 May 2008

Ossec plugin for sms notification

1. Create an account with Clickatell (they will be providing the sms gateway service)

2. Add some credit to your account

3. Create a "Connection" and note you API_ID

4. download and install smssend
wget http://www.barsnick.net/sw/smssend-3.2-1.i586.rpm
yum install smssend-3.2-1.i586.rpm

or (for Slackware users)

wget http://linuxpackages.inode.at/Slackware-10.2/Console/smssend/smssend-3.4-i486-2alt.tgz

installpkg smssend-3.4-i486-2alt.tgz

5. Create the .sms file you will be using
cat > /usr/share/smssend/clickatell.sms << "EOF"
NbParams 7
%Sessionid : Session ID
%Login : Your username
%Password : Your Pass
%ApiID : Your API ID
%Sender : API Sender Name
%Tel : Phone number To Send Message To
%Message Size=160 Convert : Your message

PostURL https://api.clickatell.com/http/sendmsg?
#GetURL https://api.clickatell.com/http/sendmsg?
#Params session_id=\%Sessionid%&from=\%Sender%&user=\%Login%&password=\%Password%&api_id=\%ApiID%&to=\%Tel%&text=\%Message%
PostData session_id=\%Sessionid%&from=\%Sender%&user=\%Login%&password=\%Password%&api_id=\%ApiID%&to=\%Tel%&text=\%Message%
Search ID:
PrintMsg message sent
Else
ErrorMsg 1 error sending message
GO
EOF

6. Add the following to your /var/ossec/etc/ossec.conf
<command>
<name>smsnotify</name>
<executable>smsnotify.sh</executable>
<expect>srcip</expect>
</command>

<active-response>
<!-- This response will notify the admin via
- sms for every event that fires a rule with
- level (severity) >= 10.
-->
<command>smsnotify</command>
<location>local</location>
<level>10</level>
</active-response>

7. Create your own smsnotify.sh
cat > /var/ossec/active-response/bin/smsnotify.sh << "EOF"
#!/bin/sh

PATH=/sbin:/bin:/usr/sbin:/usr/bin
ACTION=$1
USER=$2
IP=$3

echo "`date` $0 $1 $2 $3 $4 $5 $6 $7 $8" > /tmp/alertid
ALERTID=`cat /tmp/alertid | awk '{print $11}' | cut -d "." -f 1`

# Getting alert header
LOG=`grep -A 3 $ALERTID /var/ossec/logs/alerts/alerts.log | tail -n 2`

# Create session and save to /tmp/sessionid
wget "http://api.clickatell.com/http/auth?api_id=''yourapi_id''&user=''yourusername''&password=''yourpassword''" -O /tmp/sessionid -q

# Use only the session id for the variable
sessionid=`cat /tmp/sessionid | awk '{print $2}'`

# Send sms
smssend clickatell.sms $sessionid ''yourusername'' ''yourpassword'' ''yourapi_id'' "ServerAlert" ''yourphoneNo'' "$LOG"
EOF

chmod 755 /var/ossec/active-response/bin/smsnotify.sh

8. Do an asl -f -s to restart ossec and your are good to go



update to come...

Small reconfigurations at Plesk services

ProFTPD
ServerIdent off

php.conf
add index.html to default index

disable fstab-sync

disable prelink

httpd.conf
Serversignature off

Servertokens Prod

when installing stuff php related
(easselerator, ioncube, zend optimizer) dont forget to check the paths in /etc/php.d/

thats it so far.. not much huh? :)

rc.firewall script

A simple script to start and stop.. and restart your firewall with slackware:

The script:
  • blocks some standard workstation traffic
  • allows NAT for 192.168.0.1/24
  • allows syslog connections for a particular host
  • drops invalid and possibly bad packets

#!/bin/sh
#
# /etc/rc.d/rc.firewall
#
# Start/stop/restart the IPtables Firewall.
#
# To make IPtables Firewall start automatically at boot, make this
# file executable:  chmod 755 /etc/rc.d/rc.firewall
#

fire_start() {
if [ -x /usr/sbin/iptables ]; then
fire_stop
echo "Starting IPtables: "
iptables -A INPUT -s 0.0.0.0/0 -i ppp0 -p tcp --dport 139 -j DROP
iptables -A INPUT -s 0.0.0.0/0 -i ppp0 -p tcp --dport 445 -j DROP
iptables -A INPUT -s 0.0.0.0/0 -i ppp0 -p tcp --dport 631 -j DROP
iptables -A INPUT -s 0.0.0.0/0 -i ppp0 -p tcp --dport 6000 -j DROP
iptables -A INPUT -s source-address -p udp --dport 514 -j ACCEPT
iptables -A INPUT -s 0.0.0.0/0 -p udp --dport 514 -j DROP
iptables -N SPOOF
iptables -A INPUT -i ppp0 -s ppp0-ip.address -j SPOOF
iptables -A INPUT -i eth0 -s 192.168.0.1 -j SPOOF
iptables -A SPOOF -m limit --limit 1/second -j DROP
iptables -N PORTSCAN
iptables -A INPUT -i eth0 -p tcp ! --syn -j PORTSCAN
iptables -A INPUT -i eth0  -m state --state INVALID -j PORTSCAN
iptables -A SPOOF -j REJECT --reject-with icmp-host-unreachable
iptables -L INPUT
echo "Enabling NAT: "
iptables -A POSTROUTING -t nat -s 192.168.0.1/24 -o ppp0 -j SNAT --to-source source-address
iptables -L POSTROUTING -t nat
iptables -A FORWARD -s 192.168.0.1/24 -j ACCEPT
iptables -A FORWARD -d 192.168.0.1/24 -j ACCEPT
iptables -L FORWARD
echo 1 > /proc/sys/net/ipv4/ip_forward
echo "IPtables - NAT Configuration finished ..."
fi
}

fire_stop() {
iptables -F
iptables -F FORWARD
iptables -F SPOOF
iptables -X SPOOF
iptables -X PORTSCAN
echo "Clearing Firewall:"
iptables -L INPUT
iptables -F -t nat
echo "Clearing Forwarding"
iptables -L POSTROUTING -t nat
iptables -L FORWARD
echo 0 > /proc/sys/net/ipv4/ip_forward
}

fire_restart() {
fire_stop
sleep 2
fire_start
}

case "$1" in
'start')
fire_start
;;
'stop')
fire_stop
;;
'restart')
fire_restart
;;
*)
# Default is "start", for backwards compatibility with previous
# Slackware versions.  This may change to a 'usage' error someday.
fire_start
esac

To be updated.. soon hopefully

Windows Forensic Tools

One of the good toolkit links that certainly helps in a windows forensic investigation is the Sysinternals Suite which includes packages like:
  • PsExec
    psexec \\remote systeminfo >> d:\data.txt
    psexec \\remote ipconfig /all >> d:\data.txt
    psexec \\remote arp -a >> d:\data.txt
    psexec \\remote netstat -b >> d:\data.txt
    psexec \\remote schtasks >> d:\data.txt
  • PsFile
  • PsGetSid
  • PsInfo
  • PsKill
  • PsList
  • PsLoggedOn
  • PsLogList (psloglist -s -x security)

Another useful tool which "enables us to capture the memory space utilized by any executing process. " is included in the Microsoft OEM Support tools package available at:
Memory Dump

Forensic Acquisition Utilities (dd, memory dumps, netcat and others) can be found here

Some of the above utilities are based on the UnxUtils distribution available here

A history of logins can be obtained with the NTLast command, distributed by Foundstone

and of course never forget the Helix Toolkit

more to come...

ps. Congrats need to go out to Geraint Williams for providing most of the information here, giving us brilliant lectures and most of all making the subject fun!

Plesk Backup Script

Taken from the atomic wiki (thanks Scott):

Install the required packages
  • yum install lftp
  • create /root/scripts/remote-backup
    #!/bin/sh
    #
    # Format date YYYY-MM-DD-HH-MM
    #
    date=`/bin/date +%F-%R`
    #
    # Create plesk backup
    #
    /usr/local/psa/bin/pleskbackup all /var/backup/fullbackup-$date.bak
    #
    # Connect to FTP, change directory, send files, disconect.
    #
    lftp -u username,password@# ftp.server.ip.address <<EOF
    cd /
    lcd /var/backup/
    mirror -R
    quit 0
    EOF
  • add script to crontab

all done :)

Atomic installation

Atomic installation procedure:
  • wget -q -O - http://www.atomicorp.com/installers/atomic.sh | sh
  • add plesk.repo to /etc/yum.repos.d
    [root@www ]# cat /etc/yum.repos.d/plesk.repo
    [plesk]
    name=Plesk Server Administrator
    baseurl=http://www.atomicorp.com/channels/plesk/latest/centos/$releasever/$basearch
    gpgcheck=0
  • yum install atomic-psa
  • add license
  • add IPs
  • add locale (change /usr/local/psa/version to 7 [compatible version] and then install locale package)
  • install advanced statistics
  • add asl channel
    [root@www ]# cat /etc/yum.repos.d/asl.repo
    [asl-2.0]
    name=Atomicorp - $releasever - Atomic Secured Linux
    baseurl=http://username:password@atomicorp.com/channels/asl-2.0/centos/$releasever/$basearch
    enabled=1
    gpgcheck=1

    [asl-2.0-testing]
    name=Atomicorp - $releasever - Atomic Secured Linux
    baseurl=http://username:password@atomicorp.com/channels/asl-2.0-testing/centos/$releasever/$basearch
    enabled=1
    gpgcheck=1

    [asl-2.0-bleeding]
    name=Atomicorp - $releasever - Atomic Secured Linux
    baseurl=http://username:password@atomicorp.com/channels/asl-2.0-bleeding/centos/$releasever/$basearch
    enabled=1
    gpgcheck=1
  • yum update
  • yum install asl asl-web-gui clamd qmail-scanner dcc pyzor razor-agents
  • restart spamassassin
  • start clamd service
  • run /usr/bin/qmail-scanner-reconfigure
  • run asl -c (configure asl)
  • run asl -u (update databases)
  • run asl -s -f (enforce configuration)
  • yum install php-easselerator php-suhosin
  • yum install squirrelmail
    add "Alias /cp /var/www/vhosts/metroweb.gr/httpdocs/cp"
    for the domain.gr/cp link.. in the squirrelmail.conf configuration in httpd/conf.d
  • yum install rkhunter chkrootkit (if not installed by asl package)

all done :)

Application Vault Issues

Ok here is how the story goes...

Issue: Application Repo is not working + backup script not working + domains cannot be removed/deleted

After updating Plesk from 8.2.1 to 8.3.0 everything seemed to be working except the application repo..

I was experiensing exact error "Table 'psa.APSClientApplicationItems' doesn't exist" with that guy here http://forum.swsoft.com/showthread.php?t=50070

mostly because I did the same lame thing... copied the directory .. changed the files and woohoo created my own application on repo .. so my clients can use an up to date Greek version of joomla...

I followed the procedure the guy suggested on the forum:
mysql -u admin -p`cat /etc/psa/.psa.shadow ` psa -e "describe APSClientApplicationItems;"

mysql -u admin -p `cat /etc/psa/.psa.shadow ` psa

DROP TABLE IF EXISTS `APSClientApplicationItems`;
CREATE TABLE `APSClientApplicationItems` (
`id` int(10) unsigned NOT NULL auto_increment,
`client_id` int(10) unsigned NOT NULL default '0',
`app_item_id` int(10) unsigned NOT NULL default '0',
`instances_limit` int(11) NOT NULL default '-1',
PRIMARY KEY (`id`),
UNIQUE KEY `client_id_2` (`client_id`,`app_item_id`),
KEY `client_id` (`client_id`),
KEY `app_item_id` (`app_item_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

but it didn't fix it... it was asking for APSApplicationItems table as well...

so i did the same procedure with that... and now it asks for more.... (on the Application Vault...)
MySQL query failed: Unknown column 'categories' in 'field list'

on the backup procedure we have:
DBD::mysql::st execute failed: Table 'psa.APSLicenseTypes' doesn't exist

Plesk support this to check the situation:
rpm -q psa; rpm -qa | grep psa-hotfix; cat /usr/local/psa/version
mysql -uadmin -p`cat /etc/psa/.psa.shadow` psa -e"select * from misc where
param='version'"

After enabling mysql's debug the technical support engineer did a series of database edits that added some missing tables and removed previous entries of installed non-existing applications. Since I don't actually know what was missing, why and any possible questions even though the guys actually said "ok we added that table and removed that one and now it works" I don't know if the information is useful to be put here so... i must going to call it magic and let us move on with our lives....