Sunday, May 14, 2006

Daily Checklist ... ... ...

Check List for Server Safeguards :

Check if any rpms have been installed Lately :

[root@localhost ~]# rpm -qa --last |more
xinetd-2.3.13-2mdk Sat 13 May 2006 08:14:31 PM EDT
anonftp-3.0-31mdk Sat 13 May 2006 08:10:01 PM EDT
wu-ftpd-2.6.2-6mdk Sat 13 May 2006 08:07:00 PM EDT

Checking People Connected to a Specific PORT :
For Example (port - 21,22,25 etc ...)

I checked for Port 25

[root@localhost ram]# lsof -i :25

COMMAND PID USER FD TYPE
master 5761 root 11u IPv4
telnet 21932 root 3u IPv4
smtpd 21933 postfix 6u IPv4
smtpd 21933 postfix 10u IPv4

DEVICE SIZE NODE NAME
12532 TCP localhost:smtp (LISTEN)
106579 TCP localhost:4835->localhost:smtp (ESTABLISHED)
12532 TCP localhost:smtp (LISTEN)
106600 TCP localhost:smtp->localhost:4835 (ESTABLISHED)

Another way is to use TCPDUMP to see any active connections are made to
a Particular port :

[root@localhost ~]# tcpdump -l -i eth0 port 22

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes

This will list the IP addresses from where the connections are made.

11:28:17.137522 IP localhost.ssh > localhost.1460: F 1463822750:1463822750(0)
ack 1461555328 win 8192

11:28:17.138112 IP localhost.1460 > localhost.ssh: F 1:1(0) ack 1 win 8192

11:28:17.138133 IP localhost.ssh > localhost.1460: . ack 2 win 8192

The above shows that I have made a connection from localhost to localhost on port 22


Imagine Systems stops working suddenly, One thing that comes immediately
to our mind is what changed :

#find /ram -mmin -30

The above commmad will find Files that have been changed in /ram in the
last 30 Minutes


#find / -mtime -1

The above command will recursively list all the file from / that have changed in the last
day.

Checking Logs

Check for Log Files created in /var/log , Files will be created as per applications
used by you.

Common Log files are

/var/log/boot.log --- Systems services that has been Started/Stopped
May 14 12:46:28 localhost xinetd: xinetd shutdown succeeded
May 14 12:46:35 localhost xinetd: xinetd startup succeeded

/var/log/messages --- Check Logins

May 14 12:50:27 localhost sshd[22453]: Accepted password for sriram
from 127.0.0.1 port 3242 ssh2

/var/log/secure --- Check Xinetd Services

May 14 09:30:07 localhost xinetd[5396]: START: ftp pid=21491 from=127.0.0.1
May 14 09:31:27 localhost xinetd[5396]: EXIT: ftp pid=21491 duration=80(sec)
May 14 11:15:32 localhost xinetd[5396]: START: ftp pid=21925 from=127.0.0.1
May 14 11:17:13 localhost xinetd[5396]: EXIT: ftp pid=21925 duration=101(sec)


/var/log/auth.log --- Check Authetication Failures

May 14 11:15:32 localhost xinetd[5396]: START: ftp pid=21925 from=127.0.0.1
May 14 11:17:13 localhost xinetd[5396]: EXIT: ftp pid=21925 duration=101(sec)
May 14 11:28:17 localhost sshd[22092]: fatal: Timeout before authentication for 127.0.0.1
May 14 12:50:27 localhost sshd[22453]: Accepted password for sriram from
127.0.0.1 port 3242 ssh2
May 14 12:52:18 localhost sshd[22503]: Accepted password for sriram from 127.0.0.1
port 3245 ssh2

#dmesg

The program helps users to print out their bootup messages.

Checking Disk Space of a Particular Folder,Files

[root@localhost ram]# du -h -s Beatles/
27M Beatles/

Install Dsniff for Packet Sniffing

[root@localhost ram]# dsniff
dsniff: listening on eth0
-----------------
05/14/06 16:58:47 tcp 10.10.93.220.1785 -> distrib-coffee.ipsl.jus.21 (ftp)
USER anonymous
PASS curl_by_sriram@haha.com

-----------------
05/14/06 17:35:24 tcp 10.10.93.220.1875 -> 202.41.x.x.21 (ftp)
USER xyz
PASS xyz123

Check for Current connections made FROM and TO Server

[root@localhost ram]# netstat -apln --inet
Active Internet connections (servers and established)

Will List the Complete details of From-IP-Port to To-IP-Port live connections.

Rejecting a Particular connections with out using IPTables

Say you do not want to reject request for a particular IP Address
Incomming and Outgoing

For Rejection:
[root@localhost ram]# route add -host 202.x.x.x reject

Again to allow:
[root@localhost ram]# route del -host 202.87.41.71 reject

If for eg. if you want to block a particular site do this

[root@localhost ram]# ping xyz.com
PING xyz.com (64.146.134.38) 56(84) bytes of data.

Now to block xyz.com
[root@localhost ram]# route add -host 64.146.134.38 reject

You can also block the Domain Name,
[root@localhost ram]# route add -host xyz.com reject

after blocking try xyz.com in your browser,

The message I got in my browser was,
"Connection was refused when attempting to contact xyz.com"

To unblock Later
[root@localhost ram]# route del -host xyz.com reject


Preventing DDOS Attack- Say your Load on the server is very high
For a Particular Service say HTTP- Port 80

Then you may do the Following :


1) At command prompt execute the below command

bash#netstat -lpn|grep :80 |awk '{print $5}'|sort

2) Check each block of ips.

Like let me say , that you have more than 30 connection from a single ip.
Under normal cases there is no need for that many number of connection
requests from a single IP. Try to identify such ips/networks from the list you get.

3) If more than 5 host/ip connects from the same network then its a clear sign of DDOS .

4) Block that ips/networks using iptables /Apf

[root@localhost ~]# iptables -I INPUT -s 202.87.X.X -j REJECT

If you have apf then just add the ips which you want to block in the file /etc/apf/deny_hosts.rules

5) Keep on continuing this process untill the attack on the machine gets reduced.

To Unblock
[root@localhost ~]# iptables -D INPUT -s 202.87.x.x -j REJECT

Suppose 202.87.x.x is website than that site will also be blocked.

Some Commands :

# lspci list all your pci devices
# netstat -arn show your network route information
# netstat -ap 2 | grep EST show established connections, updates every 2 sec
# netstat -Cr print routing information from routing cache
# iptables -nL show your current iptables configuration in numeric form
# ping -c ping X times.
# ping6 ping ipv6 addresses
# dmesg print or control the kernel ring buffer, bootup messages
# uptime check your linux servers uptime and load

Try this when your System Hangs - To Trace the Problems ?

Try the following:
1. Capture an alt-sysrq-t or alt-sysrq-p or alt-sysrq-b or alt-sysrq-m backtrace when the hang occurs.
2. Capture whatever is on the screen.
3. Look for kerlnel oops in the /var/log/messages or /var/log/syslog after reboot.
Note : sysrq is the Delete Key

Securing and Hardenning Linux Check this