store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Thu Oct 23, 2014 9:02 am

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 63 posts ]  Go to page 1, 2, 3, 4, 5  Next
Author Message
 Post subject: nf_conntrack: table full, dropping packet
Unread postPosted: Wed Nov 24, 2010 2:55 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue Nov 23, 2010 7:30 am
Posts: 283
Location: Glasgow, UK
Hi,

I installed ASL yesterday after a successful hack on my server - I've went through all the steps to get it installed and now have 1 moderate and 1 low vulnerability.

Everything was operating fine until a few minute ago when the server completely stopped responding.

I could not log in via SSH, view websites, use emails or even ping.


I used my remote console to reset the power and reboot.


After inspecting /var/log/messages, i can see that the server was doing things while it was unresponsive:

Code:
Nov 24 18:27:43 xxx xinetd[3079]: EXIT: ftp status=0 pid=16002 duration=52(sec)
Nov 24 18:28:59 xxx clamd[19629]: stream(127.0.0.1@1027): ASL.MalwareBlacklist.rbcmail.ru.UNOFFICIAL FOUND
Nov 24 18:29:08 xxx kernel: nf_conntrack: table full, dropping packet.
Nov 24 18:30:01 xxx psmon[16118]: Forking background daemon, process 16119.
Nov 24 18:30:01 xxx psmon[16119]: Forking second background daemon, process 16121.
Nov 24 18:31:00 xxx kernel: nf_conntrack: table full, dropping packet.
Nov 24 18:31:07 xxx postfix/smtpd[16131]: sql_sqlite3 plugin: no result found
Nov 24 18:32:13 xxx kernel: nf_conntrack: table full, dropping packet.
Nov 24 18:33:21 xxx syslogd 1.4.1: restart.


I'm assuming here that clamd found a virus in an email - nothing unsual about that (or am I wrong?).

What concerns me is this new log entry that i've never seen before:
Code:
kernel: nf_conntrack: table full, dropping packet.


Can anyone advise on what it is doing?


Thanks


p.s. The sql_sqlite log is one i've seen for months - I tried to get rid of it, but couldn't figure out how


Last edited by chrismcb on Wed Nov 24, 2010 3:24 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Wed Nov 24, 2010 2:58 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue Nov 23, 2010 7:30 am
Posts: 283
Location: Glasgow, UK
OK, so after the restart, all websites went down amost immediately again.

The log shows:

Code:
Nov 24 18:34:51 xxx kernel: PAX: From 217.44.79.193: execution attempt in: <anonymous mapping>, 9e6556af000-9e6615cb000 9e6556af000
Nov 24 18:34:51 xxx kernel: PAX: terminating task: /usr/sbin/httpd(httpd):4076, uid/euid: 48/48, PC: 000009e660cc9c30, SP: 00007fffdee8e5f8
Nov 24 18:34:51 xxx kernel: PAX: bytes at PC: 01 00 00 00 e6 09 00 00 20 9c cc 60 e6 09 00 00 60 01 00 00
Nov 24 18:34:51 xxx kernel: PAX: bytes at SP-8: 0000627cb0711ac9 0000627cb0629d1d 000009e65c61e3f0 00007fffdee90af8 0000000000000000 000009e660cdad48 0000000000000000 000000005f562f70 000009e65f2416a0 000009e65cce9c60 0000000000000000
Nov 24 18:34:51 xxx kernel: PAX: From 67.195.115.185: execution attempt in: <anonymous mapping>, 9e6556af000-9e6613c2000 9e6556af000
Nov 24 18:34:51 xxx kernel: PAX: terminating task: /usr/sbin/httpd(httpd):3816, uid/euid: 48/48, PC: 000009e661091b20, SP: 00007fffdee91078
Nov 24 18:34:51 xxx kernel: PAX: bytes at PC: 01 00 00 00 e6 09 00 00 10 1b 09 61 e6 09 00 00 60 01 00 00
Nov 24 18:34:51 xxx kernel: PAX: bytes at SP-8: 0000627cb0711ac9 0000627cb0629d1d 000009e65c61e3f0 00007fffdee92458 0000000000000000 000009e660ac0510 0000000000000000 000000005f562f70 000009e65f2416a0 000009e65cce9c60 0000000000000000


I restarted httpd manually and it's working now.

Are these related? Could the server still be under attack?


Last edited by chrismcb on Wed Nov 24, 2010 3:23 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Wed Nov 24, 2010 3:17 pm 
Offline
Forum Regular
Forum Regular

Joined: Sat Mar 28, 2009 6:58 pm
Posts: 865
Location: Germany
One very important notice.
ASL is mainly focused on preventing attacks and securing your server with a lot of useful addtions. Of course it scans and detects a lot of bad things but it's not intended to be installed for cleaning an almost hacked server. You should definitely research your problems and fix them. You can contact atomicorp to get a quote for that.

Concerning your PAX log entries:
http://www.atomicorp.com/wiki/index.php/ASL_FAQ#What_do_the_PAX_alerts_for_software_in_.2Fusr.2Flibexec.2Fpaxtest_mean.3F


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Wed Nov 24, 2010 3:23 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue Nov 23, 2010 7:30 am
Posts: 283
Location: Glasgow, UK
Thanks - I will contact them for a quote just now.

I am, in the mean time, trying to get to the bottom of the unresponsiveness that forced me to reboot.

After the hack, everything was restored and ASL installed immediately.

I do realise that whatever vulnerability allowed the hack to tak place is possibly still there.


Thanks for the link about the PAX messages - I figured out what was producing the message, but it was the actual message and its actions which concerned me:

Code:
Nov 24 18:34:51 xxx kernel: PAX: From 217.44.79.193: execution attempt in: <anonymous mapping>, 9e6556af000-9e6615cb000 9e6556af000
Nov 24 18:34:51 xxx kernel: PAX: terminating task: /usr/sbin/httpd(httpd):4076, uid/euid: 48/48, PC: 000009e660cc9c30, SP: 00007fffdee8e5f8


To me that looks like some remote computer has tried to execute something and ASL has shut httpd down to protect itself?
Is that right?


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Wed Nov 24, 2010 4:44 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
Exactly, someone tried to exploit a vulnerability in apache and ASL stopped apache from compromising your server - so if I understand correctly (and please correct me if I am wrong) was your systems compromised previously? If so then you really should reinstall the system from a known trusted image, there is no tool that can guarantee that its removed all the malware, rootkits, etc. from your system - the best (and quickest) solution is to reinstall the box, then lock it down.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Wed Nov 24, 2010 4:54 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue Nov 23, 2010 7:30 am
Posts: 283
Location: Glasgow, UK
Yes, the server was compromised yesterday - however I cannot afford a complete reinstall as I have clients currently using the server with no current alternative.

The two IP adresses in the logs were for a computer local to my location probably looking at a website for one of my clients and the Yahoo crawl bot.

I wouldn't have expected these to cause ASL to shut apache down.

Can I find out somehow what it was that caused apache to be shut down?


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Wed Nov 24, 2010 5:21 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
Thats hard to say, if the system is compromised it could be anything, you could have a compromised library, apache itself, some module, even a script - the box is basically in a corrupted state. Someone else had root on your box and installed software on it, designed to hide itself, so the root cause of apache dying could really be anything. There are, unfortunately, no easy answers in this case. Its like your body getting possessed, no telling what it might do and no telling why either.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Thu Nov 25, 2010 5:35 am 
Offline
Forum Regular
Forum Regular

Joined: Tue Nov 23, 2010 7:30 am
Posts: 283
Location: Glasgow, UK
OK, I'm biting the bullet and purchasing a new server from my hosting company to migrate all websites/emails to using the Plesk migration tool.

What advice can you give to setting up a new server?

Update the system, install ASL and follow its guidance to secure things?
Anything else?


Your help is much appreciated.


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Thu Nov 25, 2010 6:54 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Mar 28, 2009 6:58 pm
Posts: 865
Location: Germany
Additionaly I would setup a firewall and configure it as secure as you can. afp or the plesk one.
If your provider provides a firewall in front of your server I would configure that one too.
Besides that a backup system/script/whatever you prefer is mandatory.
An antivirus solution is mandatory too. like clamav from atomicorp.
spamassassin, rbl and greylisting. rkhunter. some kind of monitoring for cpu, processes, diskspace etc.

I bet others will have some extra tipps I haven't thought of or forgot them :)


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Thu Nov 25, 2010 8:22 am 
Offline
Forum Regular
Forum Regular

Joined: Wed Jan 02, 2008 3:21 pm
Posts: 521
Location: United Kingdom
All excellent tip's from BruceLee!

As you're not sure how the bad guys got into your server, disable direct root SSH log-in and add a no privileges account which can su - to root, ensure you use NEW SSH PASSWORDS or combine this approach with certificate authentication. ASL will highlight the latter as a priority (if you've not done so already, its worth reading up before hand, there are several guides on-line covering Securing SSH with Key Based Authentication).

If you have a static IP and its feasible, only allow that addresses to connect via SSH (using Plesk/AFP firewall).

Before migrating website, run: rkhunter -c -sk & clamscan -r -i /var/www/vhosts/ - both part of the atomicorp/ASL install to ensure you do not transfer badness!

Opting for the new server is the only realistic option, the inconvenience of migrating sites is minor compared to being owned over and over again.

Good luck!


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Thu Nov 25, 2010 8:27 am 
Offline
Forum Regular
Forum Regular

Joined: Tue Nov 23, 2010 7:30 am
Posts: 283
Location: Glasgow, UK
Excellent advice guys.

Once all is put down on paper (or is it on screen?) its so obvoius - will get those done asap.


Had problems upgrading Plesk - but got them sorted now.

Will get the new server configured for a (hopefully) smooth transfer.


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Thu Nov 25, 2010 9:25 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Mar 28, 2009 6:58 pm
Posts: 865
Location: Germany
concering the security: I set all the passwords for the users at the beginning and give them the tool keepass (keepass.info) with their db. I advice them to use it and let it generate passwords.
So take strong ones for everything. Of course it doens't help if a client computer got hijacked and the password is sniffed but it prevent simple brute or guess attacks.

Your Plesk problem is that Plesk 9 has a bug where every update/installation replaces postfix with qmail and vice versa.
If you want to use qmail exclude postfix package in plesk.repo file via exclude=psa-mail-pc-driver*
If you want to use postfix exclude qmail package in plesk.repo file via exclude=psa-mail-qc-driver*


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Thu Nov 25, 2010 10:01 am 
Offline
Forum Regular
Forum Regular

Joined: Tue Nov 23, 2010 7:30 am
Posts: 283
Location: Glasgow, UK
Yes, I'm learning that now about passwords.

Any ones brought to my attention with ASL will have their passwords updated - and i'll set passwords up for them in the future, as you suggest.


Everything seems configured and happy now.

Just going to add the Atomic repo for one final update then i'll sit tight until close of business.


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Thu Nov 25, 2010 10:06 am 
Offline
Forum Regular
Forum Regular

Joined: Sat Mar 28, 2009 6:58 pm
Posts: 865
Location: Germany
if you update with atomic repo than mysql will be updated.
take a look in the wiki how to do it:
http://www.atomicorp.com/wiki/index.php/Mysql


Top
 Profile  
 
 Post subject: Re: nf_conntrack: table full, dropping packet
Unread postPosted: Thu Nov 25, 2010 10:10 am 
Offline
Forum Regular
Forum Regular

Joined: Tue Nov 23, 2010 7:30 am
Posts: 283
Location: Glasgow, UK
Thanks - i've used the atomic repo in the past for updates.

Having dependancy issues again though - this time with sqlite2... the investigation continues!


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 63 posts ]  Go to page 1, 2, 3, 4, 5  Next

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group