store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Wed Oct 22, 2014 11:46 pm

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 5 posts ] 
Author Message
 Post subject: The segfaults have gone. All hail the Bus error!
Unread postPosted: Wed Nov 17, 2010 3:02 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2082
Hehe. Very strange. Following a recent kernel update, the apache segfaults (and often associated glibc errors) I've been seeing have been vanquished.

Unfortunately they've been replaced by:

Code:
[notice] child pid 9708 exit signal Bus error (7)


Before anyone suggests it, the system's memory is 100% fine and coincidentally has been recently tested along with all other main hardware components (and in any case is ECC). This is more likely indicating a lack of memory for apache to use somehow, even though the VPS in question has 2Gb allocated to it and normally uses less than 1Gb of that, or is just a different manifestation of the original bug/oddity.

So although the words have changed, the music is still the same :-)

Anyway, the reason for my posting is just to warn people to look out for this as well as for segfaults if you run a Virtuozzo or OpenVZ-based system.

Faris.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: The segfaults have gone. All hail the Bus error!
Unread postPosted: Wed Nov 17, 2010 3:24 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
Bus Errors are caused by a program crashing, and since this is apache it could be a web app, (CGI, PHP, PERL, etc.), a bug in PHP, an accelerator, etc. - it means the program crashed. A bus error can happen for other reasons, like total incompatibility for example if someone writes a program for one architecture and then runs it on a different one (Intel vs AMD, PPC vs MIPS, etc.). This can also happen with a bug in an accelerator, like PECL or even in PHP in code is cached and fubar somehow. So check there too.

Really all this means is someone is running buggy software on your machine.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: The segfaults have gone. All hail the Bus error!
Unread postPosted: Wed Nov 17, 2010 5:41 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2082
Interesting...

I don't really know where/how to look for anything that might fit the bill. There are no executables on here that I've not installed myself, no unusual accelerators, and I've not added anything "extra" really. Well, there are a couple of things, but I've tested and neither *seem* to trigger the problem.

Also it is pretty much identical behaviour to the segfaults. After a mod_sec rules update I'll usually get a few (even after I've reduced the two big blacklists), I restart apache a few times, and then it is usually stable for a long time.

I've had more than the usual number after today's rules update, mind you. I'm guessing there have been significant additions since I last updated, which means I'll probably have to reduce the blacklists even more.

I've not seen any segfaults or bus errors on the other two systems we have runnning ASL for a very long time, but both are very lightly loaded compared to this one and both routinely have the blacklists reduced after a rules update.

I may try to provoke one of the other systems over the weekend by not reducing the blacklist to see if I get a bus error or whether I get a segfault.

I'm assuming it isn't a script on one particular system because I get segfaults (or used to get them) on all three. I always used to suspect horde because way back in the past I used to be able to reliably get a segfault after a reboot by logging in to webmail. But not these days.

Faris.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: The segfaults have gone. All hail the Bus error!
Unread postPosted: Wed Nov 17, 2010 5:59 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
Just remember bus errors are full crashes of an application - it means a web application itself crashed. So you need to look at your apps, and what they use directly. Check your PHP extension order too - and bus errors (segfaults too) can be caused by apps that dont like the version of PHP you have installed for example.

A bus error can also be caused by a bug in PHP or some PHP extension thats not keen to your app can cause it to crash and you would get a bus error. Apache modules don't cause bus errors, its an application problem in the application itself, or its interpreter (like PHP, Perl, etc.) and those libraries. Apaches just in the middle and not the cause, dont waste your time with anything other than tracing down the app thats cause the bus error. Anything else is just treating the symptom.

To that end you should look at the apache debugging modules that use the exception hooks and use them to help you track down your buggy apps:

http://people.apache.org/~trawick/exception_hook.html

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: The segfaults have gone. All hail the Bus error!
Unread postPosted: Mon Nov 22, 2010 1:14 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2082
Unfortunately there are no pre-compiled versions of those modules - and having looked into mod_whatkilledme in the past, trying to get that up and running was not something we could do on our production systems.

The only common thing between all three systems that exhibit this behaviour is the version of Apache, suhosin and Possibly Ioncube but I suspect at least one of them has a different version to the others. Horde webmail is the only common PHP script between the three.

I'll keep plodding away and maybe eventually we'll find the cause.



Faris.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group