store | blogs | forums | twitter | facebook | wiki | mailing lists | downloads | support portal
Atomic Secure Linux
It is currently Thu Apr 24, 2014 8:09 am

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 10 posts ] 
Author Message
 Post subject: CouchDB won't start under ASL kernel
Unread postPosted: Fri Dec 09, 2011 8:06 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
CouchDB, installed using the RPM package in EPEL5, won't start normally when running on the ASL kernel.

Code:
# rpm -q couchdb
couchdb-0.11.2-2.el5.x86_64
# service couchdb start
Starting couchdb:                                          [  OK  ]
# ps aux | grep couch


So, nothing in the output of 'ps aux' containing 'couch'. When CouchDB is running it should output something like this:

Code:
# ps aux | grep couch
couchdb   3249  0.0  0.0   8840  1148 ?        S    00:56   0:00 /bin/sh -e /usr/bin/couchdb -a /etc/couchdb/default.ini -a /etc/couchdb/local.ini -b -r 0 -p /var/run/couchdb/couchdb.pid -o /dev/null -e /dev/null -R
couchdb   3260  0.0  0.0   8840   672 ?        S    00:56   0:00 /bin/sh -e /usr/bin/couchdb -a /etc/couchdb/default.ini -a /etc/couchdb/local.ini -b -r 0 -p /var/run/couchdb/couchdb.pid -o /dev/null -e /dev/null -R
couchdb   3261  2.0  0.6 100708 13576 ?        Sl   00:56   0:00 /usr/lib64/erlang/erts-5.6.5/bin/beam.smp -Bd -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/couchdb -noshell -noinput -sasl errlog_type error -couch_ini /etc/couchdb/default.ini /etc/couchdb/local.ini /etc/couchdb/default.ini /etc/couchdb/local.ini -s couch -pidfile /var/run/couchdb/couchdb.pid -heart
couchdb   3270  0.0  0.0   3664   364 ?        Ss   00:56   0:00 heart -pid 3261 -ht 11
root      3287  0.0  0.0  63292   844 pts/0    S+   00:57   0:00 grep couch


In /var/log/messages I find the following message after a failed attempt to start CouchDB when the ASL kernel is active:

Quote:
kernel: grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/lib64/erlang/erts-5.6.5/bin/beam.smp[beam.smp:1750] uid/euid:118/118 gid/egid:116/116, parent /usr/bin/couchdb[couchdb:1748] uid/euid:118/118 gid/egid:116/116


As far as I know this message only says that the application crashed, but it doesn't tell me why. One other thing I noticed is that when I run 'couchdb --help' when the ASL kernel is active I see a message about failing to create a thread (operation not permitted).

What could be preventing CouchDB from starting when the ASL kernel is active?

I have verified that when the server is not running the ASL kernel CouchDB starts and works fine, but won't start when the ASL kernel is running.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: CouchDB won't start under ASL kernel
Unread postPosted: Sat Dec 10, 2011 5:26 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3548
Location: Chantilly, VA
So this:
Quote:
kernel: grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/lib64/erlang/erts-5.6.5/bin/beam.smp[beam.smp:1750] uid/euid:118/118 gid/egid:116/116, parent /usr/bin/couchdb[couchdb:1748] uid/euid:118/118 gid/egid:116/116


Is just the kernel reporting that you dont allow core dumps (not the kernel preventing them). So the first thing to do is to configure your system to allow core dumps and to do a backtrace on couchdb to see why its core dumping. If nothing else is logged by the kernel then its something in couchdb thats causing it die, best guess is that its doing something very bad on the system (like its set to have an executable stack, you'd be surprised at how many developers do this and how often its absolutely not necessary). So post the backtrace and we can see whats going on.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: CouchDB won't start under ASL kernel
Unread postPosted: Sat Dec 10, 2011 6:02 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3548
Location: Chantilly, VA
So a quick fiddle with couchdb, it looks like it wants to be able to smash your stack and the kernel is preventing it from opening that hole on your system. Just configure couchdb to remove this hole in your system with this command:

execstack -c /usr/lib64/erlang/erts-5.6.5/bin/beam

(Or whatever your path is to beam)

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: CouchDB won't start under ASL kernel
Unread postPosted: Sun Dec 11, 2011 8:28 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
Thanks for looking into this.

Code:
# rpm -qf /usr/lib64/erlang/erts-5.6.5/bin/beam.smp
erlang-R12B-5.10.el5.x86_64


So, it looks like it's actually the Erlang package that's the problem here (CouchDB is written in Erlang, beam and beam.smp are Erlang binaries). Is this something I should tell the developers or packager they should fix?

Would it be possible for the kernel to log attempts to start executables that want to be able to smash stacks? If so, OSSEC could pick up those messages and it would make it easier for users for find out that they have these executables on their system.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: CouchDB won't start under ASL kernel
Unread postPosted: Sun Dec 11, 2011 2:58 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3548
Location: Chantilly, VA
You can do that now, just turn rwxmap_logging on. In the ASL GUI just set RWXMAP_LOGGING to "yes".

Example:

Dec 11 13:57:54 asl-modsec-test kernel: grsec: From 192.168.1.250: denied RWX mmap of <anonymous mapping> by /usr/lib64/erlang/erts-5.6.5/bin/beam[beam:2729] uid/euid:108/108 gid/egid:156/156, parent /usr/bin/couchdb[couchdb:2728] uid/euid:108/108 gid/egid:156/156

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: CouchDB won't start under ASL kernel
Unread postPosted: Sun Dec 11, 2011 4:57 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
Ok, thanks. Why is RWXMAP_LOGGING off by default?

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: CouchDB won't start under ASL kernel
Unread postPosted: Wed Dec 21, 2011 6:44 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
Does enabling RWXMAP_LOGGING have a performance impact? Any other reason we wouldn't want this enabled on all boxes?

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: CouchDB won't start under ASL kernel
Unread postPosted: Wed Dec 21, 2011 6:59 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3548
Location: Chantilly, VA
No impact, just logs it.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: CouchDB won't start under ASL kernel
Unread postPosted: Wed Dec 21, 2011 7:06 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
Sounds incredibly useful for finding out why programs won't start on the ASL kernel. Might be a good idea to enable by default on new ASL installations?

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: CouchDB won't start under ASL kernel
Unread postPosted: Tue Jan 10, 2012 3:49 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
I'm still wondering why RWXMAP_LOGGING is not enabled by default. Should I file a request in the tracker?

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group