Sorry to hear about that. If I understand you correctly, as you said you were using the delayed rules, I'm going to assume that you are not using ASL on any of these systems? Just the rules?
If thats the case, that sounds like a bad configuration, overlapping/conflicting third party rules, a leak in apache (or one of its libraries, like APR, libpcre or libxml) or in cpanels modsecurity.
So heres what I would start with:
1) Make sure you are using our modsecurity builds. Cpanel builds theres in some suboptimal ways. Also, the last time I looked cpanel was only up to 2.6.5, which has bugs and a rather large vulnerability. You want to use at least 2.6.6 for those reasons alone. (And the vulnerability will let the bad guys bypass modsecurity, so upgrade!)
2) Use the real time rules. The delayed rules are not supported. You can try the real time rules out for free from this URL:https://www.atomicorp.com/amember/signu ... aysys=free
3) Make sure your modsecurity configuration exactly matches the configuration described on the wiki. I've definitely seen cases like your where a bad configuration has caused a rule loop (where a configuration is loading multiple rules over and over again). The configuration is the easiest thing to fix, so make sure it matches exactly.https://www.atomicorp.com/wiki/index.ph ... rity_Rules
4) If you are using any third party rules (including cpanels rules), or addons (like ESX), disable them.
5) Check with cloudlinux as well. Cloudlinux controls all of the things that could be also contributing to this (apache and the kernel). As they make changes to apache and the kernel that effects memory usage and garbage collection this could be a bug. I havent heard of any, so I dont think thats the case here - but you should ask just in case.
modsecurity may just be a symptom of a larger problem with a leak. modsecurity will use up memory, and if you have a broken apache, kernel, or library it will manifest this leak faster. Disabling modsecurity may not actually be removing the problem, it might just be slower with less memory intensive modules.
6) Do you have any core files for apache? If the apache processes are dying off, it sounds like possible faults and you might have core files you can look at to find the real root cause. See the link before for how to set this up (for cpanel, you'll need to contact them about getting the debuginfo for their source built apache)https://www.atomicorp.com/wiki/index.php/Apache
A backtrace would be really handy to see whats causing apache to die.
7) Have you tried strace on your apache processes to see whats actually causing this with apache using up so much memory?
I've seen a broken PHP application do this. It was calling something that just didnt agree with Apache, and caused a leak to explode. Turning of modsecurity seemed to make it go away, but it turned out it just took longer with it off and mosec wasnt the cause.
bad mod_rewrite rules can also cause weird memory problems, I've definitely seen that with the cpanel modsecurity module. I'm not sure what they do, but it causes some odd interactions. You can rule this out by disabling mod_rewrite, and that doesnt solve it you can rule out mod_rewrite/mod_security interactions.
We were hoping to roll out full ASL on some servers soon, but we've been unable to stop these servers from crashing so are worried about making the next move.
Installing ASL is actually the simplest way to fix this problem. Just install ASL, or an ASL trial. ASL will "sanitize" your modsecurity setup with a clean correct working config, and a solid correctly built and optimized modsecurity module plus all the supporting libraries. Its a quick cheat, and you can always uninstall ASL if you dont need it. I've never seen this happen with ASL, but if were the first time we'll get it fixed for you, right away.
If this is still happening you'll know its modsecurity, plus we'll be able to help you with this. We'll even log into your server, for free, to find out whats going and get it fixed. What could be easier or cheaper than that?
With one step you can rule in or out modsecurity as the true cause, plus you'll get rapid support and a supported solution if the problem continues. Like I said, we'll even log into your system, for free, if necessary to find out and fix whatever is causing this.
This is the quickest and easiest way to solve your problem.
You can get a free trial copy here:https://www.atomicorp.com/amember/signu ... aysys=free