Er, every event detail window is the same? Thats not right, and a new one for me.

So right off the bat, what happens if you clear your browsers cache? I'm wondering if its something locally cached...

As for the crash, can you expand on a little about what happened that would help me to direct you to the appropriate logs to look at.

yeah i had a feeling the event detail "issue" was not a common one, not going to even worry about it, will re install ASL tomorrow

did a little digging and I THINK it's getting jacked up when it runs it's 4am scan

-removed ASL, booted back into the orig kernel and will see if it gets past the normal plesk 4 am stuff later this afternoon (will run those cronjobs manually)

turns out is was not crashing the server - but most things were not working like http, imap, pop, smtp, etc but the machine kept chugging away

you could still ping it but that's about it...

I have atop on it, it says httpd has been using the most resources now where as before mysqld was by far the biggest hog

- can atop show usage before the last reboot?

I checked most every log (via webmin) and don't see anything REALLY bad

and I also had upgraded everything via yum after I installed your kernel (but again, right now I am back on the original one) but did not roll back any of the packages (though your uninstaller took care of all it's stuff of course)

Am not giving up or anything - only removed it to see what works differently

event detail issue was resolved with a re-install (dropped tortix db before installing asl again)

also corrected some underlying mysql issues - upgraded it through yum to 5.5.19art but didn't bring all the database up to current so there was a LOT of stuff reporting bad with mysqlcheck (which I repaired)

also removed a decent amount of unused databases from the system, just for good measure

just ran /etc/cron.daily scripts and both asl and asl-webapp-inventory ran quickly - so I believe I am good to go!

bottom line is I think having so many things wrong with mysql tables impacted how well asl could do it's job...

sounds good anyway

So far so good!

Everything is running just fine on a fully patched quad core

Last step is to add the dazuko kernel module and the ignore paths for that and we'll be in business

Thanks for your help on getting this all sorted out Mike and Happy New Year!

right after I posted this it locked up again...

back to the drawing board!

in /var/log/messages I see that clamav was the last thing to run...:

Jan 8 10:01:12 www freshclam[13874]: ClamAV update process started at Sun Jan 8 10:01:12 2012
Jan 8 10:01:12 www freshclam[13874]: main.cvd is up to date (version: 54, sigs: 1044387, f-level: 60, builder: sven)
Jan 8 10:01:12 www freshclam[13874]: daily.cld is up to date (version: 14273, sigs: 66226, f-level: 63, builder: guitar)
Jan 8 10:01:12 www freshclam[13874]: Downloading safebrowsing-35078.cdiff [100%]
Jan 8 10:01:13 www freshclam[13874]: Downloading safebrowsing-35079.cdiff [100%]
Jan 8 10:01:26 www freshclam[13874]: safebrowsing.cld updated (version: 35079, sigs: 1093994, f-level: 63, builder: google)
Jan 8 10:01:26 www freshclam[13874]: bytecode.cld is up to date (version: 160, sigs: 38, f-level: 63, builder: edwin)
Jan 8 10:01:28 www freshclam[13874]: Database updated (2204645 signatures) from db.us.clamav.net (IP: 150.214.142.197)
Jan 8 10:01:28 www freshclam[13874]: Clamd successfully notified about the update.
Jan 8 10:01:29 www clamd[18456]: Reading databases from /var/clamav
Jan 8 10:01:44 www clamd[18456]: Database correctly reloaded (2214294 signatures)
Jan 8 10:11:44 www clamd[18456]: SelfCheck: Database status OK.
Jan 8 10:15:02 www psmon[15719]: Forking background daemon, process 15720.
Jan 8 10:15:02 www psmon[15720]: Forking second background daemon, process 15721.

installed sar and it's friends...

will let it run and start there for debugging

In my experience, clamav uses huge amounts of memory and CPU when safebrowsing is enabled. On anything less than a 1Gb system I'd disable it.

I know you have more RAM than that. But maybe you have loads of incoming email? It is only a suggestion, and probably one that's really unlikely to be your problem. Your system specs are miles higher than any of ours and we have no problems like you are having.

Also remember: don't change too many things at once (Mike posted an informative message about this not so long ago).

If you do decide to disable it at some point, note that you need to set "SafeBrowsing No" at the bottom of both /etc/freshclam.conf AND /var/asl/data/templates/template-freshclam.conf

You have to make the change in the second file because ASL copies that and overwrites the one in /etc/ every several minutes. Every time you upgrade to a brand new version of ASL (e.g. 3.1.6 to 3.1.7) you'll need to change the template file again as it will get overwritten.

thanks for the 411 faris!

I am a little surprised that this machine is having so much trouble - in all the years that I have been managing servers I have never once had a machine act so funky!

Unfortunately it's a 1and1 root server so I have to reimage it before they will escalate it to check the hardware...

another day gone and it aint even 8:30am yet!

lol

time to move domains around

have a few domains back on the quad core with plesk updated and everything ASL related running with the exception of the kernel - will load that tomorrow

I don't think anything was kernel related but don't want to reboot the machine yet...

am pretty sure I screwed myself up when I updated all the OS parts via yum - I don't need to be PCI compliant and bringing everything up to the latest greatest definitely caused mare harm than good in my diverse hosting environment (I have everything from wordpress, joomla, vtiger, sugar, moodle, etc) running on multiple domains and it's more important that I keep those systems running than updating each and every package for the OS

We'll see what tomorrow brings but I am confidant that I am good to go - for real this time!

ASL is running fine without the kernel

Loaded it the other day and the machine crashed that night

removed it the following day and have been running on the standard kernel without any issues since

it wasn't the ASL kernel!

it locked up again...

the standard kernel said this about each cpu via a serial console:

BUG: soft lockup - CPU#1 stuck for 10s! [python:3485]
CPU 1:
Modules linked in: ipv6 xfrm_nalgo cpufreq_ondemand powernow_k8 freq_table dm_crypt ecb cbc aes_x86_64 testmgr_cipher testmgr aead crypto_blkcipher crypto_algapi crypto_api aes_generic cifs fuse tun ipt_LOG ipt_ecn ipt_ECN xt_string xt_connlimit xt_mark xt_tcpmss xt_DSCP xt_MARK xt_multiport xt_pkttype xt_physdev bridge xt_mac xt_limit xt_length xt_state xt_conntrack xt_tcpudp ipt_REDIRECT ipt_recent ipt_REJECT ipt_owner iptable_mangle iptable_nat ip_nat ip_conntrack nfnetlink iptable_filter ip_tables x_tables dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport snd_hda_intel snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq amd64_edac_mod snd_seq_device pcspkr edac_mc snd_pcm_oss snd_mixer_oss serio_raw sg tg3 shpchp snd_pcm snd_timer snd_page_alloc snd_hwdep snd soundcore xfs raid456 xor multipath linear sata_sis pata_sis sata_nv dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod mptspi mptscsih scsi_transport_spi mptbase ahci raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd ata_piix libata arcmsr 3w_9xxx 3w_xxxx sd_mod scsi_mod
Pid: 3485, comm: python Not tainted 2.6.18-194.26.1.el5 #1
RIP: 0010:[<ffffffff80064bfc>] [<ffffffff80064bfc>] .text.lock.spinlock+0x2/0x30
RSP: 0018:ffff8100c7e03c80 EFLAGS: 00000282
RAX: ffff810127ef10c0 RBX: ffff8100c7e03d38 RCX: 0000000000000000
RDX: ffff8100c7e03d38 RSI: ffff8101275ffed0 RDI: ffffffff803f1580
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: ffff8100c7a3b4c0 R11: ffffffff8002c456 R12: 0000000080022205
R13: 0000000000000292 R14: ffffffff882c9be9 R15: ffff810125f94800
FS: 00002b080009d190(0000) GS:ffff8101041ca7c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002abc4ab79000 CR3: 00000000c7e9b000 CR4: 00000000000006e0

Not really sure what this all means but it certainly aint good...

So a lot of things could do that, heat being probably the most common. Also a buggy motherboard, firmware, etc. Real tough to debug without the hardware vendor involved.

thanks again - 1and1 said "it seems fine now" so we'll see where this goes...



have used many of their servers over the years and have never had one misbehave like this one is

Looks like the machine is a dell. Maybe you could run Dell's diagnostics on it if you have remote console access?

will google that - thanks again!