store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Wed May 27, 2015 4:40 pm

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Uploading often fails in Wordpress
Unread postPosted: Fri Jan 04, 2013 8:28 am 
Offline
Forum Regular
Forum Regular

Joined: Tue Jul 15, 2008 2:38 pm
Posts: 786
Location: Sweden
During the last months about 20% of the uploads within Wordpress fails for my customers. The following message is found in the error log for the site.

Code:
[Fri Jan 04 13:22:40 2013] [error] [client xxx.yyy.zzz.xxx] ModSecurity:  [file "/etc/httpd/modsecurity.d/00_asl_zz_strict.conf"] [line "37"] [id "330792"] [msg "Multipart parser detected a possible unmatched boundary. This may be an impedence mismatch attack, a broken application or a broken connection.  This is not a false positive.  Check your application or client for errors."] [severity "CRITICAL"] Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [hostname "www.hostname.se"] [uri "/wp-admin/async-upload.php"] [unique_id "iOWo538AAAEAADyMvQkAAAAA"]


Often I can fixe it by changing the file a bit (sometimes the size of a picture, sometimes removing a page from a PDF). but sometimes I cannot fix it and the only way I can do it is by disable the rule for that domain.

The only error I get in Wordpress is "http error".


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Fri Jan 04, 2013 5:14 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2184
It isn't just wordpress, but yes, we see this a lot. We discussed it at length with Mike, Scott and whoever else looks at support tickets these days.

In our experience, it is triggered when the uploaded file is bigger than is allowed by some setting or other. Hence, a pdf uploads work if you remove a bit, or jpgs upload if you compress them differently.

The default upload limit in php running as factcgi is 16Kb or something, so if you don't modify that you'll see the error a lot.

Upload and post limits in php.ini, actual mangled files (PDFs in particular are easily mangled) and similar things can also cause problems.

Here's the thing....if you disable this rule, the file upload would not have worked anyway 99% of the time. The user would have seen a different error, but an error nonetheless. And in those cases where it might have worked then the resulting file would be damaged in some way.

There was one instance where a PDF just would not upload (smaller than any limit). I asked for the original file, converted it to a pdf myself, and could upload it fine. I then asked the client to use a different setting in Word to create the pdf and that worked fine too. I noted that their original PDF was many times bigger than the one I created and the one they created when trying again. This is the only time I've had a "mystery" issue.

Disabling the rule is a definite no-no. We have seen instances where this rule triggered as a result of the bad guys trying to do something nasty -- probably to bypass mod_sec. Not often, but I've seen it.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Sun Jan 06, 2013 7:36 am 
Offline
Forum Regular
Forum Regular

Joined: Tue Jul 15, 2008 2:38 pm
Posts: 786
Location: Sweden
Thanks a lot for your detailed response, faris! Much appreciated!

I'm not running php as fastcgi, so that's not the problem. There must be some other error then.


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Sun Jan 06, 2013 2:04 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2184
Check the /var/www/vhosts/domain.tld/stats/logs site logs as well as /var/log/httpd/error_log in case there's something useful in there.

If not, try turning up the log detail a bit (though I struggle to find the right settings for my own purposes. I either get too much or too little).

One thing to emphasise here -- as the log event itself says, the rule is just reporting what apache is saying. It isn't a false positive. BUT I can tell you that I didn't believe it at first. I had to go through a lot of testing and experimenting and reassurances from the ASL team to believe it.

However, we did hit a problem with it -- the IP of the uploader was shunned. This caused problems with the customers affected by it. I think the default has changed now? I'm not sure. We just whitelisted the static IPs of the wordpress (and other app) admins who triggered this a bit too often.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Sun Jan 06, 2013 4:17 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue Jul 15, 2008 2:38 pm
Posts: 786
Location: Sweden
I really, really understand the fact that it's just reporting an error. But at the same time. As soon as I disable the rule, the upload works. 100% of the time. It might be something wrong with the upload, I cannot tell, but the picture or PDF is there, readable and downloadable on the site. The "local" vhost logs I only indicates what I pasted in the first message. the next time it happens, I will check the /var/log/httpd/error_log out as well.


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Sun Jan 06, 2013 5:27 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2184
right. yes. Been there. done that. And now that you say it outloud again (disable rule and it works) I start to doubt myself.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Thu Jan 24, 2013 10:25 am 
Offline
Forum Regular
Forum Regular

Joined: Tue Jul 15, 2008 2:38 pm
Posts: 786
Location: Sweden
Now it happened again. I was unable to upload a new theme for a site, rule XXX fired. then, disabling the rule for the domain, and suddenly it worked.

Code:
Thu Jan 24 15:15:11 2013] [error] [client x.y.z.w] ModSecurity:  [file "/etc/httpd/modsecurity.d/00_asl_zz_strict.conf"] [line "37"] [id "330792"] [msg "Multipart parser detected a possible unmatched boundary. This may be an impedence mismatch attack, a broken application or a broken connection.  This is not a false positive.  Check your application or client for errors."] [severity "CRITICAL"] Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [hostname "mydomain.se"] [uri "/wp-admin/update.php"] [unique_id "b3ODpH8AAAEAABxxdYkAAAAL"]
[Thu Jan 24 15:15:11 2013] [error] [client x.y.z.w] ModSecurity: Audit log: Failed to create subdirectories: /var/asl/data/audit/20130124/20130124-1515 (Permission denied) [hostname "mydomain.se"] [uri "/wp-admin/update.php"] [unique_id "b3ODpH8AAAEAABxxdYkAAAAL"]
[Thu Jan 24 15:15:11 2013] [error] [client x.y.z.w] ModSecurity: Input filter: Failed to rename file from "/tmp/20130124-151456-b3ODpH8AAAEAABxxdYkAAAAL-file-QQt64H" to "/var/asl/data/suspicious/20130124-151456-b3ODpH8AAAEAABxxdYkAAAAL-file-QQt64H". [hostname "mydomain.se"] [uri "/wp-admin/update.php"] [unique_id "b3ODpH8AAAEAABxxdYkAAAAL"]
[Thu Jan 24 15:16:44 2013] [error] [client x.y.z.w] ModSecurity:  [file "/etc/httpd/modsecurity.d/00_asl_zz_strict.conf"] [line "37"] [id "330792"] [msg "Multipart parser detected a possible unmatched boundary. This may be an impedence mismatch attack, a broken application or a broken connection.  This is not a false positive.  Check your application or client for errors."] [severity "CRITICAL"] Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [hostname "mydomain.se"] [uri "/wp-admin/update.php"] [unique_id "dB3gDH8AAAEAABzYZ7cAAAAQ"]
[Thu Jan 24 15:16:44 2013] [error] [client x.y.z.w] ModSecurity: Audit log: Failed to create subdirectories: /var/asl/data/audit/20130124/20130124-1516 (Permission denied) [hostname "mydomain.se"] [uri "/wp-admin/update.php"] [unique_id "dB3gDH8AAAEAABzYZ7cAAAAQ"]
[Thu Jan 24 15:16:44 2013] [error] [client x.y.z.w] ModSecurity: Input filter: Failed to rename file from "/tmp/20130124-151614-dB3gDH8AAAEAABzYZ7cAAAAQ-file-26dxwV" to "/var/asl/data/suspicious/20130124-151614-dB3gDH8AAAEAABzYZ7cAAAAQ-file-26dxwV". [hostname "mydomain.se"] [uri "/wp-admin/update.php"] [unique_id "dB3gDH8AAAEAABzYZ7cAAAAQ"]


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Mon Feb 18, 2013 9:53 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2184
(sorry for the long delay)....

Yeah, we need to revisit this. But what can we do? We need some way to extract some extra information from somewhere (apache? mod_sec?) to figure out what is actually going on. At the moment nothing gets visibly logged by apache when this issue happens -- the only element that sees the problem is mod_sec - and with the rule disabled there are no errors logged anywhere that I can see.

I'm currently leaning towards this rule being triggered by some combination of characters within the post/upload which Apache really doesn't care about and happily and transparently handles but which possibly causes an error that apache doesn't log and which mod_sec somehow sees and reacts to. It could be a bug in the php scripts in question, or maybe not. Either way we need to find out what is going on.

I feel this is a serious issue. The only current solution to the problem is to disable the rule for affected sites, and this is a really bad thing to do. But in most cases there is simply no choice. You can't tell the customer "sorry -- your totally up to date [insert name of script here] installation is buggy - talk to the developers" when it would work on any other host without errors. It doesn't matter to the customer that there really is a problem somewhere and that there is a serious security issue involved - they won't care.

So let us try to get to the bottom of this.

Mike, Scott -- any ideas where to start?

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Mon Feb 18, 2013 12:37 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3732
Location: Chantilly, VA
As you may know, this rule is pretty simple and generic. It just does this:

SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0"

That means, it just triggers if modsecurity cant reassembly multiple part message because it has an unmatched boundary. Multipart messages are web requests/posts from the client that come in multiple parts. Each part has a boundary, sent by the client. The client defines these, and sends them. The server then tries to put them back together.

When, during the parsing phase of a multipart/request-body, ModSecurity encounters what the client has presented as a boundary but is not one, this variable gets set to other than 1. In other words, the client has sent the server a multipart message. These parts have boundary markers. The client didnt send a valid multipart message because one of the boundary markers either didnt exist (its missing), or its not part of one of the multiple parts (doesnt match). Its a garbled message and the WAF is hopelessly unable to reconstruct it.

Such an event may occur when evasion of ModSecurity is attempted, which is why you want to block these. It can also happen if the client is sending something garbled, broken, or incomplete. Or if the web application just doesnt care about using properly formated multupart messages. If the WAF cant put it back together, it cant figure out what it is (attack or not an attack). . This is a serious condition, and can mean only one of two things:

1) An attacker is attempting to bypass the WAF by constructing a body that will not be parsed correctly by the WAF, in hopes of bypassing the WAF.

2) The client side application (browser, application, etc.) is generating broken message bodies that the WAF is unable to assemble. This can also occur with a broken server side application thats causing the client to send broken multi part messages, or a combination of both buggy clients and buggy app(s) on the server side. It can even happen if the client has a broken connection and didnt complete the multipart message.

Most of the time buggy server side apps do this. Its pretty rare for most web applications, but word press seems to attract extensions that dont seem to care about doing things sanely.

Theres not much you can do on your end, other than to look at the debug data for the recorded event, and look into the multipart message to see why its corrupt. If its happening with one web application, you have your culprit - the application is just buggy. You could disable the rule, but that would open the server to attack. A "middle" ground would be to set the rule to not shun. That way the client just gets an error, instead of being blocked. Since its the web apps and clients that cause this, you're best solution is to report this to the web application developers and share the debug data recorded by modsecurity with them. This isnt caused by modsecurity, its just telling you the client is sending an invalid multipart message.

I wish I could tell you more than that, but its being caused by garbled multipart messages. And in general, its the server side app that usually causes this (the client can also do this if its generating broken multipart messages, but is rare). If its just wordpress doing this, then you have your root cause, something in either wordpress itself, or one of its extensions is generating invalid multipart messages. Have you discussed this with the developer(s) of those extensions?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Mon Feb 18, 2013 4:12 pm 
Offline
Forum Regular
Forum Regular

Joined: Tue Jul 15, 2008 2:38 pm
Posts: 786
Location: Sweden
Thanks Mike for explaining this in detail. I nearly understood everything. :) At least I think so. For me, no extensions is involved, it's just basic uploading of files in Wordpress. Doesn't matter which of the upload processes you use, they still fail.

Removing the shun doesn't help the situation for my customers. The result is, either way, that they cannot upload a perfectly legit file.

I would gladly post more to the Wordpress developers, but I really don't know which data to collect and post.


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Mon Feb 18, 2013 4:37 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3732
Location: Chantilly, VA
Quote:
I would gladly post more to the Wordpress developers, but I really don't know which data to collect and post.


You want to share the audit event payload with them. Heres an example of a simple event (not the same rule, but the event will look similar to this in audit_log):

[modsecurity] [client 10.23.10.6] [domain crm.progllc.com] [403] [/20130118/20130118-1354/20130118-135405-oreejQoKDgYAAGgZYYwAAAAM] [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "782"] [id "390715"] [rev "15"] [msg "Atomicorp.com WAF Rules: PHP Injection Attack"] [data "fopen("] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "\\b(f(?:tp_(?:nb_)?f?(?:ge|pu)t|get(?:s?s|c)|scanf|write|open|read)|(?:g|b)z(?:(?:encod|writ)e|compress|open|read)|scandir|read(?:(?:(?:g|b)z)?file|dir)|gzinflate|move_uploaded_file|str_rot13|(?:proc_|bz)open|call_user_func|$_(?:(?:pos|ge)t|session))\\ ..." at XML.

You can get the audit event payload via this process:

asl --show-alert <event id>

In this case the event id is "/20130118/20130118-1354/20130118-135405-oreejQoKDgYAAGgZYYwAAAAM"

asl --show-alert /20130118/20130118-1354/20130118-135405-oreejQoKDgYAAGgZYYwAAAAM

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Wed Feb 20, 2013 10:43 am 
Offline
Forum Regular
Forum Regular

Joined: Tue Jul 15, 2008 2:38 pm
Posts: 786
Location: Sweden
Only problem is that I'm running mod_ruid2 so I no events is created in the audit directory, only in the vhost error log...


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Wed Feb 20, 2013 2:15 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3732
Location: Chantilly, VA
Quote:
Only problem is that I'm running mod_ruid2 so I no events is created in the audit directory, only in the vhost error log...


Thats unfortunate, in that case, I dont see how you'll be able to get the information you need. You'll need the full payload.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Thu Feb 21, 2013 5:01 am 
Offline
Forum Regular
Forum Regular

Joined: Tue Jul 15, 2008 2:38 pm
Posts: 786
Location: Sweden
faris, are you able to step up?


Top
 Profile  
 
 Post subject: Re: Uploading often fails in Wordpress
Unread postPosted: Thu Feb 21, 2013 9:43 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2184
kinda....

The only events of this nature that I currently have easy access to are for a non WP site and that's not going to help.

We may have to wait a bit. I promised a customer that I'd install WP for them in the next couple of weeks. Once I do so I'll be able to try uploading a particular PDF, if I can find it, that I know triggered the issue on a different site. And since it will be a naked, unmodified, no-plugin WP install with the latest code, it should be "easy" to determine the issue.

** Note that the issue happens with both the fancy ajax uploader and the basic file uploader in WP. I'm not sure if they share any common code or subroutines.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: Bing [Bot] and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group