Sure you can do this, we call this class of attack the "Low and Slow" and its exactly designed to circumvent brute force detectors. There are folks that have done whole phd dissertations about the L&S approach, so there are a many strategies in academia to approach the problem.
OSSEC is going to be able to track an event over as long a period of time as you want, provided the analysis daemon is not restarted. If it does get restarted it would be starting its counters again from scratch.
The current login failure thresholds for FTP are:
* 6 fails over 120 seconds for 1 IP to accounts that exist
* 10 fails over 120 seconds for 1 IP to accounts that do not exist
We are not limited to a single thresholds for these rules. Multiple rules for the same class of event are allowed, for example we can have:
10 fails in 3 seconds == high volume brute force, and call it level 15
100 fails in 24 hours == low volume brute force, and call it level 7
10 fails in 24 hours == suspicious, but maybe not something to block so its a 5
ASL 3 has a "repeat offender" capability, which will also increase the block time each time they come back. The default method we use is for a rule to trigger a block for a fixed number of seconds (600 / 10 minutes) and then it is expired. The new system has the base block time, and a repeat attacker's block time is doubled each time they return (caveat: unless you restart ossec, then this tracking is reset).
You can also associate a specific action with a specific rule, so for example you could set a rule that just grabs login failures and dumps those to a separate file/database/script to allow something else to process it for L&S analysis.
Login failures are also captured in the mysql database (rule id 11203 & 11204 for proftpd), and in addition we have a default report that tracks login failures by IP. Screenshot here:
http://www.atomicrocketturtle.com/asl3-report.pngThe example shown are failures against SSH, but its no problem to create additional reports for FTP, SMTP_AUTH, POP, etc.