By latency, I meant the delay in getting the clients added to the clientlist via swatch.
And the idea of not using qgreylist on SMTPS submission was not enticing either, since we have a bunch of small businesses who use local Linux servers on ADSL lines, and their submission is coming from their Postfix servers with dynamic IP addresses (instead of just enabling SMTPS submission with auth in their email clients). But the following link makes it manageable.
http://wiki.redwall-firewall.com/index.php/Implementing_Upstream_SMTP_Authentication_for_Postfix
I am planning to play with this on the client side (Postfix). For regular (human!) users, they can all use SMTPS with auth.