![Relevant disclosure? - Kamaelia came out of my day work @ BBC Research - Kamaelia grey came out of my personal frustration, and has nothing to do with the BBC :-) Michael Sparks , [email_address] Kamaelia Grey](https://image.slidesharecdn.com/kamaelia-grey-1194389767886572-4/85/Kamaelia-Grey-1-320.jpg)
![But also... Debug log: /var/log/greylist-debug.log *debug* THREADS['Kamaelia.Chassis.Pipeline.Pipeline_7', 'Kamaelia.Internet.Selector.Selector_11', '__main__.GreylistServer_8', '__main__.PeriodicWakeup_5', '__main__.TCPS_10', '__main__.WakeableIntrospector_6'] Note – this says the current internal components running](https://image.slidesharecdn.com/kamaelia-grey-1194389767886572-4/85/Kamaelia-Grey-27-320.jpg)
![Internals Debug log: /var/log/greylist-debug.log ['Kamaelia.Chassis.Pipeline.Pipeline_7', 'Kamaelia.Internet.Selector.Selector_11', '__main__.GreylistServer_8', '__main__.PeriodicWakeup_5', '__main__.TCPS_10', '__main__.WakeableIntrospector_6'] Note – this says the current internal components running](https://image.slidesharecdn.com/kamaelia-grey-1194389767886572-4/85/Kamaelia-Grey-28-320.jpg)
![Internals class GreyListingPolicy(ConcreteMailHandler): allowed_senders = [] allowed_sender_nets = [] allowed_domains = [ ]](https://image.slidesharecdn.com/kamaelia-grey-1194389767886572-4/85/Kamaelia-Grey-41-320.jpg)
![Internals class GreylistServer(MoreComplexServer): logfile = config["greylist_log"] debuglogfile = config["greylist_debuglog"] socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) port = config["port"] class TCPS(TCPServer): CSA = NoActivityTimeout(ConnectedSocketAdapter, timeout=config["inactivity_timeout"], debug=False) ....](https://image.slidesharecdn.com/kamaelia-grey-1194389767886572-4/85/Kamaelia-Grey-43-320.jpg)
![Internals class GreylistServer(MoreComplexServer): (continued) # ... class protocol(GreyListingPolicy): servername = config["servername"] serverid = config["serverid"] smtp_ip = config["smtp_ip"] smtp_port = config["smtp_port"] allowed_senders = config["allowed_senders"] allowed_sender_nets = config["allowed_sender_nets"] allowed_domains = config["allowed_domains"] whitelisted_triples = config["whitelisted_triples"] whitelisted_nonstandard_triples = config["whitelisted_nonstandard_triples"]](https://image.slidesharecdn.com/kamaelia-grey-1194389767886572-4/85/Kamaelia-Grey-44-320.jpg)
Kamaelia Grey
- 1. Relevant disclosure? - Kamaelia came out of my day work @ BBC Research - Kamaelia grey came out of my personal frustration, and has nothing to do with the BBC :-) Michael Sparks , [email_address] Kamaelia Grey
- 2. Drastic Spam reduction through Greylisting Greylisting made easy through natural concurrency Natural concurrency made easy through Kamaelia ... since that's what Kamaelia is for Kamaelia Grey
- 3. Problem? Spam was getting ridiculous One weekend the number of emails to home address topped 1000 96% was spam 4% wasn't, but was mainly mailing lists Around 1% interesting Next to impossible to find NB: I run my own mail server :-)
- 4. Kamaelia Day project at the BBC since 2003 Toolkit for making concurrency easy to work with in the context of Networking (servers/clients) Multimedia TV Pretty much anything Designed to make writing servers fast, scalable & reliable
- 5. Greylisting 1 Client connects, wants to send us mail Have we let their email through before? YES! I CAN HAZ ACCEPT ALL UR EMAIL
- 6. Greylisting Client connects, wants to send us mail Have we seen them before? NO! KTHXBYE
- 7. Greylisting Client comes back after KTHXBYE Is it a resend of the same message after a reasonable delay?
- 8. Greylisting Client comes back after KTHXBYE Is it a resend of the same message after a reasonable delay? YES! I CAN HAZ ACCEPT ALL UR EMAIL
- 9. Greylisting Client comes back after KTHXBYE Is it a resend of the same message after a reasonable delay? NO! KTHXBYE
- 10. Where?
- 11. Here!
- 12. Requirements Python 2.3 onwards Must run your own mail server already Requires Kamaelia, but the tar ball includes that. I developed it under Linux I've got it deployed on Mac OS X
- 13. Mail serving UNTRUSTED INTERNET
- 14. Mail serving UNTRUSTED INTERNET MAIL SERVER 25
- 15. Mail serving UNTRUSTED INTERNET MAIL SERVER 25
- 16. Mail serving UNTRUSTED INTERNET MAIL SERVER 25
- 17. Greylisting UNTRUSTED INTERNET MAIL SERVER 8025
- 18. Greylisting UNTRUSTED INTERNET MAIL SERVER 8025 Kam Grey 25
- 19. Greylisting UNTRUSTED INTERNET MAIL SERVER 8025 Kam Grey 25
- 20. Installing Grab the code & install python setup.py install Configure edit /etc/greylist.conf then run /etc/init.d/kamaeliagrey start I actually just use App/runGreylister.sh
- 21. Basic config port = 25 servername = mail.cerenity.org serverid = Kamaelia-SMTP 1.0 smtp_ip = 192.168.2.9 smtp_port = 8025 inactivity_timeout = 60 greylist_log = /var/log/greylist.log greylist_debuglog = /var/log/greylist-debug.log attempters_db = /var/spool/attempters.dbm greylisted_db = /var/spool/greylisted.dbm
- 22. Local config allowed_senders: 127.0.0.1 allowed_sender_nets: 192.168.2 allowed_domains: private.thwackety.com thwackety.com thwackety.net yeoldeclue.com michaelsparks.info lansdowneresidents.org polinasparks.com pixienest.com kamaelia.org owiki.org cerenity.org
- 23. Exceptions # Standard triples which always allows delivery to domains we consider # accepting for. # # Format: # IP address, mail-from id, recipient id # whitelisted_triples: 213.38.186.202 <[email protected]> <[email protected]>
- 24. Exceptions # Some non-standard triples for which we always allow delivery # to domains we consider accepting for. # Format: # claimed sender name, IP prefix, recipient # whitelisted_nonstandard_triples: listmail.artsfb.org.uk 62.73.155.19 <[email protected]> mx-out.facebook.com 204.15.20 <[email protected]> mx-out.facebook.com 204.15.20 <[email protected]> fallbackmx-out.facebook.com 204.15.20 <[email protected]> fallbackmx-out.facebook.com 204.15.20 <[email protected]>
- 25. Logging? Standard log: /var/log/greylist.log 20071106172215.940 | dsl88.241-20782.ttnet.net.tr | 88.241.81.46 | <[email protected]> | <[email protected]> | DEFERRED | 20071106172231.269 | dsl88.241-20782.ttnet.net.tr | 88.241.81.46 | <[email protected]> | <[email protected]> | DEFERRED | 20071106172238.610 | 34-148.privatnet.cz | 88.146.148.34 | <[email protected]> | <[email protected]> | DEFERRED | 20071106172244.333 | dsl88.241-20782.ttnet.net.tr | 88.241.81.46 | <[email protected]> | <[email protected]> | DEFERRED | 20071106172247.885 | mta410.k.cheetahmail.com | 208.49.63.136 | <[email protected]> | <[email protected]> | ACCEPTED |
- 26. Logging? Debug log: /var/log/greylist.log 250 OK 250 ACCEPTED 451 4.7.1 Please try again later 220 mail.cerenity.org ESMTP Kamaelia-SMTP 1.0 Tue Nov 6 17:26:01 2007 500 Command Not Recognised 250 mail.cerenity.org Hello lse 89.252.24.7 250 OK Note – this is what's being sent over SMTP
- 27. But also... Debug log: /var/log/greylist-debug.log *debug* THREADS['Kamaelia.Chassis.Pipeline.Pipeline_7', 'Kamaelia.Internet.Selector.Selector_11', '__main__.GreylistServer_8', '__main__.PeriodicWakeup_5', '__main__.TCPS_10', '__main__.WakeableIntrospector_6'] Note – this says the current internal components running
- 28. Internals Debug log: /var/log/greylist-debug.log ['Kamaelia.Chassis.Pipeline.Pipeline_7', 'Kamaelia.Internet.Selector.Selector_11', '__main__.GreylistServer_8', '__main__.PeriodicWakeup_5', '__main__.TCPS_10', '__main__.WakeableIntrospector_6'] Note – this says the current internal components running
- 29. Internals Debug log: /var/log/greylist-debug.log Kamaelia.Internet.Selector.Selector_11 – wakes system on network events __main__.TCPS_10 – sits inside and handles listening for '__main__.GreylistServer_8 – A configured network server Kamaelia.Chassis.Pipeline.Pipeline_7 – For debugging '__main__.PeriodicWakeup_5 – For debugging '__main__.WakeableIntrospector_6 – For debugging Note – this says the current internal components running
- 30. No connection! TCPServer Greylist Server Selector
- 31. New connection! TCPServer Greylist Server Selector Connected Socket Adapter GreyListing Policy
- 32. More connections! TCPServer Greylist Server Selector Connected Socket Adapter GreyListing Policy Connected Socket Adapter GreyListing Policy Connected Socket Adapter GreyListing Policy Connected Socket Adapter GreyListing Policy
- 33. Accepted! GreyListing Policy TCPClient Connected Socket Adapter MAIL SERVER 8025
- 34. Rejected! Connected Socket Adapter GreyListing Policy
- 35. Greylisting GreyListing Policy Concrete Mail Handler Mail Handler
- 36. Internals The following pages are intended as a walk through of the key highlights of the code-base. It skips the body of the code since that's best looked at by scrolling through code and chatting rather than dumping into slides. In a presentation its easy to say this, and then after going through slides giving road pointers go through the code. Presentation files are a little more limited. The code referred to is here: http://tinyurl.com/2sbjxl
- 37. Internals class MailHandler(Axon.Component.component): logfile = "greylist.log" debuglogfile = "greylist-debug.log"
- 38. Internals class MailHandler(Axon.Component.component): def __init__(self,**argd): def logging_recv_connection(self): def getline(self): def handleCommand(self,command): def noteToLog(self, line): def noteToDebugLog(self, line): def netPrint(self, *args): def lastline(self): def main(self):
- 39. Internals class ConcreteMailHandler(MailHandler): Inboxes = { "inbox" : "Data from the client connecting to the server comes in here", "control" : "Shutdown & control messages regarding client side socket handling", "tcp_inbox" : "This is where we get respones from the real SMTP server", "tcp_control" : "This is where we get shutdown information from the real SMTP server", } Outboxes = { "outbox" : "Data sent here goes back the the client connecting to the server", "signal" : "Shutdown & control messages regarding client side socket handling", "tcp_outbox" : "Data sent here is sent to the real SMTP server", "tcp_signal" : "We send messages here to shutdown the connection to the real SMTP connection", } peer = "*** UNDEFINED ***" peerport = "*** UNDEFINED ***" local = "*** UNDEFINED ***" localport = "*** UNDEFINED ***" servername = "Testing.server.local" serverid = "MPS SMTP 1.0" smtp_ip = "192.168.2.9" smtp_port = 25
- 40. Internals class ConcreteMailHandler(MailHandler): def connectToRealSMTPServer(self): def __init__(self, **argv): def error(self, message): def RelayError(self): def handleConnect(self): def handleEhlo(self,command): def handleHelo(self,command): def handleHelp(self,command): def handleVrfy(self,command): def handleRset(self,command): def handleNoop(self,command): def handleMail(self,command): def handleRcpt(self,command): def handleData(self, command): def handleQuit(self,command): def shouldWeAcceptMail(self): return False # Default policy - don't accept any email def deferMail(self): def acceptMail(self): def getline_fromsmtpserver(self): def handleDisconnect(self):
- 41. Internals class GreyListingPolicy(ConcreteMailHandler): allowed_senders = [] allowed_sender_nets = [] allowed_domains = [ ]
- 42. Internals class GreyListingPolicy(ConcreteMailHandler): (continued) def shouldWeAcceptMail(self): def sentFromAllowedIPAddress(self): def sentFromAllowedNetwork(self): def sentToADomainWeForwardFor(self): def isGreylisted(self, recipient): def whiteListed(self, recipient): def logResult(self):
- 43. Internals class GreylistServer(MoreComplexServer): logfile = config["greylist_log"] debuglogfile = config["greylist_debuglog"] socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) port = config["port"] class TCPS(TCPServer): CSA = NoActivityTimeout(ConnectedSocketAdapter, timeout=config["inactivity_timeout"], debug=False) ....
- 44. Internals class GreylistServer(MoreComplexServer): (continued) # ... class protocol(GreyListingPolicy): servername = config["servername"] serverid = config["serverid"] smtp_ip = config["smtp_ip"] smtp_port = config["smtp_port"] allowed_senders = config["allowed_senders"] allowed_sender_nets = config["allowed_sender_nets"] allowed_domains = config["allowed_domains"] whitelisted_triples = config["whitelisted_triples"] whitelisted_nonstandard_triples = config["whitelisted_nonstandard_triples"]
- 45. And that... Saves me having to wade through now literally tens of thousands of spams over the past two months: # grep DEFERRED /var/log/greylist.log |wc -l 73798
- 46. Time to write? 2 days start to finish including protocol Bug fixes ~1-2 days tops
- 47. Questions? Thank you :-) I hope it's useful to you :)