Department Subject
Category Title
IT & System dev
Niceasy
Mailprocessor
System Documentation
Document control
Document information and security
Made by / Responsible for fact / Responsible for documentJohan Jonker / Johan Jonker / Johan Jonker
Security class / File name
Security / DomainManager.odt
Approved by
Date / Name / RoleRevisions
Date / Version / Name / DescriptionList of contents
Document number: YYYY-N Page 16 of 16
Revision: Revision Security class
Security
Department Subject
Category Title
1 Introduction 4
1.1 This document 4
1.2 Updates of this document 4
1.3 Abbreviations & Definition of words 4
1.4 References 4
1.5 Font 4
1.6 About .SE 4
2 Overview 6
2.1 General 6
2.2 Design & decisions 6
3 Configuration 7
3.1 mailprocessor.conf 7
3.2 SEServ::DB::Config 8
3.3 GnuPG 8
4 Classes 9
4.1 SEServ::Form 9
4.1.1 The base class 9
4.1.2 Sub-classing 9
4.1.3 Transactions 9
4.1.4 Validation 10
4.2 PGP::Mail 10
4.3 SEServ::Form::Fault 10
4.4 SEServ::Form::IP 10
5 The programs 11
5.1 seserv-localdelivery.pl 11
5.2 seserv-processordaemon.pl 11
5.3 seserv-mailprocessor.pl 11
5.4 seserv-testmail.pl 12
6 Packages 13
6.1 Ubuntu Packages 13
7 Future improvements 14
7.1 Bugs 14
7.2 General improvements 14
7.3 IPv6 14
Document number: YYYY-N Page 16 of 16
Revision: Revision Security class
Security
Department Subject
Category Title
1 Introduction
1.1 This document
This document is written as a part of an introduction to a programmer. After having read this document he/she should have a good understanding of what the NICEasy DomainManager is about and a number specific questions to be answered by those that work with the system.
1.2 Updates of this document
We have a regular release cycle (see doc…..) where new or changed functionality is implemented. This document WILL NOT be updated every time these changes are done. The document will be updated when it is considered as suitable to do it.
However ever change within the scope of this document will be listed and shortly described in the attachment #1 “Updates in program –not updated in this doc”
1.3 Abbreviations & Definition of words
Abbrev Abbreviation, this is an abbreviation and now we will see what will happen if we write stuff
1.4 References
Final location of these documents not decided.
[1] Catlyst homepage http://catalyst.perl.org
[2] Template Toolkit homepage http://template-toolkit.org
[3] Sync
[4] Catalyst::Plugin::Captcha
[5] Hantering av PGP-nycklar?
[6] Vanliga fel i PGP?
1.5 Font
In this document we use the following fonts:
Small bold style Used for library structure, file names and in- and out puts.
BLOCK LETTERS Computer names are always written with block letters.
1.6 About .SE
.SE (The Internet Infrastructure Foundation) is responsible for the Internet top-level domain for Sweden. As the central registry, .SE manages domain name registrations and the administrative and technical operation of the national domain name system for .SE.
.SE is an independent non-profit organisation, supporting the positive development of the Internet in Sweden. Through .SE's Internet Fund, the Foundation annually donates means to projects supporting the development and utilisation of the Internet.
For more information, please see: www.iis.se/english/index.shtml?lang=en
2 Overview
2.1 General
The DomainManager (DM) is the end-user interface of the NICEasy project. It is built using the Catalyst web application framework.
2.2 Design & decisions
This section contains an overview of the technical aspects of the DomainManager. For a description of the interface design, see ?
The DomainManager is developed using the MVC design pattern. MVC stands for Model, View and Controller. The idea of the MVC pattern is that the three main areas of an application, flow (controller), information processing (model) and presentation (view) are separated. This pattern enable the developer to change or replace one area without affecting the others. It also provides an easy entry point for the developer, you generally know where to look for a particular aspect. For a more in-depth description of the MVC pattern and it's origins, see ?
In general, each logical partition of the interface has a corresponding Controller class. For example, the DomainManager::C::Billing controller handles the application's billing functions. In addition to this there are some global methods inherited by all controller classes from the DomainManager::C::Base controller.
When accessing data, the DomainManager uses it's own models which in turn inherits from their corresponding classes in the logic layer. E.g. DomainManager::M::Account inherits from SEServ::DB::Account.
When manipulating data the DomainManager uses the sync facility described in doc ?. Exceptions are when manipulating accounts and account messages. In these cases the DomainManager talks directly to the logic layer using it's models.
The DomainManager uses Template Toolkit (TT) to render it's pages. Template Toolkit is a template processing system. Key features of Template Toolkit includes:
· Support for complex data types including hashes, lists, objects and subroutine references.
· Support for all standard templating directives like includes conditionals and variable substitution.
· Dynamic compilation of templates to Perl code.
3 Configuration
The main configuration file of the DomainManager is the domainmanager.yml YAML file. The following configuration options are available:
3.1 domainmanager.yml
hostname – Since we're using the pound load balancer the real hostname doesn't match that of the site. This variable tells the DomainManager which hostname to announce to the user.
root – Where template toolkit will look for it's templates.
session – Hash containing information regarding the session handling
expires – The session time-to-live in seconds.
rewrite –
dbi_dbh – The CDBI class to use for storing session data.
dbi_table – Table to use for storing session data.
cookie_expires – Number of seconds before the cookie expires. A value of 0 creates a session cookie that will die once the client browser is shut down.
flash_to_stash – Used to make catalyst copy the contents of the flash to the stash automatically.
header_adr – The header parameter to look at when determining the clients address. The current parameter, X-Forwarded- For, is set the pound load balancer.
seservsession – Information to use when creating the seserv session.
whoishost – What host to ask for whois information.
whoisport – Port to use when connecting to whoishost.
dnscheckhost – Host to fetch dnscheck information from.
dnscheck_timeout – Time in seconds to wait for an answer from the dnscheckhost.
encoding -
captcha – Options to use when creating the captcha under tools. See Catalyst::Plugin::Captcha and GD::SecurityImage for more information on the specific options.
fakeorg, fakemail and fakephone – Describes dummy information set during the migration.
pagesizes – Number of results per page available to choose by the user when performing searches.
domain_states, roles and invoiceitemstates are all temporary localization structures. Will be replaced in the future.
3.2 SEServ::DB::Config
Only one option is used by the DomainManager, shutdown.domainmanager. It is used to temporarily disable the site by causing it to render it's maintenance page.
4 Classes
4.1 Controllers
4.1.1 DomainManager::C::Root
DomainManager::C::Root
SEServ::Form manages all form handling from parsing to creating a transaction. It has validation methods for validating all parts (“objects” or “blocks”) from the form produced by the parser to creating the transaction. Errors produced from the parser and validator are stored in a object array variable containing SEServ::Form::Fault objects. Errors can be retrieved using the errors() method, and to indicate if there are errors the haveErrors() method returns a true value.
4.1.2 Sub-classing
To actually use SEServ::Form for a real electronic form you need to create a subclass to SEServ::Form and fill in the class variables $mandatory, $formid, $restricted and the $contexthash. See SEServ::Form::Reg for a proper example.
To initiate a new form object with any of the sub-classes, use the constructor new with the plain-text form as the argument. The object returned can be used to parse the form using the parseForm method.
Any methods in the base class may be overloaded with new or other functionality, for example adding other validation rules. See Validation in 4.1.4.
4.1.3 Transactions
Once the form is parsed, use the validateFormObjects() method to validate the objects the form handles, and createTransArray() to prepare a transactional array. Once having the transactional array, call the object method createTransaction() to create a transaction for the system. A transaction is a SEServ::DB::SyncTransaction containing SEServ::DB::SyncTransaction::SyncCommand objects (read more in the Sync documentation and POD). These transactions are processed by the Sync sub-system, which means that the final message to the reseller / end-user needs to be delivered by the transaction handler. To do this, we need to tell the transaction handler to send mail, and that is the reason why createTransaction() needs several arguments like the subject and the mail-address to use. The mail templates used for this purpose are ‘mailproc_trans_ok_en’ and ‘mailproc_trans_fail_en’.
To gain a complete understanding of how SEServ::Form is used, refer to the source code of ‘seserv-mailprocessor.pl’. There are some basic things being done there such as to authenticate the reseller and to make sure that the reseller exists in our system.
4.1.4 Validation
The objects that goes through the object validation are fake objects made up (blessed) by validateFormObjects() and the logic there, instead of creating the objects using the real class to avoid creating the objects in the database.
If any sub-class wants to have a different validation for any block, for example the MAIN block, it is possible to overload the _validateMain() method to add or remove validation functionality for any fields. This is done in the SEServ::Form::Reg class to add a number of fields not used anywhere else.
4.2 PGP::Mail
SEServ::Form does not use PGP at all, the PGP logic is all used by the mail handler programs.
There are three patches added to PGP::Mail as described in 2.2, design & decisions. Without these patches, e-mail and MIME messages can’t handle different character-sets and the error-handling involved, or extracting fingerprints.
4.3 SEServ::Form::Fault
The name of this class should have been SEServ::Form::Error, but since the Error class have problems with namespaces this name will do. Each Fault-object is an error produced by SEServ::Form. The error can be formatted as text using the asText() method.
4.4 SEServ::Form::IP
SEServ::Form::IP is a wrapper around Net::IP needed due to namespace collisions with the Error class.
5 The programs
The introductory text in 2.1 gave a general overview of the process and programs involved. This chapter describes the programs in more detail.
5.1 seserv-localdelivery.pl
This program is run from the Mailprocessor-users .forward-file by pipeing the mail to the program.
‘seserv-localdelivery.pl’ reads the YAML-configuration file mailprocessor.conf to figure out which directory to deliver the mail to. All Mailprocessor programs use the File::NFSLock locking mechanism to ensure proper file locking. All that this program is doing is to write the incoming e-mail on STDIN to the incoming directory specified in the configuration.
5.2 seserv-processordaemon.pl
‘seserv-processordaemon.pl’ is the daemon that runs the ‘seserv-mailprocessor.pl’. It runs the Mailprocessor and the sleeps for ‘sleep_interval’ seconds, forever. It is started and stopped using the init.d-script from the package:
/etc/init.d/seserv-mailprocessor start|stop
5.3 seserv-mailprocessor.pl
This is the main Mailprocessor program. If the daemon is not running the operator can run this program manually to process mail from the incoming directory. There are some command line arguments one can use when starting ‘seserv-mailprocessor.pl’:
· -v verbose mode
· -t test mode, not implemented
· -m max e-mails to process (given a numeric value)
· --version prints the version number
· -h prints the help
When running in verbose mode there are some debugging information printed while processing an e-mail.
One of the first things the Mailprocessor does, codewise, is to load the electronic forms modules the Mailprocessor is supposed to handle (the SEServ::Form sub-classes). If they don’t get included (using the use pragma) the Mailprocessor won’t know about the form. This is done using the Class::Inspector class to inspect what modules are loaded. The the program inspects the SEServ::Form namespace and retrieves the $formid class variable from the loaded classes and builds a hash from it. The hash is used to lookup known forms later on when checking the content of the mail for the electronic form.
‘seserv-mailprocessor.pl’ does also include a lot of its own methods for handling mail. Among the methods are functionality to decode quoted-printable and base64 encoded e-mail before handing over the mail content to PGP::Mail.
The two main methods are main() and processMail().
main() opens the incoming directory and reads the first e-mail (using receiveMail(), run it through pgpMail() and then calls receiveMail(). While reading an e-mail that file is locked using File::NFSLock, same as the deliver-program.
5.4 seserv-testmail.pl
This program is not finished. The purpose of this program is to allow the testing of forms in e-mail for resellers. The result would be similar to the normal Mailprocessor, but as a simulation of what will be done with the validation of the forms and a fake create transaction.
6 Packages
6.1 Ubuntu Packages
· seserv-mailprocessor - contains the Mailprocessor programs
· libseserv-perl - the database abstraction layer
· libpgp-mail-perl - PGP::Mail
The package seserv-mailprocessor is setting up a new user ‘servmail’ belonging to the group ‘seserv’ setup by the libseserv-perl package documented elsewhere. It also sets up the following directories for the default configuration:
/home/servmail
/var/lib/seserv/mailprocessor/failes
/var/lib/seserv/mailprocessor/incoming
/var/lib/seserv/mailprocessor/success
The default ‘sleep_interval’ for ‘seserv-processordaemon.pl’ is 20 seconds.
The package also adds the e-mail recipient “forms” for the deliver-script in /etc/aliases if the deliver-script is not already in the aliases file.
7 Future improvements
7.1 Bugs
Currently there seems to be a bug (or several?) that causes the Mailprocessor to hang indefinitely. Some of the mails that hangs the Mailprocessor contains HTML or HTML attachments, but it has happened with a perfectly normal e-mail a couple of times (where just a restart has resolved the issue). Normally the removal of the offending e-mail is the only option. There are a number of failed e-mails moved to /home/pawal on mail01.prod.iis.se that could be used for debugging.