ESSNET on SDMX II
WP4
Analysis and Design of Loader
Version 1
10-10-2011
Type of Document / Project deliverableReference:
Issue: / Revision: / Status: / [Draft;Final]
Created by: / Date:
Updated by:
Approved by:
Document Change Record
Issue/ Revision / Date / ChangeTable of contents Page
1 Introduction 5
1.1 Scope 5
1.2 Purpose 5
1.3 Structure 5
1.4 Reference Documents and Standards 6
2 Use case view 7
2.1 Requirements overview 7
2.1.1 Functional requirements 7
2.1.2 Non-functional requirements 8
2.1.3 Scenarios 9
2.1.3.1 Scenario 1 – Definition of settings 9
2.1.3.2 Scenario 2 – Change of settings (FR2, NFR1) 9
2.1.3.3 Scenario 3 – Managing of Users: Creation of a new user 10
2.1.3.4 Scenario 3 – Managing of Users: modification of an existing user 10
2.1.3.5 Scenario 4 – Managing of Users: deletion of an existing user 10
2.1.3.6 Scenario 5 – Creation of Metadata table 11
2.1.3.7 Scenario 6 – Creation of Data table – option two tables 11
2.1.3.8 Scenario 8 – Mapping of csv, gesmes data file 12
2.1.3.9 Scenario 9 – Mapping of flr data file 13
2.1.3.10 Scenario 10 – Loading of data file – option two tables 13
2.1.3.11 Scenario 11 – Loading of data file – option one table 14
2.1.3.12 Scenario 12 – Creation of data File 14
2.2 Actor specification 14
2.3 Use cases model 16
2.3.1 UseCase-UC_MS: 17
2.3.2 UseCase-UC_UM: 18
2.3.3 UseCase-UC_MT: 20
2.3.4 UseCase-UC_DT: 20
2.3.5 UseCase-UC_MDF: 22
2.3.6 UseCase-UC_LDF: 23
2.3.7 UseCase-UC_CDF: 24
2.4 Activity Diagrams 25
2.4.1 Activity Diagrams 25
2.5 Context Diagram 32
3 Package diagram 35
4 Class diagram 36
4.1 [Package Name] 36
4.1.1 Settings 37
4.1.2 ConnectionType 38
4.1.3 PathsType 38
4.1.4 NamespacesType 38
4.1.5 User 39
4.1.6 KeyFamilies 40
4.1.7 dataflows 40
4.1.8 MappFile 41
4.1.9 conceptType 41
4.1.10 DataFile 41
4.1.11 rss 42
5 Dynamic view 44
5.1 Sequence diagrams 44
6 Schema 46
6.1 Predefined schema 46
6.2 Configurable schema 48
6.3 MSDB schema 48
7 Grafical user interface 50
7.1 Setting form 50
7.1.1 Loader db Connection 50
7.1.2 Mapping Store db Connection 52
7.1.3 Directories 53
7.1.4 Namespace 54
7.1.5 General Information 56
8 Appendix 66
8.1 Mapping of csv data file 66
8.2 Mapping of flr data file 67
List of figures Page
Figure 21: Creation of schema and Managing of Users 16
Figure 22: Loading of Data – Creation of Data files 17
Figure 23: Settings 25
Figure 31: [package diagram name] 35
Figure 41: [package diagram name] 36
Figure 51: Creation of Data table - option two tables 44
Figure 51: Creation of Data table - option one table 45
List of tables Page
Table 11: Terms and Abbreviations 6
Table 21: Functional Requirement 9
Table 22: : Non - Functional Requirement 9
ii
1 Introduction
1.1 Scope
This document stands for deliverable “Analysis and Design of the Loader”, as a result of “Task 2/Subtask 2.1 of the WP4 in ESSnet on SDMX phase II “.
1.2 Purpose
The purpose of the document is to describe the functionalities of a standalone application “Loader” used to manage the loading and creation of SDMX data file (ver.2.0) in a database created from scratch. The application is integrated in the SDMX-RI environment.
NOTE: It is recommended that readers of this document should have SDMX technical knowledge.
1.3 Structure
The structure of this document is as follows:
Section 1 includes a short presentation of application’s architecture.
Section 2 describes the Use Case view. Functional and non-functional requirements are presented.
Section 3 describes the standalone packages diagram.
Section 4 shows the class diagrams.
Section 5 shows the sequence diagrams.
Section 6 presents the dbSchema created and used by the application.
Section 7 presents the GUI.
1.4 Reference Documents and Standards
Insert all references to documents or convention or standard used in the document. This part can be divided in:
Standards and Conventions
Terms and abbreviations
Acronym / Definition /LDB / Loader database. Database Created by the Loader application
MSDB / Mapping Store database. Database created by the Mapping Assistant application
AU / Administrator user
MU / Main User
Table 11: Terms and Abbreviations
Definitions:
2 Use case view
2.1 Requirements overview
2.1.1 Functional requirements
Insert all the Functional requirement of the application
FR1 / The “Loader” application must allow the creation of the database schema. The databases supported are: Oracle, SQLServer, MySQL.Part of the schema is predefined and includes the following type of tables:
· Tables for management of the users of the application
· Tables for the management of the header that must to be included in the SDMX data files
· Tables for rss application
FR2 / Part of the schema is configured by the user using the “Loader” application. These tables will contain the data.
FR3 / The Loader application must be access to the MSDB in order to retrieve the structural metadata loaded. The tables that will be used of MSDB are shown in paragraph 6.3 .
FR4 / The Loader application must have a functionality for the managing of different type of users. There are three types of users:
• “Administrator” user with all privileges
• “Main_User” user, that can manage one or more DSDs assigned to him (in this context “manage” means: to create data tables and load data corresponding to the DSDs assigned)
• “Simple_User” user that can manage one or more dataflows assigned to him (in this context “manage” means: to load data corresponding to the dataflows assigned)
The Administrator user only can create a new user.
It is possible to have more than one Administrator user.
FR5 / The “Loader” application must allow to create an association (named “mapping”) between the structural metadata contained in the data file and the concepts of DSD that is used to describe data. This mapping is necessary to load data into the LDB. The user that can make the mapping can be only a Administrator user or a main user that has the DSD assigned to him.
FR6 / The "Loader" application must allow to load data inside the database.
The input data file can have four different formats:
• csv (character separated value),
• flr (fixed length record),
• gesmes TS,
• SDMX data message vers 2.0.
The data to load are described by a DSD previously inserted in the Mapping Store database by the Mapping Assistant.
FR7 / The “Loader” application must allow to create a SDMX datafile in the formats:
• generic,
• cross sectional
• compact
and the corresponding rss file
The creation of the SDMX data files is necessary if the NSI does not want to use the Web Service to disseminate data. This is the case for example of EGR project where the SDMX data file are created and sent using another application. The rss files are created for the static file and they contain the URL where to retrieve SDMX datafiles
FR8 / The Loader application must have the functionality to check data file before loading inside the database. When a datafile is loaded inside the database the application must made a comparison between the structural metadata inside the datafile and the DSD that is associated to the datafile (see FR4). In particular for the concepts coded the check should be done between codes in datafile and corresponding codes in the DSD.
Table 21: Functional Requirement
2.1.2 Non-functional requirements
NFR1 / The Loader application must retrieve parameters for correct functioning in configuration file (Local parameters) and in database (global parameter).The connection string to the LDB and MSDB must be stored in the configuration file.
NFR2 / The Loader application can function only if the MSDB is installed and the Mapping Assistant application is used to load structural metadata
Table 22: : Non - Functional Requirement
2.1.3 Scenarios
2.1.3.1 Scenario 1 – Definition of settings
Introduction: In this scenario is described how to insert settings parameters necessary for a correct functioning of the application. All the settings will be stored inside the LDB except:
• the connection string to the LDB and to the MSDB
• The paths[1] used by the application to read or to store the files,
these information will be stored in the configuration file.
Requirements used: FR1, NFR1
Description:
Step1: The user opens the Loader application and selects the menu voice “Open → Settings”.
Step2: The Settings form will be shown empty except for the two combo box of the database provider that are filled with the three kind of database provider (SQLServer, Oracle and MySql).
Step3: The user fills insert fields in a Setting form and save them.
Step4: The system checks the correctness of information and stores into the database and in the configuration file
2.1.3.2 Scenario 2 – Change of settings (FR2, NFR1)
Introduction: This scenario describes how to modify settings parameters.
Requirements used: FR1, NFR1
Description:
Step1:The user opens the Loader application and selects the menu voice “Open → Settings”.
Step2: The "Settings" form will be shown with information stored in the configuration file and inside of the LDB (the access to the LDB will be consequent to the reading of the LDB connection string inside the configuration file)
Step3:The user modifies fields in the "Setting" form and saves them.
Step4:The system checks the correctness of information and stores into the database and in the configuration file
2.1.3.3 Scenario 3 – Managing of Users: Creation of a new user
Introduction: This scenario describes the creation of a new user. The Administrator user can create three different type of users: Administrator_User, Main_User and Simple_User.
Requirements used: FR1,FR3,FR4,NFR1
Description:
Step1:The AU selects the menu voice “Users Manager” and clicks on the “Add” button.
Step2:The "AU" inserts all the information concerning the new user (Name, Department, email, telephone number) and assigns the new user to a group (see picture 7-18).
The groups are:
• Administrator
• Main_User
• Simple_User
Step3: The AU selects the DSDs and/or dataflows that the user can manage. If the new user is a administrator user then all the DSDs and corresponding dataflows will be automatically assigned to the new user.
Step4: The AU saves the information.
Step5: The system checks the correctness of information and stores them into the LDB.
2.1.3.4 Scenario 3 – Managing of Users: modification of an existing user
Introduction: This scenario describes the modification of a existing user.
Requirements used: FR1, FR3,FR4, NFR1
Description :
Step1:The AU clicks on the “User Manger” menu voice and selects a user.
Step2:The system shows all the information associated to the user selected.
Step3: The AU modifies one or more fields of the form (the password cannot be modified) and saves the changes
Step4:The system checks correctness of changes and save all the information.
2.1.3.5 Scenario 4 – Managing of Users: deletion of an existing user
Introduction: This scenario describes the deletion of a existing user.
Requirements used: FR2, FR3,NFR1
Description:
Step1:The AU click on the “User Manger” and selects a user. It Click on the “Remove” button.
Step2:The system removes the user and all the information linked to the user from the LDB.
2.1.3.6 Scenario 5 – Creation of Metadata table
Introduction: This scenario describes how to create the metadata table. The metadata table will contain only the structural metadata that are not attached at observation level. This choice can occur when the data to be stored have a high periodicity (like short terms statistics) so to make necessary the creation of two tables (Metadata and Data) in order to make more normalized the database. This operation can be done only by a AU and MU.
Requirements used: FR1, FR2,FR3, NFR1
Description:
Step1:The user selects the menu voice “Create Data Tables”.
Step2:The system shows a form where the user selects the “Metadata Table” creation option.
Step3:The system shows to the user the DSDs that are associated to him.The user selects one DSD.
Step4:The system checks if already exist a metadata table with the same name and shows all the concepts belonging to the DSD selected that are not attached at observation level. The concepts selected will be the columns of Metadata table. The user could select only conditional concepts while the mandatory concepts (dimensions and mandatory attribute) will be automatically included in the selection.
Step5:The primary key defined for the Data table will be composed by all the mandatory concepts and by all the other concepts indicated by the user.
Step6:The user defines the order of the columns in the table and creates the table.
Step7:The system creates the Metadata table.
2.1.3.7 Scenario 6 – Creation of Data table – option two tables
Introduction: This scenario describes how to create the data table. In this case the user choose to create two tables: Metadata table and Data table, where data will be stored. This choice can occur when the data to be stored have a high periodicity (like short terms statistics) so to make necessary the creation of two tables (Metadata and Data) in order to make more normalized the database. In this case the Data Table will contain information associated to the structural metadata attached at observation level as well as time dimension. This operation can be done only by a AU and MU.
Requirements used: FR1, FR2, FR3,NFR1
Description:
Step1:The user selects the “Create Data Tables” menu voice.
Step2:The system shows a form where, the user selects the creation of “Data Table” with the option of “Creation of two tables”.
Step3: The system shows to the user the DSDs that are associated to him. The user selects one DSD.
Step4:The system check if a Metadatata table exists and shows all the data flows associated to the DSD selected
Step5:The User selects the dataflow and selects the creation of two tables.
Step6: The system checks if already exist a data table with the same name and shows all the concepts belonging to the DSD selected that are attached at observation level. The concepts selected will be the columns of data table. The user could select only conditional concepts while the mandatory concepts (dimensions and mandatory attribute) will be automatically included in the selection.