DataGrid

WP1 - WMS Software Administrator and User Guide

(PM9 Release)

Document identifier: / DataGrid-01-TEN-0118-0_4
Date: / 15/02/2002
Work package: / WP1
Partner: / Datamat SpA
Document status / DRAFT
Deliverable identifier:
Abstract: This note provides the administrator and user guide for the WP1 WMS software delivered for PM9 release.
IST-2000-25182 / PUBLIC / 91 / 102
/ WP1 - WMS Software Administrator and User Guide
(PM9 Release) / Doc. Identifier:
DataGrid-01-TEN-0118-0_4
Date: 15/02/2002
Delivery Slip
Name / Partner / Date / Signature
From / Fabrizio Pacini / Datamat SpA / 15/02/2002
Verified by / Stefano Beco / Datamat SpA / 15/02/2002
Approved by
Document Log
Issue / Date / Comment / Author
0_0 / 21/12/2001 / First draft / Fabrizio Pacini
0_1 / 14/01/2002 / Draft / Fabrizio Pacini
0_2 / 24/01/2002 / Draft / Fabrizio Pacini
0_3 / 05/02/2002 / Draft / Fabrizio Pacini
0_4 / 15/02/2002 / Draft / Fabrizio Pacini
Document Change Record /
Issue / Item / Reason for Change /
0_1 / General update / -  Take into account changes in the rpm generation procedure.
-  Add missing info about daemons (RB/JSS/CondorG) starting accounts
-  Some general corrections
0_2 / General Update / -  Add Cancelling and Cancel Reason information.
-  Add OUTPUTREADY job state.
-  Add new profile rpms.
-  Remove /etc/workload* shell scripts.
-  Add summary map table (user / daemon).
-  Add CEId format check.
-  Add new job cancel notification.
0_3 / General Update / -  Modified RB/JSS start-up procedure
-  Add gridmap-file users/groups issues
-  Add proxy certificate usage by daemons
-  Job attribute CEId changed to SubmitTo
-  Add DGLOG_TIMEOUT setting
-  Add workload-profile and userinterface-profile rpms
0_4 / General Update / -  Add configure option –enable-wl for system configuration files
-  Add installation checking option –with-globus for Globus to the Workload configure
-  Add new Information Index configure options
-  Remove edg-profile and edg-user-env rpms from II and UI dependencies
-  Add security configuration rpm’s for all the Certificate Authorities to UI dependencies
-  Add new parameters to RB configuration file
-  Add new Job Exit Code field to the returned job status info
-  Remove dependence from SWIG in the userinterface binary rpm
Files
Software Products / User files
Word 97 / DataGrid-01-TEN-0118-0_4-Document
Acrobat Exchange 4.0 / DataGrid-01-TEN-0118-0_4-Document.pdf


Content

1. Introduction 6

1.1. Objectives of this document 6

1.2. Application area 6

1.3. Applicable documents and reference documents 6

1.4. Document evolution procedure 7

1.5. Terminology 7

2. Executive summary 9

3. Build Procedure 10

3.1. Required Software 10

3.2. Build Instructions 11

3.2.1. Environment Variables 11

3.2.2. Compiling the code 13

3.3. RPM Installation 21

4. Installation and Configuration 24

4.1. Logging and Bookkeeping services 24

4.1.1. Required software 24

4.1.2. RPM installation 25

4.1.3. The installation tree structure 26

4.1.4. Configuration 27

4.1.5. Environment Variables 27

4.2. RB and JSS 29

4.2.1. Required software 29

4.2.2. RPM installation 31

4.2.3. The Installation Tree structure 32

4.2.4. Configuration 32

4.2.5. Environment variables 36

4.3. Information Index 39

4.3.1. Required software 39

4.3.2. RPM installation 39

4.3.3. The Installation tree structure 40

4.3.4. Configuration 40

4.3.5. Environment Variables 41

4.4. User Interface 42

4.4.1. Required software 42

4.4.2. RPM installation 43

4.4.3. The tree structure 44

4.4.4. Configuration 44

4.4.5. Environment variables 45

5. Operating the System 47

5.1. LB local-logger 47

5.1.1. Starting and stopping daemons 47

5.1.2. Troubleshooting 48

5.2. LB Server 49

5.2.1. Starting and stopping daemons 49

5.2.2. Purging the LB database 50

5.2.3. Troubleshooting 50

5.3. RB and JSS 51

5.3.1. Startig PostgreSQL 51

5.3.2. Starting and stopping JSS and RB daemons 51

5.3.3. RB troubleshooting 52

5.3.4. JSS troubleshooting 52

5.4. Information Index 52

5.4.1. Starting and stopping daemons 52

6. User Guide 54

6.1. User interface 54

6.1.1. Security 54

6.1.2. Common behaviours 55

6.1.3. Commands description 59

7. Annexes 91

7.1. JDL Attributes 91

7.2. Job Status Diagram 91

7.3. Job Event Types 93

7.4. wildcard patterns 95

7.5. The Match Making Algorithm 97

7.5.1. Direct Job Submission 97

7.5.2. Job submission without data-accesss requirements 97

7.5.3. Job submission with data-access requirements 99

7.6. Process/User Mapping Table 102

1. Introduction

This document provides a guide to the building, installation and usage of the WP1 WMS software released for PM9.

1.1. Objectives of this document

Goal of this document is to describe the complete process by which the WP1 WMS software can be installed and configured on the DataGrid test-bed platforms.

Guidelines for operating the whole system and accessing provided functionalities are also provided.

1.2. Application area

Administrators can use this document as a basis for installing, configuring and operating WP1 WMS software released for PM9. Users can refer to the User Guide chapter for accessing provided services through the User Interface.

1.3. Applicable documents and reference documents

Applicable documents

[A1] / Job Description Language HowTo – DataGrid-01-TEN-0102-02 – 17/12/2001
(http://www.infn.it/workload-grid/docs/DataGrid-01-TEN-0102-0_2.pdf)
[A2] / DATAGRID WP1 Job Submission User Interface for PM9 (revised presentation) – 23/03/2001
(http://www.infn.it/workload-grid/docs/20010320-JS-UI-datamat.pdf)
[A3] / WP1 meeting - CESNET presentation in Milan – 20-21/03/2001
(http://www.infn.it/workload-grid/docs/20010320-L_B-matyska.pdf)
[A4] / Logging and Bookkeeping Service – 0705/2001
(http://www.infn.it/workload-grid/docs/20010508-lb_draft-ruda.pdf)
[A5] / Results of Meeting on Workload Manager Components Interaction – 09/05/2001
(http://www.infn.it/workload-grid/docs/20010508-WM-Interactions-pacini.pdf)
[A6] / Resource Broker Architecture and APIs – 13/06/2001
(http://www.infn.it/workload-grid/docs/20010613-RBArch-2.doc)
[A7] / JDL Attributes - DataGrid-01-NOT-0101-0_6 – 04/02/2002
(http://www.infn.it/workload-grid/docs/DataGrid-01-NOT-0101-0_6.pdf)

Reference documents

[R1]
IST-2000-25182 / PUBLIC / 91 / 102
/ WP1 - WMS Software Administrator and User Guide
(PM9 Release) / Doc. Identifier:
DataGrid-01-TEN-0118-0_4
Date: 15/02/2002

1.4. Document evolution procedure

The content of this document will be subjected to modification according to the following events:

·  Comments received from Datagrid project members,

·  Changes/evolutions/additions to the WMS components.

1.5. Terminology

Definitions

Condor / Condor is a High Throughput Computing (HTC) environment that can manage very large collections of distributively owned workstations
Globus / The Globus Toolkit is a set of software tools and libraries aimed at the building of computational grids and grid-based applications.

Glossary

class-ad / Classified advertisement
CE / Computing Element
DB / Data Base
FQDN / Fully Qualified Domain Name
GDMP / Grid Data Management Pilot Project
GIS / Grid Information Service, aka MDS
GSI / Grid Security Infrastructure
job-ad / Class-ad describing a job
JDL / Job Description Language
JSS / Job Submission Service
LB / Logging and Bookkeeping Service
LRMS / Local Resource Management System
MDS / Metacomputing Directory Service, aka GIS
MPI / Message Passing Interface
PID / Process Identifier
PM / Project Month
RB / Resource Broker
RC / Replica Catalogue
SE / Storage Element
SI00 / Spec Int 2000
SMP / Symmetric Multi Processor
TBC / To Be Confirmed
TBD / To Be Defined
UI / User Interface
UID / User Identifier
WMS / Workload Management System
WP / Work Package

2. Executive summary

This document comprises the following main sections:

Section 3: Build Procedure

Outlines the software required to build the system and the actual process for building it and generating rpms for the WMS components; a step-by-step guide is included.

Section 4: Installation and Configuration

Describes changes that need to be made to the environment and the steps to be performed for installing the WMS software on the test-bed target platforms. The resulting installation tree structure is detailed for each system component.

Section 5: Operating the System

Provides actual procedures for starting/stopping WMS components processes and utilities.

Section 6: User Guide

Describes in a Unix man pages style all User Interface component commands allowing the user to access WMS provided services.

Section 7: Annexes

Deepens arguments introduced in the User Guide section that are considered useful for the user to better understand system behaviour.

3. Build Procedure

In the following section we give detailed instructions for the installation of the WP1 WMS software package. We provide a source code distribution as well as a binary distribution and explain installation procedures for both cases.

3.1. Required Software

The WP1 software runs and has been tested on platforms running Globus Toolkit 2.0 Beta Release 21 on top of Linux RedHat 6.2.

Hereafter are listed the software packages, apart from WP1 software version 1.0, that are required to be installed locally on a given site in order to be able to build the WP1 WMS on it. They are:

-  Globus Toolkit 2.0 Beta 21 or higher (download at http://datagrid.in2p3.fr/distribution/globus/beta-21)

-  Python 2.1.1 (download at http://datagrid.in2p3.fr/distribution/config/external.html)

-  Swig 1.3.7 (download at http://datagrid.in2p3.fr/distribution/config/external.html)

-  Expat 1.95.1 (download at http://datagrid.in2p3.fr/distribution/config/external.html)

-  Expat-devel 1.95.1 (download at http://datagrid.in2p3.fr/distribution/config/external.html)

-  MySQL Version 9.38 Distribution 3.22.32, for pc-linux-gnu (i686) (download at http://datagrid.in2p3.fr/distribution/config/external_services.html)

-  Postgresql 7.1.3 (http://datagrid.in2p3.fr/distribution/config/external_services.html)

-  Classads library (download at http://datagrid.in2p3.fr/distribution/external/RPMS/classads-0.0-edg2.i386.rpm)

-  CondorG 6.3.1 for INTEL-LINUX-GLIBC21 (download at

http://datagrid.in2p3.fr/distribution/external/RPMS/CondorG-6.3.1-edg4.i386.rpm)

-  Perl IO Stty 0.02, Perl IO Tty 0.04 (download at http://datagrid.in2p3.fr/distribution/config/external.html )

-  Perl 5 (download at http://datagrid.in2p3.fr/distribution/config/external.html)

-  gcc version 2.95.2

-  GNU make version 3.78.1 or higher

-  GNU autoconf version 2.13

-  GNU libtool 1.3.5

-  GNU automake 1.4

-  GNU m4 1.4 or higher

-  RPM 3.0.5

-  sendmail 8.11.6

3.2. Build Instructions

The following instructions deal with the building of the WMS software and hence apply to the source code distribution.

3.2.1. Environment Variables

Before starting the compilation, some environment variables related to the WMS components can be set or configured by means of the configure script. This is needed only if package defaults are not suitable. Involved variables are listed below:

-  GLOBUS_LOCATION base directory of the Globus installation

The default path is /opt/globus.

-  MYSQL_INSTALL_PATH base directory of the MySQL installation

The default path is /usr.

-  EXPAT_INSTALL_PATH base directory of the Expat installation.

The default path is /usr.

-  GDMP_INSTALL_PATH base directory of the Gdmp installation

The default path is /opt/edg.

-  PGSQL_INSTALL_PATH base directory of the Pgsql installation.

The default path is /usr.

-  CLASSAD_INSTALL_PATH base directory of the Classad library installation. The

default path is /opt/classads.

-  CONDORG_INSTALL_PATH base directory of the Condor installation.

The default path is /opt/CondorG.

-  PYTHON_INSTALL_PATH base directory of the Python installation.

The default path is /usr.

-  SWIG_INSTALL_PATH base directory of the Swig installation .

The default path is /usr/local.

In order to build the whole WP1 package, all the environment variables in the previous list must be set. Instead for building the User Interface module, the environment variables that need to be set are the following:

-  GLOBUS_LOCATION

-  CLASSAD_INSTALL_PATH

-  PYTHON_INSTALL_PATH

-  SWIG_INSTALL_PATH

-  EXPAT_INSTALL_PATH

If you plan to build the Job Submission and Resource Broker module, variable to set are:

-  GLOBUS_LOCATION

-  MYSQL_INSTALL_PATH

-  EXPAT_INSTALL_PATH

-  GDMP_INSTALL_PATH

-  PGSQL_INSTALL_PATH

-  CLASSAD_INSTALL_PATH

-  CONDORG_INSTALL_PATH

Whilst the LB server and Local Logger modules, to be built need the following environment variables:

-  GLOBUS_LOCATION

-  MYSQL_INSTALL_PATH

-  EXPAT_INSTALL_PATH

Finally, the LB library module needs:

-  GLOBUS_LOCATION

-  EXPAT_INSTALL_PATH

and the Information Index module only:

-  GLOBUS_LOCATION

3.2.2. Compiling the code

After having unpacked the WP1 source distribution tar file, or having downloaded the code directly from the CVS repository, change your working directory to be the WP1 base directory, i.e. the Workload directory, and run the following command:

./recoursive-bootstrap

At this point the configure command can be run. The configure script has to be invoked as follows:

./configure <options>

The list of options that are recognized by configure is reported hereafter:

--help

--prefix=<installation path>

It is used to specify the Workload installation dir. The default

installation dir is /opt/edg.

--enable-all

It is used to enable the build of the whole WP1 package. By default this option is turned on.

--enable-userinterface

It is used to enable the build of the User Interface module with Logging/Client, Broker/Client, Broker/Socket++ and ThirdParty/trio/src submodules. By default this option is turned off.

--enable-jss_rb

It is used to enable the build of the Job Submission and Resource Broker modules with Logging/Client, Common, test, and ThirdParty/trio/src submodules. By default this option is turned off.

--enable-lbserver

It is used to enable the build of the LB Server service with Logging/Client, Logging/etc, Logging/Server, Logging/InterLogger/Net, Logging/InterLogger/SSL, Logging/InterLogger/Error, Logging/InterLogger/Lbserver and ThirdParty/trio/src submodules. By default this option is turned off.

--enable-locallogger

It is used to enable the build of the LB Local Logger service with Logging/Client, Logging/InterLogger/Net, Logging/InterLogger/SSL, Logging/InterLogger/Error, Logging/InterLogger/InterLogger, Logging/LocalLogger, man and ThirdParty/trio/src submodules. By default this option is turned off.

--enable-logging_dev

It is used to enable the build of the LB Client Library with Logging/Client and ThirdParty/trio/src submodules. By default this option is turned off.

--enable-information

It is used to enable the build of the Information Index module.By default this option is turned off.

--enable-wl

It is used to enable the installation of system configuration files that are in the Workload/etc directory. By default this option is turned off.

--with-globus-install=<dir>

It allows specifying the Globus installation directory without setting the environment variable GLOBUS_LOCATION.

--with-pgsql-install=<dir>

It allows specifying the Pgsql installation directory without setting the environment variable PGSQL_INSTALL_PATH.

--with-gdmp-install=<dir>

It allows specifying the GDMP installation directory without setting the environment variable GDMP_INSTALL_PATH.

--with-expat-install=<dir>

It allows specifying the Expat installation directory without setting the environment variable EXPAT_INSTALL_PATH.

--with-mysql-install=<dir>

It allows to specify the MySQL installation directory without setting the environment variable MYSQL_INSTALL_PATH.

--with-globus=<option>

It allows either to enable or to disable the Globus installation checking. The default value is ‘yes’.

--with-expat=<option>

It allows either to enable or to disable the Expat installation checking. The default value is 'yes'.

--with-pgsql=<option>

It allows either to enable or to disable Pgsql installation checking. The default value is 'yes'.

--with-mysql=<option>

It allows either to enable or to disable MySQL installation checking. The default value is 'yes'.

--with-gdmp=<option>

It allows either to enable or to disable Gdmp installation checking. Thedefault value is 'yes'.

During the configure step, 12 spec files (i.e. wl-userinterface.spec, wl-locallogger.spec, wl lbserver.spec, wl-logging_dev.spec, wl-jss_rb.spec, wl-information.spec, wl-userinterface-profile.spec, wl-jss_rb-profile.spec, wl-information-profile.spec, wl-lbserver-profile.spec and wl-locallogger-profile.spec, wl-workload-profile.spec) are created in the following source sub-directories to produce a flavour specific version: