Hoad, Robinson and Davies

Classification of discrete event simulation models and output:

creating a sufficient model set.

Kathryn Hoad
WarwickBusinessSchool
The University of Warwick
Coventry, CV4 7AL, UK. / Stewart Robinson
WarwickBusinessSchool
The University of Warwick
Coventry, CV4 7AL, UK. / Ruth Davies
WarwickBusinessSchool
The University of Warwick
Coventry, CV4 7AL, UK.

ABSTRACT

This paper describes the creation of a representative and sufficient set of models and output data that can be used in discrete event simulation research. The motivation is to provide researchers with a representative set of models or output upon which to test their research ideas. The identification of certain DES model and output characteristics is described, as is the creation of a classification system for each general type of model and output encountered in ‘real life’ DES modelling. The processes and decisions involved in setting up this classification system are explained. The classification tables are outlined, including a collection of real and artificial models as examples of some combinations of the chosen characteristics.

1INTRODUCTION

There is much published literature on the subject of output analysis of a single scenario, for example research into warm-up methods and construction of confidence intervals. Each paper may use one or more artificial models (or possibly a real model) to test or illustrate their research findings. However, there does not seem to be a general set of models and output types in the public domain that sufficiently covers the many different types of possible models and output. The authors of this paper required such a set for use in their own specific output analysis research. As this was not readily available in the literature, we createda representative and sufficient set of models/data output that could be used in discrete event simulation research by the authors and other researchers.

A set of artificial data sets have been developed and a range of ‘real’ simulation models gathered together. Certain attributes of both the models and outputs were considered for their possible importance in the performance of output analysis methods. Appropriate categories were selected and models and outputs classified accordingly. Our aim was to categorise and collect a wide range of real models, artificial models and their associated outputs so

the collection would cover each general type of model and output encountered in ‘real life’ modelling.

2CATEGORISING A SUFFICIENT SET OF MODELS/OUTPUT

Model output falls into two main categories or groups: Transient and Steady-State. There is also output with a trend (i.e. produced by out-of-control models where traffic intensity, ρ, is greater or equal to one.); this can be described as Out-of-Control Trend. Cyclic output is characterised as steady-state with a cycle pattern.

Apart from identifying these main categories, nine other characteristics of models and output data sets were chosen to be used to categorise the models/output within these two main groups. These were divided into characteristics of the simulation model itself, characteristics of the output data that could be seen by eye (non-statistical) and statistical characteristics of the output that were determined by statistical analysis of the data sets, as follows:

Model Characteristics

1.Deterministic or stochastic (random) model

2.Significant pre-determined model changes (by time), e.g. arrival patterns

3.Dynamic internal changes (i.e. ‘feed-back’), e.g. activating additional resources in response to demand

Output Data Characteristics

Non-statistical:

4. Empty-to-empty pattern

5. Initial transient (warm-up)

6. Out of control trend ρ≥1

7. Cycle

Statistical:

8. Auto-correlation

9. Statistical distribution

3CREATION AND COLLECTION OF MODELS/OUTPUT DATA SETS

In order to find out whether the chosen classification scheme was adequate a collection of both artificial and ‘real’ models and their associated outputs were collected together and categorised by their model and/or output characteristics (as set out in section 2).

3.1COLLECTION OF ARTIFICAL MODELS/OUTPUT

A search of existing and cited literature produced 24 artificial models. It was found that authors borrow models from each other either with or without amendments. All the models produce a steady state output with or without a warm-up period. These models, which have all been recreated for this research, are as follows:

  • Cash et al (1992): AR(1); M/M/1; Markov Chain.
  • Robinson (2007): AR(1); M/M/1.
  • Goldsman et al. (1994): AR(1); M/M/1.
  • White, Cobb & Spratt (2000): AR(2).
  • Ockerman & Goldsman (1997): Random Walk; AR(1); MA(1).
  • Kelton & Law (1983): M/M/1 (FIFO); M/M/1 (LIFO); M/M/1 (SIRO); M/M/1 (initialised with 10 customers); E4/M/1; M/H2/1; M/M/2; M/M/4; M/M/1/M/1/M/1.
  • Hsieh et al (2004): M/M/1/199; M/G/1/199; M/M/1/19; Number-in-stock process single item inventory management system.

There are three main methods for creating artificial models and output data sets:

  • Create simplesimulation models where theoretical value of some attribute is known.

e.g. Model: M/M/1.

Attribute: mean waiting time.

  • Create simple simulation models where the value of some attribute is estimated but model characteristics can be controlled.

e.g. Model: Single item inventory management system.

Attribute: Number-in-stock.

  • Create data sets from known equations, which closely resemble real model output, with a known value for some specific attribute.

e.g. AR(1) with Normal(0,1) errors

3.2COLLECTION OF REAL MODELS

Real models are defined as discrete event simulation models of real existing systems, created in “real circumstances” (i.e. in business, academia, etc…).

For example:

Model / Output Result
Call Centre / Percentage of calls answered within 30 seconds
Production Line Manufacturing Plant / Throughput
Fast Food Store / Average queuing time
Hospital / Average number in system

For each model the output result chosen to be analysed was deemed to be the most likely output to be of interest to a practitioner for that type of model. When the model came with already programmed results collection then these were utilised if feasible.

3.3CLASSIFICATION OF COLLECTED MODELS / OUTPUT

After collection or creation of model/output the data output sets were identified as one of 4 sub-types: Steady–State, Steady-State Cycle, Transient, or Out of Control Trend. Each type was statistically analysed as follows:

Steady-State

  1. Subtract the mean of each replication from the data output forthat replication to create time series residuals. (Do this for 3 or more replications)
  2. Test the residuals for autocorrelation and partial autocorrelation functions (ACF and PACF)
  3. Test the residuals for normality

Steady-State Cycle

  1. If output is collected per customer/item then use time customer/item leaves system as x axis instead of customer/item index number.
  2. Run model for many cycles for each replication carried out (3 or more cycles)
  3. Take mean of each cycle to create a new time series (for each replication)
  4. Subtract mean from this new output data of each replication carried out (3or more replications)
  5. Test residuals for ACF/PACF and Normality (Normal or not Normal)

Transient

  1. Test for ACF/PACF on raw data from each replication carried out (3 or more replications)
  2. Run many replications (1000)
  3. Take mean of each replication to create new (non auto-correlated) data series.
  4. Test for what type of statistical distribution this data series is – is it normal or highly skewed etc? Find the ‘best’ fitting distributions to the data using maximum likelihood estimates of parameters and goodness of fit Anderson-Darling and Kolmogorov-Smirnov tests.

Out of Control Trend

  1. Plot data

4CREATING CLASSIFICATION TABLES

Two separate classification tables were drawn up, one showing the model characteristics only and one displaying the output characteristics. Each table is then split into two again, producing a table for transient models and a table for steady state models. This was due to the fact that these two model types had been analysed slightly differently as thought appropriate. Each of these four tables contains all combinations of important model and output characteristics that were logically possible. Each table also indicates which combinations have examples of real or artificial models existing in our collection. Table 1 shows the complete table for the transient model output characteristics. Tables 2-4 show a sample from the other three categories.Full tables can be viewed at the project website <

5CONCLUSION

This research has produced a classification of model and output types for the purpose of aiding research into simulation output analysis. A series of artificial and real models have been identified and classified. Three specific issues warrant further attention.

First, are the criteria used to categorise the models/output sufficient? Are there other criteria that should be incorporated that have thus far been missed out?

Second, there are not yet examples in our model collection for every category. In particular, there are no transient model outputs with a warm-up period, deterministic transient models or cycle output with a warm-up period. Is this because these type of models/outputs do not exist, or simply that we have failed to find such models? It is our belief that these model types do exist e.g. Beck (2004) tackles the subject of consistency of warm-up periods in cyclic models.

Table 1: Transient Output Data Characteristics

TRANSIENT OUTPUT DATA CHARACTERISTICS
NON-STATISTICAL / STATISTICAL / Model Exists in collection
ρ≥1 / Empty to empty / Warm-up? / Distribution of replication means
No / Yes / None / Left Skewed / No
Normal / Real
Other / No
Right Skewed / Real
No / None / Left Skewed / Real
Normal / Real
Other / No
Right Skewed / Real
Yes / Left Skewed / No
Normal / No
Other / No
Right Skewed / No
Yes / No / None / N/A / Real
Table 2: Steady State Output Data Characteristics (Sample)
Steady State Output Data Characteristics
Non-Statistical / Statistical / Model exists in collection
Warm-up / Cycle / Distribution / Auto-correlation
None / Yes / Normal / AR(1) / Real

Table 3: Steady State Model Characteristics (Sample)

Steady State Model Characteristics
Deterministic / Stochastic / Pre-determined model changes / Dynamic model changes / Model exists in collection
Stochastic / None / None / Artificial and Real

Table 4: Transient Model Characteristics (Sample)

Transient Model Characteristics
Deterministic / Stochastic / Pre-determined model changes / Dynamic model changes / Model exists in collection
Deterministic / Yes / None / No models

Finally, we note that the extant artificial models fall within a very limited set of categories. This suggest the need to devise a wider set of artificial models/outputs. It also raises the question of the generality of the tests previously performed on proposed output analysis methods using these models.

In terms of current and future work on this project: a group of artificially created data sets that mimic the types of transient output that we observed in our collection of models are currently being used, along with some other real models from our collection, to test and develop our output analysis algorithms. It is our intention to continue using the classification set in our research and to create artificial data sets for each category combination that is missing an example.

ACKNOWLEDGEMENTS

This work is part of the Automating Simulation Output Analysis (AutoSimOA) project that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project.

REFERENCES

Cash, C. R., B. L. Nelson, D. G. Dippold, J. M. Long, and W. P. Pollard. 1992. Evaluation of tests for initial-condition bias. Proceedings of the Winter Simulation Conference 1992.

Beck, A. D. 2004. Consistency of warm up periods for a simulation model that is cyclic in nature. Proceedings of the Simulation Study Group(The OR Society) 2004..

Goldsman, D., L. W. Schruben, and J. J. Swain. 1994. Tests for transient means in simulated time series. Naval Research Logistics. Vol. 41. pp. 171-187.

Hsieh, M-H., D. L. Iglehart, and P. W. Glynn. 2004. Empirical performance of bias-reducing estimators for regenerative steady-state simulations. ACM Transactions on Modeling and Computer Simulation. Vol. 14. pp. 325-343.

Kelton, W. D., and A. M. Law. 1983. A new approach for dealing with the startup problem in discrete event simulation. Naval Research Logistics. Vol. 30. pp. 641-658.

Ockerman, D. H., and D. Goldsman. 1997. The impact of transients on simulation variance estimators. Proceedings of the Winter Simulation Conference 1997.

Robinson, S. 2007. A statistical process control approach to selecting a warm-up period for a discrete-event simulation. Science Direct, European Journal Of Operational Research.Vol. 176.pp. 332-346.

White, K. P., M. J. Cobb, and S. C. Spratt. 2000. A comparison of five steady-state truncation heuristics for simulation. Proceedings of the Winter Simulation Conference 2000.

AUTHOR BIOGRAPHIES

KATHRYN A. HOAD is a research fellow in the Operational Research and Information Systems Group at WarwickBusinessSchool. She holds a BSc in Mathematics and its Applications from PortsmouthUniversity, an MSc in Statistics and a PhD in Operational Research from SouthamptonUniversity. Her e-mail address is

STEWART ROBINSON is a Professor of Operational Re-search at WarwickBusinessSchool. He holds a BSc and PhD in Management Science from LancasterUniversity. Previously employed in simulation consultancy, he sup-ported the use of simulation in companies throughout Europe and the rest of the world. He is author/co-author of three books on simulation. His research focuses on the practice of simulation model development and use. Key areas of interest are conceptual modelling, model valida-tion, output analysis and modelling human factors in simu-lation models. His email address is <> and his Web address is <

RUTH DAVIES is a Professor of Operational Research in WarwickBusinessSchool, University of Warwick and is head of the Operational Research and Information Systems Group. She was previously at the University of Southampton. Her expertise is in modeling health systems, using simulation to describe the interaction between the parts in order to evaluate current and potential future policies. Over the past few years she has run several substantial projects funded by the Department of Health, in order to advise on policy on: the prevention, treatment and need for resources for coronary heart disease, gastric cancer, end-stage renal failure and diabetes. Her email address is .