Verification and Validation of
Agent-Based Computational Models

Last Updated: 31 January 2009

Site Maintained By:

Leigh Tesfatsion

Professor of Economics and Mathematics

Iowa State University

Ames, Iowa 50011-1070

http://www.econ.iastate.edu/tesfatsi/

tesfatsi AT iastate.edu

Online ACE Course (Self-Study eBook)

ACE Website

Table of Contents:

• Key Issues
• Introductory Materials on Model Verification
• Introductory Materials on Empirical Validation
• Additional Readings on Verification and Validation
• Design of Computational Experiments

Key Issues

• How to verify an agent-based computational model is carrying out operations in the way a modeler intends?
• Consider the following famous quote from G.E.P. Box (1979): "All models are wrong, but some are useful." This quote raises the following important question: Must the intended purpose of a model be known before meaningful empirical validation can proceed?
• One way to approach the validation of an agent-based computational model of a real-world system is descriptive output validation, i.e., matching computationally generated output against already-acquired system data. Can this be done in a scientifically meaningful manner? If so, how?
• A second way to approach the validation of an agent-based computational model of a real-world system is predictive output validation, i.e., matching computationally generated output against yet-to-be-acquired system data. Can this be done in a scientifically meaningful manner? If so, how?
• A third way to approach the validation of an agent-based computational model of a real-world system is input validation, i.e., ensuring that the structural conditions, institutional arrangements, and behavioral dispositions incorporated into the model capture the salient aspects of the actual system. Can this be done in a scientifically meaningful manner? If so, how?
• Recently several researchers have advocated an iterative participatory modeling approach in which researchers join with stakeholders in a repeated looping through a four-stage modeling process: field study and data analysis; role-playing games; agent-based model development and implementation; and computational experiments. How might such an approach contribute to the validation of agent-based computational models?
• How can researchers provide summary reporting of model validation results to other researchers and to intended model users in an accurate, compelling, and clear manner? For example, it might be necessary to report distributions of possible outcomes rather than simple point predictions.
• How can researchers ensure the robustness of model validation findings, i.e., that model outcomes reflect persistent (locally generic) aspects of the real-world system under study rather than a modeler's particular choice of parameter settings, initial conditions, or software/hardware platform implementations?
• How to ensure the accumulation of empirically supported findings?

Introductory Materials on Model Verification

• Robert Axelrod, "Advancing the Art of Simulation in the Social Sciences" (pdf,119K), University of Michigan, August 2003.
  Abstract: The author offers advice for doing social science simulation research, focusing on the programming of a simulation model, analyzing the results, and sharing the results with others.

  Robert Axtell, Robert Axelrod, Joshua M. Epstein, and Michael D. Cohen, Aligning Simulation Models: A Case Study and Results (pdf,53K), Computational and Mathematical Organization Theory, Vol. 1, Number 1, 1996, pp. 123-141.

  Abstract: This article develops the concepts and methods of an alignment process ("docking") for computational models. Alignment is used to determine whether two models can produce the same outcomes, which in turn is the basis for critical experiments and tests to determine whether one model can subsume the other. These concepts and methods are illustrated using a model of cultural transmission due to Robert Axelrod and the Sugarscape model developed by Epstein and Axtell.

  José Manuel Galán, Luis R. Izquierdo, Segismundo S. Izquierdo, José Ignacio Santos, Ricardo del Olmo, Adolfo López-Paredes, and Bruce Edmonds, Errors and Artefacts in Agent-Based Modelling (html), Journal of Artificial Societies and Social Simulation 12(1)1, 2009.

  Abstract: "The objectives of this paper are to define and classify different types of errors and artefacts that can appear in the process of developing an agent-based model, and to propose activities aimed at avoiding them during the model construction and testing phases. To do this in a structured way, we review the main concepts of the process of developing such a model – establishing a general framework that summarises the process of designing, implementing, and using agent-based models. Within this framework we identify the various stages where different types of errors and artefacts may appear. Finally we propose activities that could be used to detect (and hence eliminate) each type of error or artefact."

  Kenneth L. Judd, "Computationally Intensive Analyses in Economics", in Leigh Tesfatsion and Kenneth L. Judd (editors), Handbook of Computational Economics, Vol. 2: Agent-Based Computational Economics (Table of Contents & Abstracts), Handbooks in Economics Series, North-Holland, Amsterdam, 2006.

  Abstract: Computer technology presents economists with new tools, but also raises novel methodological issues. This essay discusses the challenges faced by computational researchers, and proposes some solutions.

  J. Gary Polhill, Luis R. Izquierdo, and Nicholas M. Gotts, "The Ghost in the Model (and Other Effects of Floating Point Arithmetic)" (html), Journal of Artificial Societies and Social Simulation (JASSS), Vol. 8, No. 1, January 31, 2005, electronic journal.

  Abstract: This paper explores the effects of errors in floating point arithmetic in two published agent-based models: the first a model of land use change (Polhill et al. 2001; Gotts et al. 2003), the second a model of the stock market (LeBaron et al. 1999). The first example demonstrates how branching statements with floating point operands of comparison operators create a high degree of nonlinearity, leading in this case to the creation of 'ghost' agents -- visible to some parts of the program but not to others. A potential solution to this problem is proposed. The second example shows how mathematical descriptions of models in the literature are insufficient to enable exact replication of work since mathematically equivalent implementations in terms of real number arithmetic are not equivalent in terms of floating point arithmetic.

  Additional Remarks on Floating Point Arithmetic and Agent-Based Models:

  When doing arithmetic on a computer, the uncountably infinite set of real numbers must somehow be squeezed into a discrete set of isolated numbers represented in binary floating-point format. The resulting floating-point errors can lead to some unpleasant surprises for the unwary. For example, floating-point addition does not obey the associative law, e.g., (0.1 + 0.2) + 0.3 can fail to equal 0.1 + (0.2 + 0.3). Moreover, summing 0.05 twenty times can yield a total that differs from 1. In both cases the problem can be traced to the fact that simple-looking base-10 numbers such as 0.1 are not exactly representable in binary floating-point format because they correspond to infinitely repeating binary numbers.

  Researchers at the Macaulay Land Use Research Institute (MLURI) in Aberdeen, Scotland, have been conducting a number of important and interesting studies to investigate the effects of floating-point errors in agent-based models implemented in FEARLUS. A summary of these efforts, with pointers to related papers and demos, can be accessed here. (See, also, the above paper by Polhill et al.) Based on their work to date, the MLURI researchers conclude that floating-point errors are not likely to be of major importance if a model does not perform many operations and if it does not contain branching statements. However, on a personal note, I would like to warn about the need to be careful also about imported utilities such as pseudo-random number generators. In my ISU electricity research group in 2005 we discovered that a major java source file (EmpiricalWalker.java) in the well-known and frequently imported cern.jet.random package is disastrously susceptible to floating-point addition errors.

  Uri Wilenski and Willian Rand, "Making Models Match: Replicating an Agent-Based Model" (html), Journal of Artificial Societies and Social Simulation, Vol. 10, No. 4(2).

  Abstract: The authors attempt to replicate a classic agent-based model from political science developed by R. Axelrod and R. A. Hammond (2003). They detail their effort to replicate this model and the challenges that arose in recreating the model and in determining if the replication was successful. They conclude by discussing issues for (1) researchers attempting to replicate models and (2) researchers developing models in order to facilitate the replication of their results.

Introductory Materials on Empirical Validation

• Olivier Barreteau et al., "Our Companion Modelling Approach" (html), Journal of Artificial Societies and Social Simulation (JASSS), Vol. 6, No. 1, 31 March 2003, electronic journal.
  Abstract: This article discusses an iterative participatory approach to the modeling of complex systems, referred to as "companion modeling." The companion modeling approach envisions multidisciplinary researchers and stakeholders engaging together in repeated looping through a four-stage cycle: field study and data analysis; role-playing games; agent-based model design and implementation; and intensive computational experiments. The new aspect of companion modeling relative to other participatory modeling approaches is the emphasis on modeling as an open-ended collaborative learning process. The modeling objective is to help stakeholders manage complex problems over time through a continuous learning process rather than to attempt the delivery of a definitive problem solution.

  Kathleen M. Carley, "Validating Computational Models" (pdf, 105K), Working Paper, Carnegie Mellon University, September 1996.

  Abstract: This paper provides many theoretical and practical insights regarding the empirical validation of computational models for social and organizational systems.

  Joshua Epstein, "Agent-Based Computational Models and Generative Social Science", Complexity, Vol. 4/No. 5, May/June 1999, 41-60. The published article is available from Wiley Interscience. A ppt presentation of this paper by J. Epstein is available here.

  Abstract: The author argues that the agent-based computational model is a new tool for empirical research. The scientific enterprise is, first and foremost, explanatory, and the central contribution of agent-based modeling is the facilitation of generative explanation. To explain an observed macroscopic social regularity, one must show how a population of cognitively plausible agents, interacting under plausible rules, could actually arrive at the pattern on time scales of interest. In short, "if you didn't grow it, you didn't explain it."

  Giorgio Fagiolo, Christopher Birchenhall, and Paul Windrum (Eds.), Special Issue on "Empirical Validation in Agent-Based Models", Computational Economics,Volume 30, Number 3, 2007.

  Abstract: The papers included in this special issue explore the empirical validation of agent-based models (ABMs). They demonstrate that this issue is far from settled and features many difficult but exciting challenges. Some are rooted in well-known and still-open problems of scientific methodology that concern empirical validation regardless of the particular family of models under scrutiny. Others are strongly related to aspects particular to ABMs.

  Marco Janssen (Arizona State University, Tempe, AZ) organized a workshop on Empirically-Based Agent-Based Models at Indiana University (Bloomington, IN), held June 2-4, 2005.

  Abstract: The purpose of this workshop was to bring together a group of researchers who are using agent-based models (ABMs) to study empirically a variety of social-ecological and economic systems. A major goal was to derive a state-of-the-art synthesis of the various approaches currently being used to test ABMs. Another goal of the workshop was to examine the use of experimental and survey techniques for validating the assumptions made by ABM researchers regarding the internal learning and decision-making behavior of computationally modeled agents. Presentation slides for all workshop presentations can be freely accessed at the above workshop site.

  Note: See, in particular, the guest editorial of the same title by Marco A. Janssen and Elinor Ostrom subsequently publised in Ecology and Society 11(2) (2006), available here.

  Robert Marks, "Validating Simulation Models: A General Framework and Four Applied Examples" (pdf,152K), Computational Economics, 2008, to appear.

  Abstract: This paper provides a framework for discussing the empirical validation of simulation models of market phenomena, in particular of agent-based computational economics models. Such validation is difficult, perhaps because of their complexity; moreover, simulations can prove existence, but not in general necessity. The paper highlights the Energy Modeling Forum’s benchmarking studies as an exemplar for simulators. A market of competing coffee brands is used to discuss the purposes and practices of simulation, including explanation. The paper discusses measures of complexity, and derives the functional complexity of an implementation of Schelling’s segregation model. Finally, the paper discusses how courts might be convinced to trust simulation results, especially in merger policy.

  Klaus G. Troitzsch, "Validating Simulation Models" (pdf,100K), Proceedings of the 18th European Simulation Multiconference, SCS Europe, 2004.

  Abstract: This paper discusses aspects of validating simulation models designed to describe, explain, and predict real-world phenomena.

  Tamás Vicsek, "Complexity: The Bigger Picture" (pdf,71K), Nature, Vol. 418, 11 July 2002, p. 131.

  Abstract: In this short essay, Vicsek describes how computer simulation fits into the scientific enterprise. The goal is to "capture the principal laws behind the exciting variety of new phenomena that become apparent when the many units of a complex system interact."

  Paul Windrum, Giorgio Fagiolo, and Alessio Moneta, Empirical Validation of Agent-Based Models: Alternatives and Prospects (html), Journal of Artificial Societies and Social Simulation, Vol. 10, no. 2,8, March 31, 2007.

  Abstract: This paper addresses a set of methodological problems arising in the empirical validation of agent-based (AB) economics models and discusses how these are currently being tackled. These problems are generic for all those enaged in AB modelling, not just economists. The discussion is therefore of direct relevance to JASSS readers. The paper has two objectives. The first objective is the identification of a set of issues that are common to all modellers engaged in empirical validation. This gives rise to a novel taxonomy that captures the relevant dimensions along which AB modellers differ. The second objective is a focused discussion of three alternative methodological approaches being developed in AB economics -- indirect calibration, the Werker-Brenner approach, and the history-friendly approach -- and a set of (as yet) unresolved issues for empirical validation that require future research.

Additional Readings on Verification and Validation

• Simone Alfarano, Friedrich Wagner, and Thomas Lux, "Estimation of Agent-Based Models: The Case of an Asymmetric Herding Model" (pdf,854K), Computational Economics, to appear.
  Abstract: The authors introduce a simple agent-based model of herding behavior in financial markets in which the ubiquitous stylized facts of financial returns (fat tails, volatility clustering) are emergent properties of the interaction among the traders. The simplicity of the model permits the authors to estimate the underlying parameters, since it is possible to derive a closed form solution for the distribution of returns.

  Carlo Bianchi, Pasquale Cirillo, Mauro Gallegati, and Pietro A. Vagliasindi, "Validation and Calibration in ACE Models: An Investigation of the CATS Model" (pdf,217K), Working Paper, Universitá di Parma, Italy, May 1, 2005.

  Abstract: This paper discusses validation experiments performed with the CATS model proposed by Gallegati et al. (2003,2004), a model of a dynamic economy comprising many firms and banks. Staring from a sample of Italian firms included in the AIDA database, the authors perform several ex-post validation experiments for the CATS model over the simulation period 1996-2001. The simulation findings are then ex-post validated against actual data using several alternative validation techniques, with results the authors conclude are quite promising.

  H. Peter Boswijk, Cars H. Hommes, and Sebastiano Manzan, Behavioral Heterogeneity in Stock Prices (pdf,292K), CeNDEF Working Paper 05-12, University of Amsterdam, 2005.

  Abstract: The authors estimate a dynamic asset pricing model characterized by heterogenous boundedly rational agents using US stock price data from 1871 until 2003. The estimation results support the existence of two expectations regimes: a "fundamentalist regime" in which agents believe in mean reversion of stock prices towards the benchmark fundamental value, and a "chartist trend-following regime" in which agents expect the deviations from the fundamental to trend.

  Daniel G. Brown, Scott Page, Rick Riolo, Moira Zellner, and William Rand, "Path Dependence and the Validation of Agent-Based Spatial Models of Land Use" (pdf, 570K), International Journal of Geographical Information Science, Vol. 19, No. 2, February 2005, 153-174. A related slide presentation can be viewed here (pdf,253K).

  Abstract: The authors identify two distinct notions of accuracy of land-use models -- predictive (output) accuracy and process (input) accuracy -- and highlight the tension between them. To balance these two potentially conflicting motivations, they introduce the concepts of an invariant region (where land-use type is almost certain and thus path independent) and a variant region (where land-use depends on a particular series of events and is therefore path dependent). They demonstrate their methods using an agent-based land-use model based on multi-temporal land-use data collected for Washtenaw County, Michigan, USA. They conclude that their methods can help researchers improve their ability to communicate how well their models perform, the situations or instances in which their models do not perform well, and the cases in which their models are unlikely to predict well due to either path dependence or stochastic uncertainty.

  Joanna J. Bryson, Yasushi Ando, and Hagen Lehmann "Agent-based modelling as scientific method: a case study analysing primate social behaviour", (pdf,454K), Philosophical Transactions of the Royal Society, B -- Biology, 362(1485):1685-1698, Sept 2007.

  Abstract: For a methodology to be useful to science, it should provide two things: first, a means of explanation, and second, a mechanism for improving that explanation. Agent Based Modeling (ABM) is a method that facilitates exploring the collective effects of individual action selection. The explanatory force of the model is the extent to which an observed meta-level phenomenon can be accounted for by the behaviour of its micro-level actors. This article demonstrates this methodology can be applied to the biological sciences, that agent-based models like any scientific hypotheses can be tested, critiqued, generalised, or specified. We review the state of the art for ABM as a methodology for biology. We then present a case study based on the most widely-published agent-based model in the biological sciences: Hemelrijk's DomWorld, a model of primate social behaviour.

  John Duffy, "Agent-Based Models and Human-Subject Experiments", in Leigh Tesfatsion and Kenneth L. Judd (editors), Handbook of Computational Economics, Vol. 2: Agent-Based Computational Economics (Table of Contents & Abstracts), Handbooks in Economics Series, North-Holland/Elsevier, Amsterdam, 2006.

  Abstract: This chapter examines the relationship between agent-based modeling and economic decision-making experiments with human subjects. Both approaches exploit controlled "laboratory" conditions as a means of isolating the sources of aggregate phenomena. Research findings from laboratory studies of human subject behavior have inspired studies using artificial agents in "computational laboratories" and vice versa. In certain cases, both methods have been used to examine the same phenomenon. The focus of this chapter is on the empirical validity of agent-based modeling approaches in terms of explaining data from human subject experiments. We also point out synergies between the two methodologies that have been exploited as well as promising new possibilities.

  "Epistemological Perspectives on Simulation", Journal of Artificial Societies and Social Simulation (JASSS), Volume 8, Issue 4, October 2005.

  Remark: The eight articles in this JASSS special section were originally presented at a workshop of the same title held in July 2004 at the University of Koblenz. A number of the articles address the empirical validation of computer simulation models, including, in particular, the empirical validation of agent-based computer simulations. The contribution by Küppers and Lenhard seems particularly useful in clarifying the differences and common grounds between computer simulation validation in the social and natural sciences.

  Manfred Gilli and Peter Winker, "A Global Optimization Heuristic for Estimating Agent-Based Models" (pdf,410K), Computational Statistics and Data Analysis, Vol. 42, 2003, 299-312. See also the Talk Slides (pdf,359K) by Winker, Gilli, and Jeleskovic titled "An Objective Function for Simulation Based Inference on Exchange Rate Data," presented at CEF'06, Limassol, Cyprus, June 23, 2006.

  Abstract: The authors introduce a global optimization algorithm for a "goodness of fit" objective function arising from the simulation-based indirect estimation of the parameters of agent-based financial market models. As an illustration of the algorithm, the authors report parameter estimation results for a specific agent-based model of the DM/US-$ foreign exchange market.

  Volker Grimm and others, "Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology", Science, 11 November 2005, Vol. 310, No. 5750, pp. 987-991. Published article accessible here with Science subscription.

  Abstract: Agent-based complex systems are dynamic networks of many interacting agents; examples include ecosystems, financial markets, and cities. The search for general principles underlying the internal organization of such systems often uses bottom-up simulation models such as cellular automata and agent-based models. No general framework for designing, testing, and analyzing bottom-up models has yet been established, but recent advances in ecological modeling have come together in a general strategy we call pattern-oriented modeling. This strategy provides a unifying framework for decoding the internal organization of agent-based complex systems and may lead toward unifying algorithmic theories of the relation between adaptive behavior and system complexity.

  Dominique Gross and Roger Strand, "Can Agent-Based Models Assist Decisions on Large-Scale Practical Problems? A Philosophical Analysis", Complexity, Vol. 5/No. 6, July/August 2000, 26-33. Published article available from Wiley Interscience.

  Nicholas R. Jennings, "On Agent-Based Software Engineering" (pdf,257K), Artificial Intelligence 117 (2000), 277-296, copyright © 2002 Elsevier Science B.V. All rights reserved.

  Abstract: "Agent-based computing represents an exciting new synthesis both for Artificial Intelligence (AI) and, more generally, Computer Science. It has the potential to significantly improve the theory and the practice of modeling, designing, and implementing computer systems... The standpoint of this analysis is the role of agent-based software in solving complex, real-world problems. In particular, it will be argued that the development of robust and scalable software systems requires autonomous agents that can complete their objectives while situated in a dynamic and uncertain environment, that can engage in rich, high-level social interactions, and that can operate within flexible organizational structures."

  Stephen Lansing and James N. Kremer (1993), "Emergent Properties of Balinese Water Temple Networks: Coadaptation on a Rugged Fitness Landscape", American Anthropologist, Vol. 95, pp. 97-114. Published article available at JSTOR.

  Abstract: Over hundreds of years, Balinese farmers have developed an intricate hierarchical network of "water temples" dedicated to agricultural deities in parallel with physical transformations of their island deliberately undertaken to make it more suitable for growing irrigated rice. The water temple network plays an instrumental role in the coordination of activities related to rice production. Representatives of different water temple congregations meet regularly to decide cropping patterns, planting times, and water usage, thus helping to synchronize harvests and control pest populations. Lansing and Kremer develop an ecological simulation model to illuminate the system-level effects of the water temple network, both social and ecological. Their anthropological study illustrates many important agent-based modeling concepts, including emergent properties, fitness landscapes, co-adaptation, and the effects of different institutional designs.

  Note: For an analysis and critique of this article, see Marco Janssen (2007), "Coordination in Irrigation Systems: An Analysis of the Lansing-Kremer Model of Bali" (pdf,1M), Agricultural Systems 93: 170-190.

  Averill M. Law (1007), Simulation Modelling and Analysis, Fourth Edition, McGraw-Hill Higher Education.

  Abstract (From the Publisher): "This thoroughly up-to-date guide addresses all aspects of a simulation study, including modeling, simulation languages, validation, input probability distribution, and analysis of simulation output data. Full scale treatments of manufacturing systems simulation, simulation software, and animation are also included along with useful and instructive case studies."

  Roberto Leombruni, Matteo Richiardi, Nicole J. Saam, and Michele Sonnessa, "A Common Protocol for Agent-Based Social Simulation" (pdf,380K), Journal of Artificial Societies and Social Simulation 9(1), 2006. The published article is available here.

  Abstract: Traditional (i.e. analytical) modelling practices in the social sciences rely on a very well established, although implicit, methodological protocol, both with respect to the way models are presented and to the kinds of analysis that are performed. Unfortunately, computer-simulated models often lack such a reference to an accepted methodological standard. This is one of the main reasons for the scepticism among mainstream social scientists that results in low acceptance of papers with agent-based methodology in top journals. We identify some methodological pitfalls that, according to us, are common in papers employing agent-based simulations, and propose appropriate solutions. We discuss each issue with reference to a general characterization of dynamic micro models, which encompasses both analytical and simulation models. Along the way, we also clarify some confusing terminology. We then propose a three-stage process that could lead to the establishment of methodological standards in social and economic simulation.

  Franco Malerba, Richard R. Nelson, Luigi Orsenigo, and Sidney G. Winter, "`History-Friendly' Models of Industry Evolution: The Computer Industry" (html, abstract/link), Industrial and Corporate Change, Vol. 8, No. 1 (1999), 3-41.

  Abstract: "The model presented in this paper is the first of a new generation of evolutionary economic models: `history-friendly' models. History-friendly models are formal models that aim to capture, in stylized form, qualitative and `appreciative' theories about the mechanisms and factors affecting industry evolution, technological advance, and institutional change put forth by empirical scholars of industrial economics, technological change, business organization and strategy, and other social scientists. In this paper we have analyzed the long-term evolution of the computer industry... ."

  David Midgley, Robert Marks, and Dinesh Kunchamwar, "The Building and Assurance of Agent-Based Models: An Example and Challenge to the Field" (pdf,175K), Journal of Business Research, to appear.

  Abstract: "The assurance - the verification and validation - of agent-based models is difficult, because of the heterogeneity of the agents, and the possibility of the emergence of new patterns of macro behavior as a result of the interactions of agents at the micro level. We use an agent-based model of the complex interactions among consumers, retailers, and manufacturers to explore issues of model assurance. Our explorations indicate two challenges for the agent-based models field. The first challenge is to address the critical issue of software verification. The second challenge is to overcome the many methodological challenges that exist in empirically validating these models, some of which we will outline in our paper. We will also propose a method based on the Genetic Algrorithm to address both these challenges, but our experiments, and the lack of good data for many kinds of agents, suggest a minimalist approach to building and assuring agent-based models in general."

  Hassan Qudrat-Ullah, "Structural Validation of System Dynamics and Agent-Based Simulation Models" (pdf,155K), in Y. Merkuryev, R. Zobel, and E. Kerckhoffs (eds.), Proceedings, 19th European Conference on Modelling and Simulation (ECMS), 2005.

  Abstract: "Simulation models are becoming increasingly popular in the analysis of important policy issues including global warming, population dynamics, energy systems, and urban planning. The usefulness of these models is predicated on their ability to link observable patterns of behavior of a system to micro-level structures. This paper argues that structural validity of a simulation model - right behavior for the right reasons - is a stringent measure to build confidence in a simulation model regardless of how well the model passes behavior validation tests. That leads to an outline of formal structural validity procedures available but less explored in the system dynamics `repertoire.' An illustration of a set of six tests for structural validity of both system dynamics and agent-based simulation models follows."

  Juliette Rouchier, "Data Gathering to Build and Validate Small-Scale Social Models for Simulation. Two Ways: Strict Control and Stake-Holders Involvement" (pdf,75K), Working Paper, GREQAM, Marseille, October 13, 2005.

  Abstract: Currently in agent-based modeling (ABM) it has become the norm to assess results against actual data and to build hypotheses by offering some empirical facts to justify the model construction. In this paper the author discusses two distinct approaches that researchers are taking to validate ABM representations of small-scale interaction settings: quantitative statistical validation through controlled human-subject experiments; and an iterative participatory modeling approach called "companion modeling" that engages stakeholders and researchers in repeated rounds of field study and data gathering, role-playing games, agent-based model development and implementation, and computational experiments.

  Claudia Werker and Thomas Brenner, "An Advanced Methodology for Heterodox Simulation Models Based on Critical Realism" (pdf,127K), Working Paper 0401, Papers on Economics & Evolution, Max Planck Institute of Economics, Evolutionary Economics Group, Jena, Germany, 2004.

  Abstract: This paper develops an advanced methodology that makes the results of simulation models in heterodox economics more reliable and acceptable. This methodology copes with the specific characteristics of simulation models in heterodox economics, in particular with inherent uncertainty. The methodology is based on critical realism, because it treats inherent uncertainty by categorizing empirical events into underlying structural driving forces. Data is center-stage in the methodology, because it is used to infer assumptions and implications. Illustrative examples inlude the development of industries in different countries.

  Claudia Werker and Thomas Brenner, "A Practical Guide to Inference in Simulation Models" (pdf,437K), Working Paper 0602, Papers on Economics & Evolution, Max Planck Institute of Economics, Evolutionary Economics Group, Jena, Germany, 2006.

  Abstract: This paper introduces a categorization of simulation models. It provides an explicit overview of the steps that lead to a simulation model. We highlight the advantages and disadvantages of various simulation approaches by examining how they advocate different ways of constructing simulation moels. To this end, the paper discusses a number of relevant methodological issues, such as how realistic simulation models are obtained and which kinds of inference can be used in a simulation approach. Finally, the paper presents a practical guide on how simulation should and can be conducted.

Design of Computational Experiments

• Bart Husslage, Gijs Rennen, Edwin R. van Dam, and Dick den Hertog, Space-Filling Latin Hupercube Designs for Computer Experiments (pdf,338K), Working Paper No. 2006-18, ISSN 0924-7815, Department of Econometrics and Operations Research, Tilburg University, March 2006.
  Abstract: In the area of computer simulation, Latin hypercube designs play an important role. In this paper the class of maximin Latin hypercube designs is considered. Up to now only several two-dimensional designs and designs for some small number of points are known for this class. Using periodic designs and simulated annealing, we extend the known results and construct approximate maximin Latin hypercube designs for up to ten dimensions and for up to 100 design points. All of these designs can be downloaded here.

  Thomas J. Santner, Brian J. Williams, and William I. Notz, The Design and Analysis of Computer Experiments, Springer Series in Statistics, 2003, 283pp. ISBN: 978-0-387-95420-2.

  Abstract: This book describes methods for designing and analyzing experiments conducted using computer code in lieu of a physical experiment. It discusses how to select the values of the factors at which to run the code (the design of the computer experiment) in light of the research objectives of the experimenter. It also provides techniques for analyzing the resulting data so as to achieve these research goals. It illustrates these methods with code made available to the reader.

  Leigh Tesfatsion, Experimental Design: Basic Concepts and Terminology (pdf,45K)

Copyright © 2009 Leigh Tesfatsion. All Rights Reserved.