A density projection approach for non-trivial information dynamics: Adaptive management of stochastic natural resources

https://doi.org/10.1016/j.jeem.2013.07.003Get rights and content

Abstract

We demonstrate a density projection approximation method for solving resource management problems with imperfect state information. The method expands the set of partially-observed Markov decision process (POMDP) problems that can be solved with standard dynamic programming tools by addressing dimensionality problems in the decision maker's belief state. Density projection is suitable for uncertainty over both physical states (e.g. resource stock) and process structure (e.g. biophysical parameters). We apply the method to an adaptive management problem under structural uncertainty in which a fishery manager's harvest policy affects both the stock of fish and the belief state about the process governing reproduction. We solve for the optimal endogenous learning policy—the active adaptive management approach—and compare it to passive learning and non-learning strategies. We demonstrate how learning improves efficiency but typically follows a period of costly short-run investment.

Introduction

The traditional method of attempting to resolve structural uncertainty on ecosystem dynamics is to invest in natural science research (Moore and McCarthy, 2010). Because management decisions are made on a repeated basis, however, there is also the possibility of using these actions to learn about the ecosystem, i.e. to practice adaptive management. In one of the foundational works on adaptive management, biologist Carl Walters argued that the primary means of reducing uncertainty in environmental models is “through experience with management itself rather than through basic research or the development of (theory)” (Walters, 1986).

Optimal investment in learning depends on the expected value of information generated (Hartmann et al., 2007, Springborn et al., 2010). All else equal, the faster a resource manager resolves structural uncertainty, the sooner the manager is able to reduce management error resulting from the difference between the true structure of a system and the manager's beliefs about that structure. However, accelerating the process of learning, relative to the optimal non-learning policy, comes with an opportunity cost, in the form of short run losses due to learning-driven deviations. An optimal endogenous learning approach—sometimes called active adaptive management (AAM)—involves balancing these tradeoffs by pursuing informative actions, but only when the investment is justified by the value of reducing uncertainty.

We employ an adaptive control framework to model a fishery manager choosing the optimal harvest in each period subject to both irreducible uncertainty (stochasticity) and reducible (structural) uncertainty in the population dynamics as a partially observed Markov decision process (POMDP) (see also Zhou et al., 2010, McDonald-Madden et al., 2010, Bond and Loomis, 2009).1 Learning to reduce structural uncertainty is accomplished by observing the population, which a manager can influence by altering harvest policy.2 A POMDP is a sequential decision problem in which at least one underlying state is not known with certainty. Using the concept of a belief state, i.e. encoding the decision-maker's probabilistic beliefs about the uncertain state, the POMDP can be converted to a continuous-state Markov decision process (MDP), or belief MDP (Bertsekas, 1995).

Our paper makes three important contributions to the growing literature on adaptive control in natural resource economics. First, we use a hierarchical uncertainty model to flexibly represent uncertainty. Second, we relax constraints imposed in the existing AAM literature and much of the economics literature that limit the action space or belief space to a few discrete alternatives. Finally, we employ for the first time in the resource economics and adaptive management literature (to our knowledge) a density projection approximation method that enables us to solve an otherwise intractable high-dimensional control problem. The approach is general: the density projection method results in concise analytical conditions for determining information dynamics which hold across a broad set of applications (exponential family distributions).

Hierarchical modeling—also called multilevel or state-space modeling—is a framework in which complex processes, such as uncertain fish population dynamics, are broken down into a set of conditional sub-models that are coherently tied together using simple probability structures (Wikle, 2003).3 We use a hierarchical model of uncertainty to allow for a more realistic and flexible representation of uncertainty in ecosystem dynamics and therefore the beliefs of the managers. Practical usage of hierarchical models in dynamic resource management models, however, has been limited by state space dimensionality concerns.

To address the challenge of the growing dimensionality of the belief state, researchers often discretize and constrain the belief state space. For example, extending a shallow lake pollution model to incorporate learning, Bond and Loomis (2009) limit uncertainty over a critical nutrient threshold to two possible levels. Another recent example is Johnson (2011), who describes adaptive management of waterfowl harvest by the U.S. Fish and Wildlife Service, who aims to discern between four competing models of population dynamics.

Rather than constrain the belief state space that could limit the ability to map model outputs to management actions, we use a density projection method to solve for the optimal AAM policy in the presence of a hierarchical model of uncertainty. A limited set of hierarchical models are, in Bayesian terminology, conjugate, meaning that an explicit closed form solution exists for the belief state dynamics. However, hierarchical models are typically non-conjugate, as is the case here. We gain tractability for our non-conjugate hierarchical model by approximating the posterior distribution of the manager's beliefs using a function that minimizes the Kullback–Leibler (KL) divergence between true updated beliefs and an approximation (Zhou et al., 2010). The approximation of the belief state through this density projection4 method converts the belief MDP to a projected-belief MDP with lower dimensionality. Relatively simple analytical expressions can be derived to identify the parameters of the projected or approximate beliefs for a wide range of applications. A further advantage of this conversion is the ability to solve for the AAM solution using standard dynamic programming methods, such as value or policy iteration (Zhou et al., 2010).

Finally, a number of studies consider applications similar to ours (though not with a hierarchical uncertainty model) where a manager can learn about a system by observing population dynamics (e.g., Hauser and Possingham, 2008, Rout et al., 2009, McDonald-Madden et al., 2010). However, in these studies, the researchers constrain the set of management actions to a set of 2–3 discrete choices for tractability (e.g., high and low harvest levels). While the limited set of control actions can be informative for how learning might change management responses and it reduces the dimensionality of the problem, we allow for a continuous choice control variable that is more consistent with how fish populations (and many other natural resources) are managed. A notable exception to the use of limited action and/or belief spaces is Wieland (2000) who allows for continuous beliefs and actions in a generic, linear regression-style learning model (Kalman filter). However, Wieland assumes a convenient, conjugate structure using normal distributions for tractability, in contrast to the framework here which does not require selecting from a limited set of conjugate models.

We illustrate the contribution of density projection methods by solving an otherwise intractable stochastic bioeconomic model. We consider the investment decisions of a manager employing AAM in a noisy environment to maximize the expected net present value from fishing given both continuous action and belief state spaces. We find that differences in the manager's belief state can drive strong differences in resource use policy. Not surprisingly, we also find that AAM can result in significant efficiency gains relative to a non-learning approach. However, the AAM gains relative to a passive learning approach are less dramatic and we show how they depend strongly on the nature of the learning process. We also show that long-run gains associated with AAM are achieved by investing in learning, which can take the form of costly deviations from the optimal path without learning. Thus, when setting expectations, advocates of AAM must be aware that implementation of such optimal endogenous learning may bring short-run sacrifice.5

The paper proceeds as follows. After outlining the basic ecological and economic components of the bioeconomic model we develop the hierarchical model of survivorship and Bayesian learning model in detail. Next we describe how belief state approximation, or density projection, is accomplished via minimization of the KL divergence. We then discuss the solution of the stochastic dynamic programming problem and contrast the performance of various management approaches to learning.

Section snippets

Model

We present the stochastic bioeconomic model in three parts: (1) the fish population dynamics, (2) the economics and management component, and (3) the model of uncertainty and beliefs of the manager.

Numerical results

Using dynamic programming, we solve for the optimal value and policy function under an active-adaptive management (AAM) approach, which solves the full optimization problem in Eq. (3). Under AAM, the decision maker optimally adjusts harvest policy given the instantaneous returns from harvesting, beliefs about the productivity of the system, expectations about what can be learned over time, and expectations over future harvest levels. Also recall, in our system, the number of trials in the

Conclusion

We contribute to the application of AAM in natural resource economics by demonstrating the use of a hierarchical model for flexible representation of uncertainty, and by solving an otherwise intractable control problem using a density projection approximation method and standard dynamic programing techniques. We do so while at the same time relaxing constraints imposed in the existing AAM literature that limit the action space or belief space to a few discrete alternatives.

While we demonstrate

References (56)

  • W.J. Reed

    Optimal escapement levels in stochastic and deterministic harvesting models

    Journal of Environmental Economics and Management

    (1979)
  • J.N. Sanchirico

    Additivity properties in metapopulation models: implications for the assessment of marine reserves

    Journal of Environmental Economics and Management

    (2005)
  • G. Sethi et al.

    Fishery management under multiple uncertainty

    Journal of Environmental Economics and Management

    (2005)
  • M.D. Smith et al.

    Econometric modeling of fisheries with complex life histories: Avoiding biological management failures

    Journal of Environmental Economics and Management

    (2008)
  • O. Tahvonen

    Economics of harvesting age-structured fish populations

    Journal of Environmental Economics and Management

    (2009)
  • V. Wieland

    Learning by doing and the value of optimal experimentation

    Journal of Economic Dynamics and Control

    (2000)
  • Y. Bar-Shalom

    Stochastic dynamic programming: caution and probing

    IEEE Transactions on Automatic Control

    (1981)
  • J.O. Berger

    Statistical decision theory and Bayesian analysis

    (1985)
  • D.P. Bertsekas

    Dynamic programming and optimal control

    (1995)
  • C.A. Bond et al.

    Using numerical dynamic programming to compare passive and active learning in the adaptive management of nutrients in shallow lakes

    Canadian Journal of Agricultural Economics/Revue canadienne d’agroeconomie

    (2009)
  • C.A. Bond

    On the potential use of adaptive control methods for improving adaptive natural resource management

    Optimal Control Applications and Methods

    (2010)
  • A. Brooks et al.

    A Monte Carlo update for parametric POMDPs

    Robotics Research

    (2011)
  • M.H. Chen et al.

    Performance study of marginal posterior density estimation via Kullback–Leibler divergence

    Test

    (1997)
  • H. Doremus

    Precaution, science, and learning while doing in natural resource management

    Washington Law Review

    (2007)
  • H. Doremus

    Adaptive management as an information problem

    NCL Review

    (2010)
  • NM Filatov et al.

    Survey of adaptive dual control methods

    IEE Proceedings-Control Theory and Applications

    (2000)
  • A. Gelman et al.

    Bayesian Data Analysis

    (2004)
  • Gilboa, I., Marinacci, M., 2011. Ambiguity and the Bayesian Paradigm, Working Papers 379, IGIER, Innocenzo Gasparini...
  • Cited by (21)

    • Stability of international fisheries agreements under stock growth uncertainty

      2022, Journal of Environmental Economics and Management
      Citation Excerpt :

      The first strand concentrates on a renewable resource harvester’s extraction responses to several types of uncertainty. For instance, Mirman (1971), Reed (1979), Singh et al. (2006), Costello and Polasky (2008), and Springborn and Sanchirico (2013) focus on the stock growth uncertainty case whereas (Costello and Kaffine, 2008) address scenarios in which future ownership is uncertain. Clark and Kirkwood (1986) and Sethi et al. (2005) investigate scenarios in which the harvester faces managerial uncertainty (e.g., measurement errors) and environmental uncertainty (e.g., stock growth uncertainty).

    • Measure twice, cut once: Optimal inventory and harvest under volume uncertainty and stochastic price dynamics

      2020, Journal of Environmental Economics and Management
      Citation Excerpt :

      Many studies addressing resource management under uncertainty and learning focus on parameter uncertainty (e.g. Springborn and Sanchirico, 2013).

    • Optimal monitoring and control under state uncertainty: Application to lionfish management

      2017, Journal of Environmental Economics and Management
      Citation Excerpt :

      We focus on the former margin in this paper by introducing monitoring as a control variable that has a direct cost. Models in the AM literature focus on the latter margin and typically assume the capacity to learn has no direct cost (e.g., Springborn and Sanchirico, 2013). This allows an analyst to calculate the value of sophistication in a model by comparing the expected value function with and without the link between actions and future learning built into the decision maker's expectations (Williams et al., 2011a, 2011b).

    • Optimal management with reversible regime shifts

      2016, Journal of Economic Behavior and Organization
    • Risk aversion and adaptive management: Insights from a multi-armed bandit model of invasive species risk

      2014, Journal of Environmental Economics and Management
      Citation Excerpt :

      After 200 months, ADP confers an advantage of just 0.6% relative to GBayes and 1% relative to GMLE. As discussed in the introduction, such a modest relative expected gain from taking an active AM approach under risk neutrality is consistent with typical results from the existing literature (Springborn and Sanchirico, 2013; Bond and Loomis, 2009; Rout et al., 2009). Although the average performance advantage of ADP over GMLE is modest, as suggested by analytical results in Section “Analytical insights from a two-source, two-period adaptive management problem”, a more nuanced story emerges from considering the distribution of outcomes.

    • Addressing structural and observational uncertainty in resource management

      2014, Journal of Environmental Management
      Citation Excerpt :

      More generally, projection methods can be used wherein the updated (posterior) distribution is projected onto a parametric family of belief distributions. The projection approach is described in detail in Zhou et al. (2010) and is applied to a resource management problem by Springborn and Sanchirico (2013). In the current paper we concentrate on the discrete parameter case, which is sufficient to illustrate the principles involved.

    View all citing articles on Scopus
    View full text