Reinforcement learning in optimizing forest management

Pekka Malo, Olli Tahvonen, Antti Suominen, Philipp Back, and Lauri Viitasaari (Finland)

Canadian Journal of Forest Research, March 2021

Presented by Zachary Sunberg, August 30th, 2023

Motivation

  • Forests have are important sources for building materials (historically for fuel)
  • Biodiversity, carbon capture, are becoming more important factors
  • Managing involves reasoning about costs, rewards, growth dynamics, fires, etc.
  • Previous state-of-the-art-optimization was not scalable

Contributions

  • Bringing Reinforcement Learning to Forest management!
    • Biggest stochastic forestry optimization problem ever solved
    • Drastically reduced computation time
    • No need to simplify models
  • Demonstrated that RL converges to previously-known optimal strategy
  • Discovered new strategies when the uncertainty of disasters is included
  • Explained why some results were expected and some unexpected with a classic principle

Forestry 101

  • Four tree species:
    • Norway Spruce
    • Silver Birch
    • Scots Pine
    • European Aspen
  • CCF = Continuous Cover Management
  • RF = Rotation Forestry
  • Central Finland
  • Disasters

Context in Literature

  • \(j\) = species
  • \(s\) = size class
  • \(\tilde{x}_{j,s,t}\) = number of trees
  • \(\alpha_{j,s}(\tilde{x}_t)\) = fraction to new size class
  • \(\mu_{j,s}(\tilde{x}_t)\) = death fraction
  • \(\phi_j(\tilde{x}_t) \geq 0\) = ingrowth rate
  • \(\Delta\) = period length (5 years)
  • \(r\) = interest rate (discount \(b = 1/(1+r)\))
  • \(h\) = number of harvested trees
  • \(\pi\) = profit
  • \(w\) = "cost of artificial regeneration"

Important Previous Work:

  • Faustmann, 1849: Basic optimal rotation problem
  • Reed, 1984: Disasters
  • Parkatti and Tahvonen, 2020: Nonlinear Programming

Problem Setting

  • State Space: 44-dimensional continuous
  • Action Space: 44 continuous dimensions, 2 binary choices

Problem Setting

Approach

  • Hybrid PPO with GAE

"The forest state variables x for all species j and size classes s are passed (appropriately normalized) to the input layer."

  • Relu activations
  • Shared layer of 500
  • Actor networks each have a layer of 200
  • Value network has a layer of 300

Computational Resources

Numerical Results: Deterministic Model

BLV = 2336

Close to optimized BLV: 2335

Numerical Results: Initial Clearcut?

Numerical Results:
Initial Clearcut?

Numerical Results:
Stochastic Outcomes with No Disasters

Numerical Results:
Stochastic Outcomes with No Disasters

Mean: 2357 Euros

Mean: 2355 Euros

Deterministic Mean:

2336 Euros

Numerical Results:
Stochastic Outcomes with Disasters

0% Fire

1% Fire

\(r\) = 1%

2% Fire

Disaster Probability = 1%

Disaster Probability = 2%

Policy optimized with disasters

Policy optimized with no disasters

Mean 15,151

Mean 14,753

Mean 11,101

Mean 10,060

Numerical Result Summary

  • "Higher interest rate favors CCF"
  • Birch is sometimes dominant
  • With no disasters, similar value to deterministic
  • With disasters, RL significantly better
  • Confirmed Previous
  • Previously Unknown
  • Confirmed Previous
  • New Result!

RL Performance = Deterministic Optimization Performance

RL Performance > Deterministic Optimization Performance

No disasters

Disasters

Certainty Equivalence!

Critique

 

Positive

- Explains RL well to a new field

- Enough details to reproduce (especially problem parameters!)

- Not overly complex

- Well written

Negative

- Some minor details missing

- Did not release code?

- Too many results distract from story

Impact and Legacy

18 citations since 2021 = Pretty Good!

Contributions (Recap)

 

  • Brought Reinforcement Learning to Forest management!
    • Biggest stochastic forestry optimization problem ever solved
    • Drastically reduced computation time (170 h -> 6 min)
    • No need to simplify models (Could include more stochasticity)
  • Demonstrated that RL converges to previously-known optimal strategy
  • Discovered new strategies when the uncertainty of disasters is included
  • Explained why some results were expected and some unexpected with a classic principle: Certainty Equivalence