# Reinforcement learning in optimizing forest management

Pekka Malo, Olli Tahvonen, Antti Suominen, Philipp Back, and Lauri Viitasaari (Finland)

Canadian Journal of Forest Research, March 2021

Presented by Zachary Sunberg, August 30th, 2023

## Motivation

- Forests have are important sources for building materials (historically for fuel)
- Biodiversity, carbon capture, are becoming more important factors
- Managing involves reasoning about costs, rewards, growth dynamics, fires, etc.
- Previous state-of-the-art-optimization was not scalable

## Contributions

- Bringing Reinforcement Learning to Forest management!
- Biggest stochastic forestry optimization problem ever solved
- Drastically reduced computation time
- No need to simplify models

- Demonstrated that RL converges to previously-known optimal strategy
- Discovered new strategies when the uncertainty of disasters is included
- Explained why some results were expected and some unexpected with a classic principle

## Forestry 101

- Four tree species:
- Norway Spruce
- Silver Birch
- Scots Pine
- European Aspen

- CCF = Continuous Cover Management
- RF = Rotation Forestry
- Central Finland
- Disasters

## Context in Literature

- \(j\) = species
- \(s\) = size class
- \(\tilde{x}_{j,s,t}\) = number of trees
- \(\alpha_{j,s}(\tilde{x}_t)\) = fraction to new size class
- \(\mu_{j,s}(\tilde{x}_t)\) = death fraction
- \(\phi_j(\tilde{x}_t) \geq 0\) = ingrowth rate
- \(\Delta\) = period length (5 years)
- \(r\) = interest rate (discount \(b = 1/(1+r)\))
- \(h\) = number of harvested trees
- \(\pi\) = profit
- \(w\) = "cost of artificial regeneration"

Important Previous Work:

- Faustmann, 1849: Basic optimal rotation problem
- Reed, 1984: Disasters
- Parkatti and Tahvonen, 2020: Nonlinear Programming

## Problem Setting

- State Space: 44-dimensional continuous
- Action Space: 44 continuous dimensions, 2 binary choices

## Problem Setting

## Approach

- Hybrid PPO with GAE

"The forest state variables x for all species j and size classes s are passed (appropriately normalized) to the input layer."

- Relu activations
- Shared layer of 500
- Actor networks each have a layer of 200
- Value network has a layer of 300

## Computational Resources

## Numerical Results: Deterministic Model

BLV = 2336

Close to optimized BLV: 2335

## Numerical Results: Initial Clearcut?

## Numerical Results:

Initial Clearcut?

## Numerical Results:

Stochastic Outcomes with No Disasters

## Numerical Results:

Stochastic Outcomes with No Disasters

Mean: 2357 Euros

Mean: 2355 Euros

Deterministic Mean:

2336 Euros

## Numerical Results:

Stochastic Outcomes with Disasters

0% Fire

1% Fire

\(r\) = 1%

2% Fire

Disaster Probability = 1%

Disaster Probability = 2%

Policy optimized with disasters

Policy optimized with no disasters

Mean 15,151

Mean 14,753

Mean 11,101

Mean 10,060

## Numerical Result Summary

- "Higher interest rate favors CCF"
- Birch is sometimes dominant
- With no disasters, similar value to deterministic
- With disasters, RL significantly better

**Confirmed Previous****Previously Unknown****Confirmed Previous****New Result!**

RL Performance = Deterministic Optimization Performance

RL Performance > Deterministic Optimization Performance

No disasters

Disasters

**Certainty Equivalence!**

## Critique

**Positive**

- Explains RL well to a new field

- Enough details to reproduce (especially problem parameters!)

- Not overly complex

- Well written

**Negative**

- Some minor details missing

- Did not release code?

- Too many results distract from story

## Impact and Legacy

18 citations since 2021 = Pretty Good!

## Contributions (Recap)

- Brought Reinforcement Learning to Forest management!
- Biggest stochastic forestry optimization problem ever solved
- Drastically reduced computation time (170 h -> 6 min)
- No need to simplify models (Could include more stochasticity)

- Demonstrated that RL converges to previously-known optimal strategy
- Discovered new strategies when the uncertainty of disasters is included
- Explained why some results were expected and some unexpected with a classic principle:
**Certainty Equivalence**

#### RL-For-Forest-Management

By Zachary Sunberg

# RL-For-Forest-Management

- 207