深度强化学习算法PPO训练Pendulum

关注

VIP免费原创最新推荐文章于 2026-09-19 14:59:47 发布 805 阅读 AI 写同款· GEO 优化 ›

强化学习_06_pytorch-PPO实践(Pendulum-v1) TRPO(Trust Range Policy Optimate)算法每一步更新都需要大量的运算，于是便有其改进版本PPO在2017年被提出。PPO 基于 TRPO 的思想，但是其算法实现更加简单。TRPO 使用泰勒展开近似、共轭梯度、线性搜索等方法直接求解。PPO 的优化目标与 TRPO 相同，但 PPO 用了一些相对简单的方法来求解。具体来说, PPO 有两种形式，一是PPO-惩罚，二是PPO-截断，我们接下来对这两种形式进行介绍。阅读详情

PPO代码部分，训练连续动作

1. 导入必须要的包

import torch
import torch.nn as nn
import torch.nn.functional as f
import torch.optim as optim
from torch.distributions import Normal
from collections import deque
import matplotlib.pyplot as plt
import numpy as np
import gym
import copy

2. 定义actor网络和critic网络

class actor_net(nn.Module):

    def __init__(self, state_n, action_n, hidden_n):
        super(actor_net, self).__init__()
        self.fc1 = nn.Linear(state_n, hidden_n)
        self.fc2 = nn.Linear(hidden_n, hidden_n)

        self.mu = nn.Linear(hidden_n, action_n)
        self.sigma = nn.Linear(hidden_n, action_n)

    def forward(self, x):
        x = f.relu(self.fc1(x))
        x = f.relu(self.fc2(x))
        mu = torch.tanh(self.mu(x)) * 2
        sigma = f.softmax(self.sigma(x), dim = 1) + 0.001
        return mu, sigma


class critic_net(nn.Module):

    def __init__(self, state_n, hidden_n):
        super(critic_net, self).__init__()
        self.fc1 = nn.Linear(state_n, hidden_n)
        self.fc2 = nn.Linear(hidden_n, hidden_n)
        self.fc3 = nn.Linear(hidden_n, 1)

    def forward(self, x):
        x = f.relu(self.fc1(x))
        x = f.relu(self.fc2(x))
        value = self.fc3(x)
        return value

3. 定义存储数据的buffer，包括buffer添加数据，采样数据，清空数据

class buffer(object):
    def __init__(self, length):
        self.buffer_length = length
        self.buffer = deque(maxlen = self.buffer_length)
    def push(self, trans):
        self.buffer.append(trans)
    def sample(self):
        batch = list(self.buffer)
        return zip(*batch)
    def clear(self):
        self.buffer.clear()
    def length(self):
        return len(self.buffer)