增强学习（Reinforcement Learning）简介

news2025/5/10 12:30:53

增强学习（Reinforcement Learning）简介
增强学习是机器学习的一种范式，其核心目标是让智能体（Agent）通过与环境的交互，基于试错机制和延迟奖励反馈，学习如何选择最优动作以最大化长期累积回报。其核心要素包括：
• 状态（State）：描述环境的当前信息（如棋盘布局、机器人传感器数据）。

• 动作（Action）：智能体在特定状态下可执行的操作（如移动、下棋）。

• 奖励（Reward）：环境对动作的即时反馈信号（如得分增加或惩罚）。

• 策略（Policy）：从状态到动作的映射规则（如基于Q值选择动作）。

• 价值函数（Value Function）：预测某状态或动作的长期回报（如Q-Learning中的Q表）。

与监督学习不同，增强学习无需标注数据，而是通过探索-利用权衡（Exploration vs Exploitation）自主学习。

使用PyTorch实现深度Q网络（DQN）演示
以下以CartPole-v0（平衡杆环境）为例，展示完整代码及解释：

环境与依赖库

import gym
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from collections import deque
import random

# 初始化环境
env = gym.make('CartPole-v1')
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n

定义DQN网络

class DQN(nn.Module):
    def __init__(self, state_dim, action_dim):
        super(DQN, self).__init__()
        self.fc = nn.Sequential(
            nn.Linear(state_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, action_dim)
        )
    
    def forward(self, x):
        return self.fc(x)

经验回放缓冲区（Replay Buffer）

class ReplayBuffer:
    def __init__(self, capacity):
        self.buffer = deque(maxlen=capacity)
    
    def push(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))
    
    def sample(self, batch_size):
        batch = random.sample(self.buffer, batch_size)
        states, actions, rewards, next_states, dones = zip(*batch)
        return (
            torch.FloatTensor(states),
            torch.LongTensor(actions),
            torch.FloatTensor(rewards),
            torch.FloatTensor(next_states),
            torch.FloatTensor(dones)
        )

训练参数与初始化

# 超参数
batch_size = 64
gamma = 0.99        # 折扣因子
epsilon_start = 1.0
epsilon_decay = 0.995
epsilon_min = 0.01
target_update = 10  # 目标网络更新频率

# 初始化网络与优化器
policy_net = DQN(state_dim, action_dim)
target_net = DQN(state_dim, action_dim)
target_net.load_state_dict(policy_net.state_dict())
optimizer = optim.Adam(policy_net.parameters(), lr=0.001)
buffer = ReplayBuffer(10000)
epsilon = epsilon_start

训练循环

num_episodes = 500
for episode in range(num_episodes):
    state = env.reset()
    total_reward = 0
    
    while True:
        # ε-贪婪策略选择动作
        if random.random() < epsilon:
            action = env.action_space.sample()
        else:
            with torch.no_grad():
                q_values = policy_net(torch.FloatTensor(state))
                action = q_values.argmax().item()
        
        # 执行动作并存储经验
        next_state, reward, done, _ = env.step(action)
        buffer.push(state, action, reward, next_state, done)
        state = next_state
        total_reward += reward
        
        # 经验回放与网络更新
        if len(buffer.buffer) >= batch_size:
            states, actions, rewards, next_states, dones = buffer.sample(batch_size)
            
            # 计算目标Q值
            with torch.no_grad():
                next_q = target_net(next_states).max(1)[0]
                target_q = rewards + gamma * next_q * (1 - dones)
            
            # 计算当前Q值
            current_q = policy_net(states).gather(1, actions.unsqueeze(1))
            
            # 均方误差损失
            loss = nn.MSELoss()(current_q, target_q.unsqueeze(1))
            
            # 反向传播
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        
        if done:
            break
    
    # 更新目标网络与ε
    if episode % target_update == 0:
        target_net.load_state_dict(policy_net.state_dict())
    epsilon = max(epsilon_min, epsilon * epsilon_decay)
    
    print(f"Episode {episode}, Reward: {total_reward}, Epsilon: {epsilon:.2f}")