fangvv/UAV-DDPG

Code for paper "Computation Offloading Optimization for UAV-assisted Mobile Edge Computing: A Deep Deterministic Policy Gradient Approach"

What it solves

This project implements a reinforcement learning approach to optimize computation offloading in a UAV-assisted Mobile Edge Computing (MEC) system. It aims to minimize the maximum processing delay for user equipments (UEs) by jointly optimizing user scheduling, task offloading ratios, the UAV's flight angle, and its flight speed in dynamic, uncontrollable environments.

How it works

The system uses a Deep Deterministic Policy Gradient (DDPG) algorithm to handle a high-dimensional state space and continuous action space.

  • Environment Simulation: A custom simulator models the 3D area, channel bandwidth, UAV battery, and UE locations. It calculates processing delay based on transmission and computation times, considering Line-of-Sight (LOS) and Non-Line-of-Sight (NLOS) channel conditions.
  • DDPG Agent: An Actor-Critic architecture is employed. The Actor network determines the optimal action (target UE, flight angle, distance, and offloading ratio), while the Critic network evaluates the Q-value of those actions.
  • State Normalization: Raw state data (such as battery levels and coordinates) are scaled to a [0, 1] range to stabilize training and accelerate convergence.
  • Baselines: The project includes implementations of DQN (using a discretized action space) and Actor-Critic to compare performance.

Who it’s for

Researchers and engineers working on wireless communication, UAV-assisted edge computing, and the application of deep reinforcement learning for resource allocation in mobile networks.

Highlights

  • Joint Optimization: Simultaneously manages flight trajectory and task offloading ratios.
  • Continuous Action Space: Uses DDPG to avoid the limitations of discretizing actions.
  • Ablation Studies: Includes specific versions of the code to test the impact of state normalization and exploration noise.
  • Comprehensive Baselines: Provides direct comparisons against DQN and basic Actor-Critic methods.

Related

  • Project
    fangvv/VN-MADDPG

    A multi-agent deep reinforcement learning framework using MADDPG to optimize spectrum sharing and power control for V2V and V2I communications in high-mobility vehicular networks.

  • Dispatch
    OpenAI MADDPG: Learning to Cooperate, Compete, and Communicate

    OpenAI introduced MADDPG, a reinforcement learning algorithm that uses centralized learning and decentralized execution to enable multiple agents to collaborate and compete in complex environments.

  • Dispatch
    Deep Q-Learning with Space Invaders

    Hugging Face explains Deep Q-Learning (DQN), a reinforcement learning method that uses neural networks to approximate Q-values for high-dimensional state spaces like Atari games.

  • Dispatch
    OpenAI Better Exploration with Parameter Noise

    OpenAI introduces a method of adding adaptive noise to neural network parameters rather than the action space, significantly boosting reinforcement learning performance and exploration consistency.