fangvv/UAV-DDPG

Code for paper "Computation Offloading Optimization for UAV-assisted Mobile Edge Computing: A Deep Deterministic Policy Gradient Approach"

What it solves

This project addresses the problem of minimizing processing delays in UAV-assisted Mobile Edge Computing (MEC) systems. It optimizes how computing tasks are split between user equipment (UE) and a drone (UAV), while simultaneously managing the drone's flight path (angle and speed) and user scheduling to handle dynamic, non-convex environments.

How it works

The system uses a Deep Deterministic Policy Gradient (DDPG) reinforcement learning agent to manage a continuous action space. The agent observes the state of the environment—including UAV battery levels, positions of the UAV and users, and remaining task sizes—and decides on the target user, flight angle, flight distance, and the ratio of the task to be offloaded.

To ensure stability and convergence, the project implements:

  • Actor-Critic Architecture: An Actor network determines the action, while a Critic network evaluates the action's Q-value.
  • Experience Replay: Stores past transitions to break correlation in training data.
  • State Normalization: Scales raw input values (like battery Joules and coordinates) to a [0, 1] range to accelerate training.
  • Reward Function: The agent is rewarded based on the negative of the maximum processing delay, forcing it to minimize the worst-case delay across all users.

Who it’s for

Researchers and engineers working on wireless communication, drone-based edge computing, and the application of reinforcement learning to resource allocation in telecommunications.

Highlights

  • Joint Optimization: Simultaneously optimizes user scheduling, offloading ratios, and UAV flight parameters.
  • Continuous Action Space: Uses DDPG to handle continuous variables rather than discretizing them, which is more efficient than baseline methods like DQN.
  • Comprehensive Baselines: Includes implementations of DQN, Actor-Critic, and simple local/edge-only offloading strategies for performance comparison.
  • Ablation Studies: Provides specific versions of the code to test the impact of state normalization and exploration noise.

Related

  • Project
    fangvv/VN-MADDPG

    A multi-agent deep reinforcement learning implementation for optimizing spectrum sharing and resource allocation in connected vehicle networks to improve V2V and V2I link performance.

  • Dispatch
    OpenAI MADDPG: Learning to Cooperate, Compete, and Communicate

    OpenAI introduced MADDPG, a reinforcement learning algorithm that uses centralized learning and decentralized execution to enable multiple agents to collaborate and compete in complex environments.

  • Dispatch
    Deep Q-Learning with Space Invaders

    Hugging Face explains Deep Q-Learning (DQN), a reinforcement learning method that uses neural networks to approximate Q-values for high-dimensional state spaces like Atari games.

  • Dispatch
    OpenAI Better Exploration with Parameter Noise

    OpenAI introduces a method of adding adaptive noise to neural network parameters rather than the action space, significantly boosting reinforcement learning performance and exploration consistency.