Building a Scalable VPS Management System with Proxmox

As part of the Hemiron project, I took on the role of infrastructure engineer responsible for designing and implementing a robust virtual private server (VPS) management system using Proxmox Virtual Environment. This article details the architecture, implementation challenges, and lessons learned while creating a sophisticated management layer that bridged infrastructure and application teams.

Project Context and Requirements

The Hemiron project required a flexible and scalable infrastructure that could support multiple development teams while maintaining high availability, security, and operational efficiency. Key requirements included:

1. Automated provisioning: The ability to rapidly create, configure, and deploy virtual machines through API calls
2. Resource optimization: Efficient allocation and management of compute, storage, and network resources
3. High availability: Fault-tolerant infrastructure with seamless migration capabilities
4. Security isolation: Strong separation between customer environments
5. Monitoring and management: Comprehensive visibility and control of the infrastructure
6. Self-service capabilities: Developer-friendly interfaces for resource management

Architecture Overview

The solution I developed was a multi-layered system with Proxmox VE as the core virtualization platform, extended with a custom API layer, automation tools, and monitoring systems:

┌─────────────────────────────────────────────────────────────┐
│                   Custom Management Portal                  │
└───────────────────────────┬─────────────────────────────────┘
                            │
┌───────────────────────────▼─────────────────────────────────┐
│                   Custom API & Control Layer                │
└───────────────────────────┬─────────────────────────────────┘
                            │
┌───────────────────────────▼─────────────────────────────────┐
│           Infrastructure Automation (Terraform/Ansible)     │
└───────────────────────────┬─────────────────────────────────┘
                            │
┌───────────────────────────▼─────────────────────────────────┐
│                     Proxmox Cluster API                     │
└───────────────────────────┬─────────────────────────────────┘
                            │
┌──────────────┬────────────┴───────────────┬─────────────────┐
│  Proxmox     │      Proxmox               │     Proxmox     │
│  Node 1      │      Node 2                │     Node 3      │
│  ┌─────────┐ │      ┌─────────┐           │     ┌─────────┐ │
│  │ VM      │ │      │ VM      │           │     │ VM      │ │
│  └─────────┘ │      └─────────┘           │     └─────────┘ │
│  ┌─────────┐ │      ┌─────────┐           │     ┌─────────┐ │
│  │ VM      │ │      │ VM      │           │     │ VM      │ │
│  └─────────┘ │      └─────────┘           │     └─────────┘ │
│  ┌─────────┐ │      ┌─────────┐           │     ┌─────────┐ │
│  │ CT      │ │      │ CT      │           │     │ CT      │ │
│  └─────────┘ │      └─────────┘           │     └─────────┘ │
└──────────────┴────────────────────────────┴─────────────────┘
               │                           │
┌──────────────▼──────────────┬───────────▼─────────────────┐
│        Ceph Storage         │        Shared Storage       │
└─────────────────────────────┴─────────────────────────────┘

Bare Metal Configuration and Cluster Setup

Hardware Selection and Configuration

The first challenge was configuring the physical infrastructure to support our virtualization needs:

• Server hardware: Selected high-performance servers with redundant components
• Network configuration: Implemented multiple network interfaces for segregated traffic (management, storage, VM traffic)
• Storage architecture: Deployed Ceph for distributed storage with replication for data protection

Storage Configuration with Ceph

Ceph provided the distributed storage foundation for the infrastructure:

# Example of Ceph pool configuration
ceph osd pool create vm-disks 128 128
ceph osd pool set vm-disks size 3
ceph osd pool application enable vm-disks rbd

# Create RBD storage in Proxmox
pvesm add rbd vm-storage --pool vm-disks --monhost 10.0.0.1,10.0.0.2,10.0.0.3 --content images,rootdir

This configuration provided:

• Replicated storage across physical nodes
• Self-healing capabilities
• Performance optimizations for virtual machine workloads
• Snapshots and backup support

Custom API Development

While Proxmox provides its own API, we needed additional functionality and a more abstracted interface for our development teams.

Key API Features

The custom API provided several advantages over the raw Proxmox API:

1. Role-based access control: Fine-grained permissions based on user roles
2. Resource quotas: Enforced limits on resource consumption per team/project
3. Template management: Standardized VM templates with pre-installed software
4. Lifecycle hooks: Ability to trigger actions at various points in VM lifecycle
5. Advanced scheduling: Intelligent placement of VMs based on resource availability
6. API versioning: Support for multiple API versions for backwards compatibility

Backend Services

Behind the API, several services managed different aspects of the infrastructure:

• Resource Manager: Tracked and allocated resources across the cluster
• Template Service: Maintained and deployed VM templates
• Backup Manager: Scheduled and verified VM backups
• Health Monitor: Proactively checked system health
• Event Processor: Handled asynchronous operations

Infrastructure as Code

The entire infrastructure was managed using Infrastructure as Code principles:

Conclusion

Building a comprehensive VPS management system on Proxmox was a complex but rewarding challenge. By focusing on automation, abstraction, and clear communication between teams, we created an infrastructure that supported the needs of a growing application while maintaining high levels of reliability and efficiency.

The project demonstrates the value of treating infrastructure as a product with its own API, documentation, and user experience considerations. This approach bridges the traditional gap between infrastructure and application teams, enabling faster development cycles and more reliable systems.

For more information about the Hemiron project, including our zero-downtime deployment system, feel free to explore other articles or contact me through the details on my About page.