Skip to content

13 3 distributed training

Detailed Distributed Training Examples for xtorch

This document expands the "Performance and Distributed and Parallel Training -> Distributed Training" subcategory of examples for the xtorch library, a C++ deep learning framework that extends PyTorch’s LibTorch API with user-friendly abstractions. The goal is to provide a comprehensive set of beginner-to-intermediate examples that introduce users to distributed training techniques across multiple machines, with a focus on time series and graph models to align with the broader "Time Series and Graph" context. These examples showcase xtorch’s capabilities in scalability, performance, and C++ ecosystem integration, and are designed to be included in the xtorch-examples repository, helping users learn distributed training in C++.

Background and Context

xtorch simplifies deep learning for C++ developers by offering high-level model classes (e.g., XTModule, ResNetExtended, XTCNN), a streamlined training loop via the Trainer module, enhanced data utilities (e.g., ImageFolderDataset, CSVDataset, DataLoader), extended optimizers (e.g., xtorch::optim), and model serialization tools. The original distributed training example—setting up distributed training across machines—provides a solid foundation. This expansion adds seven more examples to cover additional distributed training strategies (e.g., synchronous, asynchronous, fault-tolerant, hybrid parallelism), model types (e.g., LSTM, GCN, GraphSAGE), and training scenarios (e.g., real-time training, large-scale graph processing), ensuring a broad introduction to distributed training with a focus on time series and graph applications.

The current time is 3:00 PM PDT on Monday, April 21, 2025, and all considerations are based on available information from the xtorch GitHub repository (xtorch GitHub Repository) without contradicting this timeframe.

Expanded Examples

The following table provides a detailed list of eight "Performance and Distributed and Parallel Training -> Distributed Training" examples, including the original one and seven new ones. Each example is designed to be standalone, with a clear focus on a specific distributed training concept or xtorch feature, making it accessible for users.

Category Subcategory Example Title Description
Performance and Distributed and Parallel Training Distributed Training Setting Up Distributed Training Across Machines Sets up synchronous distributed training across multiple machines to train a convolutional neural network (CNN) on the CIFAR-10 dataset (images) using xtorch’s xtorch::distributed and OpenMPI. Synchronizes gradients across machines, optimizes with SGD and cross-entropy loss, and evaluates with training speed (samples per second) and test accuracy.
Synchronous Distributed Training for Time Series Forecasting Implements synchronous distributed training across machines for an LSTM on the UCI Appliances Energy Prediction dataset using xtorch and OpenMPI. Distributes time series data and synchronizes gradients, optimizes with Adam and Mean Squared Error (MSE) loss, and evaluates with training speed (epochs per second) and generalization performance (Root Mean Squared Error, RMSE).
Asynchronous Distributed Training for Graph Node Classification Uses asynchronous distributed training across machines for a Graph Convolutional Network (GCN) on the Cora dataset (citation network) with xtorch and OpenMPI. Updates gradients asynchronously, optimizes with RMSprop and cross-entropy loss, and evaluates with training speed (batches per second) and classification accuracy.
Fault-Tolerant Distributed Training for Time Series Anomaly Detection Implements fault-tolerant distributed training across machines for an autoencoder on the PhysioNet ECG dataset (heart signals) using xtorch and OpenMPI. Includes checkpointing and recovery mechanisms, optimizes with Adagrad and MSE loss, and evaluates with training reliability (recovery success rate) and Area Under the ROC Curve (AUC-ROC).
Hybrid Distributed Training for Molecular Graph Property Prediction Combines data and model parallelism in distributed training across machines for a graph neural network on the QM9 dataset (small molecules) using xtorch and OpenMPI. Splits data and model layers, optimizes with Adam and Mean Absolute Error (MAE) loss, and evaluates with training speed (samples per second) and prediction accuracy (MAE).
Real-Time Distributed Training for Time Series Classification Implements distributed training across machines for real-time training of a CNN on a custom IoT sensor dataset (e.g., accelerometer data) using xtorch and OpenMPI. Synchronizes gradients for low-latency training, optimizes with Adam and cross-entropy loss, and evaluates with training latency (milliseconds per batch) and classification accuracy.
Large-Scale Graph Distributed Training for Node Embedding Uses synchronous distributed training across machines for a GraphSAGE model on the PPI dataset (protein interactions) with xtorch and OpenMPI. Distributes large-scale graph data, optimizes with Sparse Adam and unsupervised loss (e.g., graph reconstruction), and evaluates with training speed (epochs per second) and embedding quality (downstream classification accuracy).
Distributed Training with Visualization for Time Series Forecasting Combines synchronous distributed training across machines with OpenCV to train an LSTM for time series forecasting on streaming IoT sensor data (e.g., temperature readings). Visualizes training speed and loss curves across machines, optimizes with Adam and MSE loss, and evaluates with RMSE and visualization quality (clear plots).

Rationale for Each Example

  • Setting Up Distributed Training Across Machines: Introduces basic distributed training, using a CNN on CIFAR-10 to teach multi-machine synchronization, ideal for beginners.
  • Synchronous Distributed Training for Time Series Forecasting: Demonstrates synchronous distributed training for time series, using an LSTM on UCI data to teach scalable time series training, aligning with the time series focus.
  • Asynchronous Distributed Training for Graph Node Classification: Introduces asynchronous training for graphs, using a GCN on Cora to teach efficient graph training, aligning with the graph focus.
  • Fault-Tolerant Distributed Training for Time Series Anomaly Detection: Focuses on reliable distributed training, using an autoencoder on ECG data to teach fault tolerance, relevant for healthcare.
  • Hybrid Distributed Training for Molecular Graph Property Prediction: Demonstrates combined data and model parallelism, using a graph neural network on QM9 to teach complex graph training, relevant for cheminformatics.
  • Real-Time Distributed Training for Time Series Classification: Introduces real-time distributed training, using a CNN on IoT data to teach low-latency training, relevant for IoT applications.
  • Large-Scale Graph Distributed Training for Node Embedding: Shows scalable graph distributed training, using GraphSAGE on PPI to teach efficient large-scale training, relevant for big data applications.
  • Distributed Training with Visualization for Time Series Forecasting: Demonstrates visualization-integrated distributed training, using an LSTM on streaming IoT data to teach performance monitoring, relevant for IoT applications.

Implementation Details

Each example should be implemented as a standalone C++ program in the xtorch-examples repository, with the following structure: - Source Code: A main.cpp file containing the example code, using xtorch’s distributed utilities (e.g., xtorch::distributed for synchronous/asynchronous training), modules (e.g., xtorch::nn, xtorch::optim, xtorch::data::DataLoader), and, where applicable, OpenMPI for multi-machine parallelism, checkpointing for fault tolerance, and OpenCV for visualization. - Build Instructions: A CMakeLists.txt file to compile the example, linking against xtorch, LibTorch, OpenMPI, and OpenCV (if needed). - README.md: A detailed guide explaining the example’s purpose, prerequisites (e.g., LibTorch, dataset downloads, OpenMPI, OpenCV, multi-machine setup with GPUs), steps to run, and expected outputs (e.g., training speed, latency, accuracy, RMSE, MAE, AUC-ROC, recovery success rate, or visualization quality). - Dependencies: Ensure users have xtorch, LibTorch, datasets (e.g., CIFAR-10, UCI Appliances, Cora, PhysioNet ECG, QM9, PPI, custom IoT), and OpenMPI and optionally OpenCV installed, with download and setup instructions in each README. Multi-machine setups require appropriate hardware (e.g., cluster nodes with NVIDIA GPUs) and MPI configurations. Graph datasets may require custom utilities or integration with C++ graph libraries.

For example, the “Asynchronous Distributed Training for Graph Node Classification” might include: - Code: Train a GCN on the Cora dataset using asynchronous distributed training with xtorch::distributed and OpenMPI, optimize with xtorch::optim::RMSprop and cross-entropy loss, and output training speed and test accuracy, using xtorch’s modules and utilities. - Build: Use CMake to link against xtorch, LibTorch, and OpenMPI, specifying paths to Cora dataset. - README: Explain asynchronous distributed training for graph models, provide compilation and training commands for multi-machine setups, and show sample output (e.g., training speed of 140 batches/second, test accuracy of ~0.85).

Why These Examples?

These examples are designed to: - Cover Core Concepts: From synchronous and asynchronous distributed training to fault-tolerant, hybrid, and real-time distributed training, they introduce key distributed training paradigms for time series and graph applications. - Leverage xtorch’s Strengths: They highlight xtorch’s xtorch::distributed, xtorch::nn, xtorch::optim, and xtorch::data modules, as well as C++ performance, particularly for scalable and distributed training across machines. - Be Progressive: Examples start with simpler techniques (synchronous training) and progress to complex ones (asynchronous, fault-tolerant, hybrid parallelism), supporting a learning path. - Address Practical Needs: Techniques like fault-tolerant training, hybrid parallelism, and real-time training are widely used in real-world applications, from IoT to bioinformatics. - Encourage Exploration: Examples like visualization-integrated distributed training and large-scale graph training expose users to cutting-edge distributed training scenarios, fostering innovation.

Feasibility and Alignment with xtorch

The examples are feasible given xtorch’s features, as outlined in its GitHub repository: - Distributed Utilities: xtorch’s xtorch::distributed module (built on LibTorch’s distributed backend) supports synchronous and asynchronous training across machines, with OpenMPI integration for multi-node setups and checkpointing for fault tolerance. - Model Compatibility: xtorch::nn modules (e.g., Conv2d, LSTM, custom graph layers) support CNNs, LSTMs, GCNs, and GraphSAGE for time series and graph tasks. - Data Handling: xtorch::data::DataLoader and custom dataset classes handle image, time series, and graph datasets, with support for distributed data splitting and preprocessing (e.g., normalization, feature extraction). - Training Pipeline: The Trainer API simplifies distributed training loops, integrating with xtorch::distributed for synchronization or asynchronous updates, compatible with all examples. - Evaluation: xtorch’s utilities support metrics like training speed, latency, accuracy, RMSE, MAE, AUC-ROC, recovery success rate, and downstream task performance. - C++ Integration: xtorch’s compatibility with OpenMPI enables multi-machine parallelism, and OpenCV enables visualization of training metrics, enhancing user interaction.

The examples align with xtorch’s goal of simplifying deep learning in C++ and fit the "Time Series and Graph" context by emphasizing time series and graph distributed training, making them ideal for the xtorch-examples repository’s distributed training section.

Comparison with Existing Practices

Popular deep learning libraries like PyTorch provide distributed training tutorials, such as “Getting Started with Distributed Data Parallel in PyTorch” (PyTorch Tutorials), which cover Python-based distributed training. The proposed xtorch examples adapt this approach to C++, leveraging xtorch’s xtorch::distributed module and C++ performance. They also include time series and graph-specific distributed training (e.g., UCI, Cora, QM9) and advanced scenarios (e.g., asynchronous training, fault tolerance) to align with the category and modern distributed training trends, as seen in repositories like “pyg-team/pytorch_geometric” for graph model distributed training (GitHub - pyg-team/pytorch_geometric).

Implementation Notes

  • Directory Structure: Organize xtorch-examples with a performance_and_distributed_and_parallel_training/distributed_training/ directory, containing subdirectories for each example (e.g., distributed_cifar10/, synchronous_timeseries_uci/).
  • User Guidance: The main README.md should list all examples, suggest a learning path (e.g., start with synchronous training, then asynchronous, then fault-tolerant), and link to xtorch’s documentation.
  • C++ Focus: Ensure code uses modern C++ practices (e.g., smart pointers, exception handling) and includes detailed comments for clarity.
  • Dependencies: Note that users need LibTorch, xtorch, datasets (e.g., CIFAR-10, UCI Appliances, Cora, PhysioNet ECG, QM9, PPI, custom IoT), and OpenMPI and optionally OpenCV installed, with download and setup instructions in each README. Multi-machine setups require appropriate hardware (e.g., cluster nodes with NVIDIA GPUs) and MPI configurations. Graph datasets may require custom utilities or integration with C++ graph libraries.

Conclusion

The expanded list of eight "Performance and Distributed and Parallel Training -> Distributed Training" examples provides a comprehensive introduction to distributed training techniques with xtorch, covering synchronous training, asynchronous training, fault-tolerant training, hybrid parallelism, real-time classification, large-scale graph embedding, and visualization-integrated distributed training. These examples are beginner-to-intermediate friendly, leverage xtorch’s strengths, and align with its goal of making deep learning accessible in C++ while addressing time series and graph applications. By including them in xtorch-examples, you can help users build a solid foundation in distributed training across machines, fostering adoption and engagement with the xtorch community.

Key Citations