Automatic Cloud I/O Configurator for Parallel Applications

ACIC is a system which automatically searches for optimized I/O system configurations from many candidates for each individual HPC application running on a given cloud platform.

This is a joint work of Tsinghua University and North Carolina State University. We release the ACIC here as a community tool for free and hope it will help cloud users with I/O sub-system configuration.

ACIC's Architecture



Using ACIC

There are three steps to use ACIC to help you configuring the I/O sub-system in the cloud:

  • Collect the I/O characteristics of your HPC application with tools like HPCToolkit, mpiP etc. We also wrote a simple profiling tool named profio to automatically obtain them.
  • Build the CART model with training data. We released our bootstrap training dataset (Download!) for free, and all users are welcome to get involved to contribute more data! (How to Get Involved?)
  • Predict the optimized configurations for your application with Salford Systems CART.

INPUT: Top I/O Parameters of Your Application

Name Description Value Range * Comment
CLIENTS Number of I/O Processes 32 ~ 256
API I/O Interface {POSIX, MPI-IO}
DATA_SIZE Size of Each Block 1MB ~ 512MB
REQUEST_SIZE Transfer Size of Each I/O Request 256KB ~ 128MB
R/W Read and/or Write {read, write}
COLLECTIVE Multi-processes Read/write Collectively {True, False} Not included in bootstrap training
SEGMENT Number of I/O Iterations 1 ~ 100 Not included in bootstrap training
* Larger range will be supported as more training data are collected.

OUTPUT: What Can I Configure?

Name Description Options
FS File System {NFS, PVFS}
SSIZE Stripe Size {64KB, 4MB}
PLACEMENT The I/O Server Placement {parttime, dedicated}
IO_SERVERS I/O Server Number 1 ~ 4
DEVICE Disk Device Type {EBS, Ephemeral}

Publications

  • Jidong Zhai, Mingliang Liu, Ye Jin, Xiaosong Ma and Wenguang Chen. Automatic Cloud I/O Configurator for Parallel Applications. To appear in TPDS. 2014.
  • Mingliang Liu, Ye Jin, Jidong Zhai, Yan Zhai, Qianqian Shi, Xiaosong Ma and Wenguang Chen. ACIC: Automatic Cloud I/O Configurator for HPC Applications. In SC. 2013.
  • Mingliang Liu, Ye Jin, Jidong Zhai, Yan Zhai, Qianqian Shi, Xiaosong Ma and Wenguang Chen. Automatic Cloud I/O Configurator for Parallel Applications". In HPDC Poster Session. 2013. (Poster)
  • Mingliang Liu, Jidong Zhai, Yan Zhai, Xiaosong Ma and Wenguang Chen. One Optimized I/O Configuration per HPC Application: Leveraging the Configurability of Cloud. In APSys. 2011. (Paper | Slides)

Contacts

Get Involved

Due to time/funding constraints, we did not perform more training than the top 10 dimensions. However, more training data will both improve ACIC’s prediction accuracy and extend the coverage of cloud I/O sub-system’s configurations. Notice that it is possible to obtain additional training data without any extra monetary cost. This is because public clouds, like Amazon EC2, typically charge users at a hourly billing granularity, which enabling users to fit one or more short IOR training runs into the residual time of the last hour.

Users can download our scripts for running IOR, collecting and parsing training data in Amazon EC2 here!
Please send us the training data you collected to make ACIC even better!