The qthreads API is designed to make using large numbers of lightweight threads convenient and easy, and to allow portable access to threading constructs used in massively parallel shared memory environments. The API maps well to both MTA-style threading and PIM-style threading, and is still quite useful in a standard SMP context. The qthreads API provides access to full/empty-bit (FEB) semantics, where every word of memory can be marked either full or empty, and a thread can wait for any word to attain either state.
The qthreads library on an SMP (i.e. the POSIX implementation) is essentially a library for spawning and controlling coroutines: threads with small O(4k) stacks. The threads are entirely in user-space and use their blocked/unblocked status as part of their scheduling. The library’s metaphor is that there are many qthreads and several “shepherds”. Shepherds can be thought of as a thread mobility domain; they map to specific processors or memory regions. Qthreads are assigned to specific shepherds and do not migrate unless directed to migrate or if load balancing via work stealing is enabled.
The API includes utility functions for making threaded loops, sorting, and similar operations convenient.
The qthreads library was developed to explore innovations in highly concurrent systems where the ultimate system either does not exist, or is sufficiently hard to obtain that development of software for the system becomes difficult.
Development and releases are currently hosted on GitHub: https://github.com/sandialabs/qthreads.
Documentation is included in the repository itself and in the releases, including man pages that install with the distribution, and a user guide that can be compiled using LaTex.
Platforms and Requirements
| Architecture | Linux | Mac OS X | Solaris | SST |
| x86 64-bit | ✓ | ✓ | ||
| Arm 64-bit | New | New | ||
| PPC 64-bit | ✓ | ✓ | ||
| x86 32-bit | ✓ | ✓ | ||
| Arm 32-bit | ✓ | |||
| PPC 32-bit | ✓ | ✓ | ✓ | |
| Sparc V9+ | ✓ | |||
| Tilera (MIPS) | ✓ |
Build Requirements
To compile and run Qthreads over POSIX you will require:
- A UNIX-like shell (Qthreads uses the GNU Autotools)
- GNU Make
- C Compiler (Prior to 1.5 requires either C++ or the cprops library)
Publications
To cite Qthreads:
- Qthreads: An API for Programming with Millions of Lightweight Threads. Kyle Wheeler, Richard Murphy, Douglas Thain. In Proceedings of the 22nd IEEE International Parallel & Distributed Processing Symposium Workshops (IPDPS ’08, in the MTAAP ’08 workshop), IEEE Press, 2008.
Related Papers:
- Scheduling Chapel Tasks with Qthreads on Manycore: A Tale of Two Schedulers. Noah Evans, Stephen Olivier, Richard Barrett, George, Stelle. In Proceedings of the 7th International Workshop on Runtime and Operating Systems for Supercomputers (ROSS 2017), ACM Press, 2017.
- Kokkos/Qthreads task-parallel approach to linear algebra based graph analytics. Michael Wolf, H. Carter Edwards and Stephen Olivier. In Proceedings of the 2016 IEEE High Performance Extreme Computing Conference (HPEC 2016), IEEE Press, 2016.
- Early Experiences Co-Scheduling Work and Communication Tasks for Hybrid MPI+X Applications. D. T. Stark, R. F. Barrett, R. E. Grant, S. L. Olivier, K. T. Pedretti and C. T. Vaughan. In 2014 Workshop on Exascale MPI (ExaMPI 2014) at SC14, IEEE Press, 2014.
- Scheduling Task Parallelism on Multi-Socket Multicore Systems. Stephen Olivier, Allan Porterfield, Kyle Wheeler, and Jan Prins. In Proceedings of the 25th International Conference on Supercomputing (ICS’11, in the ROSS’11 workshop), ACM Press, 2011.
- Implementing a Portable Multi-threaded Graph Library: the MTGL on Qthreads. Brian Barrett, Jonathan Berry, Richard Murphy, Kyle Wheeler. In Proceedings of the 23rd IEEE International Parallel & Distributed Processing Symposium (IPDPS ’09, in the MTAAP ’09 workshop), IEEE Press, 2009.