Nomad has built-in support for scheduling compute resources such as CPU, memory, and networking. Nomad device plugins are used to support scheduling tasks with other devices, such as GPUs. They are responsible for fingerprinting these devices and working with the Nomad client to make them available to assigned tasks.
For a real world example of a Nomad device plugin implementation, see the Nvidia GPU plugin.
Authoring Device Plugins
Authoring a device plugin in Nomad consists of implementing the DevicePlugin interface alongside a main package to launch the plugin.
The device plugin skeleton project exists to help bootstrap the development of new device plugins. It provides most of the boilerplate necessary for a device plugin, along with detailed comments.
Lifecycle and State
A device plugin is long-lived. Nomad will ensure that one instance of the plugin is running. If the plugin crashes or otherwise terminates, Nomad will launch another instance of it.
However, unlike task drivers, device plugins do not currently have an interface for persisting state to the Nomad client. Instead, the device plugin API emphasizes fingerprinting devices and reporting their status. After helping to provision a task with a scheduled device, a device plugin does not have any responsibility (or ability) to monitor the task.
Device Plugin API
The base plugin must be implemented in addition to the following functions.
Fingerprint(context.Context) (<-chan *FingerprintResponse, error)
Fingerprint function is called by the client when the plugin is started.
It allows the plugin to provide Nomad with a list of discovered devices, along with their
attributes, for the purpose of scheduling workloads using devices.
The channel returned should immediately send an initial
FingerprintResponse, then send periodic updates at
an appropriate interval until the context is canceled.
Each fingerprint response consists of either an error or a list of device groups. A device group is a list of detected devices that are identical for the purpose of scheduling; that is, they will have identical attributes.
Stats(context.Context, time.Duration) (<-chan *StatsResponse, error)
Stats function returns a channel on which the plugin should
emit device statistics, at the specified interval, until either an error is
encountered or the specified context is cancelled. The
allows dimensioned statistics to be returned for each device in a device group.
Reserve(deviceIDs string) (*ContainerReservation, error)
Reserve function accepts a list of device IDs and returns the information
necessary for the client to make those devices available to a task. Currently,
ContainerReservation object allows the plugin to specify environment
variables for the task, as well as a list of host devices and files to be mounted
into the task's filesystem. Any orchestration required to prepare the device for
use should also be performed in this function.