Overview

DaCS for Hybrid (DaCSH) is an implementation of the DaCS API specification which supports the connection of an HE on an x86_64 system to one or more AEs on Cell Broadband Engines (CBEs). In SDK 3.0, DaCSH only supports the use of sockets to connect the HE with the AEs. Direct access to the Synergistic Processor Elements (SPEs) on the CBE is not provided. Instead DaCSH provides access to the PowerPC^® Processor Element (PPE), allowing a PPE program to be started and stopped and allowing data transfer between the x86_64 system and the PPE. The SPEs can only be used by the program running on the PPE.

A PPE program that works with the SPEs can also be a DaCS program. In this case the program will use DaCS for Cell (DaCSC - see DaCS Programmer's Guide and API Reference for Cell BE); the PPE will act as an AE for DaCSH (communicating with the x86_64 system) and as an HE for DaCSC (communicating with the SPEs). The DaCS API on the PPE is supported by a combined library which, when the PPE is being used with both DaCSH and DaCSC, will automatically use the parameters passed to the API to determine if the PPE is an AE talking to its HE (DaCSH) or an HE talking to its AEs (DaCSC).

In order to manage the interactions between the HE and the AEs DaCSH starts a service on each of them. On the host system the service is the Host DaCS daemon (hdacsd) and on the accelerator the service is the Accelerator DaCS daemon (adacsd). These services are shared between all DaCSH processes for an operating system image. For example, if the x86_64 system has multiple cores that each run a host application using DaCSH, only a single instance of the hdacsd service is needed to manage the interactions of each of the host applications with their AEs via DaCSH. Similarly, on the accelerator, if the CBE is on a Cell Blade (which has two CBEs), a single instance of the adacsd service is needed to managed both of the CBEs acting as AEs, even if they are used by different HEs.

When a host application starts using DaCSH this connects to the hdacsd service. This service manages the system topology from a DaCS perspective (managing reservations) and starts the accelerator application on the AE. Only process management requests will use the hdacsd and adacsd services. All other interactions between the host and accelerator application will flow via a direct socket connection.

The following diagram provides a summary of the DaCS daemons and their relationships:

where:

• the host application (which uses DaCS) is assumed to be started outside of the scope of DaCS;
• the broad arrow represents the flow of data (since only process management requests are handled by the hdacsd and adacsd services);
• the lines marked 'socket' are control socket connections (DaCSd will be a service that is called via ports reserved for DaCS usage. The connections that will be created are between the hdacsd and the adacsd, and between the adacsd and the host application, for stderr and stdout redirection.);
• more than one element (host and accelerator) is shown, as from the perspective of the DaCS services it is shared at an OS installation level (multiple host applications could be using the hdacsd, and multiple accelerator applications could be using the adacsd);
• The stderr and stdout streams of an AE application on the accelerator system are redirected to the HE application.
• For DaCSH the configuration information used for topology management will be retrieved from the file system on the Host Element.
• The hdacsd and adacsd can be configured using configurations files from the file system.
• The DaCSH daemons are user space programs, and are not privileged kernel modules.

User Ids

The accelerator application will be started using the user id of the host application. These user ids must already be setup on the accelerator.

DaCS for Hybrid can coexist and work with DaCS for Cell. To use them both you must install both. For instructions on installing DaCS for Cell see the SDK 3.0 Installation Guide.

When both are installed the application using DaCS on the PPU can be both an AE for the x86_64 HE and an HE for the SPU AEs. The DaCS library automatically directs the requests appropriately based on the passed parameters. For example a dacs_send() is passed the DE Id and Pid of the destination to send the message to. This is used by the DaCS library to determine if it should be sent to the x86_64 HE using DaCS for Hybrid or to a SPU AE using DaCS for Cell.

All three (x86_64, PPU and SPU) can work together with remote memory and mutexes. When these are created on the PPU they are created in a way that allow them to be used with both the x86_64 HE and the SPU AE. In this way processes on all three can be synchronized using a mutex or can share remote memory. Note, however, that this must be initiated on the PPU (which is shared between the two). It is not possible for the x86_64 HE to create the mutex and share it with the PPU AE and then for the PPU (as an HE) to share it with its SPU AEs. This does not work as a mutex can only be shared by the DE that created it - the x86_64 DE cannot see the SPU DE.

The DaCS services can be divided into the following categories:

Resource reservation

The resource reservation services allow an HE to reserve AEs below itself in the hierarchy. The APIs abstract the specifics of the reservation system (O/S, middleware, etc.) to allocate resources for an HE. Once reserved, the AEs can be used by the HE to execute tasks for accelerated applications.

Process management

The process management services provide the means for an HE to execute and manage accelerated applications on AEs, including, but not limited to, remote process launch and remote error notification.

The host system DaCSd (hdacsd) provides services to the HE applications. The accelerator DaCSD (adacsd) provides services to the hdacsd and HE application, including the launching of the AE applications on the accelerator for the HE applications.

Group management

The group management services provide the means to designate dynamic groups of processes for participation in collective operations. In SDK 3.0 this is limited to process execution synchronization (barrier).

Remote memory

The remote memory services provide the means to create, share, transfer data to, and transfer data from a remote memory segment. The data transfers are performed using a one-sided put/get remote direct memory access (rDMA) model. These services also provide the ability to scatter/gather lists of data, and provide optional enforcement of ordering for the data transfers.

Message passing

The message passing services provide the means for passing messages asynchronously, using a two-sided send/receive model. Messages are passed point-to-point from one process to another.

Mailboxes

The mailbox services provide a simple interface for synchronous transfer of small (32-bit) messages from one process to another.

Process Synchronization

The process synchronization services provide the means to coordinate or synchronize process execution. In SDK 3.0 this is limited to the barrier synchronization primitive.

Data Synchronization

The data synchronization services provide the means to synchronize and serialize data access. These include management of wait identifiers for synchronizing data transfers, as well as mutex primitives for data serialization.

Error Handling

The error handling services enable the user to register error handlers and gather error information.