Programming with DaCS

DaCS API functions

The DaCS library API services are provided as functions in the C language. The protocols and constants required are made available to the compiler by including the DaCS header file dacs.h as:

#include <dacs.h>

In general the return value from these functions is an error code (see Error codes). Data is returned within parameters passed to the functions.

Note: Implementations may provide options, restrictions and error codes that are not specified here.

Note: When more than one error condition is present it is not guaranteed which one will be reported.

Process management model

When working with the host and accelerators there has to be a way to uniquely identify the participants that are communicating. From an architectural perspective, each accelerator could have multiple processes simultaneously running, so it is not enough simply to identify the accelerator. Instead the unit of execution on the accelerator (the DaCS Process) must be identified using its DaCS Element Id (DE id) and its Process Id (Pid). The DE Id is retrieved when the accelerator is reserved (using dacs_reserve_children()) and the Pid when the process is started (using dacs_de_start()). Since the parent is not reserved, and no process is started on it, two constants are provided to identify the parent: DACS_DE_PARENT and DACS_PID_PARENT. Similarly, to identify the calling process itself, the constants DACS_DE_SELF and DACS_PID_SELF are provided.

Resource sharing model

The APIs supporting the locking primitives, remote memory access and groups follow a consistent pattern of creation, sharing, usage and destruction:

Creation: An object is created which will be shared with other DEs, for example with dacs_remote_mem_create().
Sharing: The object created is then shared by linked share and accept calls. The creator shares the item (for instance with dacs_remote_mem_share()), and the DE it is shared with accepts it (in this example with dacs_remote_mem_accept()). These calls must be paired. When one is invoked it waits for the other to occur. This is done for each DE the share is actioned with.
Usage: This may require closure (such as in the case of groups) or the object may immediately be available for use. For instance remote memory can immediately be used for put and get.
Destruction: The DEs that have accepted an item can release the item when they are done with it (for example by calling dacs_remote_mem_release()). The release does not block, but notifies the creator that it is not longer being used and cleans up any local storage. The creator does a destroy (in this case dacs_remote_mem_destroy()) which will wait for all of the DEs it has shared with to release the item, and then destroy the shared item.

API environment

To make these services accessible to the runtime code each process must create a DaCS environment. This is done by calling the special initialization service dacs_runtime_init(). When this service returns the environment is set up so that all other DaCS function calls can be invoked.

When the DaCS environment is no longer required the process must call dacs_runtime_exit() to free all resources used by the environment.

Building a DaCS application

Three versions of the DaCS libraries are provided with the DaCS packages: optimized, debug and traced. The optimized libraries have minimal error checking and are intended for production use. The debug libraries have much more error checking than the optimized libraries and are intended to be used during application development. The traced libraries are the optimized libraries with performance and debug trace hooks in them. These are intended to be used to debug functional and performance problems that might be encountered. The traced libraries use the interfaces provided by the Performance Debug Tool (PDT) and require that this tool be installed. See Performance and debug trace for more information on configuring and using traced libraries, and DaCS debugging for debug libraries.

Both static and shared libraries are provided for the x86_64 and PPU. The desired library is selected by linking to the chosen library in the appropriate path. The static library is named libdacs.a, and the shared library is libdacs.so.The locations of these libraries are the same on both the x86_64 and PPU:

Table 1.
Description	Library Path
Optimized	/prototype/usr/lib64
Debug	/prototype/usr/lib64/dacs/debug
Traced	/prototype/usr/lib64/dacs/trace

Affinity requirements for host applications

A hybrid DaCS application on the host (x86_64) must have processor affinity to start. This can be done:

on the command line,
in mpirun, or
through the sched_setaffinity function.

Here is a command line example to set affinity of the shell to the first processor:

taskset -p 0x00000001  $$

The bit mask, starting with 0 from right to left, is an index to the processor affinity setting. Bit 0 is on or off for CPU 0, bit 1 for CPU 1, and bit number x is CPU number x. $$ means the current process gets the affinity setting.

taskset -p $$

will return the mask setting as an integer. Using the -c option makes the taskset more usable. For example,

taskset -pc 7  $$

will set the processor CPU affinity to CPU 7 for the current process. The -pc parameter sets by process and CPU number.

taskset -pc  $$

will return the current CPU setting for affinity for the current process. According to the man page for taskset a user must have CAP_SYS_NICE permission to change CPU affinity. See the man page for taskset for more details.

To launch a DaCS application use a taskset call, for example:

taskset 0x00000001  HelloDaCSApp Mike

or equivalently

taskset -c 0 HelloDaCSApp Mike

where the application program is HelloDaCSApp and is passed an argument of "Mike".

On the accelerator system the adacsd launch of an AE application on a specific CBE includes setting the affinity to the CBE and its associated memory node. If the launch is on the Cell Blade as an AE no affinity is set.