Core Developer FTF 2018.09: Client access to DBRecord data

Working Draft, 02-Sept-2018

editors:
Marty Kraimer

This product is available via an open source license

Table of Contents

introduction

The EPICS core developers will have a Face To Face meeting in Lund Sweden.

This document presents some technical topics that I hope can be discussed.

All topics discuss client access to data in a DBRecord.

Since the very beginning of EPICS, channel access, both network protocol and API, provided client access to DBRecords. With V4 two new methods are available:

ca
A channel provider that uses the channel access network protocol but uses the pvAccess API,
qsrv
A server side channel provider that uses dbAccess to access DBRecord data. The pva network protocol is used for communication.

Using V4 terminology, data in a PVRecord consists of some combination of:

value
alarm
timeStamp
display
control
valueAlarm

Different types of clients want different combinations of the above.

The following discussion only refers to the C++ implementation. However any changes in ca provider should also be make in the Java implementation.

If anyone finds errors in how existing implementations are described below please speak up.

DBRecord data

Each field of a record, that can be accessed by clients, has one of the following types:

integer
Signed or unsigned integer of lenth 8, 16, 32, and 64 bits.
Array of any integer type.
float
IEEE32 or IEEE64 bit floating point.
Array of either float type.
enum
An unsigned 32 bit integer plus get_enum_strs record support.
string
A fixed length character string.
For the value field the length is MAX_STRING_SIZE.
An array of strings is supported but each element is a fixed length character string of size MAX_STRING_SIZE.

A client can access the value field of a record and some of other fields. The value field can always be read and modified. What a client can do with other fields depends on the dbd definition for the field.

For the value field clients can receive but not modify the following type of information:

DBRstatus    // all in dbCommon; no client support for getting directly
    epicsUInt16	status    // available to client via other means
    epicsUInt16	severity  // available to client via other means
    epicsUInt16	acks      // available to client only via channel.ACKS
    epicsUInt16	ackt      // available to client only via channel.ACKT
DBRunits                  // rset get_units
    char units[DB_UNITS_SIZE]
DBRprecision              // rset get_precision
    epicsInt32 precision
DBRtime                   // in dbCommon; no client support for getting directly
    epicsTimeStamp time   // available to client via other means
DBRenumStrs               // rset get_enum_strs
    epicsUInt32 no_str
    char strs[DB_MAX_CHOICES][MAX_STRING_SIZE];	/* string values    */
DBRgrLong                        //NOTE NO rset method
    epicsInt32  upper_disp_limit
    epicsInt32  lower_disp_limit
DBRgrDouble                      // rset get_graphic_double
     epicsFloat64 upper_disp_limit
     epicsFloat64 lower_disp_limit
DBRctrlLong                        //NOTE NO rset method
     epicsInt32 upper_ctrl_limit
     epicsInt32 lower_ctrl_limit
DBRctrlDouble                      // rset get_control_double
     epicsFloat64 upper_ctrl_limit
     epicsFloat64 lower_ctrl_limit
DBRalLong                        //NOTE NO rset method
     epicsInt32   upper_alarm_limit
     epicsInt32   upper_warning_limit
     epicsInt32   lower_warning_limit
     epicsInt32   lower_alarm_limit
DBRalDouble                      // rset get_alarm_double
     epicsFloat64  upper_alarm_limit
     epicsFloat64  upper_warning_limit
     epicsFloat64  lower_warning_limit
     epicsFloat64  lower_alarm_limit

The following are the rset methods related to client data access:

    long (*get_array_info)(struct dbAddr *paddr, long *no_elements, long *offset);
    long (*put_array_info)(struct dbAddr *paddr, long nNew);
    long (*get_units)(struct dbAddr *paddr, char *units);
    long (*get_precision)(const struct dbAddr *paddr, long *precision);
    long (*get_enum_str)(const struct dbAddr *paddr, char *pbuffer);
    long (*get_enum_strs)(const struct dbAddr *paddr, struct dbr_enumStrs *p);
    long (*put_enum_str)(const struct dbAddr *paddr, const char *pbuffer);
    long (*get_graphic_double)(struct dbAddr *paddr, struct dbr_grDouble *p);
    long (*get_control_double)(struct dbAddr *paddr, struct dbr_ctrlDouble *p);
    long (*get_alarm_double)(struct dbAddr *paddr, struct dbr_alDouble *p);

question about DBRgrLong, DBRctrlLong, and DBRalLong

These can not be accessed by client code. Are they ever used?

channel access API

Each field of a channel that can be accessed by a client has a chtype which is one of the following types:

integer
Signed integer of lenth 8, 16, and 32 bits.
Array of any integer type.
float
IEEE32 or IEEE64 bit floating point.
Array of either float type.
enum
An unsigned 32 bit integer plus DBR_CTRL_ENUM.
string
A fixed length character string.
For the value field the length is MAX_STRING_SIZE.
A array of strings is supported but each element is a fixed length character string of size MAX_STRING_SIZE.

For put channel access only allows the client to access the above types of data.

For get and monitor the client can access the following additional data:

DBR             one of the types above.
DBR_STATUS_*    adds status and severity to DBR.
DBR_TIME_*      adds epicsTimeStamp to DBR_STATUS.
DBR_GR_*        adds display and alarm limits to DBR_STATUS.
DBR_CTRL_*      adds control limits to DBR_GR.
DBR_CTRL_ENUM   Same info as DBRenumStrs from rset get_enum_strs 
NOTES:

In addition channel access defines a monitor event mask:

DBE_VALUE
Trigger an event when a significant change in the channel's value occurs. Relies on the monitor deadband field under DCT.
DBE_ARCHIVE (DBE_LOG)
Trigger an event when an archive significant change in the channel's value occurs. Relies on the archiver monitor deadband field under DCT.
DBE_ALARM
Trigger an event when the alarm state changes
DBE_PROPERTY
Trigger an event when a property change (control limit, graphical limit, status string, enum string ...) occurs.

A few comments about the monitor event mask:

default
If a mask is not specified the default is DBE_VALUE | DBE_ALARM.
DBE_VALUE
This will cause the most events.
DBE_PROPERTY
If only this is specified then events are normally rare.

When caqtDm connects to a channel it creates a monitor specifying DBR_CTRL_... and mask DBE_PROPERTY. When the callback for that request occurs, It then creates a monitor specifying DBR_STS_... and an empty mask. Thus for rare events it asks for a lot and for frequent events it asks only for value and alarm status and severity.

channel access and DBRecord

Code in epics-base converts between channel access data and the data in a DBRecord. Since these do not match some problems exist.

int64 and uint64

channel access does not support 64 bit integers. Will channel access clients be able to access any field in a DBRecord that is a 64 bit integer?

Two possibilities come to mind but both have problems:

double
The field type could appear to channel access as a DBR_DOUBLE. This would work for both scalar and array. But once the 64 bit value exceeds the number of bits it will loose some lower order bits.
Would this also require record support?
string
The field type could appear to channel access as a DBR_STRING. For scalar this can preserve all digits. For an array, the problem of how arrays of DBR_STRING are supported will limit the usefulness.
Would this also require record support?

unsigned integers

Since channel access does not support unsigned integers. The following is done:

uint8
The client can not determine if a DBR_CHAR signed or unsigned.
uint16
The catype appears as an int32.
uint32
The catype appears as a double.

DBR_STRING

Most DBR_STRING fields are limited to MAX_STRING_SIZE. One exception is field NAME. But unless something is done a channel access client can only access MAX_STRING_SIZE characters.

Clients have two ways to get longer strings but each relies on the client and server using the same convention. The two ways are:

int8 array
Normally a waveform record with type DBF_CHAR It has to be configured to hold enough bytes for how it is used. A client can get and put long strings by converting from/to byte arrays.
lsi and lso record types
For these record types the value field has catype DBR_STRING unless the client specifies channelName.VAL$. In this case the catype is a DBR_CHAR array.

miscellaneous issues

monitor event mask

Currently neither ca or qsrv has support for DBE_ARCHIVE. What to do?
Perhaps allow the following field option:

field(value[DBE=value])
value can be any combination of DBE_VALUE|DBE_ALARM|DBE_ARCHIVE|DBE_PROPERTY

alarm acknowledgement

Is this a problem for ca and for qsrv?
Is it handled by a client issuing gets and puts to fields ACKS and ACKT?

bitMask for standard fields

qsrv always sets the bit for each alarm, display, control, and valueAlarm field it gets from a DBRecord. ca only sets bits for alarm fields that have changed. It does not properly monitor changes in display, control, or valueAlarm.

Both qsrv and ca always set the bit for value. This seems like the correct semantics since DBRecords already have support for not raising monitors if a scalar value field does not change. It could be quite expensive to check for changes to array value fields.

alarm status

Both epics-base and pvdata have support code for alarms. The definition of status is different between them.

ca does not convert from the epics-base version to the pvdata version.
qsrv does convert but should epicsAlarmUDF be converted to undefinedAlarm?

Also should the pvdata version add support for converting between the two different versions?

DBRecord plugin support

Beginning with the 3.15 releases of epics-base channel filters are supported.

Using the channel access client API, the client requests plugins by appending to the channel name.
An example requesting the array plugin is:

caget test:channel test:channel.[3:5] test:channel.[3:2:-3]
 test:channel 10 0 1 2 3 4 5 6 7 8 9
 test:channel.[3:5] 3 3 4 5
 test:channel.[3:2:-3] 3 3 5 7

This works using the channel access client API and also for channel providers qsrv and ca.

Should any changes be made to support filters via pvRequest field options?
For qsrv and ca the answer is likely no.

standard fields

enum

enum_t value
    int index 0
    string[] 

Looks OK.

alarm

alarm_t alarm
    int severity
    int status
    string message       // not in any DBR types

Looks OK.

timeStamp

time_t timeStamp
    long secondsPastEpoch   // seconds since 0000 Jan 1, 1970 UTC
    int nanoseconds
    int userTag             // not in epicsTimeStamp

Looks OK.

Note that epicsTimeStamp is:

typedef struct epicsTimeStamp {
    epicsUInt32    secPastEpoch; /* seconds since 0000 Jan 1, 1990 UTC */
    epicsUInt32    nsec;         /* nanoseconds within second */
} epicsTimeStamp;

control

control_t control
    double limitLow
    double limitHigh
    double minStep       // not in any DBR type

Looks OK.

display

display_t display
    double limitLow
    double limitHigh
    string description   // not in any DBR type; field DESC is in dbCommon
    string format        // DBR types only define precision
    string units         // provided by DBRunits

format is a problem but what to do?
DBRecord has field PREC.

valueAlarm

valueAlarm_t valueAlarm
    boolean active           // not in any DBR type
    double lowAlarmLimit
    double lowWarningLimit
    double highWarningLimit
    double highAlarmLimit
    int lowAlarmSeverity     // not in any DBR type
    int lowWarningSeverity   // not in any DBR type
    int highWarningSeverity  // not in any DBR type
    int highAlarmSeverity    // not in any DBR type
    double hysteresis        // not in any DBR type
                             // recordTypes that support valueAlarm
                             // should have field HYST

The above is OK. But there is a version for each numeric type. control and display only have double for limit fields. Also record support only has a method for DBRalDouble
Perhaps valueAlarm should only be defined for double?

possible new support code for standard fields

enum,alarm, and timeStamp already have support code. Could some of the other types also have support code?
For example methods that check low and high limits for validity

pvRequest

Each create method of class Channel, in pvAccess, has an argument PVStructure pvRequest, i. e. it is an argument for createChannelGet, createChannelPut, etc.

A pvRequest structure allows a client to select:

an arbitrary set of fields.
When a client connects to channel, it receives a Structure that describes the PVStructure the channel supports. From the Structure, or just by anticipating what the server supports, the client can select the fields it wants to access.
global and field specific options
The client can pass global and field specific options to the server.
An option is just a name=value pair, where both name and value are a string.

The following provides a fuller description of

pvRequest

Should a better implementation of CreateRequest be implemented?

Andrew just wrote a EBNF description of the existing request syntax. Then he used a parser built into Base and pvData that turns JSON into a PVStructure. This was just a proof of concept but it did work.
Is he or someone else willing to finish the work and replace the existing implementation of CreateRequest.

Also is it possible to define:

optionValue = [A-Za-z_0-9?:|&]+ ;

record options

At present the following record options are supported:

queueSize

This is used to define the queueSize for monitors. The default is:

record[queueSize=2]

A larger size can be specified.

Who should honor this request?

client
Always Yes?
server
What if client specifies queueSize=100 for a channel that holds a gigabyte array?

Question: Is it possible to safely implement a queueSize of 0 or 1?

process

This is used to specify if records should be processed. The default is false for channelGet and true for channelPut and channelPutGet.
An example is:
record[process=false]

block

This is used to specify if a process request should block until the record completes processing. The default is false for channelPut and true for channelPutGet. An example is:
record[block=false]

field options

At the present time no standard field options have been specified.

ca provider

The primary goal for provider ca is to allow a client to use the pvAccess API instead of the channel access API.

The channel access API allows the client to pick any DBR type it wants and if the catype does not match the DBR type the server converts between the types.

With pva the client always works with pvData types that match the server types. Thus all conversions will be done on the client side.

Provider ca also follows this convention. Thus the pvData type will always be based on the catype.
Other than this ca should let the client do everthing the client can now do with the channel access API.

If the implementation can provide additional features then great!

Another goal is to do more to prevent deadlocks using the channel access API

The following presents some proposed changes.

monitor event mask

Currently ca has no support for a monitor event mask. What to do?
Perhaps allow the following field option:

field(value[DBE=value])
value can be any combination of DBE_VALUE|DBE_ALARM|DBE_ARCHIVE|DBE_PROPERTY

unsigned integer support

channel access has no support for unsigned integers. Perhaps the following field options could be provided:

field(value[dbtype=uint8])
field(value[dbtype=uint16])
field(value[dbtype=uint32])

Support for waveform, lsi and lso record types

These can be used to access strings longer than MAX_STRING_SIZE. Perhaps the following field option could be provided:

field(value[dbtype=string])  // For lsi, lso, waveform with catype char[]

Note that for lsi, lso client must specify VAL$.

DBR_CTL_... for channelGet

If a client creates a request that specifies any of display, control, or valueAlarm then a DBR_CTL_... requests is issued. But if the client also requested timeStamp there is a problem because DBR_CTL_... does not return an epicsTimeStamp.

A possible solution is to first issue a DBR_CTL_... request followed by a DBR_TIME_... request.

But maybe it is best to just document the restriction?

enum choices and description

Currently, if either is needed, a separate DBR request is made when the Structure introspection interface is created. This may cause a problem if a client only wants introspection. The separate DBR request should only be made when the first request is made to get data.

deadlock prevention

Changes have already been made so that client callbacks for get, put and monitor are called from a separate thread. This appears to fix a known channel access deadlock.

A separate thread should also be used for channel connect and state change callbacks.

ca provider and channel access simultaneous

Is this possible? Would be nice if it works.

Any Other???

qsrv provider

This is a major feature of V4. It provides the ability to access all data features of dbAccess and DBRecords. pvData has complete type support for all dbAccess data types.

Currently when a client connects to any field of a record, qsrv creates a PVStructure that contains all data that is valid for the value field of the record type. Get, put, and monitor all use the Structure introspection interface from this PVStructure. This means the implementation ignores the set of fields the client specifies in pvRequest.
This leads to significant memory usage: memoryusage

qsrv does not implement createChannelGet or createChannelProcess. Instead it uses the default implementation provided by pvAccess. The default implementation of createChannelGet uses the get method of ChannelPut.

For monitors qsrv is more efficent in regards to network usage than a channel access client issuing a DBR_CRL_... request.

When a monitor is created for a single channel, two event callbacks are created. One with an event mask of DBE_VALUE|DBE_ALARM, the other DBE_PROPERTY.

When the value or alarm changes the following is sent:

value
alarm.severity
alarm.status
alarm.message
timeStamp.secondsPastEpoch
timeStamp.nanoseconds

When a property field changes the following is sent.

timeStamp.secondsPastEpoch
timeStamp.nanoseconds
display.limitLow
display.limitHigh
display.description
display.format
display.units
control.limitLow
control.limitHigh
valueAlarm.lowAlarmLimit
valueAlarm.lowWarningLimit
valueAlarm.highWarningLimit
valueAlarm.highAlarmLimit

The following are some suggested changes to qsrv:

monitor event mask

Currently qsrv does not allow the client to specify a monitor event mask. What to do?
Perhaps allow the following field option:

field(value[DBE=value])
value can be any combination of DBE_VALUE|DBE_ALARM|DBE_ARCHIVE|DBE_PROPERTY

pvRequest

qsrv should honor the set of fields the client specifies in the pvRequest argument to createChannelGet, createChannelPut, and createMonitor.

local provider in pvDatabaseCPP

Look at "pvDatabase plugin support" in: pvRequest

Should the plugin support be implemented in pvDatabase?
Note that it has it's own version of pvCopy. pvCopy in pvData can be deprecated.