Accounting Service

Fit in NEANIAS Ecosystem

The NEANIAS Accounting service provides an aggregation functionality through which NEANIAS services can centrally log accounting information as it is gradually accumulated by their usage from the respective authorized clients. Different accumulation policies and granularity of information is supported depending on the respective services. The information consists of globally defined key point indicators and the aggregation and reports generated based on these becomes available for further usage, as appropriate

Accounting Transfer

To facilitate the aggregation of log, the Beats (https://www.elastic.co/beats/) framework for data shipping will be used. Specifically, beats modules for either file or http transfer can be provided with supportive configuration to lower the integration barrier for all NEANIAS Service Providers.

Accounting information

Acounting Templates

Templates for acounting entry formats are provided along with the respective Logstash transformation templates that will extract a limited common set of information. This way, a homogenized set of information is made available for accounting operations.

Currently supporting templates include:

• JSON accounting based

Examples of the currently supported log formats are:

JSON

{
  "timestamp": "2020-09-01T13:00:00Z", //required
  "serviceid": "ServiceId", //required
  "level": "accounting", //required value should be accounting
  "starttime": "2020-09-01T13:00:00Z", //optional
  "endtime": "2020-09-01T13:00:00Z", //optional
  "userid": "UserId", //required
  "userdelegate": "UserDelegate", //optional
  "resource": "Resource", //required
  "action": "Action", //required
  "comment": "comment", //optional
  "value": 5.1, //optional if ommited 1,
  "measure": "information" // time(ms), information(mb), throughput(mb/sec), unit, others to be considered, optional if ommited unit
  "type": "+" // + - 0, optional if ommited +
}

Accounting Model

timestamp DateTime (ISO 8601) Mandatory Date of the entry

starttime DateTime (ISO 8601) Date of the operation starts

endtime DateTime (ISO 8601) Date of the operation ends

serviceid Text Mandatory Global unique identifier for the service that the log entry refers to. It may be generated by the probe rather than be harvested from the logs.

level ENUM Mandatory Value should be “accounting”

resource Text Mandatory The resource handled by the call.

action Text Mandatory The action identifier in internal semantics.

userid Text Mandatory An identifier for the user on whos purpose the work is performed

userdelegate Text An identifier for the user who is the delegate to perform the work

value Long The measure value if omitted 1

measure Text The measure currently available messures time(ms), information(mb), throughput(mb/sec), unit, others to be considered, optional if omitted unit

type ENUM The measure type available types + - 0, optional if omitted +

comment Text A comment of accounting

Log Exploration

A Kibana dashboard is provided that offers browsing, filtering and visualization capabilities. Access to the Kibana graphical application is subject to authorization enforced by relevant access management plugins. Integration with the NEANIAS AAI is evaluated and planned in order to facilitate some degree of global policy enforcement through the central AAI service

Log Storing

A logstash service with beat input is available for pushing the accounting data loggingaggregator.dev.neanias.eu:31314. Clients can use filebeat for crawling and pushing accounting data

Filebeat configuration example:

1. Tag accounting should be set for recognition of accounting format.
2. The cerificates should be set for authentication

filebeat.inputs:
- type: log
  paths:
    - /usr/share/filebeat/log_data/*.json
  tags: ["accounting"]
  enabled: true
  reload.enabled: true
  reload.period: 10s
  processors:
  - drop_event.when.not.regexp.message: '"[lL]evel"\s*:\s*"accounting"'

output.logstash:
  hosts: ["loggingaggregator.dev.neanias.eu:31314"]
  bulk_max_size: 10
  ssl.certificate_authorities: ["/etc/filebeat/ca/root-ca.pem"]
  ssl.certificate: "/etc/filebeat/certificates/filebeat01.pem"
  ssl.key: "/etc/filebeat/certificates/filebeat01-key.pem"