diff --git a/config/config_defaults.go b/config/config_defaults.go index a7fa35f88..11731f5c6 100755 --- a/config/config_defaults.go +++ b/config/config_defaults.go @@ -280,7 +280,7 @@ const CGRATES_CFG_JSON = ` "attempts": 1, // export attempts "field_separator": ",", // used field separator in some export formats, eg: *file_csv "attributes_context": "", // attributes context - empty disables attributes processing - "fields": [ // template of the exported content fields + "fields": [ // template of the exported content fields {"path": "*exp.CGRID", "type": "*composed", "value": "~*req.CGRID"}, {"path": "*exp.RunID", "type": "*composed", "value": "~*req.RunID"}, {"path": "*exp.ToR", "type": "*composed", "value": "~*req.ToR"}, diff --git a/docs/cdre.rst b/docs/cdre.rst index 70e1bf9f7..40ee97cbf 100644 --- a/docs/cdre.rst +++ b/docs/cdre.rst @@ -1,5 +1,251 @@ +.. _AMQP: https://www.amqp.org/ +.. _SQS: https://aws.amazon.com/de/sqs/ +.. _S3: https://aws.amazon.com/de/s3/ +.. _Kafka: https://kafka.apache.org/ + + +.. _CDRe: + CDRe ==== -TBD \ No newline at end of file +**CDRe** is an extension of :ref:`CDRs`, responsible for exporting the *CDR* events processed by :ref:`CDRs`. It is accessed via :ref:`CGRateS RPC APIs` and configured within *cdre* section inside :ref:`JSON configuration `. + + +Export types +------------ + +There are two types of exports with common configuration but different data sources: + + +Online exports +^^^^^^^^^^^^^^ + +Are real-time exports, triggered by the CDR event processed by :ref:`CDRs`, and take these events as data source. + +The *online exports* are enabled via *online_cdr_exports* :ref:`JSON configuration ` option within *cdrs*. + +You can control the templates which are to be executed via the filters which are applied for each export template individually. + + +Offline exports +^^^^^^^^^^^^^^^ + +Are exports which are triggered via :ref:`CGRateS RPC APIs` and they have as data source the CDRs stored within *StorDB*. + + + +Parameters +---------- + +CDRe +^^^^ + +**CDRe** it is configured within **cdre** section from :ref:`JSON configuration `. + +One **export profile** includes the following parameters: + +export_format + Specify the type of export which will run. Possible values are: + + **\*file_csv** + Exports into a comma separated file format. + + **\*file_fwv** + Exports into a fixed width file format. + + **\*http_post** + Will post the CDR to a HTTP server. The export content will be a HTTP form encoded representation of the :ref:`internal CDR object`. + + **\*http_json_cdr** + Will post the CDR to a HTTP server. The export content will be a JSON serialized representation of the :ref:`internal CDR object`. + + **\*http_json_map** + Will post the CDR to a HTTP server. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. + + **\*amqp_json_cdr** + Will post the CDR to an AMQP_ queue. The export content will be a JSON serialized representation of the :ref:`internal CDR object`. Uses AMQP_ protocol version 0.9.1. + + **\*amqp_json_map** + Will post the CDR to an AMQP_ queue. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. Uses AMQP_ protocol version 1.0. + + **\*amqpv1_json_map** + Will post the CDR to an AMQP_ queue. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. Uses AMQP_ protocol version 1.0. + + **\*sqs_json_map** + Will post the CDR to an `Amazon SQS queue `_. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. + + **\*s3_json_map** + Will post the CDR to `Amazon S3 storage `_. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. + + **\*kafka_json_map** + Will post the CDR to an `Apache Kafka `_. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. + +export_path + Specify the export path. It has special format depending of the export type. + + **\*file_csv**, **\*file_fwv** + Standard unix-like filesystem path. + + **\*http_post**, **\*http_json_cdr**, **\*http_json_map** + Full HTTP URL + + **\*amqp_json_map**, **\*amqpv1_json_map** + AMQP URL with extra parameters. + + Sample: *amqp://guest:guest@localhost:5672/?queue_id=cgrates_cdrs&exchange=exchangename&exchange_type=fanout&routing_key=cgr_cdrs* + + **\*sqs_json_map** + SQS URL with extra parameters. + + Sample: *http://sqs.eu-west-2.amazonaws.com/?aws_region=eu-west-2&aws_key=testkey&aws_secret=testsecret&queue_id=cgrates-cdrs* + + **\*s3_json_map** + S3 URL with extra parameters. + + Sample: *http://s3.us-east-2.amazonaws.com/?aws_region=eu-west-2&aws_key=testkey&aws_secret=testsecret&queue_id=cgrates-cdrs* + + **\*kafka_json_map** + Kafka URL with extra parameters. + + Sample: *localhost:9092?topic=cgrates_cdrs* + + +filters + List of filters to pass for the export profile to execute. For the dynamic content (prefixed with *~*) following special variables are available: + + **\*req** + The *CDR* event itself. + + **\*ec** + The *EventCost* object with subpaths for all of it's nested objects. + +tenant + Tenant owning the template. It will be used mostly to match inside :ref:`FilterS`. + +synchronous + Block further exports until this one finishes. In case of *false* the control will be given to the next export template as soon as this one was started. + +attempts + Number of attempts before giving up on the export and writing the failed request to file. The failed request will be written to *failed_posts_dir* defined in *general* section. + +field_separator + Field separator to be used in some export types (ie. *\*file_csv*). + +attributes_context + The context used when sending the CDR event to :ref:`AttributeS` for modifications. If empty, there will be no event sent to :ref:`AttributeS`. + +fields + List of fields for the exported event. Not affecting templates like *\*http_json_cdr* or *\*amqp_json_cdr* with fixed content. + + +One **field template** will contain the following parameters: + +path + Path for the exported content. Possible prefixes here are: + + *\*exp* + Reference to the exported record. + + *\*hdr* + Reference to the header content. Available in case of **\*file_csv** and **\*file_fwv** export types. + + *\*trl* + Reference to the trailer content. Available in case of **\*file_csv** and **\*file_fwv** export types. + +type + The field type will give out the logic for generating the value. Values used depend on the type of prefix used in path. + + For *\*exp*, following field types are implemented: + + **\*variable** + Writes out the variable value, overwriting previous one set. + + **\*composed** + Writes out the variable value, postpending to previous value set + + **\*filler** + Fills the values with a fixed lentgh string. + + **\*constant** + Writes out a constant + + **\*datetime** + Parses the value as datetime and reformats based on the *layout* attribute. + + **\*combimed** + Writes out a combined mediation considering events with the same *CGRID*. + + **\*masked_destination** + Masks the destination using *\** as suffix. Matches the destination field against the list defined via *mask_destinationd_id* field. + + **\*http_post** + Uses a HTTP server as datasource for the value exported. + + For *\*hdr* and *\*trl*, following field types are possible: + + **\*filler** + Fills the values with a string. + + **\*constant** + Writes out a constant + + **\*handler** + Will obtain the content via a handler. This works in tandem with the attribute *handler_id*. + +value + The exported value. Works in tandem with *type* attribute. Possible prefixes for dynamic values: + + **\*req** + Data is taken from the current request coming from the *CDRs* component. + +mandatory + Makes sure that the field cannot have empty value (errors otherwise). + +tag + Used for debug purposes in logs. + +width + Used to control the formatting, enforcing the final value to a specific number of characters. + +strip + Used when the value is higher than *width* allows it, specifying the strip strategy. Possible values are: + + **\*right** + Strip the suffix. + + **\*xright** + Strip the suffix, postpending one *x* character to mark the stripping. + + **\*left** + Strip the prefix. + + **\*xleft** + Strip the prefix, prepending one *x* character to mark the stripping. + +padding + Used to control the formatting. Applied when the data is smaller than the *width*. Possible values are: + + **\*right** + Suffix with spaces. + + **\*left** + Prefix with spaces. + + **\*zeroleft** + Prefix with *0* chars. + +mask_destinationd_id + The destinations profile where we match the *masked_destinations*. + +hander_id + The identifier of the handler to be executed in case of *\*handler* *type*. + + + + + + + + diff --git a/docs/diamagent.rst b/docs/diamagent.rst index c6d69e0a8..471f5386e 100644 --- a/docs/diamagent.rst +++ b/docs/diamagent.rst @@ -283,7 +283,7 @@ asr_template templates Group fields based on their usability. Can be used in both processor templates as well as hardcoded within CGRateS functionality (ie *\*err* or *\*asr*). The IDs are unique, defining the same id in multiple configuration places/files will result into overwrite. - **\*err*** + **\*err** Is a hardcoded template used when *DiameterAgent* cannot parse the incoming message. Aside from logging the error via internal logger the message defined via *\*err* template will be sent out. **\*asr** @@ -456,3 +456,59 @@ type **\*template** Specifies a template of fields to be injected here. Value should be one of the template ids defined. +value + The captured value. Possible prefixes for dynamic values are: + + **\*req** + Take data from current request coming from diameter client. + + **\*vars** + Take data from internal container labeled *\*vars*. This is valid for the duration of the request. + + **\*cgreq** + Take data from the request being sent to :ref:`SessionS`. This is valid for one active request. + + **\*cgrep** + Take data from the reply coming from :ref:`SessionS`. This is valid for one active reply. + + **\*diamreq** + Take data from the diameter request being sent to the client (ie: *ASR*). This is valid for one active reply. + + **\*rep** + Take data from the diameter reply being sent to the client. + +mandatory + Makes sure that the field cannot have empty value (errors otherwise). + +tag + Used for debug purposes in logs. + +width + Used to control the formatting, enforcing the final value to a specific number of characters. + +strip + Used when the value is higher than *width* allows it, specifying the strip strategy. Possible values are: + + **\*right** + Strip the suffix. + + **\*xright** + Strip the suffix, postpending one *x* character to mark the stripping. + + **\*left** + Strip the prefix. + + **\*xleft** + Strip the prefix, prepending one *x* character to mark the stripping. + +padding + Used to control the formatting. Applied when the data is smaller than the *width*. Possible values are: + + **\*right** + Suffix with spaces. + + **\*left** + Prefix with spaces. + + **\*zeroleft** + Prefix with *0* chars. diff --git a/docs/ers.rst b/docs/ers.rst index 30e7a02f3..de91319c1 100644 --- a/docs/ers.rst +++ b/docs/ers.rst @@ -178,6 +178,9 @@ filters **\*vars** Request related shared variables between processors, populated especially by core functions. The data put inthere is not automatically transfered into requests sent to CGRateS, unless instructed inside templates. + **\*tmp** + Temporary container to be used when exchanging information between fields. + **\*req** Request read from the source. In case of file content without field name, the index will be passed instead of field source path. @@ -293,6 +296,58 @@ type Specifies a template of fields to be injected here. Value should be one of the template ids defined. +value + The captured value. Possible prefixes for dynamic values are: + + **\*req** + Take data from current request coming from the reader. + + **\*vars** + Take data from internal container labeled *\*vars*. This is valid for the duration of the request. + + **\*cgreq** + Take data from the request being sent to :ref:`SessionS`. This is valid for one active request. + + **\*cgrep** + Take data from the reply coming from :ref:`SessionS`. This is valid for one active reply. + +mandatory + Makes sure that the field cannot have empty value (errors otherwise). + +tag + Used for debug purposes in logs. + +width + Used to control the formatting, enforcing the final value to a specific number of characters. + +strip + Used when the value is higher than *width* allows it, specifying the strip strategy. Possible values are: + + **\*right** + Strip the suffix. + + **\*xright** + Strip the suffix, postpending one *x* character to mark the stripping. + + **\*left** + Strip the prefix. + + **\*xleft** + Strip the prefix, prepending one *x* character to mark the stripping. + +padding + Used to control the formatting. Applied when the data is smaller than the *width*. Possible values are: + + **\*right** + Suffix with spaces. + + **\*left** + Prefix with spaces. + + **\*zeroleft** + Prefix with *0* chars. + +