Update ERs docs

config/config_defaults.go

@@ -474,8 +474,8 @@ const CGRATES_CFG_JSON = `
 				// "kafkaSkipTLSVerify": false,			// if true it will skip certificate verification
 
 				// SQL
-				// "sqlDBName": "cgrates", 			// the name of the database from were the events are read
+				// "sqlDBName": "cgrates", 			// the name of the database from where the events are read
-				// "sqlTableName": "cdrs",			// the name of the table from were the events are read
+				// "sqlTableName": "cdrs",			// the name of the table from where the events are read
 				// "sqlBatchSize: 0, 				// number of SQL rows that can be selected at a time. 0 or lower for unlimited
 				// "sqlDeleteIndexedFields": [],   		// list of fields to DELETE from the table
 				// "pgSSLMode": "disable",			// the ssl mode for postgres db
@@ -487,10 +487,10 @@ const CGRATES_CFG_JSON = `
 				// "awsToken": "",
 
 				// SQS
-				// "sqsQueueID": "cgrates_cdrs", 		// the queue id for SQS readers from were the events are read
+				// "sqsQueueID": "cgrates_cdrs", 		// the queue id for SQS readers from where the events are read
 				
 				// S3
-				// "s3BucketID": "cgrates_cdrs", 		// the bucket id for S3 readers from were the events are read
+				// "s3BucketID": "cgrates_cdrs", 		// the bucket id for S3 readers from where the events are read
 
 				// nats
 				// "natsJetStream": false,			// controls if the nats reader uses the JetStream

docs/ers.rst

@@ -4,7 +4,11 @@
 .. _MSSQL: https://www.microsoft.com/en-us/sql-server/
 .. _Kamailio: https://www.kamailio.org/w/
 .. _OpenSIPS: https://opensips.org/
-.. _Kafka_: https://kafka.apache.org/
+.. _Kafka: https://kafka.apache.org/
+.. _AMQP: https://www.amqp.org/
+.. _S3: https://aws.amazon.com/s3/
+.. _SQS: https://aws.amazon.com/sqs/
+.. _NATS: https://nats.io/
 
 .. EventReaderService:
 
@@ -37,9 +41,10 @@ With explanations in the comments:
 		{
 			"id": "file_reader2",		// file_reader2 reader
 			"run_delay":  "-1",			// reading of events it is triggered outside of ERs
-			"field_separator": ";",		// field separator definition
+			"opts": {
+				"csvFieldSeparator":";" // field separator definition
+			},		
 			"type": "*file_csv",		// type of reader, *file_csv can read .csv files
-			"row_length" : 0,			// Number of fields from csv file
 			"flags": [					// influence processing logic within CGRateS workflow
 				"*cdrs",				//   *cdrs will create CDRs
 				"*log"					//   *log will log the events to syslog
@@ -140,15 +145,36 @@ type
 	**\*file_fwv**
 		Reader for *fixed width value* formatted files.
 
+	**\*file_json**
+		Reader for *json formatted files.
+
 	**\*kafka_json_map**
 		Reader for hashmaps within Kafka_ database.
 
 	**\*sql**
 		Reader for generic content out of *SQL* databases. Supported databases are: MySQL_, PostgreSQL_ and MSSQL_.
 
+	**\*amqp_json_map**
+		Reader for AMQP_ v0.9.1 messaging.
+		
+	**\*amqpv1_json_map**
+		Reader for AMQP_ v1.0 messaging.
+		
+	**\*s3_json_map**
+		Reader for S3_ events.
+		
+	**\*sqs_json_map**
+		Reader for SQS_ events.
+		
+	**\*nats_json_map**
+		Reader for NATS_ events.		
+
 run_delay
 	Duration interval between consecutive reads from source. If 0 or less, *ERs* relies on external source (ie. Linux inotify for files) for starting the reading process.
 
+start_delay
+	A duration to delay the reader from starting to read events on engine start.
+
 concurrent_requests
 	Limits the number of concurrent reads from source (ie: the number of simultaneously opened files).
 
@@ -158,9 +184,6 @@ source_path
 processed_path
 	Optional path for moving the events source to after processing.
 
-xml_root_path
-	Used in case of XML content and will specify the prefix path applied to each xml element read.
-
 tenant
 	Will auto-populate the Tenant within the API calls sent to CGRateS. It has the form of a RSRParser. If undefined, default one from *general* section will be used.
 
@@ -205,7 +228,7 @@ flags
 	**\*dryrun**
 		Together with not transfering the Event on CGRateS side will also log it, useful for troubleshooting.
 
-	**\*auth**
+	**\*authorize**
 		Sends the Event for authorization on CGRateS.
 
 		Auxiliary flags available: **\*attributes**, **\*thresholds**, **\*stats**, **\*resources**, **\*accounts**, **\*routes**, **\*routes_ignore_errors**, **\*routes_event_cost**, **\*routes_maxcost** which are used to influence the auth behavior on CGRateS side. More info on that can be found on the **SessionS** component's API behavior.
@@ -238,108 +261,320 @@ flags
 	**\*cdrs**
 		Build a CDR out of the Event on CGRateS side. Can be used simultaneously with other flags (except **\*dryrun**)
 
-path
+	**\*export**
-	Defined within field, specifies the path where the value will be written. Possible values:
+		Process the event read, and send the processed event to EEs. Can be used simultaneously with other flags. 
-
-	**\*vars**
-		Write the value in the special container, *\*vars*, available for the duration of the request.
-
-	**\*cgreq**
-		Write the value in the request object which will be sent to CGRateS side.
-
-	**\*hdr**
-		Header values (available only in case of *\*file_fwv*). In case of file content without field name, the index will be passed instead of field source path.
-
-	**\*trl**
-		Trailer values (available only in case of *\*file_fwv*). In case of file content without field name, the index will be passed instead of field source path.
 
 
-type
+reconnects
-	Defined within field, specifies the logic type to be used when writing the value of the field. Possible values:
+	The amount retries to reestablish the connection in case of connection loss for AMQP. `-1` retry indefinitely.
-
-	**\*none**
-		Pass
-
-	**\*filler**
-		Fills the values with an empty string
-
-	**\*constant**
-		Writes out a constant
-
-	**\*variable**
-		Writes out the variable value, overwriting previous one set
-
-	**\*composed**
-		Writes out the variable value, postpending to previous value set
-
-	**\*usage_difference**
-		Calculates the usage difference between two arguments passed in the *value*. Requires 2 arguments: *$stopTime;$startTime*
-
-	**\*sum**
-		Calculates the sum of all arguments passed within *value*. It supports summing up duration, time, float, int autodetecting them in this order.
-
-	**\*difference**
-		Calculates the difference between all arguments passed within *value*. Possible value types are (in this order): duration, time, float, int.
-
-	**\*value_exponent**
-		Calculates the exponent of a value. It requires two values: *$val;$exp*
-
-	**\*template**
-		Specifies a template of fields to be injected here. Value should be one of the template ids defined.
 
 
-value
+max_reconnect_interval
-	The captured value. Possible prefixes for dynamic values are:
+	The duration to wait in between retries to reconnect on a connection loss for AMQP.
 
-		**\*req**
+
-			Take data from current request coming from the reader.
+ees_ids
+	The IDs of exporters in EEs which you want to make use of, when `*export` flag is present in the reader. When an event is read and processed from the reader in use, the processed event will be sent to those specific IDs in EEs.
+
+
+ees_success_ids
+	When an ERs reader processes an event successfuly, it will send the raw(unprocessed) event that it read, to the specified EEs exporter IDs matching the `ees_success_ids`.
+
+
+ees_failed_ids
+	When an ERs reader fails to process an event, it will send the raw(unprocessed) event that it read, to the specified EEs exporter IDs matching the `ees_failed_ids`.
+
+
+opts
+	Additional options specific and non specific to reader types.
+
+	Partial:
+
+	**partialPath**
+		The path were the partial events will be sent.
+		
+	**partialCacheAction**
+		The action that will be executed for the partial events that are not matched with other events:
+
+		**\*none** - Nothing happens.
+
+		**\*post_cdr** - Try to merge partial events and post them back to ERs for processing.
+
+		**\*dump_to_file** - Apply the `cache_dump_fields` to the partial events and write the record to file in CSV format.
+		
+		**\*dump_to_json** - Apply the `cache_dump_fields` to the partial events and write the record to file in JSON format.
+
+	**partialOrderField**
+		The field after what the events are ordered when merged.
+
+	**partialcsvFieldSeparator**
+		The separator used when dumping the event fields.
+
+
+	FileCSV:
+
+	**csvRowLength**
+		Number of fields from csv file, `-1` to disable checking, `0` to inherit the lenght of first record.
+
+	**csvFieldSeparator**
+		Define what separator is used in the CSV fields that will be read.
+
+	**csvHeaderDefineChar**
+		The starting character for the header definition used in CSV files.
+
+	**csvLazyQuotes**
+		Make it true if a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field.
+
+
+	FileXML reader:
+
+	**xmlRootPath**
+		The prefix path applied to each xml element read.
+	
+
+	AMQP and AMQPv1:
+
+	**amqpQueueID**
+		ID for the primary queue where messages are consumed. (Used for AMQP 0.9.1 and 1.0)
+
+	**amqpUsername**
+		Username for SASL PLAIN auth, exclusive to AMQP 1.0, often representing the policy name.
+
+	**amqpPassword**
+		Password for authentication, exclusive to AMQP 1.0.
+
+	**amqpConsumerTag**
+		Unique tag for the consumer, useful for message tracking and consumer management. (Used for AMQP 0.9.1 and 1.0)
+
+	**amqpExchange**
+		Name of the primary exchange where messages will be published. Exclusive to AMQP 0.9.1
+
+	**amqpExchangeType**
+		Type of the primary exchange (direct, topic, fanout, headers). Exclusive to AMQP 0.9.1
+
+	**amqpRoutingKey**
+		Key used for routing messages to the primary queue. Exclusive to AMQP 0.9.1
+	
+
+	Kafka: 
+
+	**kafkaTopic**
+		The topic from were the events are read.
+
+	**kafkaGroupID**
+		The group that reads the events.
+
+	**kafkaMaxWait**
+		The maximum amount of time to wait for new data to come.
+
+	**kafkaTLS**
+		If true it will try to authenticate the client.
+
+	**kafkaCAPath**
+		Path to certificate authority file.
+	
+	**kafkaSkipTLSVerify**
+		If true it will skip certificate verification.
+	
+
+	SQL:
+
+	**sqlDBName**
+		The name of the database from where the events are read.
+
+	**sqlTableName**
+		The name of the table from where the events are read.
+	
+	**sqlBatchSize**
+		Number of SQL rows that can be selected at a time. 0 or lower for unlimited.
+
+	**sqlDeleteIndexedFields**
+		List of fields to DELETE from the table.
+
+	**pgSSLMode**
+		The SSL mode for postgres db.
+
+
+	SQS:
+
+	**sqsQueueID**
+		The queue ID for SQS readers from where the events are read.
+	
+	**awsRegion**
+		The region of the AWS SQS bucket.
+
+	**awsKey**
+		The key of the AWS SQS bucket.
+
+	**awsSecret**
+		The secret of the AWS SQS bucket.
+
+	**awsToken**
+		The token of the AWS SQS bucket.
+
+
+	S3: 
+
+	**s3BucketID**
+		The bucket ID for S3 readers from where the events are read.
+	
+	**awsRegion**
+		The region of the AWS S3 bucket.
+
+	**awsKey**
+		The key of the AWS S3 bucket.
+
+	**awsSecret**
+		The secret of the AWS S3 bucket.
+
+	**awsToken**
+		The token of the AWS S3 bucket.
+
+	
+	NATS:
+
+	**natsJetStream**
+		When true, the nats reader uses the JetStream.
+
+	**natsConsumerName**
+		Name of the JetStream consumer. Used when `natsJetStream` is enabled.
+
+	**natsStreamName**
+		JetStream stream name from which the consumer will read messages.
+
+	**natsSubject**
+		Specifies the NATS subject to subscribe to for receiving messages.
+
+	**natsQueueID**
+		Queue ID for the consumer to listen to.
+
+	**natsJWTFile**
+		Path to the NATS JWT file. Can be a chained JWT or a user JWT file.
+
+	**natsSeedFile**
+		Path to the NATS seed file used for signing the JWT. Only used if `natsJWTFile` is provided.
+
+	**natsCertificateAuthority**
+		Path to the custom certificate authority file.
+
+	**natsClientCertificate**
+		Path to the client certificate used for TLS.
+
+	**natsClientKey**
+		Path to the client private key used for TLS.
+
+	**natsJetStreamMaxWait**
+		Maximum time to wait for a JetStream response.
+
+
+fields
+	List of fields for read event. One **field template** can contain the following parameters.
+
+	**path**
+		Defined within field, specifies the path where the value will be written. Possible values:
 
 		**\*vars**
-			Take data from internal container labeled *\*vars*. This is valid for the duration of the request.
+			Write the value in the special container, *\*vars*, available for the duration of the request.
 
 		**\*cgreq**
-			Take data from the request being sent to :ref:`SessionS`. This is valid for one active request.
+			Write the value in the request object which will be sent to CGRateS side.
 
-		**\*cgrep**
+		**\*hdr**
-			Take data from the reply coming from :ref:`SessionS`. This is valid for one active reply.
+			Header values (available only in case of *\*file_fwv*). In case of file content without field name, the index will be passed instead of field source path.
 
-mandatory
+		**\*trl**
-	Makes sure that the field cannot have empty value (errors otherwise).
+			Trailer values (available only in case of *\*file_fwv*). In case of file content without field name, the index will be passed instead of field source path.
-
-tag
-	Used for debug purposes in logs.
-
-width
-	Used to control the formatting, enforcing the final value to a specific number of characters.
-
-strip
-	Used when the value is higher than *width* allows it, specifying the strip strategy. Possible values are:
-
-	**\*right**
-		Strip the suffix.
-
-	**\*xright**
-		Strip the suffix, postpending one *x* character to mark the stripping.
-
-	**\*left**
-		Strip the prefix.
-
-	**\*xleft**
-		Strip the prefix, prepending one *x* character to mark the stripping.
-
-padding
-	Used to control the formatting. Applied when the data is smaller than the *width*. Possible values are:
-
-	**\*right**
-		Suffix with spaces.
-
-	**\*left**
-		Prefix with spaces.
-
-	**\*zeroleft**
-		Prefix with *0* chars.
 
 
+	**type**
+		Defined within field, specifies the logic type to be used when writing the value of the field. Possible values:
+
+		**\*none**
+			Pass
+
+		**\*filler**
+			Fills the values with an empty string
+
+		**\*constant**
+			Writes out a constant
+
+		**\*variable**
+			Writes out the variable value, overwriting previous one set
+
+		**\*composed**
+			Writes out the variable value, postpending to previous value set
+
+		**\*usage_difference**
+			Calculates the usage difference between two arguments passed in the *value*. Requires 2 arguments: *$stopTime;$startTime*
+
+		**\*sum**
+			Calculates the sum of all arguments passed within *value*. It supports summing up duration, time, float, int autodetecting them in this order.
+
+		**\*difference**
+			Calculates the difference between all arguments passed within *value*. Possible value types are (in this order): duration, time, float, int.
+
+		**\*value_exponent**
+			Calculates the exponent of a value. It requires two values: *$val;$exp*
+
+		**\*template**
+			Specifies a template of fields to be injected here. Value should be one of the template ids defined.
 
 
+	**value**
+		The captured value. Possible prefixes for dynamic values are:
 
+			**\*req**
+				Take data from current request coming from the reader.
+
+			**\*vars**
+				Take data from internal container labeled *\*vars*. This is valid for the duration of the request.
+
+			**\*cgreq**
+				Take data from the request being sent to :ref:`SessionS`. This is valid for one active request.
+
+			**\*cgrep**
+				Take data from the reply coming from :ref:`SessionS`. This is valid for one active reply.
+
+	**mandatory**
+		Makes sure that the field cannot have empty value (errors otherwise).
+
+	**tag**
+		Used for debug purposes in logs.
+
+	**width**
+		Used to control the formatting, enforcing the final value to a specific number of characters.
+
+	**strip**
+		Used when the value is higher than *width* allows it, specifying the strip strategy. Possible values are:
+
+		**\*right**
+			Strip the suffix.
+
+		**\*xright**
+			Strip the suffix, postpending one *x* character to mark the stripping.
+
+		**\*left**
+			Strip the prefix.
+
+		**\*xleft**
+			Strip the prefix, prepending one *x* character to mark the stripping.
+
+	**padding**
+		Used to control the formatting. Applied when the data is smaller than the *width*. Possible values are:
+
+		**\*right**
+			Suffix with spaces.
+
+		**\*left**
+			Prefix with spaces.
+
+		**\*zeroleft**
+			Prefix with *0* chars.
+
+
+partial_commit_fields
+	The fields are written in the same way as import fields template. The fields are used for events which were read but werent fully processed. If the coming events that are being read, match the filters set in these partial_commit_fields, they will be used to fully create and process that partial event.
+
+
+cache_dump_fields
+	The fields are written in the same way as import fields template. The fields are used as a template to write the fields of the partial events into dump files, when the TTL expires for that partial event, or when that cache element is evicted.