22 Commits

Author SHA1 Message Date
ionutboangiu
8443af3a89 Relicense from GPLv3 to AGPLv3 2025-10-29 19:42:40 +01:00
arberkatellari
321910d181 Add WHERE statement availability and non-delete option to ERS SQL reader 2024-12-03 18:36:22 +01:00
arberkatellari
0bab7d5557 Add reader id within the *vars in ERs 2024-08-09 09:14:24 +02:00
ionutboangiu
a362511d7a Define param struct for ErSv1.RunReader API + refactoring
Rename dir field to sourceDir in file event readers for clarity.
2024-04-04 12:31:18 +03:00
ionutboangiu
85864ccaa3 Extract common dir processing logic to a func
It will be reused for all file readers.
Rename rdrDir field to dir (redundant prefix).
2024-04-04 12:31:18 +03:00
ionutboangiu
0c91933065 Revise ERs event exporting
Add ees_success_ids and ees_failed_ids fields in reader config. The
former will be used to set EeIDs when the event processing returns
no error, while the latter will be used otherwise.

Add config sanity checks for the added options.

Remove Processed opts and everything related to them since they should
not be used anymore.

Fixed test compilation errors caused by the change.
2023-11-29 16:21:51 +01:00
ionutboangiu
c282d40201 Implement reconnect logic for AMQP EventReader
Implemented functionality that handles reconnecting to the amqp server
and reinitializing the amqp channel in case of errors and timeouts. This
is handled by a goroutine created in the client constructor (it also
handles the initial connect/init).

Reconnects and reinits will use a fibonacci backoff strategy, and the
attempt amount and max waiting interval can be adjusted by the
'reconnects' and 'max_reconnect_interval' config options.

Messages that fail processing are now dropped instead of being requeued,
preventing infinite processing loops. However, this means that the
messages are lost. Handling failed messages will need to be addressed
separately.

'concurrent_requests' will now set the prefetch count. Setting the
prefetch count using the Qos function was able to replace our old
approach that was using channels. Default value is 1024 which,
according to the rabbitmq docs, 'runs into the law of diminishing
returns'. The recommended value is between 100-300. Source:
https://www.rabbitmq.com/confirms.html#channel-qos-prefetch-throughput

Fix test compilation errors and failing tests caused by these changes.

References #4160
2023-11-08 18:31:07 +01:00
ionutboangiu
a80fe9b96e Remove redundant prefixes/suffixes from rOpts field names 2023-09-28 20:48:07 +02:00
gezimbll
01d3c5d0fa Separating opts in ERS && fixing tests 2023-06-27 11:08:34 +02:00
ionutboangiu
38447e9229 Make ers opts as struct and fix compilation errors 2021-11-29 20:57:39 +01:00
Trial97
cae5b4ced5 Added partial reader 2021-05-17 18:00:31 +02:00
Trial97
051028078d Updated s3 and sqs ers configs 2021-05-17 18:00:31 +02:00
nickolasdaniel
b451279779 Coverage tests for ers 2021-04-26 13:08:22 +02:00
nickolasdaniel
742bf37366 Coverage tests for ers 2021-04-16 11:13:27 +02:00
nickolasdaniel
b9a3e1d7e6 Coverage tests for ers 2021-04-13 17:42:53 +02:00
andronache
e4c3116a23 Cover tests for functions in ers 2021-03-11 17:02:44 +01:00
porosnicuadrian
711d7d5a1a Removed error from NewDefaultCGRConfig signature's function 2020-12-02 14:00:35 +01:00
Trial97
228665989e Updated ERs and EEs options 2020-09-04 12:32:09 +02:00
TeoV
3e3fc6c457 Correctly populate ConcurrentRequest from config in EventReader fixes #1932 2020-02-18 14:24:50 +01:00
Trial97
4c337ee467 Updated ers reader tests 2020-01-06 10:00:46 +02:00
Trial97
4ce5caea76 Fixed typos and updated tests 2019-09-17 18:07:53 +02:00
TeoV
ab21f698d6 Test all cases for NewEventReader 2019-09-16 09:34:59 +02:00