Commit Graph

283 Commits

Author SHA1 Message Date
arberkatellari
a8895a6a5a Add options to support non-Amazon S3 and SQS 2025-12-05 13:16:47 +01:00
ionutboangiu
c185e46455 ees: preserve exporter attempts in failed posts 2025-10-29 19:44:04 +01:00
ionutboangiu
8443af3a89 Relicense from GPLv3 to AGPLv3 2025-10-29 19:42:40 +01:00
ionutboangiu
4c30ec4e4c ees: remove init from libcdre, rename to failedposts 2025-10-28 20:31:36 +01:00
ionutboangiu
295ddec792 cfg: move failed_posts to ees, add static_ttl 2025-10-28 20:31:36 +01:00
ionutboangiu
676900b4c3 update DB casing for InitDB helpers 2025-05-29 15:41:53 +02:00
arberkatellari
f6c300c395 Add action type *dynamic_attribute and revise clone functions 2025-05-18 16:12:20 +02:00
gezimbll
d1fe58835d added test for creating templates from account values 2025-05-15 14:56:04 +02:00
arberkatellari
5463eb61c6 Adjust tests to InternalDB constructor 2025-04-11 13:09:50 +02:00
arberkatellari
ed584364d1 Add tests for internal DB dump and restore 2025-04-06 12:08:40 +02:00
arberkatellari
a168f262e2 Add dump and restore functionality for internal DBs 2025-04-06 12:08:40 +02:00
ionutboangiu
ef4324fc0e add error handling for cron AddFunc 2025-03-18 17:51:30 +01:00
ionutboangiu
7bb065fda6 rename *dc DataProvider to *em (from EventMetrics) 2025-03-17 17:36:58 +01:00
ionutboangiu
e1c154b9e0 use Set method to update ExportPath for csv/fwv ee
prevents manual locking
2025-03-17 17:36:58 +01:00
ionutboangiu
7166795066 ees: add test for metrics reset feature
to serve only as an example for now (takes too long)
2025-03-17 17:36:58 +01:00
ionutboangiu
fdc8a1be7e ees: implement api to reset exporter metrics 2025-03-17 17:36:58 +01:00
ionutboangiu
1c00a3aad1 Add cron-based/manual reset to ExporterMetrics 2025-03-17 17:36:58 +01:00
ionutboangiu
8e4d187725 remove unused composeExp func 2025-03-17 17:36:58 +01:00
ionutboangiu
132aecd3c1 simplify TimeNow parsing for ee metrics 2025-03-17 17:36:58 +01:00
arberkatellari
d35b14b6b9 Add ability to ERS to update or move ERS SQL events to a new table & add *export flag for ERS readers 2024-12-20 13:26:06 +01:00
armirveliaj
78bfc13d37 Add coverage tests on engine 2024-12-17 18:19:59 +01:00
ionutboangiu
6270885405 Add config sanity checks for elasticsearch ee 2024-11-28 17:24:39 +01:00
ionutboangiu
2511c00db7 Add elsCAPath opt and improve opt comments 2024-11-28 17:24:39 +01:00
ionutboangiu
8d7628410e Make elasticsearch refresh opt configurable 2024-11-28 17:24:39 +01:00
ionutboangiu
f1cdf920df Remove unusable elasticsearch opts 2024-11-28 17:24:39 +01:00
ionutboangiu
2b5a3e5a5e elasticsearch: switch to fully-typed API
- index request options are now used directly during ExportEvent. They are
  passed to the request as options only if they were configured in the first
  place.
- implement PrepareMap and PrepareOrderMap methods for the elastic exporter.
  bytePreparing methods are not needed anymore as the Event map can be exported
  directly.
- elasticsearch.Client -> elasticsearch.TypedClient
- rename prepareOpts -> parseClientOpts
2024-11-27 16:50:47 +01:00
ionutboangiu
dd045148ad Revise elasticsearch exporter integration test 2024-11-27 16:50:47 +01:00
ionutboangiu
ae264deadb Slightly refactor elastic exporter
now returns error in case of invalid logger type. Could be removed
in the future in favor of handling it in configsanity.go.
2024-11-27 16:50:47 +01:00
ionutboangiu
248bf792d5 Add context to ee constructor errors 2024-11-27 16:50:47 +01:00
ionutboangiu
9d3e5e62a6 Implement getter method for EventExporterCfg (by ID)
Removed redundant getter method for default EventExporterCfg.
2024-11-27 16:50:47 +01:00
ionutboangiu
d5aa567e98 Prevent deadlock inside *els Connect 2024-11-15 20:58:40 +01:00
gezimbll
9e60ab8236 clone CGREvent when exportEventWithExporter is send asynchronusly 2024-11-08 14:59:48 +01:00
ionutboangiu
b05ed15f25 Fix vet check errors and conflicts 2024-10-16 12:24:54 +02:00
arberkatellari
bd6f5c661d Add *eventTimestamp for EES 2024-09-23 11:52:31 +02:00
ionutboangiu
6e971e954c chore: remove unused call & redundant ping methods
- Ping methods defined by engine.NewService for all services
- Call methods were previously replaced by the centralized call method
(on *birpc.Service) for all components
- All services now register V1/V2 objects from apier package for
consistency
2024-09-06 17:37:05 +02:00
armirveliaj
fb07da9924 Add new unit tests on ees 2024-09-05 20:47:27 +02:00
armirveliaj
07390d8624 Add new unit tests on ees and sessions 2024-08-21 21:07:15 +02:00
armirveliaj
f4c72b564b Add unit tests on ees 2024-08-16 11:34:45 +02:00
arberkatellari
9a79b092a5 Add exporter id within the *vars in EEs for templates 2024-08-12 18:34:01 +02:00
armirveliaj
2f9f07a976 Add new unit tests on ees and ers 2024-08-09 09:18:46 +02:00
gezimblliku
692279ef8b handle errors from exportrequest fields in ees 2024-07-21 19:05:53 +02:00
ionutboangiu
4cba4a699d Implement SharedActionsData for action groups
- holds common relevant data between actions.
- currently supports only *cdrlog, making the process more reliable
and less repetitive. It also provides access to more information about
actions when creating CDRs.
- easier to scale.
- removes the need for cloning accounts before executing a group of
actions.
- added Actions method to check if specific action types exist inside.
2024-07-21 19:04:53 +02:00
ionutboangiu
520451be7e Add precache support for exporters
Additional changes:
- removed unnecessary locking when initializing the exporter
  cache map, as the service itself is responsible for locking.
- separated setupCache method into ClearExporterCache and
  SetupExporterCache methods.
- removed idle ListenAndServe function that was only waiting for the
  stopChan to close. The reload case was unreachable due to the reload
  channel being created in Start instead of being passed down.
- removed Shutdown method on EventExporterS and replaced it with the
  exported ClearExporterCache method as it provided the same
  functionality.
2024-07-05 11:19:36 +02:00
armirveliaj
eb11b51c5e Add unit tests on ees 2024-06-30 17:51:59 +02:00
ionutboangiu
11b96de00a Add kafkaBatchSize configuration field
Avoids the default 1 second delay when the batch doesn't
reach 100 messages within that time.

Useful when the Kafka exporter is not cached, as it would
otherwise encounter that delay. Setting BatchSize to 1
prevents this.
2024-06-30 16:28:52 +02:00
ionutboangiu
9c94fbfe58 Handle logical race condition when caching exporters
- added lock to ensure only one exporter instance is cached.
- marked duplicate instances to be closed after export.
2024-06-30 16:28:52 +02:00
ionutboangiu
02ae2cce79 Revise Kafka exporter implementation
- added extra error return parameter to constructor
- failing to parse PEM certificates returns an error (previously nil
return)
- moved Connect logic inside the constructor*
- removed unnecessary fields from poster structure. Use the
configuration fields directly instead.
- removed mutex from poster structure (kafka writer is thread-safe)
- removed nil writer check. Message is exported directly.
- shortened receiver name
(https://google.github.io/styleguide/go/decisions#receiver-names)

*The Kafka Transport is not a connection but a configuration for the
connection created during the first export. The connection and its
related goroutines stay running until manually closed.
2024-06-30 16:28:52 +02:00
ionutboangiu
b7dacfe8a6 Revise APIerSv1.ReplayFailedPosts API + tests
- renamed parameter type: ArgsReplyFailedPosts -> ReplayFailedPostsParams
- renamed param fields:
  - FailedRequestsInDir -> SourcePath
  - FailedRequestsOutDir -> FailedPath
- changed param fields types from *string to string
- used the SourcePath and FailedPath params directly instead of creating separate variables
- used filepath.WalkDir instead of reading the directory and looping over the entries
- used slices.ContainsFunc to check if the file belongs to any module (if 1+ is specified)
- used filepath.Join instead of path.Join
- used the path provided by WalkFunc instead of building the file paths ourselves
- made error returns more descriptive
- added logs for directories/files that are skipped
- paths that cannot be accessed are skipped after logging the error
2024-06-06 13:30:14 +02:00
armirveliaj
8992da64b4 Revise unit test on ees 2024-06-06 13:17:59 +02:00
armirveliaj
09e9939482 Add unit tests on ees 2024-06-06 13:17:59 +02:00