Commit Graph

258 Commits

Author SHA1 Message Date
ionutboangiu
2b5a3e5a5e elasticsearch: switch to fully-typed API
- index request options are now used directly during ExportEvent. They are
  passed to the request as options only if they were configured in the first
  place.
- implement PrepareMap and PrepareOrderMap methods for the elastic exporter.
  bytePreparing methods are not needed anymore as the Event map can be exported
  directly.
- elasticsearch.Client -> elasticsearch.TypedClient
- rename prepareOpts -> parseClientOpts
2024-11-27 16:50:47 +01:00
ionutboangiu
dd045148ad Revise elasticsearch exporter integration test 2024-11-27 16:50:47 +01:00
ionutboangiu
ae264deadb Slightly refactor elastic exporter
now returns error in case of invalid logger type. Could be removed
in the future in favor of handling it in configsanity.go.
2024-11-27 16:50:47 +01:00
ionutboangiu
248bf792d5 Add context to ee constructor errors 2024-11-27 16:50:47 +01:00
ionutboangiu
9d3e5e62a6 Implement getter method for EventExporterCfg (by ID)
Removed redundant getter method for default EventExporterCfg.
2024-11-27 16:50:47 +01:00
ionutboangiu
d5aa567e98 Prevent deadlock inside *els Connect 2024-11-15 20:58:40 +01:00
gezimbll
9e60ab8236 clone CGREvent when exportEventWithExporter is send asynchronusly 2024-11-08 14:59:48 +01:00
ionutboangiu
b05ed15f25 Fix vet check errors and conflicts 2024-10-16 12:24:54 +02:00
arberkatellari
bd6f5c661d Add *eventTimestamp for EES 2024-09-23 11:52:31 +02:00
ionutboangiu
6e971e954c chore: remove unused call & redundant ping methods
- Ping methods defined by engine.NewService for all services
- Call methods were previously replaced by the centralized call method
(on *birpc.Service) for all components
- All services now register V1/V2 objects from apier package for
consistency
2024-09-06 17:37:05 +02:00
armirveliaj
fb07da9924 Add new unit tests on ees 2024-09-05 20:47:27 +02:00
armirveliaj
07390d8624 Add new unit tests on ees and sessions 2024-08-21 21:07:15 +02:00
armirveliaj
f4c72b564b Add unit tests on ees 2024-08-16 11:34:45 +02:00
arberkatellari
9a79b092a5 Add exporter id within the *vars in EEs for templates 2024-08-12 18:34:01 +02:00
armirveliaj
2f9f07a976 Add new unit tests on ees and ers 2024-08-09 09:18:46 +02:00
gezimblliku
692279ef8b handle errors from exportrequest fields in ees 2024-07-21 19:05:53 +02:00
ionutboangiu
4cba4a699d Implement SharedActionsData for action groups
- holds common relevant data between actions.
- currently supports only *cdrlog, making the process more reliable
and less repetitive. It also provides access to more information about
actions when creating CDRs.
- easier to scale.
- removes the need for cloning accounts before executing a group of
actions.
- added Actions method to check if specific action types exist inside.
2024-07-21 19:04:53 +02:00
ionutboangiu
520451be7e Add precache support for exporters
Additional changes:
- removed unnecessary locking when initializing the exporter
  cache map, as the service itself is responsible for locking.
- separated setupCache method into ClearExporterCache and
  SetupExporterCache methods.
- removed idle ListenAndServe function that was only waiting for the
  stopChan to close. The reload case was unreachable due to the reload
  channel being created in Start instead of being passed down.
- removed Shutdown method on EventExporterS and replaced it with the
  exported ClearExporterCache method as it provided the same
  functionality.
2024-07-05 11:19:36 +02:00
armirveliaj
eb11b51c5e Add unit tests on ees 2024-06-30 17:51:59 +02:00
ionutboangiu
11b96de00a Add kafkaBatchSize configuration field
Avoids the default 1 second delay when the batch doesn't
reach 100 messages within that time.

Useful when the Kafka exporter is not cached, as it would
otherwise encounter that delay. Setting BatchSize to 1
prevents this.
2024-06-30 16:28:52 +02:00
ionutboangiu
9c94fbfe58 Handle logical race condition when caching exporters
- added lock to ensure only one exporter instance is cached.
- marked duplicate instances to be closed after export.
2024-06-30 16:28:52 +02:00
ionutboangiu
02ae2cce79 Revise Kafka exporter implementation
- added extra error return parameter to constructor
- failing to parse PEM certificates returns an error (previously nil
return)
- moved Connect logic inside the constructor*
- removed unnecessary fields from poster structure. Use the
configuration fields directly instead.
- removed mutex from poster structure (kafka writer is thread-safe)
- removed nil writer check. Message is exported directly.
- shortened receiver name
(https://google.github.io/styleguide/go/decisions#receiver-names)

*The Kafka Transport is not a connection but a configuration for the
connection created during the first export. The connection and its
related goroutines stay running until manually closed.
2024-06-30 16:28:52 +02:00
ionutboangiu
b7dacfe8a6 Revise APIerSv1.ReplayFailedPosts API + tests
- renamed parameter type: ArgsReplyFailedPosts -> ReplayFailedPostsParams
- renamed param fields:
  - FailedRequestsInDir -> SourcePath
  - FailedRequestsOutDir -> FailedPath
- changed param fields types from *string to string
- used the SourcePath and FailedPath params directly instead of creating separate variables
- used filepath.WalkDir instead of reading the directory and looping over the entries
- used slices.ContainsFunc to check if the file belongs to any module (if 1+ is specified)
- used filepath.Join instead of path.Join
- used the path provided by WalkFunc instead of building the file paths ourselves
- made error returns more descriptive
- added logs for directories/files that are skipped
- paths that cannot be accessed are skipped after logging the error
2024-06-06 13:30:14 +02:00
armirveliaj
8992da64b4 Revise unit test on ees 2024-06-06 13:17:59 +02:00
armirveliaj
09e9939482 Add unit tests on ees 2024-06-06 13:17:59 +02:00
ionutboangiu
88b02a3249 Set kafka MaxAttempts to 1
This leaves it to the ExportWithAttempts function to handle the
connect attempts.
2024-04-15 10:39:33 +02:00
ionutboangiu
3062113a61 Make go vet pass
The rpcclient constructor could not see the centralized
Encoding flag because it didn't have the necessary build
constraints. Added the constraints in lib_test.go files
where it wasn't alone. In all the other cases, it was
moved to the first file where it was needed.
2024-04-08 13:14:02 +02:00
ionutboangiu
bbf441f83c Add flaky build constrait to unstable tests 2024-04-08 13:14:02 +02:00
ionutboangiu
16eb745f00 Centralize test flags in utils for shared use across tests 2024-04-08 13:14:02 +02:00
gezimbll
6f4ee914f7 revised meprofile &cpuprofile tests 2024-04-08 13:12:54 +02:00
ionutboangiu
619a1efa50 Revise ers integration tests
They also do not depend on nats server dependency anymore.
2024-03-22 15:12:04 +01:00
ionutboangiu
b457768a33 Update actions to share Time ref variable 2024-03-13 18:25:06 +01:00
ionutboangiu
58c1d639f3 Update handling of ees event after processing attributes 2024-03-11 16:41:09 +01:00
ionutboangiu
49d6b8d565 Ensure CostDetails is of *EventCost type when parsing dynamicDP
CGREventWithEeIDs has also been optimized and properly tested. Comments
have been added explaining the process.

When sending a request to AttributeS from EEs, CostDetails from the reply
will now be overwritten by the original CostDetails to preserve its type.
The downside is that we are assuming that CostDetails was not altered by
AttributeS. We might consider adding a type check against *engine.EventCost
to at least stay backwards compatible with *gob and *internal connections.

general_tests/ees_it_test.go has been updated to ensure changes are working
properly.
2024-03-04 09:49:15 +01:00
ionutboangiu
0aea8ac641 Revert "Move CGREvent from utils to engine"
This reverts commit 02195c47ed.
2024-03-04 09:49:15 +01:00
ionutboangiu
02195c47ed Move CGREvent from utils to engine 2024-02-27 17:16:57 +01:00
ionutboangiu
a9079154da Add possibility to pass conn cfg to actions 2024-02-27 16:28:48 +01:00
arberkatellari
8ab43944a2 Update readthedocs tutorial 2024-02-15 17:02:41 +01:00
ionutboangiu
9494fc6e5e Implement PrepareMap methods for virtual EE 2024-02-09 17:39:18 +01:00
ionutboangiu
c5b197db6f Simplify event cloning (ees) 2024-02-09 17:36:24 +01:00
ionutboangiu
3637e4364e Clone export event only if processed by attributes 2024-02-09 15:57:25 +01:00
ionutboangiu
0dc1567a7a Ensure event is cloned before being sent to attributes
Also the exporter ID has been added in APIOpts under the *exporterID
key.
2024-02-09 15:57:25 +01:00
ionutboangiu
36bdc41e97 Add support for kafka ssl encryption
For both poster and reader.
Added integration test.
2024-01-23 14:12:33 +01:00
gezimbll
5f801c8b67 Revise integration tests 2023-11-30 16:52:43 +01:00
gezimbll
bf89dd7650 Add default evencost when it's nil for ees
it fixes the panic on scenario  when  we define  a template containing *ec fields and exported cdr doesn't contain
a costdetails field.
2023-11-15 10:25:21 +01:00
ionutboangiu
29f58debc9 Clone header before creating exporter HTTP request
Behind http.Header is just a map and it's not safe for concurrent use.
Before this change, a panic might have occurred when doing asynchronous
HTTP exports (applies to both *http_post and *http_json_map exporters).
Cloning the header before adding it to the HTTP request has fixed this
issue.

Slightly improved the test that found this data race.
2023-11-08 17:11:40 +01:00
gezimbll
c2dacc42c1 Removed warnings from go staticcheck 2023-11-08 17:10:13 +01:00
gezimbll
9e2a629b82 Add *log flag for logging ees events 2023-11-08 16:52:38 +01:00
ionutboangiu
d1d43913c8 Replace deprecated Publish method with PublishWithContext 2023-10-27 18:51:01 +02:00
ionutboangiu
ec799dd1e9 Update expected value in kafka it test 2023-10-26 15:55:54 +02:00