Commit Graph

99 Commits

Author SHA1 Message Date
ionutboangiu
c185e46455 ees: preserve exporter attempts in failed posts 2025-10-29 19:44:04 +01:00
ionutboangiu
8443af3a89 Relicense from GPLv3 to AGPLv3 2025-10-29 19:42:40 +01:00
arberkatellari
a168f262e2 Add dump and restore functionality for internal DBs 2025-04-06 12:08:40 +02:00
ionutboangiu
7bb065fda6 rename *dc DataProvider to *em (from EventMetrics) 2025-03-17 17:36:58 +01:00
ionutboangiu
fdc8a1be7e ees: implement api to reset exporter metrics 2025-03-17 17:36:58 +01:00
ionutboangiu
1c00a3aad1 Add cron-based/manual reset to ExporterMetrics 2025-03-17 17:36:58 +01:00
arberkatellari
d35b14b6b9 Add ability to ERS to update or move ERS SQL events to a new table & add *export flag for ERS readers 2024-12-20 13:26:06 +01:00
ionutboangiu
248bf792d5 Add context to ee constructor errors 2024-11-27 16:50:47 +01:00
gezimbll
9e60ab8236 clone CGREvent when exportEventWithExporter is send asynchronusly 2024-11-08 14:59:48 +01:00
arberkatellari
bd6f5c661d Add *eventTimestamp for EES 2024-09-23 11:52:31 +02:00
ionutboangiu
6e971e954c chore: remove unused call & redundant ping methods
- Ping methods defined by engine.NewService for all services
- Call methods were previously replaced by the centralized call method
(on *birpc.Service) for all components
- All services now register V1/V2 objects from apier package for
consistency
2024-09-06 17:37:05 +02:00
arberkatellari
9a79b092a5 Add exporter id within the *vars in EEs for templates 2024-08-12 18:34:01 +02:00
gezimblliku
692279ef8b handle errors from exportrequest fields in ees 2024-07-21 19:05:53 +02:00
ionutboangiu
520451be7e Add precache support for exporters
Additional changes:
- removed unnecessary locking when initializing the exporter
  cache map, as the service itself is responsible for locking.
- separated setupCache method into ClearExporterCache and
  SetupExporterCache methods.
- removed idle ListenAndServe function that was only waiting for the
  stopChan to close. The reload case was unreachable due to the reload
  channel being created in Start instead of being passed down.
- removed Shutdown method on EventExporterS and replaced it with the
  exported ClearExporterCache method as it provided the same
  functionality.
2024-07-05 11:19:36 +02:00
ionutboangiu
9c94fbfe58 Handle logical race condition when caching exporters
- added lock to ensure only one exporter instance is cached.
- marked duplicate instances to be closed after export.
2024-06-30 16:28:52 +02:00
ionutboangiu
58c1d639f3 Update handling of ees event after processing attributes 2024-03-11 16:41:09 +01:00
ionutboangiu
49d6b8d565 Ensure CostDetails is of *EventCost type when parsing dynamicDP
CGREventWithEeIDs has also been optimized and properly tested. Comments
have been added explaining the process.

When sending a request to AttributeS from EEs, CostDetails from the reply
will now be overwritten by the original CostDetails to preserve its type.
The downside is that we are assuming that CostDetails was not altered by
AttributeS. We might consider adding a type check against *engine.EventCost
to at least stay backwards compatible with *gob and *internal connections.

general_tests/ees_it_test.go has been updated to ensure changes are working
properly.
2024-03-04 09:49:15 +01:00
ionutboangiu
0aea8ac641 Revert "Move CGREvent from utils to engine"
This reverts commit 02195c47ed.
2024-03-04 09:49:15 +01:00
ionutboangiu
02195c47ed Move CGREvent from utils to engine 2024-02-27 17:16:57 +01:00
ionutboangiu
c5b197db6f Simplify event cloning (ees) 2024-02-09 17:36:24 +01:00
ionutboangiu
3637e4364e Clone export event only if processed by attributes 2024-02-09 15:57:25 +01:00
ionutboangiu
0dc1567a7a Ensure event is cloned before being sent to attributes
Also the exporter ID has been added in APIOpts under the *exporterID
key.
2024-02-09 15:57:25 +01:00
gezimbll
bf89dd7650 Add default evencost when it's nil for ees
it fixes the panic on scenario  when  we define  a template containing *ec fields and exported cdr doesn't contain
a costdetails field.
2023-11-15 10:25:21 +01:00
gezimbll
9e2a629b82 Add *log flag for logging ees events 2023-11-08 16:52:38 +01:00
ionutboangiu
62b27b569f Make EventCost type vars satisfy the DataStorage interface 2023-10-26 15:55:54 +02:00
Bas Couwenberg
8894cc19ee Fix spelling errors.
* beacause     -> because
 * disconect    -> disconnect
 * quering      -> querying
 * retrived     -> retrieved
 * syncronous   -> synchronous
 * treshold     -> threshold
 * unsuccesfull -> unsuccessful
2023-09-11 12:24:52 +02:00
ionutboangiu
47af22c724 Update rpcclient library to latest version
`ClientConnector` is no longer defined within `rpcclient` in its latest
version. It has been changed to be obtained from the `cgrates/birpc`
library instead.

Replaced `net/rpc` with `cgrates/birpc` and `net/rpc/jsonrpc` with
`cgrates/birpc/jsonrpc` libraries.

The implementations of `CallBiRPC()` and `Handlers()` were removed,
along with the methods associated with them.

The `rpcclient.BIRPCConector` and the methods prefixed with `BiRPC` were
removed from the `BiRPClient` interface.

The `BiRPClient` interface was renamed to `BIRPCClient`, although not
sure if needed (seems useful just to test if the structure is correct).

`rpcclient.BiRPCConector` has been replaced with `context.ClientConnector`,
which is now passed alongside `context.Context` within the same struct
(`cgrates/birpc/context.Context`). Consequently, functions that were
previously relying on it are now receiving the context instead. The
changes were made in the following functions:

    - `engine/connmanager.go` - `*ConnManager.Call`
    - `engine/connmanager.go` - `*ConnManager.getConn`
    - `engine/connmanager.go` - `*ConnManager.getConnWithConfig`
    - `engine/libengine.go` - `NewRPCPool`
    - `engine/libengine.go` - `NewRPCConnection`
    - `agents/libagents.go` - `processRequest`

Compilation errors related to the `rpcclient.NewRPCClient` function were
resolved by adding the missing `context`, `max_reconnect_interval`, and
`delayFunc` parameters. Additionally, context was added to all calls made
by the client. An effort was made to avoid passing hardcoded values as
much as possible, and extra flags were added where necessary for cgr
binaries.

The `max_reconnect_interval` parameter is now passed from parent
functions, which required adjustments to the function signature.

A new context field was added to all agent objects to ensure access to
it before sending it to the `connmanager's Call`, effectively replacing
`birpcclient`. Although an alternative would have been to create the
new service and add it to the context right before passing it to the
handlers, the chosen approach is definitely more comfortable.

With the addition of a context field for the SIP servers agents, an
additional error needed to be handled, coming from the creation of the
service. Agent constructors within the services package log errors as
they occur and return. Alternate solutions considered were either
shutting down the engine instead of returning, or just logging the
occurrence as a warning, particularly when the `ctx.Client` isn't
required, especially in cases where bidirectional connections are not
needed. For the latter option, it's crucial to return the object with
the error rather than nil or to make the error nil immediately after
logging.

Context has been integrated into all internal Call implementations to
ensure the objects conform to the `birpc.ClientConnector` interface.
These implementations will be removed in the near future as all service
objects are being wrapped in a `birpc.Service` type that satisfies the
`birpc.ClientConnector` interface. Currently, they are being retained
as a reference in case of any unexpected issues that arise.

Ensured that the `birpc.Service` wrapped service objects are passed to
the internal channel getters rather than the objects themselves.

Add context.TODO() to all \*ConnManager.Call function calls. To be
replaced with the context passed to the Method, when available.

For all `*ConnManager.Call` function calls, `context.TODO()` has been
added. This will be replaced with the context passed to the method when
it becomes available.

The value returned by StringGetOpts is now passed directly to the
FirstNonEmpty function, instead of being assigned to a variable
first.

The implementation of the `*AnalyzerService.GetInternalBiRPCCodec`
function has been removed from the services package. Additionally,
the AnalyzerBiRPCConnector type definition and its associated methods
have been removed.

The codec implementation has been revised to include the following
changes:

    - `rpc.ServerCodec` -> `birpc.ServerCodec`;
    - `rpc2.ServerCodec` -> `birpc.BirpcCodec`;
    - `rpc2.Request` -> `birpc.Request`;
    - `rpc2.Response` -> `birpc.Response`;
    - The constructors for json and gob birpc codecs in `cenkalti/rpc`
    have been replaced with ones from the `birpc/jsonrpc` library;
    - The gob codec implementation has been removed in favor of the
    version already implemented in the birpc external library.

The server implementation has been updated with the following changes:

    - A field that represents a simple RPC server has been added to the
    Server struct;
    - Both the simple and bidirectional RPC servers are now initialized
    inside the Server constructor, eliminating the need for nil checks;
    - Usage of `net/rpc` and `cenkalti/rpc2` has been replaced with
    `cgrates/birpc`;
    - Additional `(Bi)RPCUnregisterName` methods have been added;
    - The implementations for (bi)json/gob servers have been somewhat
    simplified.

Before deleting the Call functions and using the `birpc.NewService`
method to register the methods for all cgrates components, update the
Call functions to satisfy the `birpc.ClientConnector` interface. This
way it will be a bit safer. Had to be done for SessionS though.

The `BiRPCCall` method has been removed from coreutils.go. The
`RPCCall` and `APIerRPCCall` methods are also to be removed in the
future.

Ensured that all methods for `SessionSv1` and `SessionS` have the
correct function signature with context. The same adjustments were made
for the session dispatcher methods and for the `SessionSv1Interface`.
Also removed sessionsbirpc.go and smgbirpc.go files.

Implemented the following methods to help with the registration of
methods across all subsystems:

    - `NewServiceWithName`;
    - `NewDispatcherService` for all dispatcher methods;
    - `NewService` for the remaining methods that are already named
    correctly.

Compared to the constructor from the external library, these also make
sure that the naming of the methods is consistent with our constants.

Added context to the Call methods for the mock client connectors (used
in tests).

Removed unused rpc fields from inside the following services:

    - EeS
    - LoaderS
    - ResourceS
    - RouteS
    - StatS
    - ThresholdS
    - SessionS
    - CoreS

Updated the methods implementing the logic for API methods to align
with the latest changes, ensuring consistency and correctness. The
modifications include:

    - Adjusting the function signature to the new format
    (ctx, args, reply).
    - Prefixing names with 'V*' to indicate that they are utilized by
    or registered as APIs.
    - Containing the complete logic within the methods, enabling APIs
    to call them and return their reply directly.

The subsystems affected by these changes are detailed as follows:

    - CoreS: Additional methods were implementing utilizing the
    existing ones. Though modifying them directly was possible, certain
    methods (e.g., StopCPUProfiling()) were used elsewhere and not as
    RPC requests.
    - CDRs: Renamed V1CountCDRs to V1GetCDRsCount.
    - StatS: V1GetQueueFloatMetrics, V1GetQueueStringMetrics,
    V1GetStatQueue accept different arguments compared to API functions
    (opted to register StatSv1 instead).
    - ResourceS: Renamed V1ResourcesForEvent to V1GetResourcesForEvent
    to align with API naming.
    - DispatcherS: Renamed V1GetProfilesForEvent to
    DispatcherSv1GetProfilesForEvent.
    - For the rest, adding context to the function signature was enough.

In the unit tests, wrapping the object within a biprc.Service is now
ensured before passing it to the internal connections map under the
corresponding key.

Some tests that are covering error cases, are also checking the other
return value besides the error. That check has been removed since it
is redundant.

Revised the RPC/BiRPC clients' constructors (for use in tests)

A different approach has been chosen for the handling of ping methods
within subsystems. Instead of defining the same structure in every file,
the ping methods were added inside the Service constructor function.
Though the existing Ping methods were left as they were, they will be
removed in the future.

An additional method has been implemented to register the Ping method
from outside of the engine package.

Implemented Sleep and CapsError methods for SessionS (before they were
exclusively for bidirectional use, I believe).

A specific issue has been fixed within the CapsError SessionSv1 API
implementation, which is designed to overwrite methods that cannot be
allocated due to the threshold limit being reached. Previously, it was
deallocating when writing the response, even when a spot hadn't been
allocated in the first place (due to the cap being hit). The reason
behind this, especially why the test was passing before, still needs
to be looked into, as the problem should have occurred from before.

Implement `*SessionSv1.RegisterInternalBiJSONConn` method in apier.

All agent methods have been registered under the SessionSv1 name. For
the correct method names, the leading "V1" prefix has been trimmed
using the `birpc.NewServiceWithMethodsRename` function.

Revise the RegisterRpcParams function to populate the parameters
while relying on the `*birpc.Service` type instead. This will
automatically also deal with the validation. At the moment,
any error encountered is logged without being returned. Might
be changed in the future.

Inside the cgrRPCAction function, `mapstructure.Decode`'s output parameter
is now guaranteed to always be a pointer.

Updated go.mod and go.sum.

Fixed some typos.
2023-09-01 11:23:54 +02:00
gezimbll
b755fcca01 Added *vars for ExportRequest && unit tests 2023-06-22 10:36:46 +02:00
gezimbll
77e9c5720d Added timestamp for els documents 2023-06-22 10:36:46 +02:00
ionutboangiu
6c16ff320f Replace interface{} with any 2023-05-31 10:23:12 +02:00
ionutboangiu
895c140209 Port dispatcher changes from 1.0 to v0.11 2022-09-25 12:17:37 +02:00
ionutboangiu
de0b0569c3 Update fibDuration function 2022-05-28 09:50:01 +02:00
ionutboangiu
3471d868b6 Add rpcAPIOpts in ees cfg and change PrepareMap structure 2022-02-24 21:29:19 +01:00
porosnicuadrian
c0c3b87b83 Updated *ec dp for ees 2022-02-23 17:16:07 +01:00
porosnicuadrian
8941b1ef0d Added *ec field to ees as DataProvider 2022-02-22 17:37:36 +01:00
ionutboangiu
14dfda53be Implement rpc exporter 2022-01-19 19:55:29 +01:00
ionutboangiu
baeb3f3c0a Change the way context is passed 2021-11-22 11:15:45 +01:00
ionutboangiu
2e241d9632 Remove AttrArgsProcessEvent type and use CGREvent in its stead 2021-11-03 21:03:33 +02:00
ionutboangiu
7250168176 Fix unit tests after adding opts field in attributes cfg struct 2021-11-03 21:03:33 +02:00
ionutboangiu
48feaf2bed Add opts field for attributes cfg struct and fix compilation errors 2021-11-03 21:03:33 +02:00
andronache
bcb564240e Corrected header in multiple files 2021-11-01 13:49:33 +02:00
Trial97
910105182b Updated integration tests 2021-10-21 14:30:20 +03:00
Trial97
239cbb7365 Updated Redis storage 2021-10-08 11:12:04 +03:00
ionutboangiu
8608541aa7 Remove duplicate INFO logger 2021-09-10 11:24:52 +02:00
Trial97
d80a5668ac Updated integration tests 2021-08-18 16:33:28 +02:00
Trial97
cd0964dfea Finished implementing all the exporters 2021-08-18 16:33:28 +02:00
Trial97
a74aacd7fe Updated the EEs with the new interface 2021-08-18 16:33:28 +02:00
Trial97
c84e861e2a Started changing the EEs 2021-08-18 16:33:28 +02:00
Trial97
f1796317a0 Updated ees metrics 2021-07-23 10:57:08 +02:00
DanB
36cf49811c Small var fix in ees 2021-07-20 13:37:13 +02:00