Commit Graph

344 Commits

Author SHA1 Message Date
arberkatellari
5463eb61c6 Adjust tests to InternalDB constructor 2025-04-11 13:09:50 +02:00
arberkatellari
ed584364d1 Add tests for internal DB dump and restore 2025-04-06 12:08:40 +02:00
arberkatellari
a168f262e2 Add dump and restore functionality for internal DBs 2025-04-06 12:08:40 +02:00
gezimbll
18644c7f0b added option to compress CostDetails before stored in db 2025-03-20 19:37:43 +01:00
gezimbll
260b32323d added option to pass UpdateSession event through stats or thresholds 2025-03-18 20:08:05 +01:00
ionutboangiu
96c2a3acee Prevent deadlocks in file reader unit tests 2025-01-22 17:37:17 +01:00
ionutboangiu
c171937c3d Add WaitGroup to ensure safe file processing in ERs
For cases when run_delay > 0.
2025-01-21 19:30:33 +01:00
arberkatellari
b8cc20bb7c Add sqlBatchSize to ERS SQL OPTS and adjust delete functionality 2025-01-14 17:04:18 +01:00
arberkatellari
2252765e93 Add 'ees_ids' field to ERs readers config 2024-12-20 17:33:45 +01:00
arberkatellari
d35b14b6b9 Add ability to ERS to update or move ERS SQL events to a new table & add *export flag for ERS readers 2024-12-20 13:26:06 +01:00
gezimbll
cf44898b98 revise tests for trends storeinterval && file readers 2024-12-07 20:59:04 +01:00
arberkatellari
6c752ac10f Improvements to ERS SQL filters 2024-12-03 18:36:22 +01:00
arberkatellari
041b14fa03 Improvements to ERS SQL filters 2024-12-03 18:36:22 +01:00
arberkatellari
321910d181 Add WHERE statement availability and non-delete option to ERS SQL reader 2024-12-03 18:36:22 +01:00
gezimbll
3974192787 added asynchronus startdelay for file readers and nats 2024-11-26 11:20:40 +01:00
gezimbll
e48eb5df13 added StartDelay for ers processing 2024-11-15 20:58:10 +01:00
ionutboangiu
aad5c4a66f Add benchmark for diameter+caps 2024-10-29 18:53:22 +01:00
ionutboangiu
ec301b1c9d Improve nats integration test
- made use of the test setup helpers.
- used t.Cleanup instead of defer.
- instead of waiting 50ms for the nats-server to start, used a helper
  hook to attempt connections in fibonacci intervals. On success it
  keeps a reference to the connection for later usage.
- handle error for stream delete function executed during cleanup.
- shorten time.Sleep durations when waiting for exports to finish.
- extract the cache itemID checking logic into a separate func
- retry failed requests in fibonacci intervals for up to 500ms
2024-10-16 18:43:34 +02:00
ionutboangiu
769293d4db XML rdr: use rowNr instead of reqVars for line nr 2024-09-26 17:33:16 +02:00
ionutboangiu
01dd4696cf Fix flaky nats integration test
Was failing on the Jetstream JWTAuth case with error:

nats: API error: code=500 err_code=10047 description=insufficient
storage resources available

The issue was with the previously generated credentials. Generated
them again and added the program used as a comment at the end of
the file for future reference"
2024-09-23 11:54:13 +02:00
armirveliaj
7560328388 Add coverage tests on ers and config 2024-09-10 13:20:39 +02:00
gezimblliku
ccc999642d added *fileLineNumber variable 2024-08-22 20:40:46 +02:00
armirveliaj
2f9f07a976 Add new unit tests on ees and ers 2024-08-09 09:18:46 +02:00
arberkatellari
0bab7d5557 Add reader id within the *vars in ERs 2024-08-09 09:14:24 +02:00
gezimbll
5f941b1f9d changed conReqs to not populate buffer at start 2024-04-24 17:09:54 +02:00
ionutboangiu
cf158a3b40 Revise client init in nats ers test
The usual newRPCClient is not reachable due to its file having
a different build constraint
2024-04-10 12:56:36 +02:00
ionutboangiu
d8946e73ee Make small revisions to a few integration tests
-update cached filter indexes count
-add the flaky constraint to a couple more tests
-fix unstable cores integration test
-fix unstable filterindexes integration test
2024-04-09 20:24:01 +02:00
ionutboangiu
3062113a61 Make go vet pass
The rpcclient constructor could not see the centralized
Encoding flag because it didn't have the necessary build
constraints. Added the constraints in lib_test.go files
where it wasn't alone. In all the other cases, it was
moved to the first file where it was needed.
2024-04-08 13:14:02 +02:00
ionutboangiu
16eb745f00 Centralize test flags in utils for shared use across tests 2024-04-08 13:14:02 +02:00
ionutboangiu
5c1e128f4f Update behaviour of file readers using inotify
Ensure that files already existing in the source path are processed
before the reader starts listening for filesystem change events.
2024-04-04 12:31:18 +03:00
ionutboangiu
a362511d7a Define param struct for ErSv1.RunReader API + refactoring
Rename dir field to sourceDir in file event readers for clarity.
2024-04-04 12:31:18 +03:00
ionutboangiu
5116a23f5b Set ChargerProfile through TPs instead of API in tests
Only for agents/ers tests. Done to prevent cyclic imports when
adding ers APIs to apier package.

In case of v2.AttrSetAccount, use a literal struct instead.
2024-04-04 12:31:18 +03:00
ionutboangiu
b9a39e233f Implement ErSv1.ProcessDir api
cores.Server, analyzer object and internal conn channel specific to ERs
are now part of the ERService struct and are passed to its constructor.
2024-04-04 12:31:18 +03:00
ionutboangiu
b9b07dc561 Remove redundant dir parameter from processFile methods
Applies to both file readers and loader (for loader, the blank statement
was used anyway).

It's redundant because for file readers, the rdr.dir value was always
passed as the parameter when it was already part of the method's object.

Parameter had to also be removed from the WatchDir function and the
functions it depends on.
2024-04-04 12:31:18 +03:00
ionutboangiu
85864ccaa3 Extract common dir processing logic to a func
It will be reused for all file readers.
Rename rdrDir field to dir (redundant prefix).
2024-04-04 12:31:18 +03:00
ionutboangiu
619a1efa50 Revise ers integration tests
They also do not depend on nats server dependency anymore.
2024-03-22 15:12:04 +01:00
gezimbll
66940db0b0 changed concurrentEvents naming in ers 2024-03-13 18:26:01 +01:00
gezimbll
21d65b3810 added concurrent event processing for ers 2024-03-13 18:26:01 +01:00
ionutboangiu
0aea8ac641 Revert "Move CGREvent from utils to engine"
This reverts commit 02195c47ed.
2024-03-04 09:49:15 +01:00
ionutboangiu
02195c47ed Move CGREvent from utils to engine 2024-02-27 17:16:57 +01:00
ionutboangiu
36bdc41e97 Add support for kafka ssl encryption
For both poster and reader.
Added integration test.
2024-01-23 14:12:33 +01:00
ionutboangiu
0c91933065 Revise ERs event exporting
Add ees_success_ids and ees_failed_ids fields in reader config. The
former will be used to set EeIDs when the event processing returns
no error, while the latter will be used otherwise.

Add config sanity checks for the added options.

Remove Processed opts and everything related to them since they should
not be used anymore.

Fixed test compilation errors caused by the change.
2023-11-29 16:21:51 +01:00
gezimbll
6161d33617 Resolve amqp test compilation error 2023-11-15 10:25:21 +01:00
ionutboangiu
a440c18c0c Prolong expiration date for balance in unit test 2023-11-14 18:26:54 +01:00
ionutboangiu
c282d40201 Implement reconnect logic for AMQP EventReader
Implemented functionality that handles reconnecting to the amqp server
and reinitializing the amqp channel in case of errors and timeouts. This
is handled by a goroutine created in the client constructor (it also
handles the initial connect/init).

Reconnects and reinits will use a fibonacci backoff strategy, and the
attempt amount and max waiting interval can be adjusted by the
'reconnects' and 'max_reconnect_interval' config options.

Messages that fail processing are now dropped instead of being requeued,
preventing infinite processing loops. However, this means that the
messages are lost. Handling failed messages will need to be addressed
separately.

'concurrent_requests' will now set the prefetch count. Setting the
prefetch count using the Qos function was able to replace our old
approach that was using channels. Default value is 1024 which,
according to the rabbitmq docs, 'runs into the law of diminishing
returns'. The recommended value is between 100-300. Source:
https://www.rabbitmq.com/confirms.html#channel-qos-prefetch-throughput

Fix test compilation errors and failing tests caused by these changes.

References #4160
2023-11-08 18:31:07 +01:00
gezimbll
09e5088262 Removed warnings from go staticcheck 2023-11-08 17:10:13 +01:00
gezimbll
5a85c119c8 Replacing IsSliceMember ,CloneStringSlice and SliceStringEqual with slices package equivalent functions 2023-11-01 21:52:50 +01:00
ionutboangiu
d1d43913c8 Replace deprecated Publish method with PublishWithContext 2023-10-27 18:51:01 +02:00
ionutboangiu
1c0ca857ee Update libraries and fix compilation errors 2023-10-17 17:40:43 +02:00
ionutboangiu
37ac937f97 Handle kafka topic creation/deletion within test 2023-10-17 17:40:43 +02:00