Commit Graph

13721 Commits

Author SHA1 Message Date
ionutboangiu
f5d69cb41b Implement ProcessStoredEvents API 2023-11-16 15:38:18 +01:00
ionutboangiu
e218ca06a8 Add consts for AdminSv1 CDRs api names 2023-11-15 17:25:21 +01:00
ionutboangiu
1ef2afaa97 Update/remove dependencies
Remove cenkalti/rpc2 dependency and update client constructor.
Commented tests that only start and stop engine.

Fixed compilation errors caused by Azure/go-amqp update.

Remove cgrates/ugocodec dependency since we have everything
we need in ugoriji/go/codec and add tests for it.
2023-11-15 14:03:21 +01:00
ionutboangiu
26cdb571b8 Upgrade MongoDB driver to 1.13
- Set (but comment) serverAPI options (currently distinct api and
create.size BSON field are deprecated + possible others that are untested)
- Remove the custom time decoder used for mongo BSON
datetime values. The custom decoder was only converting these values
into UTC and was not any different from the default time.Time
decoder in the MongoDB driver, which also handles BSON string, int64,
and document values.
- Implement 'buildURL' function to connect to mongo (can also be
used for mysql and postgres)
- Update function names, variable names, and comments for clarity
- Replace 'bsonx.Regex' with the Regex primitive (deprecated since 1.12).
- Use simple concatenation instead of Sprintf
- Declare 'decimalType' locally, replace global 'decimalType'
- Simplify several functions without altering functionality
- Converting directly from a D to an M is deprecated. We are now decoding
  directly in a M.
- Used errors.As and errors.Is for proper error comparison and assertion
- Revised sloppy reassignments and added missing error checks
2023-11-14 18:24:21 +01:00
ionutboangiu
e5ab21def5 Implement AdminSv1 Get/RemoveCDRs APIs for CDRs
Addition of storDB to AdminSv1 was required.
2023-11-09 20:50:36 +01:00
ionutboangiu
d5039b3ce8 Add CDRs CRUD queries back and remove TP functions 2023-11-09 20:50:36 +01:00
ionutboangiu
6b831d3ba3 Make Paginate function generic
Also move the paginator implementation and its tests to their
own files.
2023-11-09 20:50:36 +01:00
ionutboangiu
d7679e2e9a Add negative filters to filter type string sets 2023-11-09 20:50:36 +01:00
gezimbll
32ed816de7 Replacing IsSliceMember ,CloneStringSlice and SliceStringEqual with slices package functions 2023-11-01 21:38:27 +01:00
ionutboangiu
65310e7437 Move benchmark relying on redis to it_test file 2023-10-27 18:52:05 +02:00
ionutboangiu
46e39770a1 Replace deprecated Publish method with PublishWithContext 2023-10-27 18:52:05 +02:00
ionutboangiu
7fd7167aa5 Update expected value in kafka it test 2023-10-26 16:26:27 +02:00
ionutboangiu
c47a910386 Update README links to docs 2023-10-26 16:26:27 +02:00
ionutboangiu
b56de12ee1 Bump mongodb version in ansible role 2023-10-26 16:26:27 +02:00
gezimbll
c6571492fc Make go vet pass 2023-10-26 15:59:34 +02:00
gezimbll
7c80bd1002 Remove unreachable err cases from HandleJSONErr
json.UnmarshalError happens only when we pass a non-nil pointer, which
is not the case for us. Usage of go vet helps us make sure it also
won't be happening in the future
2023-10-26 15:59:34 +02:00
arberkatellari
0ca5d3ad20 Change default storedb password 2023-10-25 16:37:19 +02:00
ionutboangiu
6bcc374d45 Update EndOfMonth coreutils test 2023-10-19 17:19:30 +02:00
ionutboangiu
ccd4a92432 Update github workflows 2023-10-19 17:19:30 +02:00
ionutboangiu
4c34d321de Add sanity check to prevent xml reader panic
HierarchyPath parser now returns nil when
the path is empty (instead of a string slice with one
EmptyString element).

If isAbsolute is set to true, when calling the AsString
method on a nil HierarchyPath, only the separator will
be returned. Alternatively, if isAbsolute is false, it
will just return . to signal that the path currently
being processed is the one required.This avoids a nil
expr error coming from the xmlquery library.

Use the Query and QueryAll functions from the xmlquery
package to be able to handle the errors ourselves and
avoid panics.

Remove config default value for xmlRootPath. The field
will remain commented in config_defaults for reference.

Add tests for HierarchyPath.AsString function.

Add comments for XmlProvider and xmlRootPath opt.

Update XmlProvider String() func based on the comment
which specifies that it should display the already
parsed values out of cache.
2023-10-19 17:19:30 +02:00
ionutboangiu
dffc617e17 Update kafka handler to run with sudo 2023-10-19 17:19:30 +02:00
ionutboangiu
2869728de0 Handle kafka topic creation/deletion within test 2023-10-17 17:39:14 +02:00
ionutboangiu
5f079fbca8 Update kafka ansible role
- use kraft instead of zookeeper
- add handlers in case of cfg changes
- create a separate user for the kafka service
- bump kafka version
- make the role more configurable
2023-10-17 17:39:14 +02:00
ionutboangiu
2867942ebe Update AMQP message processing loop
Previously, msg.Ack(true) was used, which is mostly used for batch
processing. It mistakenly acknowledged all previously unacknowledged
messages, causing errors from the AMQP server. Now, messages are
acknowledged individually after each one is processed.

Messages that ERs failed to process are now rejected and requeued
for future processing attempts.

The reader is now closed immediately if the message delivery
channel closes. Therefore, it prevents an endless loop by avoiding
continuous consumption from empty or closed channels.

Addresses: #4146
2023-10-16 18:57:16 +03:00
ionutboangiu
20f1442683 Update rabbitmq role for latest debian/ubuntu compatibility 2023-10-16 18:57:16 +03:00
ionutboangiu
b09512a5c5 Implement *strip converter + tests 2023-10-09 21:08:28 +02:00
ionutboangiu
efcb32741a Run gofmt 2023-09-28 20:48:39 +02:00
ionutboangiu
d59025e75f Update integration_tests ansible playbook to use roles 2023-09-28 20:48:39 +02:00
ionutboangiu
1563947f9d Update default cgrates.json 2023-09-28 20:48:39 +02:00
ionutboangiu
c2caaba4e1 Import github workflows 2023-09-28 20:48:39 +02:00
ionutboangiu
bbf1eaf439 Import ansible roles 2023-09-28 20:48:39 +02:00
ionutboangiu
d0a435aa6d Migrate to new jetstream API
Upgraded go.mod nats version due to an issue caused by version mismatch
between driver and server (uncertain).

Renamed function from getProcessOptions to getProcessedOptions.

## *NatsER.Serve

- Replaced ChanQueueSubscribe with QueueSubscribe for Core NATS consumer
to handle the message processing directly.

- Since QueueSubscribe is now used regardless of jetstream status, the
message handler has been assigned to a separate variable that can be
reused.

-  The message handler is now dealing with the message processing
directly, therefore the select case listening for the channel which is
feeding NATS messages can be removed together with the channel itself
and the select. Currently, the goroutine within Serve only has to block
until the rdrExit chan is closed.

- Moved the resource check inside the handler right before starting the
message processing goroutine.

## *NatsEE.parseOpts

- Renamed function from parseOpt to parseOpts.

- Handled the error coming from GetNatsOpts function.

## *NatsEE.Connect

- Updated function to return early in case of non-nil nats.Conn value
to reduce nesting.

## *NatsEE.ExportEvent

- Use defer to release resources and RUnlock.

## *NatsEE.Close

- Use defer to Unlock.

- Update function to return early in case of nil nats.Conn value to
reduce nesting.

## ees.GetNatsOpts

- Chose switch over if else when parsing client certificate and keys
opts.

- Updated function to return the errors directly instead of assigning
them to a separate variable right before returning.

## ers.GetNatsOpts

- Chose switch over if else when parsing client certificate and keys
opts.

- Updated function to return the errors directly instead of assigning
them to a separate variable right before returning.

Removed tab from commented natsJetStreamMaxWaitProcessed option
value in config_defaults.go under ers section.

Added integration test for ERs NATS.

Updated ees/ers implementation to use the jetstream package which
separates the jetstream context from Core NATS.

Removed the jsOpts fields from the NatsEE struct. We are now using the
jetStreamMaxWait option directly through a timeout context.

Added streamName option for NATS reader since it is now required to be
specified when creating a consumer (it is not inferred based on subject
anymore).

Updated nats ers integration tests.

Updated tests to also use the new jetstream package.

Updated tests to start the nats-server using their official driver
instead of using the std go exec package. time.Sleeps are now not
required anymore to wait for the server.

In test configurations for nats readers, made sure that natsStreamName
option is populated. It is now required for consumers to know where to
subscribe.
2023-09-28 20:48:39 +02:00
arberkatellari
fdadd1ab81 Improve functionality for Drone CI 2023-09-26 21:30:25 +02:00
NikolasPetriti
8ef305d64c Add coverage tests for utils and config 2023-09-12 18:10:23 +02:00
NikolasPetriti
24dde0f351 Add coverage tests for config 2023-09-11 12:20:45 +02:00
NikolasPetriti
c474c9b8af Add coverage tests for config 2023-09-06 19:12:44 +03:00
arberkatellari
1445fa3b49 Renew Drone CI 2023-09-01 18:01:43 +02:00
NikolasPetriti
61a781675c Fix comment in models 2023-08-29 17:00:14 +02:00
NikolasPetriti
039b4269e8 Add tests for msgpack encoding/decoding for all basic go types that we are using in 1.0 2023-08-29 17:00:14 +02:00
NikolasPetriti
512f0a7c2d Revise struct tags for tp models 2023-08-29 17:00:14 +02:00
arberkatellari
770454c007 Renew DlgList, DNSAgent & add SRV/A support 2023-08-28 12:27:53 +02:00
arberkatellari
6eda4303f5 Revise DNS Agent and Service 2023-08-28 12:27:53 +02:00
gezimbll
990feb0f26 fixing tests for ees 2023-08-02 16:49:33 +02:00
gezimbll
16238fb3cb Added sentrypeercfg tests 2023-08-02 16:49:33 +02:00
gezimbll
7cb144f57b Added sentrypeer filter type 2023-08-02 16:49:33 +02:00
gezimbll
5b5a159706 Added sentrypeer filter type 2023-08-02 16:49:33 +02:00
arberkatellari
fcaa3fb360 Make RandomInteger() return int64 2023-07-21 09:03:22 +02:00
ionutboangiu
f81aba7480 Add integration test for the shared client deadlock issue 2023-07-10 13:55:06 +02:00
gezimbll
fcc9fc109b Updated go-elasticsearch client 2023-07-04 13:18:54 +02:00
gezimbll
2e96f02be4 Fix message on agents logs 2023-06-22 10:38:21 +02:00