- made use of the test setup helpers.
- used t.Cleanup instead of defer.
- instead of waiting 50ms for the nats-server to start, used a helper
hook to attempt connections in fibonacci intervals. On success it
keeps a reference to the connection for later usage.
- handle error for stream delete function executed during cleanup.
- shorten time.Sleep durations when waiting for exports to finish.
- extract the cache itemID checking logic into a separate func
- retry failed requests in fibonacci intervals for up to 500ms
Was failing on the Jetstream JWTAuth case with error:
nats: API error: code=500 err_code=10047 description=insufficient
storage resources available
The issue was with the previously generated credentials. Generated
them again and added the program used as a comment at the end of
the file for future reference"
-update cached filter indexes count
-add the flaky constraint to a couple more tests
-fix unstable cores integration test
-fix unstable filterindexes integration test
The rpcclient constructor could not see the centralized
Encoding flag because it didn't have the necessary build
constraints. Added the constraints in lib_test.go files
where it wasn't alone. In all the other cases, it was
moved to the first file where it was needed.
Only for agents/ers tests. Done to prevent cyclic imports when
adding ers APIs to apier package.
In case of v2.AttrSetAccount, use a literal struct instead.
Applies to both file readers and loader (for loader, the blank statement
was used anyway).
It's redundant because for file readers, the rdr.dir value was always
passed as the parameter when it was already part of the method's object.
Parameter had to also be removed from the WatchDir function and the
functions it depends on.
Add ees_success_ids and ees_failed_ids fields in reader config. The
former will be used to set EeIDs when the event processing returns
no error, while the latter will be used otherwise.
Add config sanity checks for the added options.
Remove Processed opts and everything related to them since they should
not be used anymore.
Fixed test compilation errors caused by the change.
Implemented functionality that handles reconnecting to the amqp server
and reinitializing the amqp channel in case of errors and timeouts. This
is handled by a goroutine created in the client constructor (it also
handles the initial connect/init).
Reconnects and reinits will use a fibonacci backoff strategy, and the
attempt amount and max waiting interval can be adjusted by the
'reconnects' and 'max_reconnect_interval' config options.
Messages that fail processing are now dropped instead of being requeued,
preventing infinite processing loops. However, this means that the
messages are lost. Handling failed messages will need to be addressed
separately.
'concurrent_requests' will now set the prefetch count. Setting the
prefetch count using the Qos function was able to replace our old
approach that was using channels. Default value is 1024 which,
according to the rabbitmq docs, 'runs into the law of diminishing
returns'. The recommended value is between 100-300. Source:
https://www.rabbitmq.com/confirms.html#channel-qos-prefetch-throughput
Fix test compilation errors and failing tests caused by these changes.
References #4160
Previously, msg.Ack(true) was used, which is mostly used for batch
processing. It mistakenly acknowledged all previously unacknowledged
messages, causing errors from the AMQP server. Now, messages are
acknowledged individually after each one is processed.
Messages that ERs failed to process are now rejected and requeued
for future processing attempts.
The reader is now closed immediately if the message delivery
channel closes. Therefore, it prevents an endless loop by avoiding
continuous consumption from empty or closed channels.
Addresses: #4146
HierarchyPath parser now returns nil when
the path is empty (instead of a string slice with one
EmptyString element).
If the prefix is set to true, when calling the AsString
method on a nil HierarchyPath, only the separator will
be returned. This avoids a nil expr error coming from
the xmlquery library.
Use the Query and QueryAll functions from the xmlquery
package to be able to handle the errors ourselves and
avoid panics.
Remove config default value for xml_root_path. The field
will remain commented in config_defaults for reference.
Add tests for HierarchyPath.AsString function.
Add comments for XmlProvider and xml_root_path opt.
Updated ees/ers implementation to use the jetstream package which
separates the jetstream context from Core NATS.
Removed the jsOpts fields from the NatsEE struct. We are now using the
jetStreamMaxWait option directly through a timeout context.
Added streamName option for NATS reader since it is now required to be
specified when creating a consumer (it is not inferred based on subject
anymore).
Updated nats ers integration test.
Removed deprecated birpc unit test.
Updated tests to also use the new jetstream package.
Updated tests to start the nats-server using their official driver
instead of using the std go exec package. time.Sleeps are now not
required anymore to wait for the server.
In test configurations for nats readers, made sure that natsStreamName
option is populated. It is now required for consumers to know where to
subscribe.
Fixed potential panic that happened when jetstreamMaxWait option would
not be set.
