- renamed parameter type: ArgsReplyFailedPosts -> ReplayEventsParams
- renamed param fields:
- FailedRequestsInDir -> SourcePath
- FailedRequestsOutDir -> FailedPath
- TypeProvider -> Provider
- changed param fields types from *string to string
- used the SourcePath and FailedPath params directly instead of creating separate variables
- used filepath.WalkDir instead of reading the directory and looping over the entries
- used slices.ContainsFunc to check if the file belongs to any module (if 1+ is specified)
- used filepath.Join instead of path.Join
- used the path provided by WalkFunc instead of building the file paths ourselves
- made error returns more descriptive
- added logs for directories/files that are skipped
- paths that cannot be accessed are skipped after logging the error
200 -> 16
Previously, considering the fibonacci back off strategy, if engine failed to
start it would have tried to attempt connections for 2.862799e+33 years.
Now, the amount of attempts is 16, equivalent to around 2 and a half seconds.
Should be more than enough, considering that usually 6-7 attempts are enough.
- optimized it (2s -> 0.03s) by lowering the cfg cache TTL to 5ms
instead of 1s, which was enough to get consistent results.
- increase margin of error by 1ms
- in case of failure, retrieve the expiry time and log it along the
current time. Makes troubleshooting easier in the future.
- made use of the test setup helpers.
- used t.Cleanup instead of defer.
- instead of waiting 50ms for the nats-server to start, used a helper
hook to attempt connections in fibonacci intervals. On success it
keeps a reference to the connection for later usage.
- handle error for stream delete function executed during cleanup.
- shorten time.Sleep durations when waiting for exports to finish.
- extract the cache itemID checking logic into a separate func
- retry failed requests in fibonacci intervals for up to 500ms
- added hook support (executed after parsing config but before
starting engine)
- made db resets configurable
- merged config parsing helper with the main Setup function
- removed engineDelay parameter
- replaced t.Log with t.Error for engine process kill error
- improved option comments
Applies to both file readers and loader (for loader, the blank statement
was used anyway).
It's redundant because for file readers, the rdr.sourceDir value was
always passed as the parameter when it was already part of the method's
object.
Parameter had to also be removed from the WatchDir function and the
functions it depends on.
They were failing after the ERs structure changes.
Added a commented helper program at the end of the test to
help generate credentials for testing nats JWT functionality
Implemented functionality that handles reconnecting to the amqp server
and reinitializing the amqp channel in case of errors and timeouts. This
is handled by a goroutine created in the client constructor (it also
handles the initial connect/init).
Reconnects and reinits will use a fibonacci backoff strategy, and the
attempt amount and max waiting interval can be adjusted by the
'reconnects' and 'max_reconnect_interval' config options.
Messages that fail processing are now dropped instead of being requeued,
preventing infinite processing loops. However, this means that the
messages are lost. Handling failed messages will need to be addressed
separately.
'concurrent_requests' will now set the prefetch count. Setting the
prefetch count using the Qos function was able to replace our old
approach that was using channels. Default value is 1024 which,
according to the rabbitmq docs, 'runs into the law of diminishing
returns'. The recommended value is between 100-300. Source:
https://www.rabbitmq.com/confirms.html#channel-qos-prefetch-throughput
Make use of the previously added ees_success_ids and ees_failed_ids
configuration fields.
Remove Processed opts and everything related to them since they should
not be used anymore.
Fixed test compilation errors caused by the change.
Add ees_success_ids and ees_failed_ids fields in reader config. The
former will be used to set EeIDs when the event processing returns
no error, while the latter will be used otherwise (implementation
will follow).
Add config sanity checks for the added options.
They are separate for each configured reader.
Additional changes:
- rearrange config_defaults fields for ers/ees;
- add comment for RunDelay config option inside struct definition;
- improve comments for amqp opts in config_defaults.
Behind http.Header is just a map and it's not safe for concurrent use.
Before this change, a panic might have occurred when doing asynchronous
HTTP exports (applies to both *http_post and *http_json_map exporters).
Cloning the header before adding it to the HTTP request has fixed this
issue.
Slightly improved the test that found this data race.
NewFileCSVStorage() now returns an error besides the storage struct itself, which is
logged and returned instead of calling log.Fatal() which was causing the engine to
crash.
Fixed compilation errors by creating the CSVStorage separately and passing it as an
argument to the TpReader constructor.
Fixes#3962
