diff --git a/docs/trends.rst b/docs/trends.rst index 40d6a2dc4..1fb19e5c7 100644 --- a/docs/trends.rst +++ b/docs/trends.rst @@ -4,17 +4,52 @@ TrendS ===== -**TrendS** is a standalone subsystem part of the **CGRateS** infrastructure, designed to store *StatS* in a time-series-like database and calculate trend percentages based on their evolution. +**TrendS** is a standalone subsystem part of the **CGRateS** infrastructure, designed to work as an extension of the :ref:`StatS`, by regularly querying it, storing it's values in a time-series-like database and calculate trend percentages based on their evolution. Complete interaction with **TrendS** is possible via `CGRateS RPC APIs `_. -Due it's real-time nature, **StatS** are designed towards high throughput being able to process thousands of *Events* per second. This is doable since each *StatQueue* is a very light object, held in memory and eventually backed up in *DataDB*. +Due it's real-time nature, **TrendS** are designed towards high throughput being able to process thousands of queries per second. This is doable since each *Trend* is a very light object, held in memory and eventually backed up in :ref:`DataDB`. Processing logic ---------------- +In order for **TrendS** to start querying the :ref:`StatS`, it will need to be *scheduled* to do that. Scheduling is being done using `Cron Expressions `_. +Once *Cron Expressions* are defined within a *TrendProfile*, internal **Cron Scheduler** needs to be triggered. This can happen in two different ways: + +Automatic Query Scheduling + The profiles needing querying will be inserted into **trends** :ref:`JSON configuration `. By leaving any part of *trend_id* or *tenat* empty, it will be interpreted as catch-all filter. + +API Scheduling + The profiles needing querying will be sent inside arguments to the `TrendSv1.ScheduleQueries API call `_. + + +Offline storage +--------------- + +Offline storage is optionally possible, by enabling profile *Stored* flag and configuring the *store_interval* inside :ref:`JSON configuration `. + + +Trend querying +-------------- + +In order to query a **Trend** (ie: to be displayed in a web interface), one should use the `TrendSv1.GetTrend API call `_ which also offers pagination parameters. + + +Trend exporting +--------------- + +On each **Trend** change, it will be possible to send a specially crafted event, *TrendUpdate* to one of the following subsystems: + +**ThresholdS** + Sending the **TrendUpdate** Event gives the administrator the possiblity to react to *Trend* changes, including escalation strategies offered by the **TresholdS** paramters. + Fine tuning parameters (ie. selecting only specific ThresholdProfiles to increase speed1) are available directly within the **TrendProfile**. + +**EEs** + **EEs** makes it possible to export the **TrendUpdate** to all the availabe outside interfaces of **CGRateS**. + +Both exporting options are enabled within :ref:`JSON configuration `. Parameters @@ -24,13 +59,30 @@ Parameters TrendS ^^^^^^ -**TrendS** is the **CGRateS** component responsible of handling the *Trend* queries. +**TrendS** is the **CGRateS** component responsible of generating the **Trend** queries. It is configured within **trends** section from :ref:`JSON configuration ` via the following parameters: enabled Will enable starting of the service. Possible values: . +store_interval + Time interval for backing up the trends into *DataDB*. 0 To completely disable the functionality, -1 to enable synchronous backup. Anything higher than 0 will give the interval for asynchronous backups. + +stats_conns + List of connections where we will query the stats. + +scheduled_ids + Limit the TrendProfiles queried. Empty to query all the available TrendProfiles or just tenants for all the available profiles on a tenant. + +thresholds_conns + Connection IDs towards *ThresholdS* component. If not defined, there will be no notifications sent to *ThresholdS* on *Trend* changes. + +ees_conns + Connection IDs towards the *EEs* component. If left empty, no exports will be performed on *Trend* changes. + +ees_exporter_ids + Limit the exporter profiles executed on *Trend* changes. TrendProfile @@ -44,15 +96,65 @@ Tenant ID Identifier for the *TrendProfile*, unique within a *Tenant*. -FilterIDs - List of *FilterProfileIDs* which should match in order to consider the profile matching the event. +Schedule + Cron expression scheduling gathering of the metrics. +StatID + StatS identifier which will be queried. +Metrics + Limit the list of metrics from the stats instance queried. + +TTL + Automatic cleanup of the queried values from inside *Trend Metrics*. + +QueueLength + Limit the size of *Trend Metrics*. Older values will be removed first. + +MinItems + Issue *TrendUpdate* events to external subsystems only if MinItems are reched to limit false alarms. + +CorrelationType + The correlation strategy to use when computing the trend. *\*average* will consider all previous query values and *\*last* only the last one. + +Tolerance + Allow a deviation of the values when computin the trend. This is defined as percentage of increase/decrease. + +Stored + Enable storing of this *Trend* for persistence. + +ThresholdIDs + Limit *TresholdProfiles* processing the *TrendUpdate* for this *TrendProfile*. Trend ^^^^^ +is made out of the following fields: + +Tenant + The tenant on the platform (one can see the tenant as partition ID). + +ID + Unique *Trend* identifier on a *Tenant* + +RunTimes + Times when the stat queries were ran by the scheduler + +Metrics + History of the queried metrics, indexed by the query time. One query stores the following values: + + ID + Metric ID on the *StatS* side + + Value + Value of the metric at the time of query + + TrendGrowth + Computed trend growth for the metric values, stored in percentage numbers. + + TrendLabel + Computed trend label for the metric values. Possible values are: *positive, *negative, *constant, N/A. Use cases diff --git a/engine/libtrends.go b/engine/libtrends.go index c581fb641..ff79326be 100644 --- a/engine/libtrends.go +++ b/engine/libtrends.go @@ -92,7 +92,8 @@ func NewTrendFromProfile(tP *TrendProfile) *Trend { ID: tP.ID, RunTimes: make([]time.Time, 0), Metrics: make(map[time.Time]map[string]*MetricWithTrend), - tPrfl: tP, + + tPrfl: tP, } } @@ -100,10 +101,11 @@ func NewTrendFromProfile(tP *TrendProfile) *Trend { type Trend struct { tMux sync.RWMutex - Tenant string - ID string - RunTimes []time.Time - Metrics map[time.Time]map[string]*MetricWithTrend + Tenant string + ID string + RunTimes []time.Time + Metrics map[time.Time]map[string]*MetricWithTrend + CompressedMetrics []byte // if populated, Metrics and RunTimes will be emty // indexes help faster processing mLast map[string]time.Time // last time a metric was present