Completing TrendS documentation for readthedocs

docs/trends.rst

@@ -4,17 +4,52 @@ TrendS
 =====
 
 
-**TrendS** is a standalone subsystem part of the **CGRateS** infrastructure, designed to store *StatS* in a time-series-like database and calculate trend percentages based on their evolution.
+**TrendS** is a standalone subsystem part of the **CGRateS** infrastructure, designed to work as an extension of the :ref:`StatS`, by regularly querying it, storing it's values in a time-series-like database and calculate trend percentages based on their evolution.
 
 Complete interaction with **TrendS** is possible via `CGRateS RPC APIs <https://pkg.go.dev/github.com/cgrates/cgrates/apier@master/>`_.
 
-Due it's real-time nature, **StatS** are designed towards high throughput being able to process thousands of *Events* per second. This is doable since each *StatQueue* is a very light object, held in memory and eventually backed up in *DataDB*.
+Due it's real-time nature, **TrendS** are designed towards high throughput being able to process thousands of queries per second. This is doable since each *Trend* is a very light object, held in memory and eventually backed up in :ref:`DataDB`.
 
 
 Processing logic
 ----------------
 
+In order for **TrendS** to start querying the :ref:`StatS`, it will need to be *scheduled* to do that. Scheduling is being done using `Cron Expressions <https://en.wikipedia.org/wiki/Cron>`_. 
 
+Once *Cron Expressions* are defined within a *TrendProfile*, internal **Cron Scheduler** needs to be triggered. This can happen in two different ways:
+
+Automatic Query Scheduling
+	The profiles needing querying will be inserted into **trends** :ref:`JSON configuration <configuration>`. By leaving any part of *trend_id* or *tenat* empty, it will be interpreted as catch-all filter.
+
+API Scheduling
+	The profiles needing querying will be sent inside arguments to the `TrendSv1.ScheduleQueries API call <https://pkg.go.dev/github.com/cgrates/cgrates/apier@master/>`_.
+
+
+Offline storage
+---------------
+
+Offline storage is optionally possible, by enabling profile *Stored* flag and configuring the *store_interval* inside :ref:`JSON configuration <configuration>`.
+
+
+Trend querying
+--------------
+
+In order to query a **Trend** (ie: to be displayed in a web interface), one should use the `TrendSv1.GetTrend API call <https://pkg.go.dev/github.com/cgrates/cgrates/apier@master/>`_ which also offers pagination parameters.
+
+
+Trend exporting
+---------------
+
+On each **Trend** change, it will be possible to send a specially crafted event, *TrendUpdate* to one of the following subsystems:
+
+**ThresholdS**
+	Sending the **TrendUpdate** Event gives the administrator the possiblity to react to *Trend* changes, including escalation strategies offered by the **TresholdS** paramters. 
+	Fine tuning parameters (ie. selecting only specific ThresholdProfiles to increase speed1) are available directly within the **TrendProfile**.
+
+**EEs**
+	**EEs** makes it possible to export the **TrendUpdate** to all the availabe outside interfaces of **CGRateS**.
+
+Both exporting options are enabled within :ref:`JSON configuration <configuration>`.
 
 
 Parameters
@@ -24,13 +59,30 @@ Parameters
 TrendS
 ^^^^^^
 
-**TrendS** is the **CGRateS** component responsible of handling the *Trend* queries. 
+**TrendS** is the **CGRateS** component responsible of generating the **Trend** queries. 
 
 It is configured within **trends** section from :ref:`JSON configuration <configuration>` via the following parameters:
 
 enabled
 	Will enable starting of the service. Possible values: <true|false>.
 
+store_interval
+	Time interval for backing up the trends into *DataDB*. 0 To completely disable the functionality, -1 to enable synchronous backup. Anything higher than 0 will give the interval for asynchronous backups.
+
+stats_conns
+	List of connections where we will query the stats.
+
+scheduled_ids
+	Limit the TrendProfiles queried. Empty to query all the available TrendProfiles or just tenants for all the available profiles on a tenant.
+
+thresholds_conns
+	Connection IDs towards *ThresholdS* component. If not defined, there will be no notifications sent to *ThresholdS* on *Trend* changes.
+
+ees_conns
+	Connection IDs towards the *EEs* component. If left empty, no exports will be performed on *Trend* changes.
+
+ees_exporter_ids
+	Limit the exporter profiles executed on *Trend* changes.
 
 
 TrendProfile
@@ -44,15 +96,65 @@ Tenant
 ID
 	Identifier for the *TrendProfile*, unique within a *Tenant*.
 
-FilterIDs
+Schedule
-	List of *FilterProfileIDs* which should match in order to consider the profile matching the event.
+	Cron expression scheduling gathering of the metrics.
 
+StatID
+	StatS identifier which will be queried.
 
+Metrics
+	Limit the list of metrics from the stats instance queried.
+
+TTL 
+	Automatic cleanup of the queried values from inside *Trend Metrics*.
+	
+QueueLength 
+	Limit the size of *Trend Metrics*. Older values will be removed first.
+
+MinItems 
+	Issue *TrendUpdate* events to external subsystems only if MinItems are reched to limit false alarms.
+
+CorrelationType 
+	The correlation strategy to use when computing the trend. *\*average* will consider all previous query values and *\*last* only the last one.
+
+Tolerance
+	Allow a deviation of the values when computin the trend. This is defined as percentage of increase/decrease.
+
+Stored
+	Enable storing of this *Trend* for persistence.
+
+ThresholdIDs
+	Limit *TresholdProfiles* processing the *TrendUpdate* for this *TrendProfile*.
 
 
 Trend
 ^^^^^
 
+is made out of the following fields:
+
+Tenant 
+	The tenant on the platform (one can see the tenant as partition ID).
+
+ID 
+	Unique *Trend* identifier on a *Tenant*
+
+RunTimes 
+	Times when the stat queries were ran by the scheduler
+
+Metrics
+	History of the queried metrics, indexed by the query time. One query stores the following values:
+
+	ID 
+		Metric ID on the *StatS* side 
+
+	Value 
+		Value of the metric at the time of query 
+
+	TrendGrowth 
+		Computed trend growth for the metric values, stored in percentage numbers.
+
+	TrendLabel 
+		Computed trend label for the metric values. Possible values are: *positive, *negative, *constant, N/A.
 
 
 Use cases

engine/libtrends.go

@@ -92,7 +92,8 @@ func NewTrendFromProfile(tP *TrendProfile) *Trend {
 		ID:       tP.ID,
 		RunTimes: make([]time.Time, 0),
 		Metrics:  make(map[time.Time]map[string]*MetricWithTrend),
-		tPrfl:    tP,
+
+		tPrfl: tP,
 	}
 }
 
@@ -100,10 +101,11 @@ func NewTrendFromProfile(tP *TrendProfile) *Trend {
 type Trend struct {
 	tMux sync.RWMutex
 
-	Tenant   string
+	Tenant            string
-	ID       string
+	ID                string
-	RunTimes []time.Time
+	RunTimes          []time.Time
-	Metrics  map[time.Time]map[string]*MetricWithTrend
+	Metrics           map[time.Time]map[string]*MetricWithTrend
+	CompressedMetrics []byte // if populated, Metrics and RunTimes will be emty
 
 	// indexes help faster processing
 	mLast   map[string]time.Time // last time a metric was present