mirror of
https://github.com/cgrates/cgrates.git
synced 2026-02-11 18:16:24 +05:00
[Docs] Adding AttributeS, ChargerS, StatS, updating FilterS
This commit is contained in:
DanB
@@ -1,5 +1,95 @@
|
||||
AttributeS
|
||||
==========
|
||||
|
||||
**AttributeS** is a standalone subsystem within **CGRateS** and it is the equivalent of a key-value store. It is accessed via :ref:`CGRateS RPC APIs<remote-management>`.
|
||||
|
||||
As most of the other subsystems, it is performance oriented, stored inside *DataDB* but cached inside the *cgr-engine* process.
|
||||
Caching can be done dynamically/on-demand or at start-time/precached and it is configurable within "cache" section in the .json configuration file.
|
||||
|
||||
Selection
|
||||
---------
|
||||
|
||||
It is able to process generic events (hashmaps) and decision for matching it is outsourced to :ref:`FilterS`.
|
||||
|
||||
If there are multiple profiles (configurations) matching, the one with highest *Weight* will be the winner. There can be only one *AttributeProfile* processing the event per *process run*. If one configures multiple *process runs* either in :ref:`JSON configuration <configuration>` or as parameter to the *.ProcessEvent* API call, the output event from one *process run* will be forwarded as input to the next selected profile. There will be independent *AttributeProfile* selection performed for each run, hence the event fields modified in one run can be applied as filters to the next *process run*, giving out the possibility to chain *AttributeProfiles* and have multiple replacements with a minimum of performance penalty (in-memory matching).
|
||||
|
||||
|
||||
Parameters
|
||||
----------
|
||||
|
||||
|
||||
AttributeProfile
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
Tenant
|
||||
The tenant on the platform (one can see the tenant as partition ID)
|
||||
|
||||
ID
|
||||
Identifier for the *AttributeProfile*, unique within a *Tenant*
|
||||
|
||||
Context
|
||||
A list of *contexts* applying to this profile. A *context* is usually associated with a logical phase during event processing (ie: *\*sessions* or *\*cdrs* for events parsed by :ref:`SessionS` or :ref:`CDRs`)
|
||||
|
||||
FilterIDs
|
||||
List of *FilterProfiles* which should match in order to consider the *AttributeProfile* matching the event.
|
||||
|
||||
ActivationInterval
|
||||
The time interval when this profile becomes active. If undefined, the profile is always active. Other options are start time, end time or both.
|
||||
|
||||
Blocker
|
||||
In case of multiple *process runs* are allowed, this flag will break further processing.
|
||||
|
||||
Weight
|
||||
Used in case of multiple profiles matching an event. The higher, the better (0 has lowest possible priority).
|
||||
|
||||
Attributes
|
||||
List of [Attribute](#attribute) objects part of this profile
|
||||
|
||||
|
||||
Attribute
|
||||
^^^^^^^^^
|
||||
|
||||
FilterIDs
|
||||
List of *FilterProfiles* which should match in order to consider the *Attribute* matching the event.
|
||||
|
||||
Path
|
||||
Identifying the targeted absolute path within the processed event.
|
||||
|
||||
Type
|
||||
Represents the type of substitution which will be performed on the Event. The following *Types* are available:
|
||||
|
||||
**\*constant**
|
||||
The *Value* is a constant value, it will just set the *FieldName* to this value as it is.
|
||||
|
||||
**\*variable**
|
||||
The *Value* is a *RSRParser* which will be able to capture the value out of one or more fields in the event (also combined with other constants) and write it to *Path*.
|
||||
|
||||
**\*composed**
|
||||
Same as *\*variable* but instead of overwriting *Path*, it will append to it.
|
||||
|
||||
**\*usage_difference**
|
||||
Will calculate the duration difference between two field names defined in the *Value*. If the number of fields in the *Value* are different than 2, it will error.
|
||||
|
||||
**\*sum**
|
||||
Will sum up the values in the *Value*.
|
||||
|
||||
**\*value_exponent**
|
||||
Will compute the exponent of the first field in the *Value*.
|
||||
|
||||
Value
|
||||
The value which will be set for *Path*. It can be a list of :ref:`RSRParsers` capturing even from multiple sources in the same event. If the *Value* is *\*none* the field with *Path* will be removed from *Event*
|
||||
|
||||
|
||||
Use cases
|
||||
---------
|
||||
|
||||
* Fields aliasing
|
||||
* Number portability (replacing a dialed number with it's translation)
|
||||
* Roaming (using *Category* to point out the zone where the user is roaming in so we can apply different rating or consume out of restricted account bundles).
|
||||
|
||||
* Appending new fields
|
||||
* Adding separate header with location information
|
||||
* Adding additional rating information (ie: SMS only contains origin+destination, add *Tenant*, *Account*, *Subject*, *RequestType*)
|
||||
* Using as query language (ie: append user password for a given user so we can perform authorization on SIP Proxy side).
|
||||
|
||||
|
||||
TBD
|
||||
@@ -1,5 +1,64 @@
|
||||
ChargerS
|
||||
========
|
||||
|
||||
**ChargerS** is a **CGRateS** subsystem designed to produce billing runs via *DerivedCharging* mechanism.
|
||||
|
||||
TBD
|
||||
It works as standalone component of **CGRateS**, accessible via :ref:`CGRateS RPC<remote-management>` via a rich set of *APIs*. As input **ChargerS** is capable of receiving generic events (hashmaps) with dynamic types for fields.
|
||||
|
||||
**ChargerS** is an **important** part of the charging process within **CGRateS** since with no *ChargingProfile* matching, there will be no billing run performed.
|
||||
|
||||
|
||||
DerivedCharging
|
||||
---------------
|
||||
|
||||
Is a process of receiving an event as input and *deriving* that into multiples (unlimited) out. The *derived* event will be a standalone clone of original with possible modifications of individual event fields. In case of billing, this will translate into multiple Events or CDRs being billed simultaneously for the same input.
|
||||
|
||||
|
||||
Processing logic
|
||||
----------------
|
||||
|
||||
For the received *Event* we will retrieve the list of matching *ChargingProfiles' via :ref:`FilterS`. These profiles will be then ordered based on their *Weight* - higher *Weight* will have more priority. If no profile will match due to *Filter* or *ActivationInterval*, *NOT_FOUND* will be returned back to the RPC client.
|
||||
|
||||
Each *ChargingProfile* matching the *Event* will produce a standalone event based on configured *RunID*. These events will each have a special field added (or overwritten), the *RunID*, which is taken from the applied *ChargingProfile*.
|
||||
|
||||
If *AttributeIDs* are different than *\*none*, the newly created *Event* will be sent to [AttributeS](AttributeS) and fields replacement will be performed based on the logic there. If the *AttributeIDs* is populated, these profile IDs will be selected directly for faster processing, otherwise (if empty) the *AttributeProfiles* will be selected using :ref:`FilterS`.
|
||||
|
||||
|
||||
Parameters
|
||||
----------
|
||||
|
||||
ChargerProfile
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
A *ChargerProfile* is the configuration producing the *DerivedCharging* for the Event received. It's made of the following fields:
|
||||
|
||||
Tenant
|
||||
Is the tenant on the platform (one can see the tenant as partition ID)
|
||||
|
||||
ID
|
||||
Identifier for the ChargerProfile, unique within a *Tenant*.
|
||||
|
||||
FilterIDs
|
||||
List of *FilterProfiles* which should match in order to consider the ChargerProfile matching the event.
|
||||
|
||||
ActivationInterval
|
||||
Is the time interval when this profile becomes active. If undefined, the profile is always active. Other options are start time, end time or both.
|
||||
|
||||
RunID
|
||||
The identifier for a single bill run / charged output *Event*.
|
||||
|
||||
AttributeIDs
|
||||
List of *AttributeProfileIDs* which will be applied for the output *Event* in order to change some of it's fields. If empty, the list is discovered via [FilterS](FilterS) (*AttributeProfiles* matching the event). If *\*none, no AttributeProfile will be applied, event will be a simple clone of the one at input with just *RunID* being different.
|
||||
|
||||
Weight
|
||||
Used in case of multiple profiles matching an event. The higher, the better (0 has lowest possible priority).
|
||||
|
||||
|
||||
Use cases
|
||||
---------
|
||||
|
||||
* Calculating standard charges for the *Customer* calling as well as for the *Reseller*/*Distributor*. One can build chains of charging rules if multiple *Resellers* are involved.
|
||||
* Calculating revenue based on *Customer* vs *Supplier* pricing.
|
||||
* Calculating pricing for multiple *Suppliers* for revenue protection.
|
||||
* Adding *local* vs *mobile* charges for *premium numbers* when accessed from mobile headsets.
|
||||
* etc.
|
||||
@@ -1,3 +1,5 @@
|
||||
.. _FilterS:
|
||||
|
||||
FilterS
|
||||
=======
|
||||
|
||||
@@ -7,7 +9,7 @@ A Tenant will define multiple Filter profiles via .csv or API calls. The Filter
|
||||
|
||||
In order to be used in event processing, a Filter profile will be attached inside another subsystem profile definition, otherwise Filter profile will have no effect on it's own.
|
||||
|
||||
A subsystem can use a Filter via Filter profile or in-line (ad-hock in the same place where subsystem profile is defined).
|
||||
A subsystem can use a *Filter* via *FilterProfile* or in-line (ad-hock in the same place where subsystem profile is defined).
|
||||
|
||||
Filter profile
|
||||
--------------
|
||||
@@ -44,39 +46,57 @@ The matching logic of each FilterRule is given by it's type.
|
||||
|
||||
The following types are implemented:
|
||||
|
||||
- *\*string* will match in full the *Element* with at least one value defined inside *Values*. Any of the values matching will have the FilterRule as *matched*.
|
||||
\*string*
|
||||
Will match in full the *Element* with at least one value defined inside *Values*.
|
||||
Any of the values matching will have the FilterRule as *matched*.
|
||||
|
||||
- *\*notstring* is the negation of *\*string*.
|
||||
\*notstring
|
||||
Is the negation of *\*string*.
|
||||
|
||||
- *\*prefix* will match at beginning of *Element* one of the values defined inside *Values*.
|
||||
\*prefix
|
||||
Will match at beginning of *Element* one of the values defined inside *Values*.
|
||||
|
||||
- *\*notprefix* is the negation of *\*prefix*.
|
||||
\*notprefix
|
||||
Is the negation of *\*prefix*.
|
||||
|
||||
- *\*suffix* will match at end of *Element* one of the values defined inside *Values*.
|
||||
\*suffix
|
||||
Will match at end of *Element* one of the values defined inside *Values*.
|
||||
|
||||
- *\*notsuffix* is the negation of *\*suffix*.
|
||||
\*notsuffix*
|
||||
Is the negation of *\*suffix*.
|
||||
|
||||
- *\*empty* will make sure that *Element* is empty or it does not exist in the event.
|
||||
\*empty
|
||||
Will make sure that *Element* is empty or it does not exist in the event.
|
||||
|
||||
- *\*notempty* is the negation of *\*empty*.
|
||||
\*notempty
|
||||
Is the negation of *\*empty*.
|
||||
|
||||
- *\*exists* will make sure that *Element* exists in the event.
|
||||
\*exists
|
||||
Will make sure that *Element* exists in the event.
|
||||
|
||||
- *\*notexists* is the negation of *\*exists*.
|
||||
\*notexists
|
||||
Is the negation of *\*exists*.
|
||||
|
||||
- *\*timings* will compare the time contained in *Element* with one of the TimingIDs defined in Values.
|
||||
\*timings
|
||||
Will compare the time contained in *Element* with one of the TimingIDs defined in Values.
|
||||
|
||||
- *\*nottimings* is the negation of *\*timings*.
|
||||
\*nottimings
|
||||
Is the negation of *\*timings*.
|
||||
|
||||
- *\*destinations* will make sure that the *Element* is a prefix contained inside one of the destination IDs as *Values*.
|
||||
\*destinations
|
||||
Will make sure that the *Element* is a prefix contained inside one of the destination IDs as *Values*.
|
||||
|
||||
- *\*notdestinations* is the negation of *\*destinations*.
|
||||
\*notdestinations
|
||||
Is the negation of *\*destinations*.
|
||||
|
||||
- *\*rsr* will match the *RSRRules* defined in Values.
|
||||
\*rsr
|
||||
Will match the *RSRRules* defined in Values.
|
||||
|
||||
- *\*notrsr* is the negation of *\*rsr*.
|
||||
\*notrsr*
|
||||
Is the negation of *\*rsr*.
|
||||
|
||||
- *\*lt* (less than), *\*lte* (less than or equal), *\*gt* (greather than), *\*gte* (greather than or equal) are comparison operators and they pass if at least one of the values defined in *Values* are passing for the *Element* of event. The operators are able to compare string, float, int, time.Time, time.Duration, however both types need to be the same, otherwise the filter will raise *incomparable* as error.
|
||||
*\*lt* (less than), *\*lte* (less than or equal), *\*gt* (greather than), *\*gte* (greather than or equal)
|
||||
Are comparison operators and they pass if at least one of the values defined in *Values* are passing for the *Element* of event. The operators are able to compare string, float, int, time.Time, time.Duration, however both types need to be the same, otherwise the filter will raise *incomparable* as error.
|
||||
|
||||
|
||||
Inline Filter
|
||||
@@ -93,18 +113,19 @@ Example::
|
||||
*string:WebsiteName:CGRateS.org
|
||||
|
||||
|
||||
|
||||
Subsystem profiles selection based on Filters
|
||||
---------------------------------------------
|
||||
|
||||
|
||||
When a subsystem will process an event it will need to find fast enough (close to real-time and most preferably with constant speed) all the profiles having filters matching the event. For low number of profiles (tens of) we can go through all available profiles and check their filters but as soon as the number of profiles is growing, processing time will exponentially grow also. As an example, the *AttributeS* need to deal with 20 mil+ profiles for a number portability implementation.
|
||||
When a subsystem will process an event it will need to find fast enough (close to real-time and most preferably with constant speed) all the profiles having filters matching the event. For low number of profiles (tens of) we can go through all available profiles and check their filters but as soon as the number of profiles is growing, processing time will exponentially grow also. As an example, the *AttributeS* need to deal with 20 mil+ profiles in case of number portability implementation.
|
||||
|
||||
In order to guarantee constant processing time - **O(1)** - *CGRateS* will use internally a profile selection mechanism based on indexed filters which can be enabled within *.json* configuration file via *indexed_selects*. When *indexed_selects* is disabled, the indexes will not be used at all and profiles will be checked one by one. On the other hand, if *indexed_selects* is enabled, each FilterProfile needs to have at least one *\*string* or *\*prefix* type in order to be visible to the indexes (otherwise being completely ignored).
|
||||
|
||||
Following settings are further applied once *indexed_selects* is enabled:
|
||||
The following settings are further applied once *indexed_selects* is enabled:
|
||||
|
||||
*string_indexed_fields - list of field names in the event which will be checked against string indexes (defaults to nil which means check all fields)
|
||||
*prefix_indexed_fields - list of field names in the event which will be checked against prefix indexes (default is empty, hence prefix matching is disabled inside indexes - small optimization since for prefixes there are multiple queries done for one field)
|
||||
string_indexed_fields
|
||||
list of field names in the event which will be checked against string indexes (defaults to nil which means check all fields)
|
||||
|
||||
prefix_indexed_fields
|
||||
list of field names in the event which will be checked against prefix indexes (default is empty, hence prefix matching is disabled inside indexes - small optimization since for prefixes there are multiple queries done for one field)
|
||||
|
||||
|
||||
|
||||
108
docs/stats.rst
108
docs/stats.rst
@@ -2,4 +2,110 @@ StatS
|
||||
=====
|
||||
|
||||
|
||||
TBD
|
||||
**StatS** is a standalone subsystem part of the **CGRateS** infrastructure, designed to aggregate and calculate statistical metrics for the generic *Events* (hashmaps) it receives.
|
||||
|
||||
Both receiving of *Events* as well as *Metrics* displaying is performed via a complete set of :ref:`CGRateS RPC APIs<remote-management>`.
|
||||
|
||||
Due it's real-time nature, **StatS** are designed towards high throughput being able to process thousands of *Events* per second. This is doable since each *StatQueue* is a very light object, held in memory and eventually backed up in *DataDB*.
|
||||
|
||||
Processing logic
|
||||
----------------
|
||||
|
||||
When a new *Event* is received, **StatS** will pass it to :ref:`FilterS` in order to find all *StatProfiles* matching the *Event*.
|
||||
|
||||
As a result of the selection process we will further get an ordered list of *StatProfiles* which are matching the *Event* and are active at the request time.
|
||||
|
||||
For each of these profiles we will further calculate the metrics it has configured for the *Event* received. If *ThresholdIDs* are not *\*none*, we will include the *Metrics* into special *StatUpdate* events, defined internally, and pass them further to the [ThresholdS](ThresholdS) for processing.
|
||||
|
||||
Depending on configuration each *StatQueue* can be backed up regularly and asynchronously to DataDB so it can survive process restarts.
|
||||
|
||||
|
||||
Parameters
|
||||
----------
|
||||
|
||||
|
||||
StatQueueProfile
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
Ís made of the following fields:
|
||||
|
||||
Tenant
|
||||
The tenant on the platform (one can see the tenant as partition ID).
|
||||
|
||||
ID
|
||||
Identifier for the *StatQueueProfile*, unique within a *Tenant*.
|
||||
|
||||
FilterIDs
|
||||
List of *FilterProfiles* which should match in order to consider the *StatQueueProfile* matching the event.
|
||||
|
||||
ActivationInterval
|
||||
The time interval when this profile becomes active. If undefined, the profile is always active. Other options are start time, end time or both.
|
||||
|
||||
QueueLength
|
||||
Maximum number of items stored in the queue. Once the *QueueLength* is reached, new items entering will cause oldest one to be dropped (FIFO mode).
|
||||
|
||||
TTL
|
||||
Time duration causing items in the queue to expire and be removed automatically from the queue.
|
||||
|
||||
Metrics
|
||||
List of statistical metrics to build for items within this *StatQueue*. See [bellow](#statqueue-metrics) for possible values here.
|
||||
|
||||
ThresholdIDs
|
||||
List of threshold IDs to check on when new items are updating the queue metrics.
|
||||
|
||||
Blocker
|
||||
Do not process further *StatQueues*.
|
||||
|
||||
Stored
|
||||
Enable offline backups for this *StatQueue*
|
||||
|
||||
Weight
|
||||
Order the *StatQueues* matching the event. Higher value - higher priority.
|
||||
|
||||
MinItems
|
||||
Display metrics only if the number of items in the queue is higher than this.
|
||||
|
||||
|
||||
StatQueue Metrics
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
Following metrics are implemented:
|
||||
|
||||
\*asr
|
||||
`Answer-seizure ratio <https://en.wikipedia.org/wiki/Answer-seizure_ratio>`_. Relies on *AnswerTime* field in the *Event*.
|
||||
\*acd
|
||||
`Average call duration <https://en.wikipedia.org/wiki/Average_call_duration>`. Uses *AnswerTime* and *Usage* fields in the *Event*.
|
||||
\*tcd
|
||||
Total call duration. Uses *Usage* out of *Event*.
|
||||
|
||||
\*acc
|
||||
Average call cost. Uses *Cost* field out of *Event*.
|
||||
|
||||
\*tcc
|
||||
Total call cost. Uses *Cost* field out of *Event*.
|
||||
|
||||
\*pdd
|
||||
`Post dial delay <https://www.voip-info.org/pdd/>`. Uses *PDD* field in the event.
|
||||
|
||||
\*ddc
|
||||
Distinct destination count will keep the number of unique destinations found in *Events*. Relies on *Destination* field in the *Event*.
|
||||
|
||||
\*sum
|
||||
Generic metric to calculate mathematical sum for a specific field in the *Events*. Format: <*\*sum#FieldName*>.
|
||||
|
||||
\*average
|
||||
Generic metric to calculate the mathematical average of a specific field in the *Events*. Format: <*\*average#FieldName*>.
|
||||
|
||||
\*distinct
|
||||
Generic metric to return the distinct number of appearance of a field name within *Events*. Format: <*\*distinct#FieldName*>.
|
||||
|
||||
|
||||
Use cases
|
||||
---------
|
||||
|
||||
* Aggregate various traffic metrics for traffic transparency.
|
||||
* Revenue assurance applications.
|
||||
* Fraud detection by aggregating specific billing metrics during sensitive time intervals (*acc, *tcc, *tcd).
|
||||
* Building call patterns.
|
||||
* Building statistical information to train systems capable of artificial intelligence.
|
||||
* Building quality metrics used in traffic routing.
|
||||
|
||||
Reference in New Issue
Block a user