From 6a2654e6d8e24c4c8425fcceae56bbf5cbefa311 Mon Sep 17 00:00:00 2001 From: ionutboangiu Date: Thu, 8 May 2025 21:39:08 +0300 Subject: [PATCH] port doc changes from v0.11 --- .readthedocs.yaml | 10 +- docs/agents.rst | 4 +- docs/apicalls.rst | 2 +- docs/architecture.rst | 2 +- docs/attributes.rst | 12 +- docs/cdrs.rst | 27 +- docs/cgr-console.rst | 9 + docs/cgr-engine.rst | 46 ++-- docs/cgr-loader.rst | 52 ++-- docs/cgr-migrator.rst | 50 ++-- docs/cgr-tester.rst | 52 +++- docs/chargers.rst | 7 +- docs/conf.py | 16 +- docs/datadb.rst | 286 +++++++++++++++++++++- docs/diamagent.rst | 14 +- docs/dispatchers.rst | 204 +++++++++++++++- docs/ees.rst | 251 +++++++++++++++++++ docs/ers.rst | 34 +-- docs/filters.rst | 7 + docs/guardian.rst | 39 +++ docs/index.rst | 5 +- docs/installation.rst | 477 ++++++++++++++++++++++-------------- docs/janusagent.rst | 39 +++ docs/overview.rst | 6 +- docs/prometheus.rst | 105 ++++++++ docs/rals.rst | 54 +++- docs/rankings.rst | 166 +++++++++++++ docs/requirements.in | 4 + docs/requirements.txt | 79 +++++- docs/resources.rst | 7 +- docs/routes.rst | 21 +- docs/rpcconns.rst | 161 ++++++++++++ docs/sessions.rst | 4 +- docs/stats.rst | 7 +- docs/thresholds.rst | 9 +- docs/trends.rst | 168 +++++++++++++ docs/troubleshooting.rst | 308 +++++++++++++++++++++++ docs/tutorial.rst | 515 +++++++++++++++++++++++++++++++++++++++ 38 files changed, 2934 insertions(+), 325 deletions(-) create mode 100644 docs/ees.rst create mode 100644 docs/guardian.rst create mode 100644 docs/janusagent.rst create mode 100644 docs/prometheus.rst create mode 100644 docs/rankings.rst create mode 100644 docs/requirements.in create mode 100644 docs/rpcconns.rst create mode 100644 docs/trends.rst create mode 100644 docs/troubleshooting.rst create mode 100644 docs/tutorial.rst diff --git a/.readthedocs.yaml b/.readthedocs.yaml index 24d4f738c..f334b70c2 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -1,5 +1,13 @@ version: 2 +build: + os: ubuntu-22.04 + tools: + python: "3.11" + +sphinx: + configuration: docs/conf.py + python: install: - - requirements: docs/requirements.txt \ No newline at end of file + - requirements: docs/requirements.txt diff --git a/docs/agents.rst b/docs/agents.rst index e474abebb..2f6825191 100644 --- a/docs/agents.rst +++ b/docs/agents.rst @@ -19,4 +19,6 @@ Following *Agents* are implemented within CGRateS: astagent fsagent kamagent - ers \ No newline at end of file + ers + janusagent + prometheus diff --git a/docs/apicalls.rst b/docs/apicalls.rst index a53e75a08..c49369443 100644 --- a/docs/apicalls.rst +++ b/docs/apicalls.rst @@ -3,5 +3,5 @@ API Calls API calls are documented in the following GoDoc_ -.. _GoDoc : https://godoc.org/github.com/cgrates/cgrates/apier +.. _GoDoc : https://pkg.go.dev/github.com/cgrates/cgrates/apier@master diff --git a/docs/architecture.rst b/docs/architecture.rst index 4cfafd3a0..7c876e391 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -13,4 +13,4 @@ The CGRateS framework consists of functionality packed within **five** software cgr-loader cgr-migrator cgr-tester - \ No newline at end of file + diff --git a/docs/attributes.rst b/docs/attributes.rst index 45319e2e5..f6e976a93 100644 --- a/docs/attributes.rst +++ b/docs/attributes.rst @@ -3,7 +3,7 @@ AttributeS ========== -**AttributeS** is a standalone subsystem within **CGRateS** and it is the equivalent of a key-value store. It is accessed via `CGRateS RPC APIs `_. +**AttributeS** is a standalone subsystem within **CGRateS** and it is the equivalent of a key-value store. It is accessed via `CGRateS RPC APIs `_. As most of the other subsystems, it is performance oriented, stored inside *DataDB* but cached inside the *cgr-engine* process. Caching can be done dynamically/on-demand or at start-time/precached and it is configurable within *cache* section in the :ref:`JSON configuration `. @@ -58,9 +58,15 @@ Tenant ID Identifier for the *AttributeProfile*, unique within a *Tenant* +Context + A list of *contexts* applying to this profile. A *context* is usually associated with a logical phase during event processing (ie: *\*sessions* or *\*cdrs* for events parsed by :ref:`SessionS` or :ref:`CDRs`) + FilterIDs List of *FilterProfiles* which should match in order to consider the *AttributeProfile* matching the event. +ActivationInterval + The time interval when this profile becomes active. If undefined, the profile is always active. Other options are start time, end time or both. + Blocker In case of multiple *process runs* are allowed, this flag will break further processing. @@ -94,13 +100,13 @@ Type **\*composed** Same as *\*variable* but instead of overwriting *Path*, it will append to it. - **\*usageDifference** + **\*usage_difference** Will calculate the duration difference between two field names defined in the *Value*. If the number of fields in the *Value* are different than 2, it will error. **\*sum** Will sum up the values in the *Value*. - **\*valueExponent** + **\*value_exponent** Will compute the exponent of the first field in the *Value*. Value diff --git a/docs/cdrs.rst b/docs/cdrs.rst index 17e9ca9dc..8d69b4463 100644 --- a/docs/cdrs.rst +++ b/docs/cdrs.rst @@ -4,7 +4,7 @@ CDRs ==== -**CDRs** is a standalone subsystem within **CGRateS** responsible to process *CDR* events. It is accessed via `CGRateS RPC APIs `_ or separate *HTTP handlers* configured within *http* section inside :ref:`JSON configuration `. +**CDRs** is a standalone subsystem within **CGRateS** responsible to process *CDR* events. It is accessed via `CGRateS RPC APIs `_ or separate *HTTP handlers* configured within *http* section inside :ref:`JSON configuration `. Due to multiple interfaces exposed, the **CDRs** is designed to function as centralized server for *CDRs* received from various sources. Examples of such sources are: *\*real-time events* from interfaces like *Diameter*, *Radius*, *Asterisk*, *FreeSWITCH*, *Kamailio*, *OpenSIPS* @@ -48,9 +48,30 @@ stats_conns Connections towards :ref:`StatS` component to compute stat metrics for CDR events. Empty to disable the functionality. online_cdr_exports - List of :ref:`CDRe` profiles which will be processed for each CDR event. Empty to disable online CDR exports. + List of :ref:`EEs` profiles which will be processed for each CDR event. Empty to disable online CDR exports. +Export types +------------ + +There are two types of exports with common configuration but different data sources: + + +Online exports +^^^^^^^^^^^^^^ + +Are real-time exports, triggered by the CDR event processed by :ref:`CDRs`, and take these events as data source. + +The *online exports* are enabled via *online_cdr_exports* :ref:`JSON configuration ` option within *cdrs*. + +You can control the templates which are to be executed via the filters which are applied for each export template individually. + + +Offline exports +^^^^^^^^^^^^^^^ + +Are exports which are triggered via `CGRateS RPC APIs `_ and they have as data source the CDRs stored within *StorDB*. + APIs logic ---------- @@ -79,7 +100,7 @@ Receives the CDR in the form of *CGRateS Event* together with processing flags a Will store the *CDR* to *StorDB*. Defaults to *store_cdrs* parameter within :ref:`JSON configuration `. If store process fails for one of the CDRs, an automated refund is performed for all derived. \*export - Will export the event matching export profiles. These profiles are defined within *cdre* section inside :ref:`JSON configuration `. Defaults to *true* if there is at least one *online_cdr_exports* profile configured within :ref:`JSON configuration `. + Will export the event matching export profiles. These profiles are defined within *ees* section inside :ref:`JSON configuration `. Defaults to *true* if there is at least one *online_cdr_exports* profile configured within :ref:`JSON configuration `. \*thresholds Will process the event with the :ref:`ThresholdS`, allowing us to execute actions based on filters set for matching profiles. Defaults to *true* if there are connections towards :ref:`ThresholdS` within :ref:`JSON configuration `. diff --git a/docs/cgr-console.rst b/docs/cgr-console.rst index 2edb28eed..1600dfef9 100644 --- a/docs/cgr-console.rst +++ b/docs/cgr-console.rst @@ -13,10 +13,18 @@ Configurable via command line arguments. Usage of cgr-console: -ca_path string path to CA for tls connection(only for self sign certificate) + -connect_attempts int + Connect attempts (default 3) + -connect_timeout int + Connect timeout in seconds (default 1) -crt_path string path to certificate for tls connection -key_path string path to key for tls connection + -max_reconnect_interval int + Maximum reconnect interval + -reconnects int + Reconnect attempts (default 3) -reply_timeout int Reply timeout in seconds (default 300) -rpc_encoding string @@ -31,4 +39,5 @@ Configurable via command line arguments. Prints the application version. + .. hint:: # cgr-console status \ No newline at end of file diff --git a/docs/cgr-engine.rst b/docs/cgr-engine.rst index 1a2ea2e9e..a010f4351 100644 --- a/docs/cgr-engine.rst +++ b/docs/cgr-engine.rst @@ -9,36 +9,43 @@ Customisable through the use of *json* :ref:`JSON configuration ` Able to read the configuration from either a local directory of *.json* files with an unlimited number of subfolders (ordered alphabetically) or a list of http paths (separated by ";"). + :: $ cgr-engine -help Usage of cgr-engine: -config_path string - Configuration directory path. (default "/etc/cgrates/") + Configuration directory path (default "/etc/cgrates/") -cpuprof_dir string - write cpu profile to files - -httprof_path string - http address used for program profiling + Directory for CPU profiles -log_level int - Log level (0-emergency to 7-debug) (default -1) + Log level (0=emergency to 7=debug) (default -1) -logger string - logger <*syslog|*stdout> + Logger type <*syslog|*stdout> -memprof_dir string - write memory profile to file + Directory for memory profiles -memprof_interval duration - Time betwen memory profile saves (default 5s) - -memprof_nrfiles int - Number of memory profile to write (default 1) + Interval between memory profile saves (default 15s) + -memprof_maxfiles int + Number of memory profiles to keep (most recent) (default 1) + -memprof_timestamp + Add timestamp to memory profile files -node_id string - The node ID of the engine + Node ID of the engine -pid string - Write pid file - -scheduled_shutdown string - shutdown the engine after this duration + Path to write the PID file + -preload string + Loader IDs used to load data before engine starts + -print_config + Print configuration object in JSON format + -scheduled_shutdown duration + Shutdown the engine after the specified duration + -set_versions + Overwrite database versions (equivalent to cgr-migrator -exec=*set_versions) -singlecpu - Run on single CPU core + Run on a single CPU core -version - Prints the application version. + Print application version and exit .. hint:: $ cgr-engine -config_path=/etc/cgrates @@ -61,12 +68,13 @@ The components from the diagram can be found documented in the links bellow: sessions rals cdrs - cdre + ees attributes chargers resources routes stats + trends thresholds filters dispatchers @@ -74,6 +82,8 @@ The components from the diagram can be found documented in the links bellow: apiers loaders caches + guardian datadb stordb - \ No newline at end of file + rpcconns + diff --git a/docs/cgr-loader.rst b/docs/cgr-loader.rst index 85da28e22..fe17e118a 100644 --- a/docs/cgr-loader.rst +++ b/docs/cgr-loader.rst @@ -23,6 +23,8 @@ Customisable through the use of :ref:`JSON configuration ` or com CacheS component to contact for cache reloads, empty to disable automatic cache reloads (default "*localhost") -caching string Caching strategy used when loading TP + -caching_delay duration + Adds delay before cache reload -config_path string Configuration directory path. -datadb_host string @@ -34,7 +36,7 @@ Customisable through the use of :ref:`JSON configuration ` or com -datadb_port string The DataDb port to bind to. (default "6379") -datadb_type string - The type of the DataDB database <*redis|*mongo> (default "redis") + The type of the DataDB database <*redis|*mongo> (default "*redis") -datadb_user string The DataDb user to sign in as. (default "cgrates") -dbdata_encoding string @@ -51,20 +53,38 @@ Customisable through the use of :ref:`JSON configuration ` or com Load the tariff plan from storDb to dataDb -import_id string Uniquely identify an import/load, postpended to some automatic fields + -mongoConnScheme string + Scheme for MongoDB connection (default "mongodb") + -mongoQueryTimeout duration + The timeout for queries (default 10s) -path string The path to folder containing the data files (default "./") - -recursive - Loads data from folder recursive. + -redisCACertificate string + Path to the CA certificate + -redisClientCertificate string + Path to the client certificate + -redisClientKey string + Path to the client key + -redisCluster + Is the redis datadb a cluster + -redisClusterOndownDelay duration + The delay before executing the commands if the redis cluster is in the CLUSTERDOWN state + -redisClusterSync duration + The sync interval for the redis cluster (default 5s) + -redisConnectAttempts int + The maximum amount of dial attempts (default 20) + -redisConnectTimeout duration + The amount of wait time until timeout for a connection attempt + -redisMaxConns int + The connection pool size (default 10) + -redisReadTimeout duration + The amount of wait time until timeout for reading operations -redisSentinel string The name of redis sentinel - -redisCluster bool - Is the redis datadb a cluster - -cluster_sync string - The sync interval for the redis cluster - -cluster_ondown_delay string - The delay before executing the commands if thredis cluster is in the CLUSTERDOWN state - -mongoQueryTimeout string - The timeout for queries + -redisTLS + Enable TLS when connecting to Redis + -redisWriteTimeout duration + The amount of wait time until timeout for writing operations -remove Will remove instead of adding data from DB -route_id string @@ -78,15 +98,17 @@ Customisable through the use of :ref:`JSON configuration ` or com -stordb_name string The name/number of the storDb to connect to. (default "cgrates") -stordb_passwd string - The storDb user's password. + The storDb user's password. (default "CGRateS.org") -stordb_port string The storDb port to bind to. (default "3306") -stordb_type string - The type of the storDb database <*mysql|*postgres|*mongo> (default "mysql") + The type of the storDb database <*mysql|*postgres|*mongo> (default "*mysql") -stordb_user string The storDb user to sign in as. (default "cgrates") + -tenant string + (default "cgrates.org") -timezone string - Timezone for timestamps where not specified <""|UTC|Local|$IANA_TZ_DB> + Timezone for timestamps where not specified <""|UTC|Local|$IANA_TZ_DB> (default "Local") -to_stordb Import the tariff plan from files to storDb -tpid string @@ -94,4 +116,4 @@ Customisable through the use of :ref:`JSON configuration ` or com -verbose Enable detailed verbose logging output -version - Prints the application version. \ No newline at end of file + Prints the application version. diff --git a/docs/cgr-migrator.rst b/docs/cgr-migrator.rst index d05c8e38a..2a7070546 100644 --- a/docs/cgr-migrator.rst +++ b/docs/cgr-migrator.rst @@ -22,7 +22,7 @@ Customisable through the use of :ref:`JSON configuration ` or com -datadb_port string the DataDB port (default "6379") -datadb_type string - the type of the DataDB Database <*redis|*mongo> (default "redis") + the type of the DataDB Database <*redis|*mongo> (default "*redis") -datadb_user string the DataDB user (default "cgrates") -dbdata_encoding string @@ -31,11 +31,17 @@ Customisable through the use of :ref:`JSON configuration ` or com parse loaded data for consistency and errors, without storing it -exec string fire up automatic migration <*set_versions|*cost_details|*accounts|*actions|*action_triggers|*action_plans|*shared_groups|*filters|*stordb|*datadb> + -mongoConnScheme string + Scheme for MongoDB connection (default "mongodb") + -mongoQueryTimeout duration + The timeout for queries (default 10s) + -out_datadb_encoding string + the encoding used to store object Data in strings in move mode (default "*datadb") -out_datadb_host string output DataDB host to connect to (default "*datadb") -out_datadb_name string output DataDB name/number (default "*datadb") - -out_datadb_passwd string + -out_datadb_password string output DataDB password (default "*datadb") -out_datadb_port string output DataDB port (default "*datadb") @@ -43,15 +49,13 @@ Customisable through the use of :ref:`JSON configuration ` or com output DataDB type <*redis|*mongo> (default "*datadb") -out_datadb_user string output DataDB user (default "*datadb") - -out_dbdata_encoding string - the encoding used to store object Data in strings in move mode (default "*datadb") -out_redis_sentinel string the name of redis sentinel (default "*datadb") -out_stordb_host string output StorDB host (default "*stordb") -out_stordb_name string output StorDB name/number (default "*stordb") - -out_stordb_passwd string + -out_stordb_password string output StorDB password (default "*stordb") -out_stordb_port string output StorDB port (default "*stordb") @@ -59,26 +63,42 @@ Customisable through the use of :ref:`JSON configuration ` or com output StorDB type for move mode <*mysql|*postgres|*mongo> (default "*stordb") -out_stordb_user string output StorDB user (default "*stordb") + -redisCACertificate string + Path to the CA certificate + -redisClientCertificate string + Path to the client certificate + -redisClientKey string + Path to the client key + -redisCluster + Is the redis datadb a cluster + -redisClusterOndownDelay duration + The delay before executing the commands if the redis cluster is in the CLUSTERDOWN state + -redisClusterSync duration + The sync interval for the redis cluster (default 5s) + -redisConnectAttempts int + The maximum amount of dial attempts (default 20) + -redisConnectTimeout duration + The amount of wait time until timeout for a connection attempt + -redisMaxConns int + The connection pool size (default 10) + -redisReadTimeout duration + The amount of wait time until timeout for reading operations -redisSentinel string the name of redis sentinel - -redisCluster bool - Is the redis datadb a cluster - -cluster_sync string - The sync interval for the redis cluster - -cluster_ondown_delay string - The delay before executing the commands if thredis cluster is in the CLUSTERDOWN state - -mongoQueryTimeout string - The timeout for queries + -redisTLS + Enable TLS when connecting to Redis + -redisWriteTimeout duration + The amount of wait time until timeout for writing operations -stordb_host string the StorDB host (default "127.0.0.1") -stordb_name string the name/number of the StorDB (default "cgrates") -stordb_passwd string - the StorDB password + the StorDB password (default "CGRateS.org") -stordb_port string the StorDB port (default "3306") -stordb_type string - the type of the StorDB Database <*mysql|*postgres|*mongo> (default "mysql") + the type of the StorDB Database <*mysql|*postgres|*mongo> (default "*mysql") -stordb_user string the StorDB user (default "cgrates") -verbose diff --git a/docs/cgr-tester.rst b/docs/cgr-tester.rst index eaad73e93..e62c61c83 100644 --- a/docs/cgr-tester.rst +++ b/docs/cgr-tester.rst @@ -9,10 +9,14 @@ Command line stress testing tool configurable via command line arguments. $ cgr-tester -h Usage of cgr-tester: + -calls int + run n number of calls (default 100) -category string The Record category to test. (default "call") -config_path string Configuration directory path. + -cps int + run n requests in parallel (default 100) -cpuprofile string write cpu profile to file -datadb_host string @@ -24,44 +28,70 @@ Command line stress testing tool configurable via command line arguments. -datadb_port string The DataDb port to bind to. (default "6379") -datadb_type string - The type of the DataDb database (default "redis") + The type of the DataDb database (default "*redis") -datadb_user string The DataDb user to sign in as. (default "cgrates") -dbdata_encoding string The encoding used to store object data in strings. (default "msgpack") -destination string The destination to use in queries. (default "1002") + -digits int + Number of digits Account and Destination will have (default 10) + -exec string + Pick what you want to test <*sessions|*cost> -file_path string read requests from file with path -json Use JSON RPC + -max_usage duration + Maximum usage a session can have (default 5s) -memprofile string write memory profile to this file - -parallel int - run n requests in parallel + -min_usage duration + Minimum usage a session can have (default 1s) + -mongoConnScheme string + Scheme for MongoDB connection (default "mongodb") + -mongoQueryTimeout duration + The timeout for queries (default 10s) -rater_address string Rater address for remote tests. Empty for internal rater. + -redisCluster + Is the redis datadb a cluster + -redisClusterOndownDelay duration + The delay before executing the commands if the redis cluster is in the CLUSTERDOWN state + -redisClusterSync duration + The sync interval for the redis cluster (default 5s) + -redisConnectAttempts int + The maximum amount of dial attempts (default 20) + -redisConnectTimeout duration + The amount of wait time until timeout for a connection attempt + -redisMaxConns int + The connection pool size (default 10) + -redisReadTimeout duration + The amount of wait time until timeout for reading operations -redisSentinel string The name of redis sentinel - -redisCluster bool - Is the redis datadb a cluster - -cluster_sync string - The sync interval for the redis cluster - -cluster_ondown_delay string - The delay before executing the commands if thredis cluster is in the CLUSTERDOWN state - -mongoQueryTimeout string - The timeout for queries + -redisWriteTimeout duration + The amount of wait time until timeout for writing operations -req_separator string separator for requests in file (default "\n\n") + -request_type string + Request type of the call (default "*rated") -runs int stress cycle number (default 100000) -subject string The rating subject to use in queries. (default "1001") -tenant string The type of record to use in queries. (default "cgrates.org") + -timeout duration + After last call, time out after this much duration (default 10s) -tor string The type of record to use in queries. (default "*voice") + -update_interval duration + Time duration added for each session update (default 1s) -usage string The duration to use in call simulation. (default "1m") + -verbose + Enable detailed verbose logging output -version Prints the application version. diff --git a/docs/chargers.rst b/docs/chargers.rst index a115a9705..ff403f1ac 100644 --- a/docs/chargers.rst +++ b/docs/chargers.rst @@ -5,7 +5,7 @@ ChargerS **ChargerS** is a **CGRateS** subsystem designed to produce billing runs via *DerivedCharging* mechanism. -It works as standalone component of **CGRateS**, accessible via `CGRateS RPC `_ via a rich set of *APIs*. As input **ChargerS** is capable of receiving generic events (hashmaps) with dynamic types for fields. +It works as standalone component of **CGRateS**, accessible via `CGRateS RPC `_ via a rich set of *APIs*. As input **ChargerS** is capable of receiving generic events (hashmaps) with dynamic types for fields. **ChargerS** is an **important** part of the charging process within **CGRateS** since with no *ChargingProfile* matching, there will be no billing run performed. @@ -19,7 +19,7 @@ Is a process of receiving an event as input and *deriving* that into multiples ( Processing logic ---------------- -For the received *Event* we will retrieve the list of matching *ChargingProfiles' via :ref:`FilterS`. These profiles will be then ordered based on their *Weight* - higher *Weight* will have more priority. If no profile will match due to *Filter*, *NOT_FOUND* will be returned back to the RPC client. +For the received *Event* we will retrieve the list of matching *ChargingProfiles' via :ref:`FilterS`. These profiles will be then ordered based on their *Weight* - higher *Weight* will have more priority. If no profile will match due to *Filter* or *ActivationInterval*, *NOT_FOUND* will be returned back to the RPC client. Each *ChargingProfile* matching the *Event* will produce a standalone event based on configured *RunID*. These events will each have a special field added (or overwritten), the *RunID*, which is taken from the applied *ChargingProfile*. @@ -43,6 +43,9 @@ ID FilterIDs List of *FilterProfiles* which should match in order to consider the ChargerProfile matching the event. +ActivationInterval + Is the time interval when this profile becomes active. If undefined, the profile is always active. Other options are start time, end time or both. + RunID The identifier for a single bill run / charged output *Event*. diff --git a/docs/conf.py b/docs/conf.py index 6f3e695b6..0023a12b1 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -25,7 +25,12 @@ import sys, os # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['sphinx.ext.todo'] +extensions = [ + 'sphinx.ext.todo', + 'sphinx_copybutton', + 'sphinx_tabs.tabs', + 'sphinx_rtd_theme' +] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] @@ -41,16 +46,16 @@ master_doc = 'index' # General information about the project. project = u'CGRateS' -copyright = u'2012-2020, ITsysCOM GmbH' +copyright = u'2012-2023, ITsysCOM GmbH' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. -version = '1.0' +version = '0.11' # The full version, including alpha/beta/rc tags. -release = '1.0~dev' +release = '0.11.0~dev' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. @@ -91,7 +96,7 @@ pygments_style = 'sphinx' # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. -html_theme = 'default' +html_theme = 'sphinx_rtd_theme' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the @@ -217,3 +222,4 @@ man_pages = [ # extensions = ['autoapi.extension'] # autoapi_type = 'go' +# autoapi_dirs = ['../apier/v1'] diff --git a/docs/datadb.rst b/docs/datadb.rst index b63abc40a..ffa6bbc07 100644 --- a/docs/datadb.rst +++ b/docs/datadb.rst @@ -3,5 +3,289 @@ DataDB ====== +**DataDB** is the subsystem within **CGRateS** responsible for storing runtime data like accounts, rating plans, and other objects that the engine needs for its operation. It supports various database backends to fit different deployment needs. -TBD \ No newline at end of file +Database Types +-------------- + +DataDB supports the following database types through the ``db_type`` parameter: + +* ``*redis``: Uses Redis as the storage backend +* ``*mongo``: Uses MongoDB as the storage backend +* ``*internal``: Uses in-memory storage within the CGRateS process + +When using ``*internal`` as the ``db_type``, **CGRateS** leverages your machine's memory to store all **DataDB** records directly inside the engine. This drastically increases read/write performance, as no data leaves the process, avoiding the overhead associated with external databases. The configuration supports periodic data dumps to disk to enable persistence across reboots. + +The internal database is ideal for: + +* Environments with extremely high performance requirements +* Systems where external database dependencies should be avoided +* Lightweight deployments or containers requiring self-contained runtime +* Temporary setups or testing environments + +Remote and Replication Functionality +------------------------------------ + +Remote Functionality +~~~~~~~~~~~~~~~~~~~~ + +DataDB supports fetching data from remote CGRateS instances when items are not found locally. This allows for distributed setups where data can be stored across multiple instances. + +To use remote functionality: + +1. Define RPC connections to remote engines +2. Configure which data items should be fetched remotely by setting ``remote: true`` for specific items +3. Optionally set an identifier for the local connection with ``remote_conn_id`` + +Replication +~~~~~~~~~~~ + +DataDB supports replicating data changes to other CGRateS instances. When enabled, modifications (Set/Remove operations) are propagated to configured remote engines, ensuring data consistency across a distributed deployment. + +Unlike the remote functionality which affects Get operations, replication applies to Set and Remove operations, pushing changes outward to other nodes. + +Configuration +------------- + +A complete DataDB configuration includes database connection details, remote functionality settings, replication options, and item-specific settings. For reference, the full default configuration can be found in the :ref:`configuration` section. + +.. code-block:: json + + "data_db": { + "db_type": "*redis", + "db_host": "127.0.0.1", + "db_port": 6379, + "db_name": "10", + "db_user": "cgrates", + "db_password": "", + "remote_conns": ["engine2", "engine3"], + "remote_conn_id": "engine1", + "replication_conns": ["engine2", "engine3"], + "replication_filtered": false, + "replication_cache": "", + "replication_failed_dir": "/var/lib/cgrates/failed_replications", + "replication_interval": "1s", + "items": { + "*accounts": {"limit": -1, "ttl": "", "static_ttl": false, "remote": false, "replicate": true}, + "*rating_plans": {"limit": -1, "ttl": "", "static_ttl": false, "remote": true, "replicate": true} + // Other items... + }, + "opts": { + // Database-specific options... + } + } + +Parameters +---------- + +Basic Connection +~~~~~~~~~~~~~~~~ + +db_type + The database backend to use. Values: <*redis|*mongo|*internal> + +db_host + Database host address (e.g., "127.0.0.1") + +db_port + Port to reach the database (e.g., 6379 for Redis) + +db_name + Database name to connect to (e.g., "10" for Redis database number) + +db_user + Username for database authentication + +db_password + Password for database authentication + +Remote Functionality +~~~~~~~~~~~~~~~~~~~~ + +remote_conns + Array of connection IDs (defined in rpc_conns) that will be queried when items are not found locally + +remote_conn_id + Identifier sent to remote connections to identify this engine + +Replication Parameters +~~~~~~~~~~~~~~~~~~~~~~ + +replication_conns + Array of connection IDs (defined in rpc_conns) to which data will be replicated + +replication_filtered + When enabled, replication occurs only to connections that previously received a Get request for the item. Values: + +replication_cache + Caching action to execute on replication targets when items are replicated + +replication_failed_dir + Directory to store failed batch replications when using intervals. This directory must exist before launching CGRateS. + +replication_interval + Interval between batched replications: + - Empty/0: Immediate replication after each operation + - Duration (e.g., "1s"): Batches replications and sends them at the specified interval + +Items Configuration +~~~~~~~~~~~~~~~~~~~ + +DataDB manages multiple data types through the ``items`` map, with these configuration options for each item: + +limit + Maximum number of items of this type to store. -1 means no limit. Only applies to *internal database. + +ttl + Time-to-live for items before automatic removal. Empty string means no expiration. Only applies to *internal database. + +static_ttl + Controls TTL behavior. When true, TTL is fixed from initial creation. When false, TTL resets on each update. Only applies to *internal database. + +remote + When true, enables fetching this item type from remote connections if not found locally. + +replicate + When true, enables replication of this item type to configured remote connections. + +Internal Database Options +~~~~~~~~~~~~~~~~~~~~~~~~~ + +When using ``*internal`` as the database type, additional options are available in the ``opts`` section: + +internalDBDumpPath + Defines the path to the folder where the memory-stored **DataDB** will be dumped. This path is also used for recovery during engine startup. Ensure the folder exists before launching the engine. + +internalDBBackupPath + Path where backup copies of the dump folder will be stored. Backups are triggered via the `APIerSv1.BackupDataDBDump `_ API call. This API can also specify a custom path for backups, otherwise the default `internalDBBackupPath` is used. Backups serve as a fallback in case of dump file corruption or loss. The created folders are timestamped in UNIX time for easy identification of the latest backup. To recover using a backup, simply transfer the folders from a backup in internalDBBackupPath to internalDBDumpPath and start the engine. If backups are zipped, they need to be unzipped manually when restoring. + +internalDBStartTimeout + Specifies the maximum amount of time the engine will wait to recover the in-memory **DataDB** state from the dump files during startup. If this duration is exceeded, the engine will timeout and an error will be returned. + +internalDBDumpInterval + Specifies the time interval at which **DataDB** will be dumped to disk. This duration should be chosen based on the machine's capacity and data load. If the interval is set too long and a lot of data changes during that period, the dumping process will take longer, and in the event of an engine crash, any data not dumped will be lost. Conversely, if the interval is too short, and a high number of queries are done often to **DataDB**, some of the needed processing power for the queries will be used by the dump process. Since machine resources and data loads vary, it is recommended to simulate the load on your system and determine the optimal "sweet spot" for this interval. At engine shutdown, any remaining undumped data will automatically be written to disk, regardless of the interval setting. + + - Setting the interval to ``0s`` disables the periodic dumping, meaning any data in **DataDB** will be lost when the engine shuts down. + - Setting the interval to ``-1`` enables immediate dumping—whenever a record in **DataDB** is added, changed, or removed, it will be dumped to disk immediately. + + Manual dumping can be triggered using the `APIerSv1.DumpDataDB `_ API. + +internalDBRewriteInterval + Defines the interval for rewriting files that are not currently being used for dumping data, converting them into an optimized, streamlined version and improving recovery time. Similar to ``internalDBDumpInterval``, the rewriting will trigger based on specified intervals: + + - Setting the interval ``0s`` disables rewriting. + - Setting the interval ``-1`` triggers rewriting only once when the engine starts. + - Setting the interval ``-2`` triggers rewriting only once when the engine shuts down. + + Rewriting should be used sparingly, as the process temporarily loads the entire ``internalDBDumpPath`` folder into memory for optimization, and then writes it back to the dump folder once done. This results in a surge of memory usage, which could amount to the size of the dump file itself during the rewrite. As a rule of thumb, expect the engine's memory usage to approximately double while the rewrite process is running. Manual rewriting can be triggered at any time via the `APIerSv1.RewriteDataDB `_ API. + +internalDBFileSizeLimit + Specifies the maximum size a single dump file can reach. Upon reaching the limit, a new dump file is created. Limiting file size improves recovery time and allows for limit reached files to be rewritten. + +Redis-Specific Options +~~~~~~~~~~~~~~~~~~~~~~ + +The following options in the ``opts`` section apply when using Redis: + +redisMaxConns + Connection pool size + +redisConnectAttempts + Maximum number of connection attempts + +redisSentinel + Sentinel name when using Redis Sentinel + +redisCluster + Enables Redis Cluster mode + +redisClusterSync + Sync interval for Redis Cluster + +redisClusterOndownDelay + Delay before executing commands when Redis Cluster is in CLUSTERDOWN state + +redisConnectTimeout, redisReadTimeout, redisWriteTimeout + Timeout settings for various Redis operations + +redisTLS, redisClientCertificate, redisClientKey, redisCACertificate + TLS configuration for secure Redis connections + +MongoDB-Specific Options +~~~~~~~~~~~~~~~~~~~~~~~~ + +The following options in the ``opts`` section apply when using MongoDB: + +mongoQueryTimeout + Timeout for MongoDB queries + +mongoConnScheme + Connection scheme for MongoDB () + +Configuration Examples +---------------------- + +Persistent Internal Database +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: json + + "data_db": { + "db_type": "*internal", + "opts": { + "internalDBDumpPath": "/var/lib/cgrates/internal_db/datadb", + "internalDBBackupPath": "/var/lib/cgrates/internal_db/backup/datadb", + "internalDBStartTimeout": "5m", + "internalDBDumpInterval": "1m", + "internalDBRewriteInterval": "15m", + "internalDBFileSizeLimit": "1GB" + } + } + +Replication Setup +~~~~~~~~~~~~~~~~~ + +First, define connections to the engines you want to replicate to: + +.. code-block:: json + + "rpc_conns": { + "rpl_engine": { + "conns": [ + { + "address": "127.0.0.1:2012", + "transport": "*json", + "connect_attempts": 5, + "reconnects": -1, + "max_reconnect_interval": "", + "connect_timeout": "1s", + "reply_timeout": "2s" + } + ] + } + } + +Then configure DataDB replication (showing only replication-related parameters): + +.. code-block:: json + + "data_db": { + "replication_conns": ["rpl_engine"], + "replication_failed_dir": "/var/lib/cgrates/failed_replications", + "replication_interval": "1s", + "items": { + "*accounts": {"replicate": true}, + "*reverse_destinations": {"replicate": true}, + "*destinations": {"replicate": true}, + "*rating_plans": {"replicate": true} + // Other items... + } + } + +Notes +----- + +* By default, both replication and remote functionality are disabled for all items and must be explicitly enabled by setting ``replicate: true`` or ``remote: true`` for each desired item +* When using replication with intervals, make sure to configure a ``replication_failed_dir`` to handle failed replications +* Failed replications can be manually replayed using the `APIerSv1.ReplayFailedReplications `_ API call +* Remote functionality and replication can be used independently or together, depending on your deployment needs diff --git a/docs/diamagent.rst b/docs/diamagent.rst index c22932063..6aa2ac6a1 100644 --- a/docs/diamagent.rst +++ b/docs/diamagent.rst @@ -251,13 +251,13 @@ flags **\*none** Disable transfering the request from *Diameter* to *CGRateS* side. Used mostly to pasively answer *Diameter* requests or troubleshoot (mostly in combination with *\*log* flag). - **\*dryRun** + **\*dryrun** Together with not transfering the request on CGRateS side will also log the *Diameter* request/reply, useful for troubleshooting. **\*auth** Sends the request for authorization on CGRateS. - Auxiliary flags available: **\*attributes**, **\*thresholds**, **\*stats**, **\*resources**, **\*accounts**, **\*routes**, **\*rouIgnoreErrors**, **\*routesEventCost**, **\*rouMaxCost** which are used to influence the auth behavior on CGRateS side. More info on that can be found on the **SessionS** component's API behavior. + Auxiliary flags available: **\*attributes**, **\*thresholds**, **\*stats**, **\*resources**, **\*accounts**, **\*routes**, **\*routes_ignore_errors**, **\*routes_event_cost**, **\*routes_maxcost** which are used to influence the auth behavior on CGRateS side. More info on that can be found on the **SessionS** component's API behavior. **\*initiate** Initiates a session out of request on CGRateS side. @@ -277,7 +277,7 @@ flags **\*message** Process the request as individual message charging on CGRateS side. - Auxiliary flags available: **\*attributes**, **\*thresholds**, **\*stats**, **\*resources**, **\*accounts**, **\*routes**, **\*rouIgnoreErrors**, **\*routesEventCost**, **\*rouMaxCost** which are used to influence the behavior on CGRateS side. + Auxiliary flags available: **\*attributes**, **\*thresholds**, **\*stats**, **\*resources**, **\*accounts**, **\*routes**, **\*routes_ignore_errors**, **\*routes_event_cost**, **\*routes_maxcost** which are used to influence the behavior on CGRateS side. **\*event** @@ -286,7 +286,7 @@ flags Auxiliary flags available: all flags supported by the "SessionSv1.ProcessEvent" generic API **\*cdrs** - Build a CDR out of the request on CGRateS side. Can be used simultaneously with other flags (except *\*dry_run) + Build a CDR out of the request on CGRateS side. Can be used simultaneously with other flags (except **\*dryrun**) path @@ -328,10 +328,10 @@ type **\*group** Writes out the variable value, postpending to the list of variables with the same path - **\*usageDifference** + **\*usage_difference** Calculates the usage difference between two arguments passed in the *value*. Requires 2 arguments: *$stopTime;$startTime* - **\*ccUsage** + **\*cc_usage** Calculates the usage out of *CallControl* message. Requires 3 arguments: *$reqNumber;$usedCCTime;$debitInterval* **\*sum** @@ -340,7 +340,7 @@ type **\*difference** Calculates the difference between all arguments passed within *value*. Possible value types are (in this order): duration, time, float, int. - **\*valueExponent** + **\*value_exponent** Calculates the exponent of a value. It requires two values: *$val;$exp* **\*template** diff --git a/docs/dispatchers.rst b/docs/dispatchers.rst index 08afcf15a..9c234f356 100644 --- a/docs/dispatchers.rst +++ b/docs/dispatchers.rst @@ -1,5 +1,207 @@ +.. _dispatchers: + DispatcherS =========== +**DispatcherS** is the **CGRateS** component that handles request routing and load balancing. When enabled, it manages all requests to other CGRateS subsystems by wrapping their methods with additional features like :ref:`authorization `, request routing, load balancing, broadcasting, and loop prevention. -TBD \ No newline at end of file +Processing Logic +---------------- + +.. _dispatcher-authorization: + +Authorization +~~~~~~~~~~~~~ + +Optional step when :ref:`AttributeS ` connections are configured: + +- Looks for ``*apiKey`` in APIOpts (returns mandatory missing error if not present) +- Sends key to AttributeS, which adds it as "APIKey" to the Event +- AttributeS processes the event and adds allowed methods to APIMethods (e.g., "method1&method2&method3") +- Checks if the requested method is in the allowed list +- Continues only after successful authorization + +Dispatch +~~~~~~~~ + +The dispatcher processes requests through these steps: + +* Check for bypass conditions: + * Presence of ``*dispatchers: false`` in APIOpts + * Request source is another dispatcher and ``prevent_loops`` is enabled + +* Check cached routes: + * Search for ``*routeID`` in APIOpts + * If found, use cached dispatch data (tenant, profile ID, host ID) + * Fall back to full dispatch on network errors or timeouts + +* Run full dispatch sequence: + * Get matching dispatcher profiles + * Try each profile until dispatch succeeds + +.. _dispatcher-types: + +Dispatcher Types and Strategies +------------------------------- + +Load Dispatchers +~~~~~~~~~~~~~~~~ + +Used for ratio-based request distribution. Hosts are sorted in three steps: + +1. Initial sorting by weight +2. Secondary sorting by load ratio (current active requests/configured ratio), where lower ratios have priority +3. Final sorting based on the specified strategy: + + * ``*random``: Randomizes host selection + * ``*round_robin``: Sequential host selection with weight consideration + * ``*weight``: Skips final sorting, maintains weight and load-based ordering + +Configuration through: + +- ``*defaultRatio`` in StrategyParams +- Direct ratio specification in Host configuration + + +Simple Dispatchers +~~~~~~~~~~~~~~~~~~ + +Standard request distribution where hosts are sorted first by weight, followed by the chosen strategy (*random, *round_robin, *weight). + +Broadcast Dispatchers +~~~~~~~~~~~~~~~~~~~~~ + +Handles scenarios requiring multi-host distribution. Supports three broadcast strategies: + +* ``*broadcast``: Sends to all hosts, uses first response +* ``*broadcast_sync``: Sends to all hosts, waits for all responses +* ``*broadcast_async``: Sends to all hosts without waiting (fire-and-forget) + +Parameters +---------- + +Configure the dispatcher in the **dispatchers** section of the :ref:`JSON configuration `: + +enabled + Enables/disables the DispatcherS component. Values: + +indexed_selects + Enables profile matching exclusively on indexes for improved performance + +string_indexed_fields + Fields used for string-based index querying + +prefix_indexed_fields + Fields used for prefix-based index querying + +suffix_indexed_fields + Fields used for suffix-based index querying + +nested_fields + Controls indexed filter matching depth. Values: + - true: checks all levels + - false: checks only first level + +attributes_conns + Connections to :ref:`AttributeS ` for API authorization + - Empty: disables authorization + - "*internal": uses internal connection + - Custom connection ID + +any_subsystem + Enables matching of *any subsystem. Values: + +prevent_loops + Prevents request loops between dispatcher nodes. Values: + +DispatcherHost +~~~~~~~~~~~~~~ + +Defines individual dispatch destinations with the following parameters: + +Tenant + The tenant on the platform + +ID + Unique identifier for the host + +Address + Host address (use *internal for internal connections) + +Transport + Protocol used for communication (*gob, *json) + +ConnectAttempts + Number of connection attempts + +Reconnects + Maximum number of reconnection attempts + +MaxReconnectInterval + Maximum interval between reconnection attempts + +ConnectTimeout + Connection timeout (e.g., "1m") + +ReplyTimeout + Response timeout (e.g., "2m") + +TLS + TLS connection settings: + - ClientKey: Path to client key file + - ClientCertificate: Path to client certificate + - CaCertificate: Path to CA certificate + +DispatcherProfile +~~~~~~~~~~~~~~~~~ + +Defines routing rules and strategies. See :ref:`dispatcher-types` for available strategies. + +Tenant + The tenant on the platform + +ID + Profile identifier + +Subsystems + Target subsystems (*any for all) + +FilterIDs + List of filters for request matching + +ActivationInterval + Time interval when profile is active + +Strategy + Dispatch strategy (*weight, *random, *round_robin, *broadcast, *broadcast_sync) + +StrategyParameters + Additional strategy configuration (e.g., *default_ratio) + +ConnID + Target host identifier + +ConnFilterIDs + Filters for connection selection + +ConnWeight + Priority weight for connection selection within the profile + +ConnBlocker + Blocks connection if true + +ConnParameters + Additional connection parameters (e.g., *ratio) + +Weight + Priority weight used when selecting between multiple matching profiles + +Use Cases +--------- + +- Load balancing between multiple CGRateS nodes +- High availability setups with automatic failover +- Request authorization and access control +- Broadcasting requests for data collection +- Traffic distribution based on weight or custom metrics +- System scaling and performance optimization diff --git a/docs/ees.rst b/docs/ees.rst new file mode 100644 index 000000000..c957d6ea7 --- /dev/null +++ b/docs/ees.rst @@ -0,0 +1,251 @@ +.. _AMQP: https://www.amqp.org/ +.. _SQS: https://aws.amazon.com/de/sqs/ +.. _S3: https://aws.amazon.com/de/s3/ +.. _Kafka: https://kafka.apache.org/ + + +.. _EEs: + +EEs +==== + + +**EventExporterService/EEs** is a subsystem designed to convert internal, already processed events into external ones and then export them to a defined destination. It is accessible via `CGRateS RPC APIs `_. + + +Configuration +------------- + +**EEs** is configured within **ees** section from :ref:`JSON configuration `. + +Config params +^^^^^^^^^^^^^ + +Most of the parameters are explained in :ref:`JSON configuration `, hence we mention here only the ones where additional info is necessary or there will be particular implementation for *EventExporterService*. + +One **exporters** instance includes the following parameters: + +id + Exporter identificator, used mostly for debug. The id should be unique per each exporter since it can influence updating configuration from different *.json* configuration. + +type + Specify the type of export which will run. Possible values are: + + **\*file_csv** + Exports into a comma separated file format. + + **\*file_fwv** + Exports into a fixed width file format. + + **\*http_post** + Will post the CDR to a HTTP server. The export content will be a HTTP form encoded representation of the `internal CDR object `_. + + + **\*http_json_map** + Will post the CDR to a HTTP server. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. + + **\*amqp_json_map** + Will post the CDR to an AMQP_ queue. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. Uses AMQP_ protocol version 1.0. + + **\*amqpv1_json_map** + Will post the CDR to an AMQP_ queue. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. Uses AMQP_ protocol version 1.0. + + **\*sqs_json_map** + Will post the CDR to an `Amazon SQS queue `_. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. + + **\*s3_json_map** + Will post the CDR to `Amazon S3 storage `_. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. + + **\*kafka_json_map** + Will post the CDR to an `Apache Kafka `_. The export content will be a JSON serialized hmap with fields defined within the *fields* section of the template. + + **\*nats_json_map** + Exporter for publishing messages to NATS (Message Queue) in JSON format. + + **\*virt** + In-memory exporter. + + **\*els** + Exporter for Elasticsearch. + + **\*sql** + Exporter for generic content to *SQL* databases. Supported databases are: MySQL_, PostgreSQL_ and MSSQL_. + + **\*rpc** + Exporter for calling APIs through node connections. + +export_path + Specify the export path. It has special format depending of the export type. + + **\*file_csv**, **\*file_fwv** + Standard unix-like filesystem path. + + **\*http_post**, **\*http_json_map** + Full HTTP URL + + **\*amqp_json_map**, **\*amqpv1_json_map** + AMQP URL with extra parameters. + + Sample: *amqp://guest:guest@localhost:5672/?queue_id=cgrates_cdrs&exchange=exchangename&exchange_type=fanout&routing_key=cgr_cdrs* + + **\*sqs_json_map** + SQS URL with extra parameters. + + Sample: *http://sqs.eu-west-2.amazonaws.com/?aws_region=eu-west-2&aws_key=testkey&aws_secret=testsecret&queue_id=cgrates-cdrs* + + **\*s3_json_map** + S3 URL with extra parameters. + + Sample: *http://s3.us-east-2.amazonaws.com/?aws_region=eu-west-2&aws_key=testkey&aws_secret=testsecret&queue_id=cgrates-cdrs* + + **\*kafka_json_map** + Kafka URL with extra parameters. + + Sample: *localhost:9092?topic=cgrates_cdrs* + + **\*sql** + SQL URL with extra parameters. + + Sample: *mysql://cgrates:CGRateS.org@127.0.0.1:3306* + + **\*nats** + NATS URL. + + Sample: *nats://localhost:4222* + + **\*els** + Elasticsearch URL + + Sample: *http://localhost:9200* + +filters + List of filters to pass for the export profile to execute. For the dynamic content (prefixed with *~*) following special variables are available: + + **\*req** + The *CDR* event itself. + + **\*ec** + The *EventCost* object with subpaths for all of it's nested objects. + +tenant + Tenant owning the template. It will be used mostly to match inside :ref:`FilterS`. + +synchronous + Block further exports until this one finishes. In case of *false* the control will be given to the next export template as soon as this one was started. + +attempts + Number of attempts before giving up on the export and writing the failed request to file. The failed request will be written to *failed_posts_dir*. + +fields + List of fields for the exported event. + + +One **field template** will contain the following parameters: + +path + Path for the exported content. Possible prefixes here are: + + *\*exp* + Reference to the exported record. + + *\*hdr* + Reference to the header content. Available in case of **\*file_csv** and **\*file_fwv** export types. + + *\*trl* + Reference to the trailer content. Available in case of **\*file_csv** and **\*file_fwv** export types. + +type + The field type will give out the logic for generating the value. Values used depend on the type of prefix used in path. + + For *\*exp*, following field types are implemented: + + **\*variable** + Writes out the variable value, overwriting previous one set. + + **\*composed** + Writes out the variable value, postpending to previous value set + + **\*filler** + Fills the values with a fixed lentgh string. + + **\*constant** + Writes out a constant + + **\*datetime** + Parses the value as datetime and reformats based on the *layout* attribute. + + **\*combimed** + Writes out a combined mediation considering events with the same *CGRID*. + + **\*masked_destination** + Masks the destination using *\** as suffix. Matches the destination field against the list defined via *mask_destinationd_id* field. + + **\*http_post** + Uses a HTTP server as datasource for the value exported. + + For *\*hdr* and *\*trl*, following field types are possible: + + **\*filler** + Fills the values with a string. + + **\*constant** + Writes out a constant + + **\*handler** + Will obtain the content via a handler. This works in tandem with the attribute *handler_id*. + +value + The exported value. Works in tandem with *type* attribute. Possible prefixes for dynamic values: + + **\*req** + Data is taken from the current request coming from the *CDRs* component. + +mandatory + Makes sure that the field cannot have empty value (errors otherwise). + +tag + Used for debug purposes in logs. + +width + Used to control the formatting, enforcing the final value to a specific number of characters. + +strip + Used when the value is higher than *width* allows it, specifying the strip strategy. Possible values are: + + **\*right** + Strip the suffix. + + **\*xright** + Strip the suffix, postpending one *x* character to mark the stripping. + + **\*left** + Strip the prefix. + + **\*xleft** + Strip the prefix, prepending one *x* character to mark the stripping. + +padding + Used to control the formatting. Applied when the data is smaller than the *width*. Possible values are: + + **\*right** + Suffix with spaces. + + **\*left** + Prefix with spaces. + + **\*zeroleft** + Prefix with *0* chars. + +mask_destinationd_id + The destinations profile where we match the *masked_destinations*. + +hander_id + The identifier of the handler to be executed in case of *\*handler* *type*. + + + + + + + + diff --git a/docs/ers.rst b/docs/ers.rst index d97269e11..cd9582df3 100644 --- a/docs/ers.rst +++ b/docs/ers.rst @@ -38,7 +38,7 @@ With explanations in the comments: "id": "file_reader2", // file_reader2 reader "run_delay": "-1", // reading of events it is triggered outside of ERs "field_separator": ";", // field separator definition - "type": "*fileCSV", // type of reader, *fileCSV can read .csv files + "type": "*file_csv", // type of reader, *file_csv can read .csv files "row_length" : 0, // Number of fields from csv file "flags": [ // influence processing logic within CGRateS workflow "*cdrs", // *cdrs will create CDRs @@ -46,7 +46,7 @@ With explanations in the comments: ], "source_path": "/tmp/ers2/in", // location of the files "processed_path": "/tmp/ers2/out", // move the files here once processed - "content_fields":[ // mapping definition between line index in the file and CGRateS field + "fields":[ // mapping definition between line index in the file and CGRateS field { "tag": "OriginID", // OriginID together with OriginHost will "path": "*cgreq.OriginID", // uniquely identify the session on CGRateS side @@ -131,16 +131,16 @@ id type Reader type. Following types are implemented: - **\*fileCSV** + **\*file_csv** Reader for *comma separated* files. - **\*fileXML** + **\*file_xml** Reader for *.xml* formatted files. - **\*fileFWV** + **\*file_fwv** Reader for *fixed width value* formatted files. - **\*kafkaJSONMap** + **\*kafka_json_map** Reader for hashmaps within Kafka_ database. **\*sql** @@ -158,7 +158,7 @@ source_path processed_path Optional path for moving the events source to after processing. -xmlRootPath +xml_root_path Used in case of XML content and will specify the prefix path applied to each xml element read. tenant @@ -180,10 +180,10 @@ filters Request read from the source. In case of file content without field name, the index will be passed instead of field source path. **\*hdr** - Header values (available only in case of *\*fileFWV*). In case of file content without field name, the index will be passed instead of field source path. + Header values (available only in case of *\*file_fwv*). In case of file content without field name, the index will be passed instead of field source path. **\*trl** - Trailer values (available only in case of *\*fileFWV*). In case of file content without field name, the index will be passed instead of field source path. + Trailer values (available only in case of *\*file_fwv*). In case of file content without field name, the index will be passed instead of field source path. flags Special tags enforcing the actions/verbs done on an event. There are two types of flags: **main** and **auxiliary**. @@ -202,13 +202,13 @@ flags **\*none** Disable transfering the Event from *Reader* to *CGRateS* side. - **\*dryRun** + **\*dryrun** Together with not transfering the Event on CGRateS side will also log it, useful for troubleshooting. **\*auth** Sends the Event for authorization on CGRateS. - Auxiliary flags available: **\*attributes**, **\*thresholds**, **\*stats**, **\*resources**, **\*accounts**, **\*routes**, **\*rouIgnoreErrors**, **\*routesEventCost**, **\*rouMaxCost** which are used to influence the auth behavior on CGRateS side. More info on that can be found on the **SessionS** component's API behavior. + Auxiliary flags available: **\*attributes**, **\*thresholds**, **\*stats**, **\*resources**, **\*accounts**, **\*routes**, **\*routes_ignore_errors**, **\*routes_event_cost**, **\*routes_maxcost** which are used to influence the auth behavior on CGRateS side. More info on that can be found on the **SessionS** component's API behavior. **\*initiate** Initiates a session out of Event on CGRateS side. @@ -228,7 +228,7 @@ flags **\*message** Process the Event as individual message charging on CGRateS side. - Auxiliary flags available: **\*attributes**, **\*thresholds**, **\*stats**, **\*resources**, **\*accounts**, **\*routes**, **\*rouIgnoreErrors**, **\*routesEventCost**, **\*rouMaxCost** which are used to influence the behavior on CGRateS side. + Auxiliary flags available: **\*attributes**, **\*thresholds**, **\*stats**, **\*resources**, **\*accounts**, **\*routes**, **\*routes_ignore_errors**, **\*routes_event_cost**, **\*routes_maxcost** which are used to influence the behavior on CGRateS side. **\*event** Process the Event as generic event on CGRateS side. @@ -236,7 +236,7 @@ flags Auxiliary flags available: all flags supported by the "SessionSv1.ProcessEvent" generic API **\*cdrs** - Build a CDR out of the Event on CGRateS side. Can be used simultaneously with other flags (except *\*dry_run) + Build a CDR out of the Event on CGRateS side. Can be used simultaneously with other flags (except **\*dryrun**) path Defined within field, specifies the path where the value will be written. Possible values: @@ -248,10 +248,10 @@ path Write the value in the request object which will be sent to CGRateS side. **\*hdr** - Header values (available only in case of *\*fileFWV*). In case of file content without field name, the index will be passed instead of field source path. + Header values (available only in case of *\*file_fwv*). In case of file content without field name, the index will be passed instead of field source path. **\*trl** - Trailer values (available only in case of *\*fileFWV*). In case of file content without field name, the index will be passed instead of field source path. + Trailer values (available only in case of *\*file_fwv*). In case of file content without field name, the index will be passed instead of field source path. type @@ -272,7 +272,7 @@ type **\*composed** Writes out the variable value, postpending to previous value set - **\*usageDifference** + **\*usage_difference** Calculates the usage difference between two arguments passed in the *value*. Requires 2 arguments: *$stopTime;$startTime* **\*sum** @@ -281,7 +281,7 @@ type **\*difference** Calculates the difference between all arguments passed within *value*. Possible value types are (in this order): duration, time, float, int. - **\*valueExponent** + **\*value_exponent** Calculates the exponent of a value. It requires two values: *$val;$exp* **\*template** diff --git a/docs/filters.rst b/docs/filters.rst index 2f4f9be00..a91d9e0f9 100644 --- a/docs/filters.rst +++ b/docs/filters.rst @@ -21,6 +21,7 @@ Definition:: Tenant string ID string Rules []*FilterRule + ActivationInterval *utils.ActivationInterval } A Filter profile can be shared between multiple subsystem profile definitions. @@ -77,6 +78,12 @@ The following types are implemented: \*notexists Is the negation of *\*exists*. +\*timings + Will compare the time contained in *Element* with one of the TimingIDs defined in Values. + +\*nottimings + Is the negation of *\*timings*. + \*destinations Will make sure that the *Element* is a prefix contained inside one of the destination IDs as *Values*. diff --git a/docs/guardian.rst b/docs/guardian.rst new file mode 100644 index 000000000..255c2fdf8 --- /dev/null +++ b/docs/guardian.rst @@ -0,0 +1,39 @@ +.. _guardian: + +Guardian +======== + +Guardian is CGRateS' internal locking mechanism that ensures data consistency during concurrent operations. + +What Guardian Does +------------------ + +Guardian prevents race conditions when multiple processes try to access or modify the same data. It uses string-based locks, typically created using some variation of the tenant and ID of the resource being protected, often with a type prefix. Guardian can use either explicit IDs or generate UUIDs internally for reference-based locking when no specific ID is provided. + +When CGRateS Uses Guardian +-------------------------- + +Guardian protects: + +* Account balance operations (debits/topups) - the most critical use case +* ResourceProfiles, Resources, StatQueueProfiles, StatQueues, ThresholdProfiles, and Thresholds while they're being used or loaded into cache +* Filter index updates +* There are other cases, but the ones listed above are the most frequent applications + +Performance Implications +------------------------ + +Guardian affects system performance in these ways: + +* Operations on the same resource are processed one after another, not simultaneously +* Under heavy load on the same resources, operations may queue up and wait +* System throughput is better when operations are distributed across different resources + +Configuration +------------- + +Guardian has a single configuration option: + +The `locking_timeout` setting in the general configuration determines how long Guardian will hold a lock before forcing it to release. Zero timeout (no timeout) is the default and recommended setting. However, setting a reasonable timeout can help prevent system hangs if a process fails to release a lock. + +When a timeout occurs, Guardian logs a warning and forces the lock to release. This keeps the system running, but the operation that timed out may fail. diff --git a/docs/index.rst b/docs/index.rst index dae6cdb2d..a6eaeb9c9 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -10,14 +10,15 @@ Welcome to `CGRateS`_'s documentation! .. toctree:: :maxdepth: 4 - + overview architecture installation configuration administration advanced - tutorials + tutorial + troubleshooting miscellaneous diff --git a/docs/installation.rst b/docs/installation.rst index ff8985816..0a445815b 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -1,270 +1,383 @@ +.. _Docker: https://docs.docker.com/get-started/get-docker/ .. _Redis: https://redis.io/ .. _MySQL: https://dev.mysql.com/ .. _PostgreSQL: https://www.postgresql.org/ .. _MongoDB: https://www.mongodb.com/ - - - .. _installation: Installation ============ -CGRateS can be installed via packages as well as Go automated source install. -We recommend using source installs for advanced users familiar with Go programming and packages for users not willing to be involved in the code building process. +.. contents:: + :local: + :depth: 2 + +CGRateS can be installed either via packages or through an automated Go source installation. We recommend the latter for advanced users familiar with Go programming, and package installations for those not wanting to engage in the code building process. + +After completing the installation, you need to perform the :ref:`post-install configuration ` steps to set up CGRateS properly and prepare it to run. After these steps, CGRateS will be configured in **/etc/cgrates/cgrates.json** and the service can be managed using the **systemctl** command. + +Package Installation +-------------------- + +Package installation method varies according to the Linux distribution: + +Debian or Debian-based Distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You can add the CGRateS repository to your system's sources list, depending of the Debian version you are running, as follows: + +.. tabs:: + + .. group-tab:: Bookworm + + .. code-block:: bash + + # Install dependencies + sudo apt-get install wget gnupg -y + + # Download and move the GPG Key to the trusted area + wget https://apt.cgrates.org/apt.cgrates.org.gpg.key -O apt.cgrates.org.asc + sudo mv apt.cgrates.org.asc /etc/apt/trusted.gpg.d/ + + # Add the repository to the apt sources list + echo "deb http://apt.cgrates.org/debian/ master-bookworm main" | sudo tee /etc/apt/sources.list.d/cgrates.list + + # Update the system repository and install CGRateS + sudo apt-get update -y + sudo apt-get install cgrates -y + + Alternatively, you can manually install a specific .deb package as follows: + + .. code-block:: bash + + wget http://pkg.cgrates.org/deb/master/bookworm/cgrates_current_amd64.deb + sudo dpkg -i ./cgrates_current_amd64.deb + + .. group-tab:: Bullseye + + .. code-block:: bash + + # Install dependencies + sudo apt-get install wget gnupg -y + + # Download and move the GPG Key to the trusted area + wget https://apt.cgrates.org/apt.cgrates.org.gpg.key -O apt.cgrates.org.asc + sudo mv apt.cgrates.org.asc /etc/apt/trusted.gpg.d/ + + # Add the repository to the apt sources list + echo "deb http://apt.cgrates.org/debian/ master-bullseye main" | sudo tee /etc/apt/sources.list.d/cgrates.list + + # Update the system repository and install CGRateS + sudo apt-get update -y + sudo apt-get install cgrates -y + + Alternatively, you can manually install a specific .deb package as follows: + + .. code-block:: bash + + wget http://pkg.cgrates.org/deb/master/bullseye/cgrates_current_amd64.deb + sudo dpkg -i ./cgrates_current_amd64.deb -Using packages --------------- - -Depending on the packaged distribution, the following methods are available: +.. note:: + A complete archive of CGRateS packages is available at http://pkg.cgrates.org/deb/. -Debian -^^^^^^ +Redhat-based Distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^ -There are two main ways of installing the maintained packages: +For .rpm distros, we are using copr to manage the CGRateS packages: + +- If using a version of Linux with dnf: + + .. code-block:: bash + + # sudo yum install -y dnf-plugins-core on RHEL 8 or CentOS Stream + sudo dnf install -y dnf-plugins-core + sudo dnf copr -y enable cgrates/master + sudo dnf install -y cgrates + +- For older distributions: + + .. code-block:: bash + + sudo yum install -y yum-plugin-copr + sudo yum copr -y enable cgrates/master + sudo yum install -y cgrates + +To install a specific version of the package, run: + +.. code-block:: bash + + sudo dnf install -y cgrates-.x86_64 + +Alternatively, you can manually install a specific .rpm package as follows: + +.. code-block:: bash + + wget http://pkg.cgrates.org/rpm/nightly/epel-9-x86_64/cgrates-current.rpm + sudo dnf install ./cgrates_current.rpm -Aptitude repository -~~~~~~~~~~~~~~~~~~~ +.. note:: + The entire archive of CGRateS rpm packages is available at https://copr.fedorainfracloud.org/coprs/cgrates/master/packages/ or http://pkg.cgrates.org/rpm/nightly/. +Installing from Source +---------------------- -Add the gpg key: +Prerequisites: +^^^^^^^^^^^^^^ -:: +- **Git** - sudo wget -O - https://apt.cgrates.org/apt.cgrates.org.gpg.key | sudo apt-key add - +.. code-block:: bash -Add the repository in apt sources list: + sudo apt-get install -y git + # sudo dnf install -y git for .rpm distros -:: +- **Go** (refer to the official Go installation docs: https://go.dev/doc/install) - echo "deb http://apt.cgrates.org/debian/ nightly main" | sudo tee /etc/apt/sources.list.d/cgrates.list +To install the latest Go version at the time of writing this documentation, run: -Update & install: - -:: - - sudo apt-get update - sudo apt-get install cgrates - - -Once the installation is completed, one should perform the :ref:`post_install` section in order to have the CGRateS properly set and ready to run. -After *post-install* actions are performed, CGRateS will be configured in **/etc/cgrates/cgrates.json** and enabled in **/etc/default/cgrates**. - - -Manual installation of .deb package out of archive server -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - -Run the following commands: - -:: - - wget http://pkg.cgrates.org/deb/nightly/cgrates_current_amd64.deb - dpkg -i cgrates_current_amd64.deb - -As a side note on http://pkg.cgrates.org/deb/ one can find an entire archive of CGRateS packages. - - -Redhat/Fedora/CentOS -^^^^^^^^^^^^^^^^^^^^ - -There are two main ways of installing the maintained packages: - - -YUM repository -~~~~~~~~~~~~~~ - - -To install CGRateS out of YUM execute the following commands - -:: - - sudo tee -a /etc/yum.repos.d/cgrates.repo > /dev/null <` - * Git + sudo dnf install -y rpm-build wget curl tar - :: + # Create build directories + mkdir -p $HOME/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} - sudo apt-get install git + # Fetch source code + cd $HOME/go/src/github.com/cgrates/cgrates + export gitLastCommit=$(git rev-parse HEAD) + export rpmTag=$(git log -1 --format=%ci | date +%Y%m%d%H%M%S)+$(git rev-parse --short HEAD) + #Create the tarball from the source code + cd .. + tar -czvf $HOME/rpmbuild/SOURCES/$gitLastCommit.tar.gz cgrates - * RPM - - :: - - sudo apt-get install rpm - -Execute the following commands - -:: - - cd $HOME/go/src/github.com/cgrates/cgrates - export gitLastCommit=$(git rev-parse HEAD) - export rpmTag=$(git log -1 --format=%ci | date +%Y%m%d%H%M%S)+$(git rev-parse --short HEAD) - mkdir -p $HOME/cgr_build/{BUILD,RPMS,SOURCES,SPECS,SRPMS} - wget -P $HOME/cgr_build/SOURCES https://github.com/cgrates/cgrates/archive/$gitLastCommit.tar.gz - cp $HOME/go/src/github.com/cgrates/cgrates/packages/redhat_fedora/cgrates.spec $HOME/cgr_build/SPECS - cd $HOME/cgr_build - rpmbuild -bb --define "_topdir $HOME/cgr_build" SPECS/cgrates.spec + # Copy RPM spec file + cp $HOME/go/src/github.com/cgrates/cgrates/packages/redhat_fedora/cgrates.spec $HOME/rpmbuild/SPECS + # Build RPM package + cd $HOME/rpmbuild + rpmbuild -bb SPECS/cgrates.spec .. _post_install: -Post-install ------------- +Post-install Configuration +-------------------------- - -Database setup +Database Setup ^^^^^^^^^^^^^^ -For its operation CGRateS uses **one or more** database types, depending on its nature, install and configuration being further necessary. +CGRateS supports multiple database types for various operations, based on your installation and configuration. -At present we support the following databases: +Currently, we support the following databases: `Redis`_ - Can be used as :ref:`DataDB`. - Optimized for real-time information access. - Once installed there should be no special requirements in terms of setup since no schema is necessary. + This can be used as :ref:`DataDB`. It is optimized for real-time information access. Post-installation, no additional setup is required as Redis doesn't require a specific schema. `MySQL`_ - Can be used as :ref:`StorDB`. - Optimized for CDR archiving and offline Tariff Plan versioning. - Once MySQL is installed, CGRateS database needs to be set-up out of provided scripts. (example for the paths set-up by debian package) + This can be used as :ref:`StorDB` and is optimized for CDR archiving and offline Tariff Plan versioning. Post-installation, you need to set up the CGRateS database using the provided scripts: - :: +.. code-block:: bash - cd /usr/share/cgrates/storage/mysql/ - ./setup_cgr_db.sh root CGRateS.org localhost + cd /usr/share/cgrates/storage/mysql/ + sudo ./setup_cgr_db.sh root CGRateS.org localhost `PostgreSQL`_ - Can be used as :ref:`StorDB`. - Optimized for CDR archiving and offline Tariff Plan versioning. - Once PostgreSQL is installed, CGRateS database needs to be set-up out of provided scripts (example for the paths set-up by debian package). + Like MySQL, PostgreSQL can be used as :ref:`StorDB`. Post-installation, you need to set up the CGRateS database using the provided scripts: - :: +.. code-block:: bash - cd /usr/share/cgrates/storage/postgres/ - ./setup_cgr_db.sh + cd /usr/share/cgrates/storage/postgres/ + ./setup_cgr_db.sh `MongoDB`_ - Can be used as :ref:`DataDB` as well as :ref:`StorDB`. - It is the first database that can be used to store all kinds of data stored from CGRateS from accounts, tariff plans to cdrs and logs. - Once MongoDB is installed, CGRateS database needs to be set-up out of provided scripts (example for the paths set-up by debian package) + MongoDB can be used as both :ref:`DataDB` and :ref:`StorDB`. This is the first database that can store all types of data from CGRateS - from accounts, tariff plans to CDRs and logs. Post-installation, you need to set up the CGRateS database using the provided scripts: - :: +.. code-block:: bash - cd /usr/share/cgrates/storage/mongo/ - ./setup_cgr_db.sh + cd /usr/share/cgrates/storage/mongo/ + ./setup_cgr_db.sh - -Set versions data +Set Versions Data ^^^^^^^^^^^^^^^^^ -Once database setup is completed, we need to write the versions data. To do this, run migrator tool with the parameters specific to your database. +After completing the database setup, you need to write the versions data. To do this, run the migrator tool with the parameters specific to your database. Sample usage for MySQL: -:: + +.. code-block:: bash cgr-migrator -stordb_passwd="CGRateS.org" -exec="*set_versions" - diff --git a/docs/janusagent.rst b/docs/janusagent.rst new file mode 100644 index 000000000..61cd95579 --- /dev/null +++ b/docs/janusagent.rst @@ -0,0 +1,39 @@ +.. _JanusAgent: + +JanusAgent +============= + + +**JanusAgent** is an api endpoint that connects to JanusServer through **CGRateS**. +It authorizes webrtc events in **CGRateS** for each user and after managing and creating sessions in JanusServer. + +The **JanusAgent** is configured within *janus_agent* section from :ref:`JSON configuration `. +It will listen on http port 2080 in /janus endpoint as specified in config ,it will accept same http requests that would be sent normally to JanusServer. + +Sample config + +:: + + "janus_agent": { + "enabled": false, // enables the Janus agent: + "url": "/janus", + "sessions_conns": ["*internal"], + "janus_conns": [{ // instantiate connections to multiple Janus Servers + "address": "127.0.0.1:8088", // janus API address + "type": "*ws", // type of the transport to interact via janus API + "admin_address": "localhost:7188", // janus admin address used to retrive more information for sessions and handles + "admin_password": "", // secret to pass restriction to communicate to the endpoint + }], + "request_processors": [], // request processors to be applied to Janus messages +}, + +Config params +^^^^^^^^^^^^^ + +Most of the parameters are explained in :ref:`JSON configuration `, hence we mention here only the ones where additional info is necessary or there will be particular implementation for *JanusAgent*. + +Software Installation +--------------------- + + For detailed information on installing JanusServer on Debian, please refer to its official `repository `_. + diff --git a/docs/overview.rst b/docs/overview.rst index f391c77d1..242e4fb23 100644 --- a/docs/overview.rst +++ b/docs/overview.rst @@ -4,7 +4,7 @@ .. _Kafka: https://kafka.apache.org/ .. _redis: https://redis.io/ .. _mongodb: https://www.mongodb.com/ -.. _api docs: https://godoc.org/github.com/cgrates/cgrates/apier +.. _api docs: https://pkg.go.dev/github.com/cgrates/cgrates/apier@master .. _SQS: https://aws.amazon.com/de/sqs/ .. _AMQP: https://www.amqp.org/ .. _Asterisk: https://www.asterisk.org/ @@ -187,7 +187,7 @@ Links - CGRateS home page ``_ - Documentation ``_ -- API docs ``_ +- API docs ``_ - Source code ``_ - Travis CI ``_ - Google group ``_ @@ -197,4 +197,4 @@ Links License ------- -`CGRateS`_ is released under the terms of the `[GNU GENERAL PUBLIC LICENSE Version 3] `_. \ No newline at end of file +`CGRateS`_ is released under the terms of the `[GNU GENERAL PUBLIC LICENSE Version 3] `_. diff --git a/docs/prometheus.rst b/docs/prometheus.rst new file mode 100644 index 000000000..51c48832b --- /dev/null +++ b/docs/prometheus.rst @@ -0,0 +1,105 @@ +.. _prometheus_agent: + +PrometheusAgent +=============== + +**PrometheusAgent** is a CGRateS component that exposes metrics for Prometheus monitoring systems. It serves as a bridge between CGRateS and Prometheus by collecting and exposing metrics from: + +1. **Core metrics** - collected from configured CGRateS engines via CoreSv1.Status API +2. **StatQueue metrics** - values from CGRateS :ref:`StatS ` component, collected via StatSv1.GetQueueFloatMetrics API + +For core metrics, the agent computes real-time values on each Prometheus scrape request. For StatQueue metrics, it retrieves the current state of the stored StatQueues without additional calculations. + +Configuration +------------- + +Example configuration in the JSON file: + +.. code-block:: json + + "prometheus_agent": { + "enabled": true, + "path": "/prometheus", + "cores_conns": ["*internal", "external"], + "stats_conns": ["*internal", "external"], + "stat_queue_ids": ["cgrates.org:SQ_1", "SQ_2"] + } + +The default configuration can be found in the :ref:`configuration` section. + +Parameters +---------- + +enabled + Enable the PrometheusAgent module. Possible values: + +path + HTTP endpoint path where Prometheus metrics will be exposed, e.g., "/prometheus" or "/metrics" + +cores_conns + List of connection IDs to CoreS components for collecting core metrics. Empty list disables core metrics collection. Possible values: <""|*internal|$rpc_conns_id> + +stats_conns + List of connection IDs to StatS components for collecting StatQueue metrics. Empty list disables StatQueue metrics collection. Possible values: <""|*internal|$rpc_conns_id> + +stat_queue_ids + List of StatQueue IDs to collect metrics from. Can include tenant in format <[tenant]:ID>. If tenant is not specified, default tenant from general configuration is used. + +Available Metrics +----------------- + +The PrometheusAgent exposes the following metrics: + +1. **StatQueue Metrics** + - Uses the naming format ``cgrates_stats_metrics`` with labels for tenant, queue, and metric type + - Obtained from StatS services on each scrape request + + Example of StatQueue metrics output: + + .. code-block:: none + + # HELP cgrates_stats_metrics Current values for StatQueue metrics + # TYPE cgrates_stats_metrics gauge + cgrates_stats_metrics{metric="*acc",queue="SQ_1",tenant="cgrates.org"} 7.73779 + cgrates_stats_metrics{metric="*tcc",queue="SQ_1",tenant="cgrates.org"} 23.21337 + cgrates_stats_metrics{metric="*acc",queue="SQ_2",tenant="cgrates.org"} 11.34716 + cgrates_stats_metrics{metric="*tcc",queue="SQ_2",tenant="cgrates.org"} 34.04147 + +.. note:: + StatQueue metrics don't include node_id labels since StatQueues can be shared between CGRateS instances. Users should ensure StatQueue IDs are unique across their environment. + +2. **Core Metrics** (when cores_conns is configured) + - Standard Go runtime metrics (go_goroutines, go_memstats_*, etc.) + - Standard process metrics (process_cpu_seconds_total, process_open_fds, etc.) + - Node identification via "node_id" label, allowing multiple CGRateS engines to be monitored + + Example of core metrics output: + + .. code-block:: none + + # HELP go_goroutines Number of goroutines that currently exist. + # TYPE go_goroutines gauge + go_goroutines{node_id="e94160b"} 40 + + # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds. + # TYPE process_cpu_seconds_total counter + process_cpu_seconds_total{node_id="e94160b"} 0.34 + + # HELP go_memstats_alloc_bytes Number of bytes allocated in heap and currently in use. + # TYPE go_memstats_alloc_bytes gauge + go_memstats_alloc_bytes{node_id="e94160b"} 1.1360808e+07 + +How It Works +------------ + +The PrometheusAgent operates differently than other CGRateS components that use connection failover: + +- When multiple connections are configured in stats_conns, the agent collects metrics from **all** connections, not just the first available one +- When multiple connections are configured in cores_conns, the agent attempts to collect metrics from **all** connections, labeling them with their respective node_id +- The agent processes metrics requests only when Prometheus sends a scrape request to the configured HTTP endpoint + +You can view all exported metrics and see what Prometheus would scrape by making a simple curl request to the HTTP endpoint: + +.. code-block:: bash + + curl http://localhost:2080/prometheus diff --git a/docs/rals.rst b/docs/rals.rst index f631c9640..dba4aef61 100644 --- a/docs/rals.rst +++ b/docs/rals.rst @@ -4,7 +4,7 @@ RALs ==== -**RALs** is a standalone subsystem within **CGRateS** designed to handle two major tasks: :ref:`Rating` and :ref:`Accounting`. It is accessed via `CGRateS RPC APIs `_. +**RALs** is a standalone subsystem within **CGRateS** designed to handle two major tasks: :ref:`Rating` and :ref:`Accounting`. It is accessed via `CGRateS RPC APIs `_. @@ -51,7 +51,7 @@ FallbackSubjects RatingPlan ^^^^^^^^^^ -Groups together rates per destination. Configured via the following parameters: +Groups together rates per destination and relates them to event timing. Configured via the following parameters: ID The tag uniquely idenfying each RatingPlan. There can be multiple entries grouped by the same ID. @@ -59,8 +59,11 @@ ID DestinationRatesID The identifier of the :ref:`DestinationRate` set. +TimingID + The itentifier of the :ref:`Timing` profile. + Weight - Priority of matching rule (*DestinationRatesID*). Higher value equals higher priority. + Priority of matching rule (*DestinationRatesID*+*TimingID*). Higher value equals higher priority. .. _DestinationRate: @@ -147,6 +150,36 @@ GroupIntervalStart Activates the rate at specific usage within the event. +.. _Timing: + +Timing +^^^^^^ + +A *Timing* profile is giving time awarness to an event. Configured via the following parameters: + +ID + The tag uniquely idenfying each *Timing* profile. + +Years + List of years to match within the event. Defaults to the catch-all meta: *\*any*. + +Months + List of months to match within the event. Defaults to the catch-all meta: *\*any*. + +MonthDays + List of month days to match within the event. Defaults to the catch-all meta: *\*any*. + +WeekDays + List of week days to match within the event as integer values. Special case for *Sunday* which matches for both 0 and 7. + +Time + The exact time to match (mostly as time start). Defined in the format: *hh:mm:ss* + + + +.. Note:: Due to optimization, CGRateS encapsulates and stores the rating information into just three objects: *Destinations*, *RatingProfiles* and *RatingPlan* (composed out of *RatingPlan*, *DestinationRate*, *Rate* and *Timing* objects). + + .. _Accounting: @@ -210,10 +243,10 @@ The following *BalanceTypes* are supported: Coupled with MMS events, represents number of MMS units. \*generic - Matching all types of events after specific ones, represents generic units (ie: for each x *voice minutes, y *sms units, z *data units will have ) + Matching all types of events after specific ones, representing generic units (i.e., for each x \*voice minutes, y \*sms units, and z \*data units will have their respective usage) \*monetary - Matching all types of events after specific ones, represents monetary units (can be interpreted as virtual currency). + Matching all types of events after specific ones, representing monetary units (can be interpreted as virtual currency). @@ -254,10 +287,13 @@ Categories SharedGroup Pointing towards a shared balance ID. +TimingIDs + List of :ref:`Timing` profiles this *Balance* will match for, considering event's *AnswerTime* field. + Disabled Makes the *Balance* invisible to charging. -Factor +Factors Used in case of of *\*generic* *BalanceType* to specify the conversion factors for different type of events. Blocker @@ -345,7 +381,7 @@ Action Actions are routines executed on demand (ie. by one of the three subsystems: :ref:`SchedulerS`, :ref:`ThresholdS` or :ref:`ActionTriggers `) or called by API by external scripts. -An *Action has the following parameters: +An \*Action has the following parameters: ID *ActionSet* identifier. @@ -398,7 +434,7 @@ ActionType **\*disable_account** Set the :ref:`Account` *Disabled* flag. - **\*httpPost** + **\*http_post** Post data over HTTP protocol to configured HTTP URL. **\*http_post_async** @@ -443,7 +479,7 @@ ActionType **\*remove_expired** Removes expired balances of type matching the filter. - **\*cdr_account** + **\*reset_account_cdr** Creates the account out of last *CDR* saved in :ref:`StorDB` matching the account details in the filter. The *CDR* should contain *AccountSummary* within it's *CostDetails*. diff --git a/docs/rankings.rst b/docs/rankings.rst new file mode 100644 index 000000000..cd448e072 --- /dev/null +++ b/docs/rankings.rst @@ -0,0 +1,166 @@ +.. _RankingS: + + +RankingS +======== + + +**RankingS** is a standalone subsystem part of the **CGRateS** infrastructure, designed to work as an extension of the :ref:`StatS`, by regularly querying it for a list of predefined StatProfiles and ordering them based on their metrics. + +Complete interaction with **RankingS** is possible via `CGRateS RPC APIs `_. + +Due it's real-time nature, **RankingS** are designed towards high throughput being able to process thousands of queries per second. This is doable since each *Ranking* is a very light object, held in memory and eventually backed up in :ref:`DataDB`. + + +Processing logic +---------------- + +In order for **RankingS** to start querying the :ref:`StatS`, it will need to be *scheduled* to do that. Scheduling is being done using `Cron Expressions `_. + +Once *Cron Expressions* are defined within a *RankingProfile*, internal **Cron Scheduler** needs to be triggered. This can happen in two different ways: + +Automatic Query Scheduling + The profiles needing querying will be inserted into **RankingS** :ref:`JSON configuration `. By leaving any part of *ranking_id* or *tenat* empty, it will be interpreted as catch-all filter. + +API Scheduling + The profiles needing querying will be sent inside arguments to the `RankingSv1.ScheduleQueries API call `_. + + +Offline storage +--------------- + +Offline storage is optionally possible, by enabling profile *Stored* flag and configuring the *store_interval* inside :ref:`JSON configuration `. + + +Ranking querying +---------------- + +In order to query a **Ranking** (ie: to be displayed in a web interface), one should use the `RankingSv1.GetRanking API call `_ or `RankingSv1.GetRankingSummary API call `. + + +Ranking exporting +--------------- + +On each **Ranking** change, it will be possible to send a specially crafted *RankingSummary* event to one of the following subsystems: + +**ThresholdS** + Sending the **RankingUpdate** Event gives the administrator the possiblity to react to *Ranking* changes, including escalation strategies offered by the **TresholdS** paramters. + Fine tuning parameters (ie. selecting only specific ThresholdProfiles to increase speed) are available directly within the **TrendProfile**. + +**EEs** + **EEs** makes it possible to export the **RankingUpdate** to all the availabe outside interfaces of **CGRateS**. + +Both exporting options are enabled within :ref:`JSON configuration `. + + +Parameters +---------- + + +RankingS +^^^^^^ + +**RankingS** is the **CGRateS** service component responsible of generating the **Ranking** queries. + +It is configured within **RankingS** section from :ref:`JSON configuration ` via the following parameters: + +enabled + Will enable starting of the service. Possible values: . + +store_interval + Time interval for backing up the RankingS into *DataDB*. 0 To completely disable the functionality, -1 to enable synchronous backup. Anything higher than 0 will give the interval for asynchronous backups. + +stats_conns + List of connections where we will query the stats. + +scheduled_ids + Limit the RankingProfiles generating queries towards **StatS**. Empty to enable all available RankingProfiles or just tenants for all the available profiles on a tenant. + +thresholds_conns + Connection IDs towards *ThresholdS* component. If not defined, there will be no notifications sent to *ThresholdS* on *Trend* changes. + +ees_conns + Connection IDs towards the *EEs* component. If left empty, no exports will be performed on *Trend* changes. + +ees_exporter_ids + Limit the exporter profiles executed on *Ranking* changes. + + +RankingProfile +^^^^^^^^^^^^^^ + +Ís made of the following fields: + +Tenant + The tenant on the platform (one can see the tenant as partition ID). + +ID + Identifier for the *RankingProfile*, unique within a *Tenant*. + +Schedule + Cron expression scheduling gathering of the metrics. + +StatIDs + List of **StatS** instances to query. + +MetricIDs + Limit the list of metrics from the stats instance queried. + +Sorting + Sorting strategy for the StatIDs. Possible values: + + \*asc + Sort the StatIDs ascendent based on list of MetricIDs provided in SortParameters. One or more MetricIDs can be specified in hte SortingParameters for the cases when one level sort is not enough to differentiate them. If all metrics will be equal, a random sort will be applied. + + \*desc + Sort the StatIDs descendat based on list of MetricIDs provided in SortParameters. One or more MetricIDs can be specified in hte SortingParameters for the cases when one level sort is not enough to differentiate them. If all metrics will be equal, a random sort will be applied. + +SortingParameters + List of sorting parameters. For the current sorting strategies (\*asc/\*desc) there will be one or more MetricIDs defined. + Metric can be defined in compressed mode (ie. ["Metric1","Metric2"]) or extended mode (ie: ["Metric1:true", "Metric2:false"]) where *false* will reverse the sorting logic for that particular metric (ie: ["\*tcc:true","\*pdd:false"] with \*desc sorting strategy). + +Stored + Enable storing of this *Ranking* intance for persistence. + +ThresholdIDs + Limit *TresholdProfiles* processing the *RankingUpdate* for this *RankingProfile*. + + +Ranking +^^^^^^^ + +instance is made out of the following fields: + +Tenant + The tenant on the platform (one can see the tenant as partition ID). + +ID + Unique *Ranking* identifier on a *Tenant*. + +LastUpdate + Time of the last Metrics update. + +Metrics + Stat Metrics and their values at the query time. + +Sorting + Archived sorting strategy from the profile. + +SortingParameters + Archived list of sorted parameters from the profile. + +SortedStatIDs + List of queried stats, sorted based on sorting strategy and parameters. + + +Use cases +--------- + +* Ranking computation for commercial and monitoring applications. +* Revenue assurance applications. +* Fraud detection by ranking specific billing metrics during sensitive time intervals (\*acc, \*tcc, \*tcd). +* Building call patterns. +* Building statistical information to train systems capable of artificial intelligence. +* Building quality metrics used in traffic routing. + + diff --git a/docs/requirements.in b/docs/requirements.in new file mode 100644 index 000000000..5e7703c92 --- /dev/null +++ b/docs/requirements.in @@ -0,0 +1,4 @@ +sphinx_rtd_theme==1.2.0 +sphinx==6.2.1 +sphinx_copybutton==0.5.2 +sphinx_tabs==3.4.1 \ No newline at end of file diff --git a/docs/requirements.txt b/docs/requirements.txt index 769b0639c..f84b0fe0a 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,9 +1,70 @@ -mock==1.0.1 -pillow==5.4.1 -alabaster>=0.7,<0.8,!=0.7.5 -commonmark==0.8.1 -recommonmark==0.5.0 -sphinx<2 -sphinx-rtd-theme<0.5 -readthedocs-sphinx-ext<2.2 -docutils>=0.14,<0.18 +# +# This file is autogenerated by pip-compile with Python 3.9 +# by the following command: +# +# pip-compile --output-file=requirements.txt --resolver=backtracking requirements.in +# +alabaster==0.7.13 + # via sphinx +babel==2.12.1 + # via sphinx +certifi==2023.5.7 + # via requests +charset-normalizer==3.1.0 + # via requests +docutils==0.18.1 + # via + # sphinx + # sphinx-rtd-theme + # sphinx-tabs +idna==3.4 + # via requests +imagesize==1.4.1 + # via sphinx +importlib-metadata==6.6.0 + # via sphinx +jinja2==3.1.2 + # via sphinx +markupsafe==2.1.2 + # via jinja2 +packaging==23.1 + # via sphinx +pygments==2.15.1 + # via + # sphinx + # sphinx-tabs +requests==2.30.0 + # via sphinx +snowballstemmer==2.2.0 + # via sphinx +sphinx==6.2.1 + # via + # -r requirements.in + # sphinx-copybutton + # sphinx-rtd-theme + # sphinx-tabs + # sphinxcontrib-jquery +sphinx-copybutton==0.5.2 + # via -r requirements.in +sphinx-rtd-theme==1.2.0 + # via -r requirements.in +sphinx-tabs==3.4.1 + # via -r requirements.in +sphinxcontrib-applehelp==1.0.4 + # via sphinx +sphinxcontrib-devhelp==1.0.2 + # via sphinx +sphinxcontrib-htmlhelp==2.0.1 + # via sphinx +sphinxcontrib-jquery==4.1 + # via sphinx-rtd-theme +sphinxcontrib-jsmath==1.0.1 + # via sphinx +sphinxcontrib-qthelp==1.0.3 + # via sphinx +sphinxcontrib-serializinghtml==1.1.5 + # via sphinx +urllib3==2.0.2 + # via requests +zipp==3.15.0 + # via importlib-metadata diff --git a/docs/resources.rst b/docs/resources.rst index 70344f75d..75000faf2 100644 --- a/docs/resources.rst +++ b/docs/resources.rst @@ -6,7 +6,7 @@ ResourceS **ResourceS** is a standalone subsystem part of the **CGRateS** infrastructure, designed to allocate virtual resources for the generic *Events* (hashmaps) it receives. -Both receiving of *Events* as well as operational commands on the virtual resources is performed via a complete set of `CGRateS RPC APIs `_. +Both receiving of *Events* as well as operational commands on the virtual resources is performed via a complete set of `CGRateS RPC APIs `_. Due it's real-time nature, **ResourceS** are designed towards high throughput being able to process thousands of *Events* per second. This is doable since each *Resource* is a very light object, held in memory and eventually backed up in *DataDB*. @@ -46,7 +46,7 @@ nested_fields ResourceProfile ^^^^^^^^^^^^^^^ -The **ResourceProfile** is the configuration of a *Resource*. This will be performed over `CGRateS RPC APIs `_ or *.csv* files. A profile is comprised out of the following parameters: +The **ResourceProfile** is the configuration of a *Resource*. This will be performed over `CGRateS RPC APIs `_ or *.csv* files. A profile is comprised out of the following parameters: Tenant The tenant on the platform (one can see the tenant as partition ID). @@ -57,6 +57,9 @@ ID FilterIDs List of *FilterProfiles* which should match in order to consider the *ResourceProfile* matching the event. +ActivationInterval + The time interval when this profile becomes active. If undefined, the profile is always active. Other options are start time, end time or both. + UsageTTL Autoexpire resource allocation after this time duration. diff --git a/docs/routes.rst b/docs/routes.rst index 7164b056c..38c245d5d 100644 --- a/docs/routes.rst +++ b/docs/routes.rst @@ -10,7 +10,7 @@ RouteS ========= -**RouteS** is a standalone subsystem within **CGRateS** responsible to compute a list of routes which can be used for a specific event received to process. It is accessed via `CGRateS RPC APIs `_. +**RouteS** is a standalone subsystem within **CGRateS** responsible to compute a list of routes which can be used for a specific event received to process. It is accessed via `CGRateS RPC APIs `_. As most of the other subsystems, it is performance oriented, stored inside *DataDB* but cached inside the *cgr-engine* process. Caching can be done dynamically/on-demand or at start-time/precached and it is configurable within *cache* section in the :ref:`JSON configuration `. @@ -73,13 +73,13 @@ attributes_conns Connections to AttributeS for altering events before supplier queries. If undefined, fields modifications are disabled. resources_conns - Connections to ResourceS for *res sorting, empty to disable functionality. + Connections to ResourceS for \*res sorting, empty to disable functionality. stats_conns - Connections to StatS for *stats sorting, empty to disable stats functionality. + Connections to StatS for \*stats sorting, empty to disable stats functionality. default_ratio - Default ratio used in case of *load strategy + Default ratio used in case of \*load strategy .. _SupplierProfile: @@ -98,6 +98,9 @@ ID FilterIDs List of *FilterProfileIDs* which should match in order to consider the profile matching the event. +ActivationInterval + The time interval when this profile becomes active. If undefined, the profile is always active. Other options are start time, end time or both. + Sorting Sorting strategy applied when ordering the individual *Routes* defined bellow. Possible values are: @@ -156,19 +159,19 @@ FilterIDs List of *FilterProfileIDs* which should match in order to consider the *Supplier* in use/active. AccountIDs - List of account IDs which should be checked in case of some strategies (ie: *lc, *hc). + List of account IDs which should be checked in case of some strategies (ie: \*lc, \*hc). RatingPlanIDs - List of RatingPlanIDs which should be checked in case of some strategies (ie: *lc, *hc). + List of RatingPlanIDs which should be checked in case of some strategies (ie: \*lc, \*hc). ResourceIDs - List of ResourceIDs which should be checked in case of some strategies (ie: *reas or *reds). + List of ResourceIDs which should be checked in case of some strategies (ie: \*reas or \*reds). StatIDs - List of StatIDs which should be checked in case of some strategies (ie: *qos or *load). Can also be defined as *StatID:MetricID*. + List of StatIDs which should be checked in case of some strategies (ie: \*qos or \*load). Can also be defined as *StatID:MetricID*. Weight - Used for sorting in some strategies (ie: *weight, *lc or *hc). + Used for sorting in some strategies (ie: \*weight, \*lc or \*hc). Blocker No more routes are provided after this one. diff --git a/docs/rpcconns.rst b/docs/rpcconns.rst new file mode 100644 index 000000000..7926d4746 --- /dev/null +++ b/docs/rpcconns.rst @@ -0,0 +1,161 @@ +.. _rpc_conns: + +RPCConns +======== + +**RPCConns** defines connection pools used by CGRateS components for inter-service communication. These pools enable services to interact both within a single CGRateS instance or across multiple instances. + + +Configuration Structure +----------------------- + +Example configuration in the JSON file: + +.. code-block:: json + + { + "rpc_conns": { + "conn1": { + "strategy": "*first", + "pool_size": 0, + "conns": [{ + "address": "192.168.122.210:2012", + "transport": "*json", + "connect_attempts": 5, + "reconnects": -1, + "connect_timeout": "1s", + "reply_timeout": "2s" + }] + } + } + } + + +Predefined Connection Pools +--------------------------- + +\*internal + Direct in-process communication + +\*birpc_internal + Bidirectional in-process communication + +\*localhost + JSON-RPC connection to local cgr-engine on port 2012 + +\*bijson_localhost + Bidirectional JSON-RPC connection to local cgr-engine on port 2014 + +Bidirectional Communication with SessionS +----------------------------------------- + +Bidirectional connections are specifically designed and used for communication between agents and the :ref:`SessionS ` component. While agents can send requests using standard connections, bidirectional connections are necessary when SessionS needs to communicate back to the agents. + +When using bidirectional connections, SessionS maintains references to all connected agents, allowing it to send requests back to specific agents when needed (for example, to force disconnect a session or query active sessions). + +.. note:: + Bidirectional connections (``*birpc_internal``, ``*birpc_json``, ``*birpc_gob``) are exclusively used for Agent-SessionS communication. All other service interactions use standard one-way connections. + + +Parameters +---------- + + +Pool Parameters +^^^^^^^^^^^^^^^ + +Strategy + Controls connection selection within the pool. Possible values: + + * ``*first``: Uses first available connection, fails over on network/timeout/missing service errors + * ``*next``: Round-robin between connections with same failover as ``*first`` + * ``*random``: Random connection selection with same failover as ``*first`` + * ``*first_positive``: Tries connections in order until getting any successful response + * ``*first_positive_async``: Async version of ``*first_positive`` + * ``*broadcast``: Sends to all connections, returns first successful response + * ``*broadcast_sync``: Sends to all, waits for completion, logs errors that wouldn't trigger failover in ``*first`` + * ``*broadcast_async``: Sends to all without waiting for responses + * ``*parallel``: Pool that creates and reuses connections up to a limit + +.. note:: + Connections attempt failover to the next available connection in the pool on connection errors, timeouts, or service errors. Service errors (usually referring to "can't find service" errors) occur when attempting to reach services that are either temporarily unavailable during engine initialization or disabled in that particular instance. + +PoolSize + Sets the connection limit for ``*parallel`` strategy (0 means unlimited) + + +Connection Parameters +^^^^^^^^^^^^^^^^^^^^^ + +Address + Network address, ``*internal``, or ``*birpc_internal`` + +Transport + Protocol (``*json``, ``*gob``, ``*birpc_json``, ``*birpc_gob``, ``*http_jsonrpc``). When using ``*internal`` or ``*birpc_internal`` addresses, defaults to the address value. Otherwise defaults to ``*gob``. + +ConnectAttempts + Number of initial connection attempts + +Reconnects + Max number of reconnection attempts (-1 for infinite) + +MaxReconnectInterval + Maximum delay between reconnects + +ConnectTimeout + Connection timeout (e.g., "1s") + +ReplyTimeout + Response timeout (e.g., "2s") + +TLS + Enable TLS encryption + +ClientKey + Path to TLS client key file + +ClientCertificate + Path to TLS client certificate + +CaCertificate + Path to CA certificate + + +Transport Performance +--------------------- + +\*internal, \*birpc_internal + In-process communication (by far the fastest) + +\*gob, \*birpc_gob + Binary protocol that provides better performance at the cost of being harder to troubleshoot + +\*json, \*birpc_json + Standard JSON protocol - slower but easier to debug since you can read the traffic + +\*http_jsonrpc + HTTP-based JSON-RPC protocol - slower than direct JSON-RPC due to HTTP overhead, but can integrate with web infrastructure and provides easy debugging through standard HTTP tools + +.. note:: + While the "transport" parameter name is used in the configuration, it actually specifies the codec (*json, *gob) used for data encoding. All network connections use TCP, while internal ones skip networking completely. + +Using Connection Pools +---------------------- + +Components reference connection pools through "_conns" configuration fields: + +.. code-block:: json + + { + "cdrs": { + "enabled": true, + "rals_conns": ["*internal"], + "ees_conns": ["conn1"] + } + } + +This configuration approach allows: + +* Deploying services across single or multiple instances +* Selecting transports based on performance requirements +* Automatic failover between connections diff --git a/docs/sessions.rst b/docs/sessions.rst index dc40ea27e..b3ccb8709 100644 --- a/docs/sessions.rst +++ b/docs/sessions.rst @@ -4,7 +4,7 @@ SessionS ======== -**SessionS** is a standalone subsystem within **CGRateS** responsible to manage virtual sessions based on events received. It is accessed via `CGRateS RPC APIs `_. +**SessionS** is a standalone subsystem within **CGRateS** responsible to manage virtual sessions based on events received. It is accessed via `CGRateS RPC APIs `_. Parameters @@ -341,7 +341,7 @@ Instead of arguments, the options for enabling various functionaity will come in \*routes Process the event with :ref:`Routes`. Auxiliary flags available: - **\*ignoreErrors** + **\*ignore_errors** Ignore the routes with errors instead of failing the request completely. **\*event_cost** diff --git a/docs/stats.rst b/docs/stats.rst index 835c836c5..3e06dcb4b 100644 --- a/docs/stats.rst +++ b/docs/stats.rst @@ -6,7 +6,7 @@ StatS **StatS** is a standalone subsystem part of the **CGRateS** infrastructure, designed to aggregate and calculate statistical metrics for the generic *Events* (hashmaps) it receives. -Both receiving of *Events* as well as *Metrics* displaying is performed via a complete set of `CGRateS RPC APIs `_. +Both receiving of *Events* as well as *Metrics* displaying is performed via a complete set of `CGRateS RPC APIs `_. Due it's real-time nature, **StatS** are designed towards high throughput being able to process thousands of *Events* per second. This is doable since each *StatQueue* is a very light object, held in memory and eventually backed up in *DataDB*. @@ -73,6 +73,9 @@ ID FilterIDs List of *FilterProfileIDs* which should match in order to consider the profile matching the event. +ActivationInterval + The time interval when this profile becomes active. If undefined, the profile is always active. Other options are start time, end time or both. + QueueLength Maximum number of items stored in the queue. Once the *QueueLength* is reached, new items entering will cause oldest one to be dropped (FIFO mode). @@ -137,7 +140,7 @@ Use cases * Aggregate various traffic metrics for traffic transparency. * Revenue assurance applications. -* Fraud detection by aggregating specific billing metrics during sensitive time intervals (*acc, *tcc, *tcd). +* Fraud detection by aggregating specific billing metrics during sensitive time intervals (\*acc, \*tcc, \*tcd). * Building call patterns. * Building statistical information to train systems capable of artificial intelligence. * Building quality metrics used in traffic routing. diff --git a/docs/thresholds.rst b/docs/thresholds.rst index 05d1a3c58..9add710e5 100644 --- a/docs/thresholds.rst +++ b/docs/thresholds.rst @@ -4,7 +4,7 @@ ThresholdS ========== -**ThresholdS** is a standalone subsystem within **CGRateS** responsible to execute a list of *Actions* for a specific event received to process. It is accessed via `CGRateS RPC APIs `_. +**ThresholdS** is a standalone subsystem within **CGRateS** responsible to execute a list of *Actions* for a specific event received to process. It is accessed via `CGRateS RPC APIs `_. As most of the other subsystems, it is performance oriented, stored inside *DataDB* but cached inside the *cgr-engine* process. Caching can be done dynamically/on-demand or at start-time/precached and it is configurable within *cache* section in the :ref:`JSON configuration `. @@ -93,6 +93,9 @@ ID FilterIDs List of *FilterProfileIDs* which should match in order to consider the profile matching the event. +ActivationInterval + The time interval when this profile becomes active. If undefined, the profile is always active. Other options are start time, end time or both. + MaxHits Limit number of hits for this threshold. Once this is reached, the threshold is considered disabled. @@ -108,8 +111,8 @@ Blocker Weight Sorts the execution of multiple thresholds matching the event. The higher the *Weight* is, the higher the priority to be executed. -ActionProfileIDs - List of *ActionProfiles* to execute for this threshold. +ActionIDs + List of *Actions* to execute for this threshold. Async If true, do not wait for actions to complete. diff --git a/docs/trends.rst b/docs/trends.rst new file mode 100644 index 000000000..1fb19e5c7 --- /dev/null +++ b/docs/trends.rst @@ -0,0 +1,168 @@ +.. _trends: + +TrendS +===== + + +**TrendS** is a standalone subsystem part of the **CGRateS** infrastructure, designed to work as an extension of the :ref:`StatS`, by regularly querying it, storing it's values in a time-series-like database and calculate trend percentages based on their evolution. + +Complete interaction with **TrendS** is possible via `CGRateS RPC APIs `_. + +Due it's real-time nature, **TrendS** are designed towards high throughput being able to process thousands of queries per second. This is doable since each *Trend* is a very light object, held in memory and eventually backed up in :ref:`DataDB`. + + +Processing logic +---------------- + +In order for **TrendS** to start querying the :ref:`StatS`, it will need to be *scheduled* to do that. Scheduling is being done using `Cron Expressions `_. + +Once *Cron Expressions* are defined within a *TrendProfile*, internal **Cron Scheduler** needs to be triggered. This can happen in two different ways: + +Automatic Query Scheduling + The profiles needing querying will be inserted into **trends** :ref:`JSON configuration `. By leaving any part of *trend_id* or *tenat* empty, it will be interpreted as catch-all filter. + +API Scheduling + The profiles needing querying will be sent inside arguments to the `TrendSv1.ScheduleQueries API call `_. + + +Offline storage +--------------- + +Offline storage is optionally possible, by enabling profile *Stored* flag and configuring the *store_interval* inside :ref:`JSON configuration `. + + +Trend querying +-------------- + +In order to query a **Trend** (ie: to be displayed in a web interface), one should use the `TrendSv1.GetTrend API call `_ which also offers pagination parameters. + + +Trend exporting +--------------- + +On each **Trend** change, it will be possible to send a specially crafted event, *TrendUpdate* to one of the following subsystems: + +**ThresholdS** + Sending the **TrendUpdate** Event gives the administrator the possiblity to react to *Trend* changes, including escalation strategies offered by the **TresholdS** paramters. + Fine tuning parameters (ie. selecting only specific ThresholdProfiles to increase speed1) are available directly within the **TrendProfile**. + +**EEs** + **EEs** makes it possible to export the **TrendUpdate** to all the availabe outside interfaces of **CGRateS**. + +Both exporting options are enabled within :ref:`JSON configuration `. + + +Parameters +---------- + + +TrendS +^^^^^^ + +**TrendS** is the **CGRateS** component responsible of generating the **Trend** queries. + +It is configured within **trends** section from :ref:`JSON configuration ` via the following parameters: + +enabled + Will enable starting of the service. Possible values: . + +store_interval + Time interval for backing up the trends into *DataDB*. 0 To completely disable the functionality, -1 to enable synchronous backup. Anything higher than 0 will give the interval for asynchronous backups. + +stats_conns + List of connections where we will query the stats. + +scheduled_ids + Limit the TrendProfiles queried. Empty to query all the available TrendProfiles or just tenants for all the available profiles on a tenant. + +thresholds_conns + Connection IDs towards *ThresholdS* component. If not defined, there will be no notifications sent to *ThresholdS* on *Trend* changes. + +ees_conns + Connection IDs towards the *EEs* component. If left empty, no exports will be performed on *Trend* changes. + +ees_exporter_ids + Limit the exporter profiles executed on *Trend* changes. + + +TrendProfile +^^^^^^^^^^^^ + +Ís made of the following fields: + +Tenant + The tenant on the platform (one can see the tenant as partition ID). + +ID + Identifier for the *TrendProfile*, unique within a *Tenant*. + +Schedule + Cron expression scheduling gathering of the metrics. + +StatID + StatS identifier which will be queried. + +Metrics + Limit the list of metrics from the stats instance queried. + +TTL + Automatic cleanup of the queried values from inside *Trend Metrics*. + +QueueLength + Limit the size of *Trend Metrics*. Older values will be removed first. + +MinItems + Issue *TrendUpdate* events to external subsystems only if MinItems are reched to limit false alarms. + +CorrelationType + The correlation strategy to use when computing the trend. *\*average* will consider all previous query values and *\*last* only the last one. + +Tolerance + Allow a deviation of the values when computin the trend. This is defined as percentage of increase/decrease. + +Stored + Enable storing of this *Trend* for persistence. + +ThresholdIDs + Limit *TresholdProfiles* processing the *TrendUpdate* for this *TrendProfile*. + + +Trend +^^^^^ + +is made out of the following fields: + +Tenant + The tenant on the platform (one can see the tenant as partition ID). + +ID + Unique *Trend* identifier on a *Tenant* + +RunTimes + Times when the stat queries were ran by the scheduler + +Metrics + History of the queried metrics, indexed by the query time. One query stores the following values: + + ID + Metric ID on the *StatS* side + + Value + Value of the metric at the time of query + + TrendGrowth + Computed trend growth for the metric values, stored in percentage numbers. + + TrendLabel + Computed trend label for the metric values. Possible values are: *positive, *negative, *constant, N/A. + + +Use cases +--------- + +* Aggregate various traffic metrics for traffic transparency. +* Revenue assurance applications. +* Fraud detection by aggregating specific billing metrics during sensitive time intervals (\*acc, \*tcc, \*tcd). +* Building call patterns. +* Building statistical information to train systems capable of artificial intelligence. +* Building quality metrics used in traffic routing. diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst new file mode 100644 index 000000000..5db138afb --- /dev/null +++ b/docs/troubleshooting.rst @@ -0,0 +1,308 @@ +Troubleshooting +=============== + +.. contents:: + :local: + :depth: 3 + +Profiling +--------- + +This section covers how to set up and use profiling tools for ``cgrates`` to diagnose performance issues. You can enable profiling through configuration, runtime flags, or APIs. + +For more information on profiling in Go, see the `Go Diagnostics: Profiling `_ documentation. + +Configuration +~~~~~~~~~~~~~ + +There are two ways to set up profiling: + +Using JSON Configuration +^^^^^^^^^^^^^^^^^^^^^^^^ + +Enable profiling by adding the ``pprof_path`` under the ``http`` section in your JSON config file: + +.. code-block:: json + + { + "listen": { + "http": ":2080", + "http_tls": ":2280" + }, + "http": { + "pprof_path": "/debug/pprof/" + } + } + +Profiling is enabled by default and exposes the ``/debug/pprof/`` endpoint. You can access it through the address set in the ``listen`` section (``http`` or ``http_tls``). To turn off profiling, set ``pprof_path`` to an empty string ``""``. + +Using Runtime Flags +^^^^^^^^^^^^^^^^^^^ + +You can also control profiling with runtime flags when starting ``cgr-engine``: + +.. code-block:: console + + $ cgr-engine -help + Usage of cgr-engine: + -cpuprof_dir string + Directory for CPU profiles + -memprof_dir string + Directory for memory profiles + -memprof_interval duration + Interval between memory profile saves (default 15s) + -memprof_maxfiles int + Number of memory profiles to keep (most recent) (default 1) + -memprof_timestamp + Add timestamp to memory profile files + +Generating Profile Data +~~~~~~~~~~~~~~~~~~~~~~~ + +Let's assume the profiling interface is available at ``http://localhost:2080/debug/pprof/``. + +.. note:: + Profiling started with flags or APIs can be stopped using the corresponding API calls. If you start profiling on startup using flags and don't stop it manually, a profile will be automatically generated when the engine shuts down. The same applies if you start profiling via API and forget to stop it before shutting down the engine. + +CPU Profiling +^^^^^^^^^^^^^ + +Here's how to generate CPU profile data: + +- **Web Browser**: Go to ``http://localhost:2080/debug/pprof/`` in your browser. Click "profile" to start a 30-second CPU profile. +- **Custom Duration**: Add the ``seconds`` parameter to set a different duration: ``http://localhost:2080/debug/pprof/profile?seconds=5``. +- **Command Line**: Use ``curl`` to download the profile: + + .. code-block:: console + + curl -o cpu.prof http://localhost:2080/debug/pprof/profile?seconds=5 + +- **APIs**: Use ``CoreSv1.StartCPUProfiling`` and ``CoreSv1.StopCPUProfiling`` APIs: + + .. code-block:: json + + { + "method": "CoreSv1.StartCPUProfiling", + "params": [{ + "DirPath": "/tmp" + }], + "id": 1 + } + + { + "method": "CoreSv1.StopCPUProfiling", + "params": [], + "id": 1 + } + +- **Startup Profiling**: Profile the entire runtime by specifying a directory with the ``-cpuprof_dir`` flag: + + .. code-block:: console + + cgr-engine -cpuprof_dir=/tmp [other flags] + +Memory Profiling +^^^^^^^^^^^^^^^^ + +Generate memory profile data like this: + +- **Web Browser**: Visit ``http://localhost:2080/debug/pprof/`` to create a memory snapshot. Use ``?debug=2`` (or ``?debug=1``) for human-readable output. If the ``debug`` parameter is omitted or set to ``0``, the output will be in binary format. +- **Command Line**: Use ``curl`` to download the memory profile: + + .. code-block:: console + + curl -o mem.prof http://localhost:2080/debug/pprof/heap + +- **Automated Profiling**: Use the ``CoreSv1.StartMemoryProfiling`` API for periodic memory snapshots: + + .. code-block:: json + + { + "method": "CoreSv1.StartMemoryProfiling", + "params": [{ + "DirPath": "/tmp", + "Interval": 5000000000, + "MaxFiles": 5, + "UseTimestamp": true + }], + "id": 1 + } + + .. note:: + + ``Interval`` is in nanoseconds. Future updates will allow using time strings (e.g., ``5s``, ``1h``) or seconds as an integer. + +Other Useful Profiles +^^^^^^^^^^^^^^^^^^^^^ + +The ``/debug/pprof/`` endpoint offers more useful profiles: + +- **Goroutine Profile** (``/debug/pprof/goroutine``): View or download goroutine stack dumps. +- **Mutex Profile** (``/debug/pprof/mutex``): Find bottlenecks where goroutines wait for locks. +- **Block Profile** (``/debug/pprof/block``): Identify where goroutines block waiting on synchronization primitives. +- **Thread Create Profile** (``/debug/pprof/threadcreate``): Show stack traces that led to the creation of new OS threads. +- **Execution Trace**: For information on generating and analyzing execution traces, see the `Tracing`_ section below. + +Analyzing Profiles +~~~~~~~~~~~~~~~~~~ + +The main tool for analyzing profiles is ``go tool pprof``. It helps visualize and analyze profiling data. You can use it with both downloaded profile files and directly with URLs. + +Command-Line Analysis +^^^^^^^^^^^^^^^^^^^^^ + +For CPU profiles: + +.. code-block:: console + + go tool pprof cpu.prof + # or + go tool pprof http://localhost:2080/debug/pprof/profile + +For memory profiles: + +.. code-block:: console + + go tool pprof mem.prof + # or + go tool pprof http://localhost:2080/debug/pprof/heap + +This opens an interactive terminal. Use commands like ``top``, ``list``, ``web``, and ``svg`` to explore the profile. + +.. hint:: + Run ``go tool pprof -h`` for more information on available commands and options. + +Visual Analysis +^^^^^^^^^^^^^^^ + +Create visual representations of your profiling data: + +- **SVG**: Generate an SVG graph: + + .. code-block:: console + + go tool pprof -svg cpu.prof > cpu.svg + # or + go tool pprof -svg mem.prof > mem.svg + +- **Web Interface**: Use ``-http`` for an interactive visualization in your browser: + + .. code-block:: console + + go tool pprof -http=:8080 cpu.prof + # or + go tool pprof -http=:8080 mem.prof + + .. note:: + + You might need to install the ``graphviz`` package. + +Tracing +------- + +Execution tracing provides a detailed view of runtime behavior of your Go program. + +For detailed information on tracing in Go, see the `Go Diagnostics: Execution Tracing `_ documentation. + +To generate and analyze trace data: + +.. code-block:: console + + # Generate trace data + curl -o trace.out http://localhost:2080/debug/pprof/trace?seconds=5 + + # Analyze trace data + go tool trace trace.out + +This opens a browser interface for detailed execution analysis. + +Debugging +--------- + +This section covers how to set up and use Delve, a Go debugger, with ``cgrates``. + +For detailed information on debugging Go programs, see the `Go Diagnostics: Debugging `_ documentation. + +Installation +~~~~~~~~~~~~ + +To install Delve, run: + +.. code-block:: console + + go install github.com/go-delve/delve/cmd/dlv@latest + +Basic Usage +~~~~~~~~~~~ + +There are several ways to use Delve with ``cgrates``: + +1. Start ``cgr-engine`` in debug mode: + + .. code-block:: console + + dlv exec /path/to/cgr-engine -- --config_path=/etc/cgrates --logger=*stdout + +2. Attach to a running instance: + + .. code-block:: console + + ENGINE_PID=$(pidof cgr-engine) + dlv attach $ENGINE_PID + +3. Debug tests: + + .. code-block:: console + + dlv test github.com/cgrates/cgrates/apier/v1 -- -test.run=TestName + + +.. hint:: + For better debugging, disable optimizations (``-N``) and inlining (``-l``) when building ``cgr-engine``: + + .. code-block:: console + + go install -gcflags="all=-N -l" -ldflags "-X 'github.com/cgrates/cgrates/utils.GitLastLog=$GIT_LAST_LOG'" github.com/cgrates/cgrates/cmd/cgr-engine + +Handling Crashes +~~~~~~~~~~~~~~~~ + +To capture more information when ``cgrates`` crashes: + +1. Enable core dump generation: + + .. code-block:: console + + ulimit -c unlimited + GOTRACEBACK=crash cgr-engine -config_path=/etc/cgrates + +2. Analyze core dumps with Delve: + + .. code-block:: console + + dlv core /path/to/cgr-engine core + +Common Debugging Commands +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once in a Delve debug session, you can use these common commands: + +- ``break`` or ``b``: Set a breakpoint +- ``continue`` or ``c``: Run until breakpoint or program termination +- ``next`` or ``n``: Step over to next line +- ``step`` or ``s``: Step into function call +- ``print`` or ``p``: Evaluate an expression +- ``goroutines``: List current goroutines +- ``help``: Show help for commands + +For more information on using Delve, refer to the `Delve Documentation `_. + +Further Reading +--------------- + +For more comprehensive information on Go diagnostics, profiling, and debugging, check out these resources: + +- `Go Diagnostics `_: Official documentation on diagnostics in Go. +- `Profiling Go Programs `_: In-depth blog post on profiling in Go. +- `net/http/pprof godoc `_: Documentation for the pprof package. +- `Delve Debugger `_: GitHub repository for the Delve debugger. diff --git a/docs/tutorial.rst b/docs/tutorial.rst new file mode 100644 index 000000000..2fe838906 --- /dev/null +++ b/docs/tutorial.rst @@ -0,0 +1,515 @@ +Tutorial +======== + +.. contents:: + :local: + :depth: 3 + +Introduction +------------ + +This tutorial provides detailed instructions for setting up a SIP Server and managing communication between the server and the CGRateS instance. + +.. note:: + + The development and testing of the instructions in this tutorial has been done on a Debian 11 (Bullseye) virtual machine. + + +Scenario Overview +----------------- + +The tutorial comprises the following steps: + +1. **SIP Server Setup**: + Select and install a SIP Server. The tutorial supports the following options: + + - FreeSWITCH_ + - Asterisk_ + - Kamailio_ + - OpenSIPS_ + +2. **CGRateS Initialization**: + Launch a CGRateS instance with the corresponding agent configured. In this context, an "agent" refers to a component within CGRateS that manages communication between CGRateS and the SIP Servers. + +3. **Account Configuration**: + Establish user accounts for different request types. + +4. **Balance Addition**: + Allocate suitable balances to the user accounts. + +5. **Call Simulation**: + Use Zoiper_ (or any other SIP UA of your choice) to register the user accounts and simulate calls between the configured accounts, and then verify the balance updates post-calls. + +6. **Fraud Detection Setup**: + Implement a fraud detection mechanism to secure and maintain the integrity of the service. + +As we progress through the tutorial, each step will be elaborated in detail. Let's embark on this journey with the SIP Server Setup. + + + +Software Installation +--------------------- + +*CGRateS* already has a section within this documentation regarding installation. It can be found :ref:`here`. + +Regarding the SIP Servers, click on the tab corresponding to the choice you made and follow the steps in order to set up: + +.. tabs:: + + .. group-tab:: FreeSWITCH + + For detailed information on installing FreeSWITCH_ on Debian, please refer to its official `installation wiki `_. + + Before installing FreeSWITCH_, you need to authenticate by creating a SignalWire Personal Access Token. To generate your personal token, follow the instructions in the `SignalWire official wiki on creating a personal token `_. + + To install FreeSWITCH_ and configure it, we have chosen the simplest method using *vanilla* packages. + + .. code-block:: bash + + TOKEN=YOURSIGNALWIRETOKEN # Insert your SignalWire Personal Access Token here + sudo apt-get update && apt-get install -y gnupg2 wget lsb-release + wget --http-user=signalwire --http-password=$TOKEN -O /usr/share/keyrings/signalwire-freeswitch-repo.gpg https://freeswitch.signalwire.com/repo/deb/debian-release/signalwire-freeswitch-repo.gpg + echo "machine freeswitch.signalwire.com login signalwire password $TOKEN" > /etc/apt/auth.conf + chmod 600 /etc/apt/auth.conf + echo "deb [signed-by=/usr/share/keyrings/signalwire-freeswitch-repo.gpg] https://freeswitch.signalwire.com/repo/deb/debian-release/ `lsb_release -sc` main" > /etc/apt/sources.list.d/freeswitch.list + echo "deb-src [signed-by=/usr/share/keyrings/signalwire-freeswitch-repo.gpg] https://freeswitch.signalwire.com/repo/deb/debian-release/ `lsb_release -sc` main" >> /etc/apt/sources.list.d/freeswitch.list + + # If /etc/freeswitch does not exist, the standard vanilla configuration is deployed + sudo apt-get update && apt-get install -y freeswitch-meta-all + + .. group-tab:: Asterisk + + To install Asterisk_, follow these steps: + + .. code-block:: bash + + # Install the necessary dependencies + sudo apt-get install -y build-essential libasound2-dev autoconf \ + openssl libssl-dev libxml2-dev \ + libncurses5-dev uuid-dev sqlite3 \ + libsqlite3-dev pkg-config libedit-dev \ + libjansson-dev + + # Download Asterisk + wget https://downloads.asterisk.org/pub/telephony/asterisk/asterisk-20-current.tar.gz -P /tmp + + # Extract the downloaded archive + sudo tar -xzvf /tmp/asterisk-20-current.tar.gz -C /usr/src + + # Change the working directory to the extracted Asterisk source + cd /usr/src/asterisk-20*/ + + # Compile and install Asterisk + sudo ./configure --with-jansson-bundled + sudo make menuselect.makeopts + sudo make + sudo make install + sudo make samples + sudo make config + sudo ldconfig + + # Create the Asterisk system user + sudo adduser --quiet --system --group --disabled-password --shell /bin/false --gecos "Asterisk" asterisk + + .. group-tab:: Kamailio + + Kamailio_ can be installed using the commands below, as documented in the `Kamailio Debian Installation Guide `_. + + .. code-block:: bash + + wget -O- http://deb.kamailio.org/kamailiodebkey.gpg | sudo apt-key add - + echo "deb http://deb.kamailio.org/kamailio57 bullseye main" > /etc/apt/sources.list.d/kamailio.list + sudo apt-get update + sudo apt-get install kamailio kamailio-extra-modules kamailio-json-modules + + .. group-tab:: OpenSIPS + + We got OpenSIPS_ installed via following commands: + + .. code-block:: bash + + curl https://apt.opensips.org/opensips-org.gpg -o /usr/share/keyrings/opensips-org.gpg + echo "deb [signed-by=/usr/share/keyrings/opensips-org.gpg] https://apt.opensips.org bookworm 3.4-releases" >/etc/apt/sources.list.d/opensips.list + echo "deb [signed-by=/usr/share/keyrings/opensips-org.gpg] https://apt.opensips.org bookworm cli-nightly" >/etc/apt/sources.list.d/opensips-cli.list + sudo apt-get update + sudo apt-get install opensips opensips-mysql-module opensips-cgrates-module opensips-cli + +Configuration and initialization +-------------------------------- + +This section will be dedicated to configuring both the chosen SIP Server, as well as CGRateS and then get them running. + +Regarding the SIP Servers, we have prepared custom configurations in advance, as well as an init scripts that can be used to start the services using said configurations. It can also be used to stop/restart/check on the status of the services. Another way to do that would be to copy the configuration in the default folder, where the Server will be searching for the configuration before starting, with it usually being /etc/. + +.. tabs:: + + .. group-tab:: FreeSWITCH + + + The FreeSWITCH_ setup consists of: + + - *vanilla* configuration + "mod_json_cdr" for CDR generation; + - configurations for the following users (found in *etc/freeswitch/directory/default*): 1001-prepaid, 1002-postpaid, 1003-pseudoprepaid, 1004-rated, 1006-prepaid, 1007-rated; + - addition of CGRateS' own extensions befoure routing towards users in the dialplan (found in *etc/freeswitch/dialplan/default.xml*). + + + To start FreeSWITCH_ with the prepared custom configuration, run: + + .. code-block:: bash + + sudo /usr/share/cgrates/tutorials/fs_evsock/freeswitch/etc/init.d/freeswitch start + + To verify that FreeSWITCH_ is running, run the following command: + + .. code-block:: bash + + sudo fs_cli -x status + + + .. group-tab:: Asterisk + + + The Asterisk_ setup consists of: + + - *basic-pbx* configuration sample; + - configurations for the following users: 1001-prepaid, 1002-postpaid, 1003-pseudoprepaid, 1004-rated, 1007-rated. + + + To start Asterisk_ with the prepared custom configuration, run: + + .. code-block:: bash + + sudo /usr/share/cgrates/tutorials/asterisk_ari/asterisk/etc/init.d/asterisk start + + + To verify that Asterisk_ is running, run the following commands: + + .. code-block:: bash + + sudo asterisk -r -s /tmp/cgr_asterisk_ari/asterisk/run/asterisk.ctl + ari show status + + .. group-tab:: Kamailio + + The Kamailio_ setup consists of: + + - default configuration with small modifications to add **CGRateS** interaction; + - for script maintainability and simplicity, we have separated **CGRateS** specific routes in *kamailio-cgrates.cfg* file which is included in main *kamailio.cfg* via include directive; + - configurations for the following users: 1001-prepaid, 1002-postpaid, 1003-pseudoprepaid, stored using the CGRateS AttributeS subsystem. + + + To start Kamailio_ with the prepared custom configuration, run: + + .. code-block:: bash + + sudo /usr/share/cgrates/tutorials/kamevapi/kamailio/etc/init.d/kamailio start + + To verify that Kamailio_ is running, run the following command: + + .. code-block:: bash + + sudo kamctl moni + + .. group-tab:: OpenSIPS + + The OpenSIPS_ setup consists of: + - *residential* configuration; + - user accounts configuration not needed since it's enough for them to only be defined within CGRateS; + - for simplicity, no authentication was configured (WARNING: Not suitable for production). + - creating database for the DRouting module, using the following command: + + .. code-block:: bash + + opensips-cli -x database create + + After creating the database for DRouting module populate the tables with routing info: + + .. code-block:: bash + + insert into dr_gateways (gwid,type,address) values("gw2_1",0,"sip:127.0.0.1:5082"); + insert into dr_gateways (gwid,type,address) values("gw1_1",0,"sip:127.0.0.1:5081"); + insert into dr_carriers (carrierid,gwlist) values("route1","gw1_1"); + insert into dr_carriers (carrierid,gwlist) values("route2","gw2_1"); + + + To start OpenSIPS_ with the prepared custom configuration, run: + + .. code-block:: bash + + sudo mv /etc/opensips /etc/opensips.old + sudo cp -r /usr/share/cgrates/tutorials/osips/opensips/etc/opensips /etc + sudo systemctl restart opensips + + + To verify that OpenSIPS_ is running, run the following command: + + .. code-block:: bash + + opensips-cli -x mi uptime + + + Since we are using OpenSIPS_ with DRouting module we have to set up a SIP entity that OpenSIPS_ can forward the calls to for our setup. + In this example we use SIPp a free Open Source test tool / traffic generator for the SIP protocol. + The install SiPp use commands below : + + .. code-block:: bash + + apt update + apt install git pkg-config dh-autoreconf ncurses-dev build-essential libssl-dev libpcap-dev libncurses5-dev libsctp-dev lksctp-tools cmake + git clone https://github.com/SIPp/sipp.git + cd sipp + git checkout v3.7.0 + git submodule init + git submodule update + ./build.sh --common + cmake . -DUSE_SSL=1 -DUSE_SCTP=0 -DUSE_PCAP=1 -DUSE_GSL=1 + make all + make install + + + + Write SIPp XML scenario named uas.xml or to your liking with the content below,this scenario will simulate calls with OpenSIPS_ . + Change "OpenSIPS_IP" in the line ** with your OpenSIPS_ IP. + + .. code-block:: XML + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Content-Length: 0 ]]> + + + Content-Type: application/sdp Content-Length: [len] v=0 o=user1 53655765 2353687637 IN IP[local_ip_type] [local_ip] s=- c=IN IP[media_ip_type] [media_ip] t=0 0 m=audio [media_port] RTP/AVP 0 a=rtpmap:0 PCMU/8000 ]]> + + + + + Content-Length: 0 ]]> + + + + + + + + + + + + Run the SIPp with the command below: + + .. code-block:: bash + + sipp -sf uas.xml -p 5082 + + + + +**CGRateS** will be configured with the following subsystems enabled: + + - **SessionS**: started as gateway between the SIP Server and rest of CGRateS subsystems; + - **ChargerS**: used to decide the number of billing runs for customer/supplier charging; + - **AttributeS**: used to populate extra data to requests (ie: prepaid/postpaid, passwords, paypal account, LCR profile); + - **RALs**: used to calculate costs as well as account bundle management; + - **SupplierS**: selection of suppliers for each session (in case of OpenSIPS_, it will work in tandem with their DRouting module); + - **StatS**: computing statistics in real-time regarding sessions and their charging; + - **ThresholdS**: monitoring and reacting to events coming from above subsystems; + - **EEs**: exporting rated CDRs from CGR StorDB (export path: */tmp*). + +Just as with the SIP Servers, we have also prepared configurations and init scripts for CGRateS. And just as well, you can manage the CGRateS service using systemctl if you prefer. You can even start it using the cgr-engine binary, like so: + + .. code-block:: bash + + cgr-engine -config_path= -logger=*stdout + +.. note:: + The logger flag from the command above is optional, it's usually more convenient for us to check for logs in the terminal that cgrates was started in rather than checking the syslog. + + +.. tabs:: + + .. group-tab:: FreeSWITCH + + .. code-block:: bash + + sudo /usr/share/cgrates/tutorials/fs_evsock/cgrates/etc/init.d/cgrates start + + .. group-tab:: Asterisk + + .. code-block:: bash + + sudo /usr/share/cgrates/tutorials/asterisk_ari/cgrates/etc/init.d/cgrates start + + .. group-tab:: Kamailio + + .. code-block:: bash + + sudo /usr/share/cgrates/tutorials/kamevapi/cgrates/etc/init.d/cgrates start + + .. group-tab:: OpenSIPS + + .. code-block:: bash + + sudo systemctl restart opensips + +.. note:: + If you have chosen OpenSIPS_, CGRateS has to be started first since the dependency is reversed. + + +Loading **CGRateS** Tariff Plans +-------------------------------- + +Now that we have **CGRateS** installed and started with one of the custom configurations, we can load the prepared data out of the shared folder, containing the following rules: + +- Create the necessary timings (always, asap, peak, offpeak). +- Configure 3 destinations (1002, 1003 and 10 used as catch all rule). +- As rating we configure the following: + + - Rate id: *RT_10CNT* with connect fee of 20cents, 10cents per minute for the first 60s in 60s increments followed by 5cents per minute in 1s increments. + - Rate id: *RT_20CNT* with connect fee of 40cents, 20cents per minute for the first 60s in 60s increments, followed by 10 cents per minute charged in 1s increments. + - Rate id: *RT_40CNT* with connect fee of 80cents, 40cents per minute for the first 60s in 60s increments, follwed by 20cents per minute charged in 10s increments. + - Rate id: *RT_1CNT* having no connect fee and a rate of 1 cent per minute, chargeable in 1 minute increments. + - Rate id: *RT_1CNT_PER_SEC* having no connect fee and a rate of 1 cent per second, chargeable in 1 second increments. + +- Accounting part will have following configured: + + - Create 3 accounts: 1001, 1002, 1003. + - 1001, 1002 will receive 10units of **\*monetary** balance. + + +.. code-block:: bash + + cgr-loader -verbose -path=/usr/share/cgrates/tariffplans/tutorial + +To verify that all actions successfully performed, we use following *cgr-console* commands: + +- Make sure all our balances were topped-up: + + .. code-block:: bash + + cgr-console 'accounts Tenant="cgrates.org" AccountIds=["1001"]' + cgr-console 'accounts Tenant="cgrates.org" AccountIds=["1002"]' + +- Query call costs so we can see our calls will have expected costs (final cost will result as sum of *ConnectFee* and *Cost* fields): + + .. code-block:: bash + + cgr-console 'cost Category="call" Tenant="cgrates.org" Subject="1001" Destination="1002" AnswerTime="2014-08-04T13:00:00Z" Usage="20s"' + cgr-console 'cost Category="call" Tenant="cgrates.org" Subject="1001" Destination="1002" AnswerTime="2014-08-04T13:00:00Z" Usage="1m25s"' + cgr-console 'cost Category="call" Tenant="cgrates.org" Subject="1001" Destination="1003" AnswerTime="2014-08-04T13:00:00Z" Usage="20s"' + + +Test calls +---------- + + +1001 -> 1002 +~~~~~~~~~~~~ + +Since the user 1001 is marked as *prepaid* inside the telecom switch, calling between 1001 and 1002 should generate pre-auth and prepaid debits which can be checked with *accounts* command integrated within *cgr-console* tool. Charging will be done based on time of day as described in the tariff plan definition above. + +.. note:: + + An important particularity to note here is the ability of **CGRateS** SessionManager to refund units booked in advance (eg: if debit occurs every 10s and rate increments are set to 1s, the SessionManager will be smart enough to refund pre-booked credits for calls stoped in the middle of debit interval). + +Check that 1001 balance is properly deducted, during the call, and moreover considering that general balance has priority over the shared one debits for this call should take place at first out of general balance. + +.. code-block:: bash + + cgr-console 'accounts Tenant="cgrates.org" AccountIds=["1001"]' + + +1002 -> 1001 +~~~~~~~~~~~~ + +The user 1002 is marked as *postpaid* inside the telecom switch hence his calls will be debited at the end of the call instead of during a call and his balance will be able to go on negative without influencing his new calls (no pre-auth). + +To check that we had debits we use again console command, this time not during the call but at the end of it: + +.. code-block:: bash + + cgr-console 'accounts Tenant="cgrates.org" AccountIds=["1002"]' + + +1001 -> 1003 +~~~~~~~~~~~~ +The user 1001 call user 1003 and after 12 seconds the call will be disconnected. + +CDR Processing +-------------- + + - The SIP Server generates a CDR event at the end of each call (i.e., FreeSWITCH_ via HTTP Post and Kamailio_ via evapi) + - The event is directed towards the port configured inside cgrates.json due to the automatic handler registration built into the SessionS subsystem. + - The event reaches CGRateS through the SessionS subsystem in close to real-time. + - Once inside CGRateS, the event is instantly rated and ready for export. + + +CDR Exporting +------------- + +Once the CDRs are mediated, they are available to be exported. To export them, you first need to configure your EEs in configs (already done by the cgrates script from earlier). Important fields to populate are "id" (sample: tutorial_export), "type" (sample: *file_csv), "export_path" (sample: /tmp), and "fields" where you define all the data that you want to export. After that, you can use available RPC APIs or directly call export_cdrs from the console to export them: + +.. code-block:: bash + + cgr-console 'export_cdrs ExporterIDs=["tutorial_export"]' + +Your exported files will be appear on your defined "export_path" folder after the command is executed. In this case the folder is /tmp +For all available parameters you can check by running ``cgr-console help export_cdrs``. + +Fraud detection +--------------- + +We have configured some action triggers for our tariffplans where more than 20 units of balance topped-up triggers a notification over syslog, and most importantly, an action trigger to monitor for 100 or more units topped-up which will also trigger an account disable together with killing it's calls if prepaid debits are used. + +To verify this mechanism simply add some random units into one account's balance: + +.. code-block:: bash + + cgr-console 'balance_set Tenant="cgrates.org" Account="1003" Value=23 BalanceType="*monetary" Balance={"ID":"MonetaryBalance"}' + tail -f /var/log/syslog -n 20 + + cgr-console 'balance_set Tenant="cgrates.org" Account="1001" Value=101 BalanceType="*monetary" Balance={"ID":"MonetaryBalance"}' + tail -f /var/log/syslog -n 20 + +On the CDRs side we will be able to integrate CdrStats monitors as part of our Fraud Detection system (eg: the increase of average cost for 1001 and 1002 accounts will signal us abnormalities, hence we will be notified via syslog). + + +.. _Zoiper: https://www.zoiper.com/ +.. _Asterisk: http://www.asterisk.org/ +.. _FreeSWITCH: https://freeswitch.com/ +.. _Kamailio: https://www.kamailio.org/w/ +.. _OpenSIPS: https://opensips.org/ +.. _CGRateS: http://www.cgrates.org/ \ No newline at end of file