mirror of
https://github.com/cgrates/cgrates.git
synced 2026-02-11 10:06:24 +05:00
Revise troubleshooting section in docs
This commit is contained in:
ionutboangiu
committed by
Dan Christian Bogos
Dan Christian Bogos
parent
6f65f09e4f
commit
0748f09758
@@ -1,34 +1,308 @@
|
||||
Troubleshooting
|
||||
===============
|
||||
|
||||
In case of **troubleshooting**, CGRateS can monitor memory profiling and CPU profiling.
|
||||
.. contents::
|
||||
:local:
|
||||
:depth: 3
|
||||
|
||||
Profiling
|
||||
---------
|
||||
|
||||
This section covers how to set up and use profiling tools for ``cgrates`` to diagnose performance issues. You can enable profiling through configuration, runtime flags, or APIs.
|
||||
|
||||
For more information on profiling in Go, see the `Go Diagnostics: Profiling <https://go.dev/doc/diagnostics#profiling>`_ documentation.
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
There are two ways to set up profiling:
|
||||
|
||||
Using JSON Configuration
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Enable profiling by adding the ``pprof_path`` under the ``http`` section in your JSON config file:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"listen": {
|
||||
"http": ":2080",
|
||||
"http_tls": ":2280"
|
||||
},
|
||||
"http": {
|
||||
"pprof_path": "/debug/pprof/"
|
||||
}
|
||||
}
|
||||
|
||||
Profiling is enabled by default and exposes the ``/debug/pprof/`` endpoint. You can access it through the address set in the ``listen`` section (``http`` or ``http_tls``). To turn off profiling, set ``pprof_path`` to an empty string ``""``.
|
||||
|
||||
Using Runtime Flags
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
You can also control profiling with runtime flags when starting ``cgr-engine``:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cgr-engine -help
|
||||
Usage of cgr-engine:
|
||||
-cpuprof_dir string
|
||||
Directory for CPU profiles
|
||||
-memprof_dir string
|
||||
Directory for memory profiles
|
||||
-memprof_interval duration
|
||||
Interval between memory profile saves (default 15s)
|
||||
-memprof_maxfiles int
|
||||
Number of memory profiles to keep (most recent) (default 1)
|
||||
-memprof_timestamp
|
||||
Add timestamp to memory profile files
|
||||
|
||||
Generating Profile Data
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Let's assume the profiling interface is available at ``http://localhost:2080/debug/pprof/``.
|
||||
|
||||
.. note::
|
||||
Profiling started with flags or APIs can be stopped using the corresponding API calls. If you start profiling on startup using flags and don't stop it manually, a profile will be automatically generated when the engine shuts down. The same applies if you start profiling via API and forget to stop it before shutting down the engine.
|
||||
|
||||
CPU Profiling
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
Here's how to generate CPU profile data:
|
||||
|
||||
- **Web Browser**: Go to ``http://localhost:2080/debug/pprof/`` in your browser. Click "profile" to start a 30-second CPU profile.
|
||||
- **Custom Duration**: Add the ``seconds`` parameter to set a different duration: ``http://localhost:2080/debug/pprof/profile?seconds=5``.
|
||||
- **Command Line**: Use ``curl`` to download the profile:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
curl -o cpu.prof http://localhost:2080/debug/pprof/profile?seconds=5
|
||||
|
||||
- **APIs**: Use ``CoreSv1.StartCPUProfiling`` and ``CoreSv1.StopCPUProfiling`` APIs:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"method": "CoreSv1.StartCPUProfiling",
|
||||
"params": [{
|
||||
"DirPath": "/tmp"
|
||||
}],
|
||||
"id": 1
|
||||
}
|
||||
|
||||
{
|
||||
"method": "CoreSv1.StopCPUProfiling",
|
||||
"params": [],
|
||||
"id": 1
|
||||
}
|
||||
|
||||
- **Startup Profiling**: Profile the entire runtime by specifying a directory with the ``-cpuprof_dir`` flag:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
cgr-engine -cpuprof_dir=/tmp [other flags]
|
||||
|
||||
Memory Profiling
|
||||
----------------
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
Creating the memory profile files
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Generate memory profile data like this:
|
||||
|
||||
Firstly, go to the main config directory and run the engine with the flag named ```-memprof_dir```. For this example, I choosed ```tmp``` directory from my machine.
|
||||
Also, there are other flags that we can use:
|
||||
- **Web Browser**: Visit ``http://localhost:2080/debug/pprof/`` to create a memory snapshot. Use ``?debug=2`` (or ``?debug=1``) for human-readable output. If the ``debug`` parameter is omitted or set to ``0``, the output will be in binary format.
|
||||
- **Command Line**: Use ``curl`` to download the memory profile:
|
||||
|
||||
```-memprof_interval``` - Time between memory profile saves. By default, the time between writing into files is 5 seconds.
|
||||
```-memprof_nrfiles``` - Number of memory profile to write. By default, the numbers of files is 1.
|
||||
.. code-block:: console
|
||||
|
||||
::
|
||||
curl -o mem.prof http://localhost:2080/debug/pprof/heap
|
||||
|
||||
cgr-engine -config_path=. -logger=*stdout -memprof_dir=/tmp/
|
||||
- **Automated Profiling**: Use the ``CoreSv1.StartMemoryProfiling`` API for periodic memory snapshots:
|
||||
|
||||
In the running process, this will create a file named ```mem1_prof.prof```. Let the engine run for some time, and then you can kill the process. When the process is killed, it will create another file named ```mem_final.prof``` containing the final memory profiling information.
|
||||
.. code-block:: json
|
||||
|
||||
Reading the memory profile files
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
{
|
||||
"method": "CoreSv1.StartMemoryProfiling",
|
||||
"params": [{
|
||||
"DirPath": "/tmp",
|
||||
"Interval": 5000000000,
|
||||
"MaxFiles": 5,
|
||||
"UseTimestamp": true
|
||||
}],
|
||||
"id": 1
|
||||
}
|
||||
|
||||
Next step is to read properly the memory profiling. We will use the tool from go package, pprof. Go to the directory where the files were written and read the files.
|
||||
.. note::
|
||||
|
||||
::
|
||||
``Interval`` is in nanoseconds. Future updates will allow using time strings (e.g., ``5s``, ``1h``) or seconds as an integer.
|
||||
|
||||
cd /tmp
|
||||
go tool pprof mem1.prof
|
||||
Other Useful Profiles
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
It will open a console, and to check the memory usage, type ```top``` to see. For more documentation about this tool, we recommand you to read the pprof package documentation from golang, or simply type ```help```.
|
||||
The ``/debug/pprof/`` endpoint offers more useful profiles:
|
||||
|
||||
- **Goroutine Profile** (``/debug/pprof/goroutine``): View or download goroutine stack dumps.
|
||||
- **Mutex Profile** (``/debug/pprof/mutex``): Find bottlenecks where goroutines wait for locks.
|
||||
- **Block Profile** (``/debug/pprof/block``): Identify where goroutines block waiting on synchronization primitives.
|
||||
- **Thread Create Profile** (``/debug/pprof/threadcreate``): Show stack traces that led to the creation of new OS threads.
|
||||
- **Execution Trace**: For information on generating and analyzing execution traces, see the `Tracing`_ section below.
|
||||
|
||||
Analyzing Profiles
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The main tool for analyzing profiles is ``go tool pprof``. It helps visualize and analyze profiling data. You can use it with both downloaded profile files and directly with URLs.
|
||||
|
||||
Command-Line Analysis
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
For CPU profiles:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
go tool pprof cpu.prof
|
||||
# or
|
||||
go tool pprof http://localhost:2080/debug/pprof/profile
|
||||
|
||||
For memory profiles:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
go tool pprof mem.prof
|
||||
# or
|
||||
go tool pprof http://localhost:2080/debug/pprof/heap
|
||||
|
||||
This opens an interactive terminal. Use commands like ``top``, ``list``, ``web``, and ``svg`` to explore the profile.
|
||||
|
||||
.. hint::
|
||||
Run ``go tool pprof -h`` for more information on available commands and options.
|
||||
|
||||
Visual Analysis
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
Create visual representations of your profiling data:
|
||||
|
||||
- **SVG**: Generate an SVG graph:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
go tool pprof -svg cpu.prof > cpu.svg
|
||||
# or
|
||||
go tool pprof -svg mem.prof > mem.svg
|
||||
|
||||
- **Web Interface**: Use ``-http`` for an interactive visualization in your browser:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
go tool pprof -http=:8080 cpu.prof
|
||||
# or
|
||||
go tool pprof -http=:8080 mem.prof
|
||||
|
||||
.. note::
|
||||
|
||||
You might need to install the ``graphviz`` package.
|
||||
|
||||
Tracing
|
||||
-------
|
||||
|
||||
Execution tracing provides a detailed view of runtime behavior of your Go program.
|
||||
|
||||
For detailed information on tracing in Go, see the `Go Diagnostics: Execution Tracing <https://go.dev/doc/diagnostics#tracing>`_ documentation.
|
||||
|
||||
To generate and analyze trace data:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# Generate trace data
|
||||
curl -o trace.out http://localhost:2080/debug/pprof/trace?seconds=5
|
||||
|
||||
# Analyze trace data
|
||||
go tool trace trace.out
|
||||
|
||||
This opens a browser interface for detailed execution analysis.
|
||||
|
||||
Debugging
|
||||
---------
|
||||
|
||||
This section covers how to set up and use Delve, a Go debugger, with ``cgrates``.
|
||||
|
||||
For detailed information on debugging Go programs, see the `Go Diagnostics: Debugging <https://go.dev/doc/diagnostics#debugging>`_ documentation.
|
||||
|
||||
Installation
|
||||
~~~~~~~~~~~~
|
||||
|
||||
To install Delve, run:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
go install github.com/go-delve/delve/cmd/dlv@latest
|
||||
|
||||
Basic Usage
|
||||
~~~~~~~~~~~
|
||||
|
||||
There are several ways to use Delve with ``cgrates``:
|
||||
|
||||
1. Start ``cgr-engine`` in debug mode:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
dlv exec /path/to/cgr-engine -- --config_path=/etc/cgrates --logger=*stdout
|
||||
|
||||
2. Attach to a running instance:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
ENGINE_PID=$(pidof cgr-engine)
|
||||
dlv attach $ENGINE_PID
|
||||
|
||||
3. Debug tests:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
dlv test github.com/cgrates/cgrates/apier/v1 -- -test.run=TestName
|
||||
|
||||
|
||||
.. hint::
|
||||
For better debugging, disable optimizations (``-N``) and inlining (``-l``) when building ``cgr-engine``:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
go install -gcflags="all=-N -l" -ldflags "-X 'github.com/cgrates/cgrates/utils.GitLastLog=$GIT_LAST_LOG'" github.com/cgrates/cgrates/cmd/cgr-engine
|
||||
|
||||
Handling Crashes
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
To capture more information when ``cgrates`` crashes:
|
||||
|
||||
1. Enable core dump generation:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
ulimit -c unlimited
|
||||
GOTRACEBACK=crash cgr-engine -config_path=/etc/cgrates
|
||||
|
||||
2. Analyze core dumps with Delve:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
dlv core /path/to/cgr-engine core
|
||||
|
||||
Common Debugging Commands
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Once in a Delve debug session, you can use these common commands:
|
||||
|
||||
- ``break`` or ``b``: Set a breakpoint
|
||||
- ``continue`` or ``c``: Run until breakpoint or program termination
|
||||
- ``next`` or ``n``: Step over to next line
|
||||
- ``step`` or ``s``: Step into function call
|
||||
- ``print`` or ``p``: Evaluate an expression
|
||||
- ``goroutines``: List current goroutines
|
||||
- ``help``: Show help for commands
|
||||
|
||||
For more information on using Delve, refer to the `Delve Documentation <https://github.com/go-delve/delve/tree/master/Documentation>`_.
|
||||
|
||||
Further Reading
|
||||
---------------
|
||||
|
||||
For more comprehensive information on Go diagnostics, profiling, and debugging, check out these resources:
|
||||
|
||||
- `Go Diagnostics <https://go.dev/doc/diagnostics>`_: Official documentation on diagnostics in Go.
|
||||
- `Profiling Go Programs <https://go.dev/blog/pprof>`_: In-depth blog post on profiling in Go.
|
||||
- `net/http/pprof godoc <https://pkg.go.dev/net/http/pprof>`_: Documentation for the pprof package.
|
||||
- `Delve Debugger <https://github.com/go-delve/delve>`_: GitHub repository for the Delve debugger.
|
||||
|
||||
Reference in New Issue
Block a user