Files
a955b351ff [NEW] Introduce lttng based tracing (#2070)
## Introduce

In a production environment, it's quite challenging to figure out why a
Valkey is under high load. Right now, tools like INFO or slowlog can
offer some clues. But if the Valkey can't respond, we might not get any
information at all.
Usually, we have to rely on tools like `strace` or `perf` to find the
root cause. If we set up trace points in advance during the project
development, we can quickly pinpoint performance issues.

In this current PR, support has been added for all latency sampling
points. Also, information reporting for command execution has been
added. At the same time, it supports dynamically turning on or off the
information reporting as required. The trace feature is implemented
based on LTTng, and this capability is supported in projects like QEMU,
Ceph.

## How to use

Building Valkey with LTTng support:

```
USE_LTTNG=yes make
```

Open event report:
```
config set trace-events "sys server db cluster aof commands"
```

Events are classified as follows:
- sys (System-level operations)
- server (Server core logic)
- db (Database core operations)
- cluster (Cluster configuration operations)
- aof (AOF persistence operations)
- commands(Command execution information)

## How to trace

Enable lttng trace events dynamically:
```
~# lttng destroy valkey
~# lttng create valkey
~# lttng enable-event -u valkey:*
~# lttng track -u -p `pidof valkey-server`
~# lttng start
~# lttng stop
~# lttng view
```

Examples (a client run 'SET', another run 'keys'):

```
[15:30:19.334463706] (+0.000001243) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 0 }
[15:30:19.334465183] (+0.000001477) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 1 }
[15:30:19.334466516] (+0.000001333) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 0 }
[15:30:19.334467738] (+0.000001222) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 0 }
[15:30:19.334469105] (+0.000001367) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 1 }
[15:30:19.334470327] (+0.000001222) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 0 }
[15:30:19.369348485] (+0.034878158) libai valkey:command_call: { cpu_id = 15 }, { name = "keys", duration = 34874 }
[15:30:19.369698322] (+0.000349837) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 4 }
[15:30:19.369702327] (+0.000004005) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 2 }

```

Then we can use another script to analyze topN slow commands and other
system
level events.

About performance overhead (valkey-benchmark -t get -n 1000000 --threads
4):
1> no lttng builtin: 285632.69 requests per second
2> lttng builtin, no trace: 285551.09 requests per second (almost 0
overhead)
3> lttng builtin, trace commands: 266595.59 requests per second (about
~6.6 overhead)

Generally valkey-server would not run in full utilization, the overhead
is acceptable.

## Problem analysis

Add prot and conn field into trace command

Run benchmark tool:
```
GET: rps=227428.0 (overall: 222756.2) avg_msec=0.114 (overall: 0.117)
GET: rps=225248.0 (overall: 223005.2) avg_msec=0.115 (overall: 0.117)
GET: rps=167474.1 (overall: 217942.2) avg_msec=0.193 (overall: 0.122) --> performance drop
GET: rps=220192.0 (overall: 218129.5) avg_msec=0.118 (overall: 0.122)
GET: rps=222868.0 (overall: 218493.7) avg_msec=0.117 (overall: 0.121)

```
Run another 'keys *' command in another connection, lead benchmark
performance
drop.

At the same time, lttng traces events:
```
[21:16:30.420997167] (+0.000004064) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54668", name = "get", duration = 1 }
[21:16:30.421001262] (+0.000004095) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54782", name = "get", duration = 1 }
[21:16:30.485562459] (+0.064561197) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54386", name = "keys", duration = 64551 } --> root cause
[21:16:30.485583101] (+0.000020642) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54522", name = "get", duration = 1 }
[21:16:30.485763891] (+0.000180790) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54542", name = "get", duration = 1 }
[21:16:30.485766451] (+0.000002560) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54438", name = "get", duration = 1 }
```

From this change, we can see that connection
127.0.0.1:6379-127.0.0.1:54386
affects other connections.

---------

Signed-off-by: zhenwei pi <[email protected]>
Signed-off-by: artikell <[email protected]>
Signed-off-by: skyfirelee <[email protected]>
Co-authored-by: zhenwei pi <[email protected]>
2025-06-08 14:39:57 -07:00

117 lines
5.1 KiB
C

/* latency.h -- latency monitor API header file
* See latency.c for more information.
*
* ----------------------------------------------------------------------------
*
* Copyright (c) 2014, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __LATENCY_H
#define __LATENCY_H
#include "trace/trace.h"
#define LATENCY_TS_LEN 160 /* History length for every monitored event. */
#ifndef LATENCY_TRACE_SWITCH
#define LATENCY_TRACE_SWITCH 0
#endif
/* Representation of a latency sample: the sampling time and the latency
* observed in milliseconds. */
struct latencySample {
int32_t time; /* We don't use time_t to force 4 bytes usage everywhere. */
uint32_t latency; /* Latency in milliseconds. */
};
/* The latency time series for a given event. */
struct latencyTimeSeries {
int idx; /* Index of the next sample to store. */
uint32_t max; /* Max latency observed for this event. */
uint32_t sum; /* Sum latency observed for this event. */
uint32_t cnt; /* Count observed for this event. */
struct latencySample samples[LATENCY_TS_LEN]; /* Latest history. */
};
/* Latency statistics structure. */
struct latencyStats {
uint32_t all_time_high; /* Absolute max observed since latest reset. */
uint32_t avg; /* Average of current samples. */
uint32_t min; /* Min of current samples. */
uint32_t max; /* Max of current samples. */
uint32_t mad; /* Mean absolute deviation. */
uint32_t samples; /* Number of non-zero samples. */
time_t period; /* Number of seconds since first event and now. */
};
void latencyMonitorInit(void);
void latencyAddSample(const char *event, ustime_t latency);
/* Latency monitoring macros. */
/* Start monitoring an event. We just set the current time. */
#define latencyStartMonitor(var) \
if (server.latency_monitor_threshold || LATENCY_TRACE_SWITCH) { \
var = ustime(); \
} else { \
var = 0; \
}
/* End monitoring an event, compute the difference with the current time
* to check the amount of time elapsed. */
#define latencyEndMonitor(var) \
if (server.latency_monitor_threshold || LATENCY_TRACE_SWITCH) { \
var = ustime() - var; \
}
/* Add the sample only if the elapsed time is >= to the configured threshold. */
#define latencyAddSampleIfNeeded(event, var) \
if (server.latency_monitor_threshold && (var) >= server.latency_monitor_threshold * 1000) latencyAddSample((event), (var));
/* Remove time from a nested event. */
#define latencyRemoveNestedEvent(event_var, nested_var) event_var += nested_var;
typedef struct durationStats {
unsigned long long cnt;
unsigned long long sum;
unsigned long long max;
} durationStats;
typedef enum {
EL_DURATION_TYPE_EL = 0, // cumulative time duration metric of the whole eventloop
EL_DURATION_TYPE_CMD, // cumulative time duration metric of executing commands
EL_DURATION_TYPE_AOF, // cumulative time duration metric of flushing AOF in eventloop
EL_DURATION_TYPE_CRON, // cumulative time duration metric of cron (serverCron and beforeSleep, but excluding IO and
// AOF)
EL_DURATION_TYPE_NUM
} DurationType;
void durationAddSample(int type, monotime duration);
#endif /* __LATENCY_H */