Files
redis/deps/tre/tests/test-str-source.c
1ac3bc07d6 Implement the new Redis Array type (#15162)
# Redis Array

For years, Redis has been missing a real indexed data structure for the
use cases where the index and the spatial relationship of elements are
semantic. Hashes give you random lookups, but you have to store an index
as a key, and have no range visibility. Lists give you appending and
trimming, but what is in the middle remains hard to access. Streams give
you append-only events, which is another (useful, indeed) beast. None of
these is what you want when the *position itself* has business meaning —
slot 37, step 4, row 18552, day from 2934 to 2949, file line 11, 12, 15
and so forth. And, all those types, for different reasons, are all
suboptimal when you want a **ring buffer** able to store the latest N
observed samples of something.

Up to now, users found ways (they always do \o/) using the fact that the
data structures that are obvious in this universe are also extremely
powerful, if well implemented. But this forces compromises. Arrays
handle these index-first requirements natively, and usually with much
better memory and CPU usage than the workarounds. If the use case is the
right one, Arrays often provide much better space, time and usability at
the same time.

## Internal encoding

1. When dense, an Array is essentially a more fancy C array. You don't
pay anything for storing the index.
2. Yet, instead of going really flat, arrays are sliced into
4096-element slices, and each slice, when it contains just a few
elements, uses a special sparse encoding. When a slice is empty it's
just a `NULL` stored in the directory.
3. Small ints, floats, and short strings are pointer-tagged, so they
cost zero additional memory beyond the pointer slot itself.
4. When very sparse, a super-directory of windowed directories is used.
This allows the data type to be safe, instead of exhibiting pathological
space or time behavior. This representation is only triggered when there
are more than 8 million elements or very high indexes set.

## Use cases

Arrays are mostly stateless if not for the fact that each array
remembers the index of the latest added item, allowing `ARINSERT` and
`ARRING` to work properly. Otherwise it is a set/get at this index game,
with solid support for both setting / getting ranges, server-side
scanning, returning only populated elements in a time which is
proportional not to the range size, but to the population size.

A few concrete examples, that may work as mental models for the set of
problems that are similar to them (from the POV of the data modeling).

**Thermometer.** A sensor reporting once per minute, with gaps:

```
ARSET       temp:room12:day7 123 22.3
ARGETRANGE  temp:room12:day7 600 660     # the 10:00–11:00 window, with NULLs
ARSCAN      temp:room12:day7 600 660     # only populated elements
AROP        temp:room12:day7 0 1439 MAX  # peak of the day, server-side
```

Missing minutes cost little to nothing. Numeric aggregation runs inside
Redis. Telemetry, IoT, meter readings, KPI rollups.

**Calendar.** A clinic with 96 fifteen-minute slots per day:

```
ARSET       sched:room12:day 32 booking:991
ARSCAN      sched:room12:day 0 95      # only occupied slots
ARGETRANGE  sched:room12:day 48 63     # the afternoon full view to render
```

The slot number is the business key in this case. Room booking, parking
spaces, warehouse bins, lockers, ...

**Ring buffer.** ARRING replaces the classic LPUSH+LTRIM pattern.
Imagine remote `dmesg`.

```
ARRING       machine:123 200 "[141087.430123]: arm_cpu_init(): cpu 14 online" # Capped to 200 entries
ARLASTITEMS  machine:123 50 REV       # 50 newest first
```

Faster than LPUSH+LTRIM, keep indexed access to past elements. Last-N
alarms, recent fraud scores, access history, remote logs, device events.
Ok here the use cases are mainly the ones of the old pattern: it is just
a better fit and allows to access random items in the middle, aggregate
server-side, and so forth.

**Workflow.** Step number is the index, value is the status. Gaps are
meaningful:

```
ARSET       claim:99172 0 received
ARSET       claim:99172 3 waiting:reviewer42
ARSET       claim:99172 5 approved
ARGETRANGE  claim:99172 0 5     # full workflow view, with NULLs for missing steps
ARSCAN      claim:99172 0 5     # only steps that have a state
ARCOUNT     claim:99172         # number of recorded steps
ARLEN       claim:99172         # highest reached step + 1
```

**Skills knowledge base for agents.** Arrays are good at representing /
grepping into Markdown files:

```
ARSET   skill:metal_gpu 0 "...."
ARSET   skill:metal_gpu 1 "...."
ARSET   skill:metal_gpu 2 "...."
ARGREP  skill:metal_gpu - + RE "M3|M4" WITHVALUES
```

ARGREP has EXACT, MATCH, GLOB, RE, you can have multiple predicates, can
select AND or OR behavior.

**Bulk import results.** Sparse row annotations over millions of rows /
CSV / ...:

```
ARSET  import:job551 18552 ERR:bad_email
ARSCAN import:job551 0 1000000        # Provides only rows that have something
```

## TLDR

If the position is part of the meaning, use an Array. If you want to
aggregate or grep remotely, use an Array. Feedback welcome :)

---------

Co-authored-by: debing.sun <[email protected]>
Co-authored-by: Shubham S Taple <[email protected]>
Co-authored-by: Yuan Wang <[email protected]>
Co-authored-by: Marc Gravell <[email protected]>
2026-05-14 00:56:44 +08:00

193 lines
4.4 KiB
C

/*
test-str-source.c - Sample program for using tre_reguexec()
This software is released under a BSD-style license.
See the file LICENSE for details and copyright.
*/
#ifdef HAVE_CONFIG_H
#include <config.h>
#endif /* HAVE_CONFIG_H */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* look for getopt in order to use a -o option for output. */
#if defined(HAVE_UNISTD_H)
#include <unistd.h>
#elif defined(HAVE_GETOPT_H)
#include <getopt.h>
#endif
#include "tre-internal.h"
static FILE *outf = NULL;
/* Context structure for the tre_str_source wrappers. */
typedef struct {
/* Our string. */
const char *str;
/* Current position in the string. */
size_t pos;
} str_handler_ctx;
/* The get_next_char() handler. Sets `c' to the value of the next character,
and increases `pos_add' by the number of bytes read. Returns 1 if the
string has ended, 0 if there are more characters. */
static int
str_handler_get_next(tre_char_t *c, unsigned int *pos_add, void *context)
{
str_handler_ctx *ctx = context;
unsigned char ch = ctx->str[ctx->pos];
#ifdef TRE_DEBUG
fprintf(outf, "str[%lu] = %d\n", (unsigned long)ctx->pos, ch);
#endif /* TRE_DEBUG */
*c = ch;
if (ch)
ctx->pos++;
*pos_add = 1;
return ch == '\0';
}
/* The rewind() handler. Resets the current position in the input string. */
static void
str_handler_rewind(size_t pos, void *context)
{
str_handler_ctx *ctx = context;
#ifdef TRE_DEBUG
fprintf(outf, "rewind to %lu\n", (unsigned long)pos);
#endif /* TRE_DEBUG */
ctx->pos = pos;
}
/* The compare() handler. Compares two substrings in the input and returns
0 if the substrings are equal, and a nonzero value if not. */
static int
str_handler_compare(size_t pos1, size_t pos2, size_t len, void *context)
{
str_handler_ctx *ctx = context;
#ifdef TRE_DEBUG
fprintf(outf, "comparing %lu-%lu and %lu-%lu\n",
(unsigned long)pos1, (unsigned long)pos1 + len,
(unsigned long)pos2, (unsigned long)pos2 + len);
#endif /* TRE_DEBUG */
return strncmp(ctx->str + pos1, ctx->str + pos2, len);
}
/* Creates a tre_str_source wrapper around the string `str'. Returns the
tre_str_source object or NULL if out of memory. */
static tre_str_source *
make_str_source(const char *str)
{
tre_str_source *s;
str_handler_ctx *ctx;
s = calloc(1, sizeof(*s));
if (!s)
return NULL;
ctx = malloc(sizeof(str_handler_ctx));
if (!ctx)
{
free(s);
return NULL;
}
ctx->str = str;
ctx->pos = 0;
s->context = ctx;
s->get_next_char = str_handler_get_next;
s->rewind = str_handler_rewind;
s->compare = str_handler_compare;
return s;
}
/* Frees the memory allocated for `s'. */
static void
free_str_source(tre_str_source *s)
{
free(s->context);
free(s);
}
/* Run one test with tre_reguexec. Returns 1 if the regex matches, 0 if
it doesn't, and -1 if an error occurs. */
static int
test_reguexec(const char *str, const char *regex)
{
regex_t preg;
tre_str_source *source;
regmatch_t pmatch[5];
int ret;
if ((source = make_str_source(str)) == NULL)
{
fprintf(stderr, "Out of memory\n");
ret = -1;
}
else
{
if (tre_regcomp(&preg, regex, REG_EXTENDED) != REG_OK)
{
fprintf(stderr, "Failed to compile /%s/\n", regex);
ret = -1;
}
else
{
if (tre_reguexec(&preg, source, elementsof(pmatch), pmatch, 0) == 0)
{
fprintf(outf, "Match: /%s/ matches \"%.*s\" in \"%s\"\n", regex,
(int)(pmatch[0].rm_eo - pmatch[0].rm_so),
str + pmatch[0].rm_so, str);
ret = 1;
}
else
{
fprintf(outf, "No match: /%s/ in \"%s\"\n", regex, str);
ret = 0;
}
tre_regfree(&preg);
}
free_str_source(source);
}
return ret;
}
int
main(int argc, char **argv)
{
int ret = 0;
outf = stdout;
#if defined(HAVE_UNISTD_H) || defined(HAVE_GETOPT_H)
int opt;
while ((opt = getopt(argc, argv, "o:")) != EOF)
{
switch (opt)
{
case 'o':
if ((outf = fopen(optarg, "w")) == NULL)
{
perror(optarg);
exit(1);
}
break;
default:
/* getopt() will have printed an error message already */
exit(1);
}
}
#endif
ret += test_reguexec("xfoofofoofoo", "(foo)\\1") != 1;
ret += test_reguexec("catcat", "(cat|dog)\\1") != 1;
ret += test_reguexec("catdog", "(cat|dog)\\1") != 0;
ret += test_reguexec("dogdog", "(cat|dog)\\1") != 1;
ret += test_reguexec("dogcat", "(cat|dog)\\1") != 0;
return ret;
}