mirror of
https://github.com/hanzoai/redis.git
synced 2026-11-02 10:30:08 +00:00
# Redis Array For years, Redis has been missing a real indexed data structure for the use cases where the index and the spatial relationship of elements are semantic. Hashes give you random lookups, but you have to store an index as a key, and have no range visibility. Lists give you appending and trimming, but what is in the middle remains hard to access. Streams give you append-only events, which is another (useful, indeed) beast. None of these is what you want when the *position itself* has business meaning — slot 37, step 4, row 18552, day from 2934 to 2949, file line 11, 12, 15 and so forth. And, all those types, for different reasons, are all suboptimal when you want a **ring buffer** able to store the latest N observed samples of something. Up to now, users found ways (they always do \o/) using the fact that the data structures that are obvious in this universe are also extremely powerful, if well implemented. But this forces compromises. Arrays handle these index-first requirements natively, and usually with much better memory and CPU usage than the workarounds. If the use case is the right one, Arrays often provide much better space, time and usability at the same time. ## Internal encoding 1. When dense, an Array is essentially a more fancy C array. You don't pay anything for storing the index. 2. Yet, instead of going really flat, arrays are sliced into 4096-element slices, and each slice, when it contains just a few elements, uses a special sparse encoding. When a slice is empty it's just a `NULL` stored in the directory. 3. Small ints, floats, and short strings are pointer-tagged, so they cost zero additional memory beyond the pointer slot itself. 4. When very sparse, a super-directory of windowed directories is used. This allows the data type to be safe, instead of exhibiting pathological space or time behavior. This representation is only triggered when there are more than 8 million elements or very high indexes set. ## Use cases Arrays are mostly stateless if not for the fact that each array remembers the index of the latest added item, allowing `ARINSERT` and `ARRING` to work properly. Otherwise it is a set/get at this index game, with solid support for both setting / getting ranges, server-side scanning, returning only populated elements in a time which is proportional not to the range size, but to the population size. A few concrete examples, that may work as mental models for the set of problems that are similar to them (from the POV of the data modeling). **Thermometer.** A sensor reporting once per minute, with gaps: ``` ARSET temp:room12:day7 123 22.3 ARGETRANGE temp:room12:day7 600 660 # the 10:00–11:00 window, with NULLs ARSCAN temp:room12:day7 600 660 # only populated elements AROP temp:room12:day7 0 1439 MAX # peak of the day, server-side ``` Missing minutes cost little to nothing. Numeric aggregation runs inside Redis. Telemetry, IoT, meter readings, KPI rollups. **Calendar.** A clinic with 96 fifteen-minute slots per day: ``` ARSET sched:room12:day 32 booking:991 ARSCAN sched:room12:day 0 95 # only occupied slots ARGETRANGE sched:room12:day 48 63 # the afternoon full view to render ``` The slot number is the business key in this case. Room booking, parking spaces, warehouse bins, lockers, ... **Ring buffer.** ARRING replaces the classic LPUSH+LTRIM pattern. Imagine remote `dmesg`. ``` ARRING machine:123 200 "[141087.430123]: arm_cpu_init(): cpu 14 online" # Capped to 200 entries ARLASTITEMS machine:123 50 REV # 50 newest first ``` Faster than LPUSH+LTRIM, keep indexed access to past elements. Last-N alarms, recent fraud scores, access history, remote logs, device events. Ok here the use cases are mainly the ones of the old pattern: it is just a better fit and allows to access random items in the middle, aggregate server-side, and so forth. **Workflow.** Step number is the index, value is the status. Gaps are meaningful: ``` ARSET claim:99172 0 received ARSET claim:99172 3 waiting:reviewer42 ARSET claim:99172 5 approved ARGETRANGE claim:99172 0 5 # full workflow view, with NULLs for missing steps ARSCAN claim:99172 0 5 # only steps that have a state ARCOUNT claim:99172 # number of recorded steps ARLEN claim:99172 # highest reached step + 1 ``` **Skills knowledge base for agents.** Arrays are good at representing / grepping into Markdown files: ``` ARSET skill:metal_gpu 0 "...." ARSET skill:metal_gpu 1 "...." ARSET skill:metal_gpu 2 "...." ARGREP skill:metal_gpu - + RE "M3|M4" WITHVALUES ``` ARGREP has EXACT, MATCH, GLOB, RE, you can have multiple predicates, can select AND or OR behavior. **Bulk import results.** Sparse row annotations over millions of rows / CSV / ...: ``` ARSET import:job551 18552 ERR:bad_email ARSCAN import:job551 0 1000000 # Provides only rows that have something ``` ## TLDR If the position is part of the meaning, use an Array. If you want to aggregate or grep remotely, use an Array. Feedback welcome :) --------- Co-authored-by: debing.sun <[email protected]> Co-authored-by: Shubham S Taple <[email protected]> Co-authored-by: Yuan Wang <[email protected]> Co-authored-by: Marc Gravell <[email protected]>
129 lines
3.4 KiB
C
129 lines
3.4 KiB
C
/*
|
|
tre-ast.h - Abstract syntax tree (AST) definitions
|
|
|
|
This software is released under a BSD-style license.
|
|
See the file LICENSE for details and copyright.
|
|
|
|
*/
|
|
|
|
|
|
#ifndef TRE_AST_H
|
|
#define TRE_AST_H 1
|
|
|
|
#include "tre-mem.h"
|
|
#include "tre-internal.h"
|
|
#include "tre-compile.h"
|
|
|
|
/* The different AST node types. */
|
|
typedef enum {
|
|
LITERAL,
|
|
CATENATION,
|
|
ITERATION,
|
|
UNION
|
|
} tre_ast_type_t;
|
|
|
|
/* Special subtypes of TRE_LITERAL. */
|
|
#define EMPTY -1 /* Empty leaf (denotes empty string). */
|
|
#define ASSERTION -2 /* Assertion leaf. */
|
|
#define TAG -3 /* Tag leaf. */
|
|
#define BACKREF -4 /* Back reference leaf. */
|
|
#define PARAMETER -5 /* Parameter. */
|
|
|
|
#define IS_SPECIAL(x) ((x)->code_min < 0)
|
|
#define IS_EMPTY(x) ((x)->code_min == EMPTY)
|
|
#define IS_ASSERTION(x) ((x)->code_min == ASSERTION)
|
|
#define IS_TAG(x) ((x)->code_min == TAG)
|
|
#define IS_BACKREF(x) ((x)->code_min == BACKREF)
|
|
#define IS_PARAMETER(x) ((x)->code_min == PARAMETER)
|
|
|
|
|
|
/* A generic AST node. All AST nodes consist of this node on the top
|
|
level with `obj' pointing to the actual content. */
|
|
typedef struct {
|
|
tre_ast_type_t type; /* Type of the node. */
|
|
void *obj; /* Pointer to actual node. */
|
|
int nullable;
|
|
int submatch_id;
|
|
unsigned int num_submatches;
|
|
unsigned int num_tags;
|
|
tre_pos_and_tags_t *firstpos;
|
|
tre_pos_and_tags_t *lastpos;
|
|
} tre_ast_node_t;
|
|
|
|
|
|
/* A "literal" node. These are created for assertions, back references,
|
|
tags, matching parameter settings, and all expressions that match one
|
|
character. */
|
|
typedef struct {
|
|
long code_min;
|
|
long code_max;
|
|
int position;
|
|
union {
|
|
tre_ctype_t class;
|
|
int *params;
|
|
} u;
|
|
tre_ctype_t *neg_classes;
|
|
} tre_literal_t;
|
|
|
|
/* A "catenation" node. These are created when two regexps are concatenated.
|
|
If there are more than one subexpressions in sequence, the `left' part
|
|
holds all but the last, and `right' part holds the last subexpression
|
|
(catenation is left associative). */
|
|
typedef struct {
|
|
tre_ast_node_t *left;
|
|
tre_ast_node_t *right;
|
|
} tre_catenation_t;
|
|
|
|
/* An "iteration" node. These are created for the "*", "+", "?", and "{m,n}"
|
|
operators. */
|
|
typedef struct {
|
|
/* Subexpression to match. */
|
|
tre_ast_node_t *arg;
|
|
/* Minimum number of consecutive matches. */
|
|
int min;
|
|
/* Maximum number of consecutive matches. */
|
|
int max;
|
|
/* If 0, match as many characters as possible, if 1 match as few as
|
|
possible. Note that this does not always mean the same thing as
|
|
matching as many/few repetitions as possible. */
|
|
unsigned int minimal:1;
|
|
/* Approximate matching parameters (or NULL). */
|
|
int *params;
|
|
} tre_iteration_t;
|
|
|
|
/* An "union" node. These are created for the "|" operator. */
|
|
typedef struct {
|
|
tre_ast_node_t *left;
|
|
tre_ast_node_t *right;
|
|
} tre_union_t;
|
|
|
|
tre_ast_node_t *
|
|
tre_ast_new_node(tre_mem_t mem, tre_ast_type_t type, size_t size);
|
|
|
|
tre_ast_node_t *
|
|
tre_ast_new_literal(tre_mem_t mem, int code_min, int code_max);
|
|
|
|
tre_ast_node_t *
|
|
tre_ast_new_iter(tre_mem_t mem, tre_ast_node_t *arg, int min, int max,
|
|
int minimal);
|
|
|
|
tre_ast_node_t *
|
|
tre_ast_new_union(tre_mem_t mem, tre_ast_node_t *left, tre_ast_node_t *right);
|
|
|
|
tre_ast_node_t *
|
|
tre_ast_new_catenation(tre_mem_t mem, tre_ast_node_t *left,
|
|
tre_ast_node_t *right);
|
|
|
|
#ifdef TRE_DEBUG
|
|
void
|
|
tre_ast_print(tre_ast_node_t *tree);
|
|
|
|
/* XXX - rethink AST printing API */
|
|
void
|
|
tre_print_params(int *params);
|
|
#endif /* TRE_DEBUG */
|
|
|
|
#endif /* TRE_AST_H */
|
|
|
|
/* EOF */
|