mirror of
https://github.com/zap-proto/zip.git
synced 2026-10-19 15:38:57 +00:00
Count a form's separators in one pass
A url-encoded form is refused when it carries 10000 or more '&', so more than 10000 pieces whether fields or empty, by one bytes.Count before anything is split. Counting only non-empty pieces split the body one bytes.Cut per separator, so 4 KiB of gzip that decoded to 4 MiB of '&' cost about 60 ms of CPU before fasthttp parsed it again.
This commit is contained in:
@@ -73,7 +73,7 @@ Patch tags, one above the highest `v*` on GitHub. Push the commit to main, read
|
||||
- The `/docs` page loads Swagger UI from cdn.jsdelivr.net. Embedding swagger-ui-dist 5 would add about 1.74 MB (bundle 1,552,209 bytes, CSS 185,784 bytes, Apache-2.0) to every binary that links zip.
|
||||
|
||||
- A manifest op may say `"stream": "sse"`, `"bytes"` or `"socket"` (the answer is written as produced, or the connection upgrades; `ProjectOpenAPI` publishes `text/event-stream` with `x-events`, binary media, or `101` with `x-socket`) and `"raw": true` (the request body is bytes, bound to no type; the Rust front end's). A stream beside an `out` means either. `Check` refuses a socket beside an `out` and a raw body beside an `in`.
|
||||
- Every wire is a typed op. A request body that is not JSON is a `zip.Body` field (bytes and Content-Type as sent), a `Parser` In (decodes itself; the document keeps In's schema), or a form: `*zip.File` parts make multipart, and `form:` fields bind only when the op `Consumes` a form media type (otherwise the tags are another binder's and JSON binds as before). A `cookie:` field is an `in: cookie` parameter set by the request's cookie and nothing else: the body, an argument object and the URL never set it, and no schema lists it as a body field. An answer that is not one JSON value is a `Body` (bytes, filename, its own status and headers; a `Reader` streams chunked; no bytes and no Type sends no Content-Type), `Verbatim[T]` (a relay's bytes documented as T; the upstream's 4xx/5xx pass undeclared), `Sse[T]` (`Send(ctx, emit)` on its own goroutine; ctx ends with the stream), `Socket[M]` (GET only; the handshake is decided before the handler runs: 501 over ZAP, 426 without an upgrade or for another version, 400 without a key, 403 for a browser origin the op does not admit — its own, plus `zip.Origins(...)`, `"*"` for any), `Redirect` or a `CookieCoder`; a type embedding `Body` or `Redirect` is that answer, so one with a `Cookies` method sets them beside it. A JSON answer keeps fiber's `application/json; charset=utf-8` unless the op `Produces` another. Every declared status carries the Out's schema, except one a union's alternative claims with its value-receiver `StatusCode()`. A union is a type whose `OneOf()` result types are its alternatives; `Or[A,B]` is the stock one. Over MCP bytes are base64 (in) and tool content (out); bytes, a Reader and a stream (as JSON lines) are bounded at `toolBound` and a Reader or stream at `toolWait`, and a socket op is no tool (`_meta.refused`). 204, 304 and a redirecting op's 3xx carry no body in the document; any other declared status carries the Out and its example. The call plane writes an `Sse` as the REST door does, refuses a `Socket` before its handler, and carries a union that holds a value as the union. A header field is a CLI flag under its field's name, as before; there are no header flags. An op that takes the body itself (a `Body` field, a form, a `Parser`) reads its content through `content`: each Content-Encoding (every header line) undone and held to BodyLimit, 415 for a coding it does not undo or a chain of more than three, 413 past the limit, 400 for one that does not decode, all before the handler; a multipart form is parsed in memory from that content, a url-encoded one of more than 10000 fields is 400 before any is decoded, and a form keeps only the keys In binds. Over HTTP fasthttp has read a multipart body as a form before routing, so a `Body` field holds that form re-encoded (what `c.Body()` gives a raw handler), not the bytes as sent; ZAP carries them as sent. A JSON op still reads fiber's `Body` as it always has. An answer refused after its handler ran closes its `Reader`. The graph (`/.well-known/graph`) and `Here` carry only plain ops (`registeredOp.plain`: one JSON value in, one out); a `cookie:` field is never a graph argument, and `Here` refuses a socket op before its handler.
|
||||
- Every wire is a typed op. A request body that is not JSON is a `zip.Body` field (bytes and Content-Type as sent), a `Parser` In (decodes itself; the document keeps In's schema), or a form: `*zip.File` parts make multipart, and `form:` fields bind only when the op `Consumes` a form media type (otherwise the tags are another binder's and JSON binds as before). A `cookie:` field is an `in: cookie` parameter set by the request's cookie and nothing else: the body, an argument object and the URL never set it, and no schema lists it as a body field. An answer that is not one JSON value is a `Body` (bytes, filename, its own status and headers; a `Reader` streams chunked; no bytes and no Type sends no Content-Type), `Verbatim[T]` (a relay's bytes documented as T; the upstream's 4xx/5xx pass undeclared), `Sse[T]` (`Send(ctx, emit)` on its own goroutine; ctx ends with the stream), `Socket[M]` (GET only; the handshake is decided before the handler runs: 501 over ZAP, 426 without an upgrade or for another version, 400 without a key, 403 for a browser origin the op does not admit — its own, plus `zip.Origins(...)`, `"*"` for any), `Redirect` or a `CookieCoder`; a type embedding `Body` or `Redirect` is that answer, so one with a `Cookies` method sets them beside it. A JSON answer keeps fiber's `application/json; charset=utf-8` unless the op `Produces` another. Every declared status carries the Out's schema, except one a union's alternative claims with its value-receiver `StatusCode()`. A union is a type whose `OneOf()` result types are its alternatives; `Or[A,B]` is the stock one. Over MCP bytes are base64 (in) and tool content (out); bytes, a Reader and a stream (as JSON lines) are bounded at `toolBound` and a Reader or stream at `toolWait`, and a socket op is no tool (`_meta.refused`). 204, 304 and a redirecting op's 3xx carry no body in the document; any other declared status carries the Out and its example. The call plane writes an `Sse` as the REST door does, refuses a `Socket` before its handler, and carries a union that holds a value as the union. A header field is a CLI flag under its field's name, as before; there are no header flags. An op that takes the body itself (a `Body` field, a form, a `Parser`) reads its content through `content`: each Content-Encoding (every header line) undone and held to BodyLimit, 415 for a coding it does not undo or a chain of more than three, 413 past the limit, 400 for one that does not decode, all before the handler; a multipart form is parsed in memory from that content, a url-encoded one of 10000 or more `&` (more than 10000 pieces, fields or empty) is 400 before any is decoded, and a form keeps only the keys In binds. Over HTTP fasthttp has read a multipart body as a form before routing, so a `Body` field holds that form re-encoded (what `c.Body()` gives a raw handler), not the bytes as sent; ZAP carries them as sent. A JSON op still reads fiber's `Body` as it always has. An answer refused after its handler ran closes its `Reader`. The graph (`/.well-known/graph`) and `Here` carry only plain ops (`registeredOp.plain`: one JSON value in, one out); a `cookie:` field is never a graph argument, and `Here` refuses a socket op before its handler.
|
||||
|
||||
## Rust
|
||||
|
||||
|
||||
@@ -695,38 +695,25 @@ func undo(coding string, b []byte, limit int) ([]byte, error) {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// maxFormFields is the most fields a url-encoded form may carry: the bound Go's
|
||||
// own query parser holds (net/url, urlmaxqueryparams). A multipart form is held
|
||||
// to mime/multipart's own bounds on parts and part headers.
|
||||
// maxFormFields is the most pieces a url-encoded form may carry between its
|
||||
// '&'s, a field or empty: the bound Go's own query parser holds (net/url,
|
||||
// urlmaxqueryparams). Counting separators rather than fields keeps the check
|
||||
// one pass over the bytes, so a form of nothing but '&' costs no more to refuse
|
||||
// than any other. A multipart form is held to mime/multipart's own bounds on
|
||||
// parts and part headers.
|
||||
const maxFormFields = 10000
|
||||
|
||||
// fewFields reports whether a url-encoded form holds at most maxFormFields
|
||||
// fields, counting the non-empty pieces between its '&'s without decoding any.
|
||||
func fewFields(form []byte) bool {
|
||||
n := 0
|
||||
for len(form) > 0 {
|
||||
var field []byte
|
||||
field, form, _ = bytes.Cut(form, []byte("&"))
|
||||
if len(field) > 0 {
|
||||
if n++; n > maxFormFields {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// readForm reads a REST request's form from its content: url-encoded fields,
|
||||
// or a multipart form's fields and parts, each part read whole, keeping only
|
||||
// the keys the op binds (names). The content is already decoded and held to the
|
||||
// body limit, and a multipart form is parsed from it in memory, so no part is
|
||||
// larger than the body that carried it. A url-encoded form is decoded as fiber
|
||||
// decodes one, and one of more than [maxFormFields] fields is refused before
|
||||
// decodes one, and one of more than [maxFormFields] pieces is refused before
|
||||
// any is decoded.
|
||||
func readForm(c fiber.Ctx, media string, body []byte, names map[string]bool) (formData, error) {
|
||||
fd := formData{values: map[string][]string{}, files: map[string][]File{}}
|
||||
if m, _, _ := mime.ParseMediaType(media); m != mimeMultipart {
|
||||
if !fewFields(body) {
|
||||
if bytes.Count(body, []byte("&")) >= maxFormFields {
|
||||
return fd, fmt.Errorf("a form carries more than %d fields", maxFormFields)
|
||||
}
|
||||
var args fasthttp.Args
|
||||
|
||||
+13
-1
@@ -294,7 +294,7 @@ type sessionOut struct {
|
||||
}
|
||||
|
||||
// A url-encoded form binds the fields the op declares and no others, and one
|
||||
// of more fields than Go's own query parser takes is refused before any is
|
||||
// of more pieces than Go's own query parser takes is refused before any is
|
||||
// decoded or the handler runs. The refusal is cheap: a few kilobytes that
|
||||
// decode to millions of fields cost the decode and nothing per field.
|
||||
func TestContent_AFormKeepsItsFieldsAndIsBounded(t *testing.T) {
|
||||
@@ -333,6 +333,18 @@ func TestContent_AFormKeepsItsFieldsAndIsBounded(t *testing.T) {
|
||||
if alloc := after.TotalAlloc - before.TotalAlloc; alloc > 64<<20 {
|
||||
t.Errorf("refusing a %d-byte flood allocated %d MiB; it should cost the decode, not a value per field", len(flood), alloc>>20)
|
||||
}
|
||||
|
||||
// Empty pieces count: a form of nothing but separators is refused in the
|
||||
// same one pass, not split piece by piece.
|
||||
ran.Store(0)
|
||||
seps := gzipped(t, bytes.Repeat([]byte("&"), 4<<20-16))
|
||||
if code, body = send(t, big, "/v1/form", form, "gzip", seps); code != 400 || ran.Load() != 0 {
|
||||
t.Errorf("a %d-byte flood of separators: %d %s, want 400 and no run", len(seps), code, body)
|
||||
}
|
||||
padded := strings.Repeat("a=1&&", 5000) + "note=padded"
|
||||
if code, body = send(t, big, "/v1/form", form, "", []byte(padded)); code != 400 {
|
||||
t.Errorf("5001 fields padded past 10000 pieces: %d %s, want 400", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// Every Content-Encoding line is a coding, as one comma-separated line is, and
|
||||
|
||||
Reference in New Issue
Block a user