mirror of
https://github.com/hanzoai/semantic.git
synced 2026-10-30 15:04:19 +00:00
zeekayandHanzo Dev
997e8a55bf
The package comment named Entity, Date, Number, Measure and Rows, none of which existed. Four kinds now do, each a Kind[T]: Parse reads text under an Anchor, Canon writes the one canonical form, and every Kind is also a Datatype for code that holds several side by side. - Date reads into a half-open Span with a Grain: "March 2023" is [2023-03-01, 2023-04-01). Relative dates count from Anchor.At, the source's own date; nothing calls time.Now. - Number reads exactly into a big.Rat; Quantity adds a unit, folded to its symbol only where the word has one meaning; Money adds an ISO 4217 code. What the text leaves open is refused with ErrAmbiguous, never guessed: day and month with no Anchor.Order, a clock time with no offset or Anchor.Zone, or one a zone repeats, "1,234" with no Anchor.Point, a two-digit year, and a sign several currencies write. One generic test holds Canon∘Parse idempotent for every kind. The package comment now names what is there, and the normalize example shows the kinds. Co-authored-by: Hanzo Dev <[email protected]>
48 lines
2.0 KiB
Go
48 lines
2.0 KiB
Go
// Package normalize is the cleanup between parsing a document and splitting
|
|
// it: the same sentence written with a non-breaking space, a curly apostrophe
|
|
// or a decomposed accent should reach the extractor as one string, not three.
|
|
//
|
|
// Every transform is a [Step] — a function from string to string — and steps
|
|
// compose into a [Chain], so a caller assembles the pipeline the corpus needs
|
|
// instead of arguing with an options struct. [Text] and [Clean] are the two
|
|
// chains worth having by default.
|
|
//
|
|
// The package also reads the values that end up as graph nodes: dates
|
|
// ([Date]), numbers ([Number]), quantities ([Quantity]) and sums of money
|
|
// ([Money]). Each is a [Kind]: it reads text under an [Anchor] — what the
|
|
// source says about how to read it — into a typed value, and writes that value
|
|
// in one canonical form. A reading returns an error rather than a string,
|
|
// because a date that failed to parse is not a date, and text with two
|
|
// readings is refused with [ErrAmbiguous] rather than read one way.
|
|
package normalize
|
|
|
|
// Step is one text transform. Every exported function of the form
|
|
// func(string) string in this package is a Step and can be used as one.
|
|
type Step func(string) string
|
|
|
|
// Chain applies steps in order. The zero Chain returns its input unchanged.
|
|
type Chain []Step
|
|
|
|
// Run applies every step in the chain to s.
|
|
func (c Chain) Run(s string) string {
|
|
for _, step := range c {
|
|
s = step(s)
|
|
}
|
|
return s
|
|
}
|
|
|
|
var (
|
|
prose = Chain{NFC, Lines, Control, Quotes, Space}
|
|
plain = Chain{Tags, NFC, Control, Quotes, Flat}
|
|
)
|
|
|
|
// Text is the default cleanup for prose that keeps its shape: compose
|
|
// characters, settle line endings, drop control codes, fold typographic
|
|
// punctuation to ASCII and regularize spacing. Paragraph breaks survive.
|
|
func Text(s string) string { return prose.Run(s) }
|
|
|
|
// Clean is the default cleanup for text headed into a single field or a
|
|
// similarity key: everything Text does, plus HTML removal, and the line
|
|
// structure flattened to single spaces.
|
|
func Clean(s string) string { return plain.Run(s) }
|