feat!: declarative datetime engine, AstroDate/DateSpan type system, and native calendars (1.0.0)#284
Draft
JarbasAl wants to merge 66 commits into
Draft
feat!: declarative datetime engine, AstroDate/DateSpan type system, and native calendars (1.0.0)#284JarbasAl wants to merge 66 commits into
JarbasAl wants to merge 66 commits into
Conversation
Add ovos_date_parser.eras with AstroDate (astronomical year numbering, proleptic Gregorian, date-only precision beyond the datetime range), astro_year_range for BC/deep-future decade/century/millennium buckets, and a language-agnostic era registry (anno domini, BC, before present, unix, julian day, holocene, anno mundi, French Republican, Baha'i, Buddhist) with resolve_era. In-range results stay plain datetime types; AstroDate appears only where datetime.date cannot represent the year. Epochs are cited to canonical sources: Stuiver & Polach 1977 (BP = AD 1950), USNO Julian-date conversion (JD 0 = 4713 BC = astronomical -4712, resolved via the Fliegel & Van Flandern integer algorithm), and POSIX.1-2017 (the Epoch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
get_date_ordinal's BEFORE_PRESENT_* branches previously raised OverflowError for any count reaching before year 1. Day and week counts now use Julian-day arithmetic (exact at any depth), month and year counts use integer month/year arithmetic anchored at the 1950 epoch; in-range results are byte-identical to the old relativedelta behaviour, and deeper counts return an AstroDate. Negative counts still raise OverflowError, unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract the date-scanning and time-scanning halves of the monolithic scanner into internal helpers and expose extract_date_en / extract_time_en with date-only / time-only contracts. Behaviour of extract_datetime_en is byte-identical, verified against a 1112-case frozen oracle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a language-agnostic era-phrase scanner (eras_scan.py) where a
language contributes only an ordered pattern vocabulary and its
spelled-number normaliser; extraction, epoch resolution and the
in-range/AstroDate return rule are shared. English (eras_en.py) and
Portuguese (eras_pt.py) vocabularies cover BC/BCE and a.C./a.e.c.,
AD/CE and d.C./e.c., radiocarbon before-present (BP/AP), unix time,
Julian day, Holocene/human era, anno mundi, out-of-range bare years
("in the year 12000" / "no ano 12000") and BC-axis century/millennium
ordinals. extract_date_en tries the era layer before the ordinary
scanner; representable bare years fall through unchanged.
Ambiguity guards: era words that double as ordinary language (ad, ce,
he, am, ap) only match adjacent to a number, and bare HE additionally
requires a 5+ digit year so the English pronoun never matches.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pec-tools
Move all era/scoped surface forms into .voc phrase sets under
ovos_date_parser/locale/<lang>/, loaded with ovos-spec-tools, so a new
language's era support is a translation task. Add the calendar-scoped
ordinal and season layer (scoped_scan.py + locale forms): absolute
periods ("the 21st century"), month/year-scoped ordinals ("the 3rd week
of june", "the 100th day of the year"), one-level nesting ("the first
decade of the 21st century") and hemisphere-aware seasons ("summer of
1969", "next winter"), wired into extract_date_en after the era layer.
Language modules keep only their spelled-number normaliser and
non-translatable guards (English bare HE needs a 5+ digit year).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same vote logic as the number parser's multi-reference harness: both
references agreeing against us is a bug lead, references disagreeing is
contested and adjudicated by hand. First run already surfaced the
past-marker direction bug class ("2 weeks ago" equivalents resolving
forward in pt/es/fr/de/it) and missing bare yesterday-words in fr/de.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… Italian Locale phrase sets and thin normaliser modules over the shared era scanner: a. de C./d. de C., av. J.-C./ap. J.-C., v. Chr./n. Chr., a.C./d.C. and their secular-era equivalents, before-present, unix time, Julian day, Holocene and anno mundi phrasing, deep-future bare years and BC-axis century/millennium ordinals, with natural-language tests per language. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Design doc for extending date extraction to named eras, epochs, named dates and calendar-scoped ordinals. Records what is already landed (the ranges.py / DateTimeResolution / season primitives, currently orphaned), the parsing-layer gap, the salvageable prior art from upstream lingua-franca PR #96, a target architecture (eras.py + extract_date/extract_time split), a phased no-regression plan, and the open decisions — chiefly the representation of BC / out-of-range years.
BC and out-of-range years are represented by AstroDate (astronomical year numbering, proleptic Gregorian, date-only); new APIs return plain datetime types whenever representable. Calendar plurality, location scope and ambiguity policy keep their recommended defaults. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…er, matcher, resolver, loader, explain)
JarbasAl
force-pushed
the
feat/declarative-engine
branch
from
July 21, 2026 12:24
fd3eaa4 to
5a33a3f
Compare
JarbasAl
changed the title
Add ovos_date_parser/calendars.py: pure integer conversions between five arithmetic calendars and the astronomical Julian Day Number, plus the Fliegel & Van Flandern Gregorian/Julian pair. Registry covers tabular Islamic (civil), Hebrew (molad + dechiyot), proleptic Julian, French Republican (Romme variant) and arithmetic Bahai (Gregorian-locked Naw-Ruz). Every algorithm is transcribed from Dershowitz & Reingold (1990) or a published conversion table, cited in module and test docstrings; no conversion library is used. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dars Wire the calendars module into the engine. The loader reads month_<calendar>_<n>.voc into LangSpec.calendar_months (unknown calendar key or a surface claimed by two calendars is a load error). A new CAL_MONTH matcher slot binds the union of calendar-month surfaces and tags the Match with the calendar it matched; nongregorian_date sits just above calendar_date in precedence. The resolver converts through the calendar's JDN functions, defaulting the day to 1, deriving a missing year in calendar space and applying prefer_future by bumping the calendar year; impossible days and out-of-range dates resolve to None. explain() names the calendar in the slot bindings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add Islamic-civil month vocab to the synthetic zz locale and real ar (Hijri) and he (Hebrew) locales with nongregorian_date stanzas, exercised through the DateTimeEngine facade. Engine tests assert exact Gregorian values against the downloaded conversion tables (anchor 2017-06-27), cover yearless prefer-future bumping in calendar space, and confirm impossible days and garbage never raise. Loader tests cover calendar_months parsing, the unknown-calendar guard, and cross-calendar surface collisions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a Calendar systems section: the JDN interchange hub, the arithmetic-only scope, the nongregorian_date construction, the month_<calendar>_<n>.voc filename convention, the out-of-scope list (observational calendars, holidays, calendar-aware output) and the AstroDate/era-registry fusion planned once the named-epochs work lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…larative-engine # Conflicts: # ovos_date_parser/dates_en.py
AstroDate drops the Optional month/day and the resolution tag (imprecision now belongs to DateSpan) and becomes a frozen, slotted, tz-naive point with the full year..microsecond field set, month/day defaulting to 1. It fully duck-types datetime's public API: replace, weekday/isoweekday/isocalendar, toordinal (proleptic Gregorian ordinal, plain int), date/time, isoformat/fromisoformat (year-expanded), from_date/from_datetime, strftime over the year-width-safe subset, timedelta arithmetic and AstroDate differences, and comparisons plus equality/hash interoperating with date/datetime (equal instant => equal; hash-consistent with datetime for in-range values). Weekday/ordinal math routes through calendars.gregorian_to_jdn / jdn_to_gregorian, and eras.julian_day_to_date now delegates to the same hub, deduplicating the Fliegel & Van Flandern transcription. resolve_era, ranges BP fallback, and eras_scan drop the resolution kwarg; era/ranges tests are restated against the new shape and a protocol test walks datetime's public API.
Add a roman module and roman_date construction resolving inclusive backward counting from the three monthly anchors over the Julian calendar: 'a.d. III Kal. Apr.' -> 30 March, 'pridie Idus Martias' -> 14 March, 'Idibus Martiis' -> 15 March. Nones/Ides shift to the 7th/15th in March/May/July/October; Kalends counting crosses into the previous month; an ordinal past the previous anchor yields None. Day-wide spans with the Julian-calendar labels. Adds zz anchor vocab and worked-example + adversarial tests, cited to a downloaded Roman calendar reference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Instantiate the regnal-sequence kind with a dozen well-attested Roman consular year pairs, resolving 'the consulship of <pair>' to that eponymous year's span (each consulship a one-year segment, BC years astronomical). Add the Olympiad era as a four-year-cycle era via item 4's year-span machinery: Era gains year_length (4) and a 1 July midsummer year_start, so Olympiad 1 spans 776 BC..772 BC midsummer. Both cited to downloaded fasti and Olympiad references; the datasets are deliberately small to prove the mechanism. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the vendored astrodate/calendars/cycles/regnal/roman/eras modules with thin re-export shims over the chronologia library, and share its DateTimeResolution enum through ranges so a single enum identity is used everywhere. Expose the newer reckoning surface (TIMELINES, PERIODS, resolve_bp, calibrate_c14, civil_add, resolve_wall_clock) from the package. Drop the four pure-core test files now covered by chronologia's own suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire locale/en/lang.json and the engine-format vocab (months, weekdays, named days, relative markers, clock fractions/meridiems, scope units, hijri/hebrew calendar months, nengo regnal names, Roman anchors) so extract_timespan works for English. Add a spelled-number folding pre-pass (numfold.fold_en), applied via the language hook, folding runs of English number-words and digit+ordinal suffixes into digit tokens so NUM/DAY/YEAR/ORD slots bind whether a value is typed or spelled.
96 hand-derived cases over 'in N units' / 'N units ago' (digit and spelled), named days, and next/last/this weekday; 5 idiomatic compounds (day-after-tomorrow, week-from-tuesday, fortnight, couple-of-weeks, half-an-hour) xfail'd pending new constructions.
81 cases over every order/style (month-first, day-first, ordinal, ISO, month+year, bare month), prefer_future roll semantics, and impossible dates that must be rejected; bare-year and current-month idioms xfail'd.
Add the '3 pm' order (hour + meridiem, no 'at') to the English clock construction, and compose a lone date with a lone clock in extract_timespan so 'june 5th at 3pm' yields the minute-wide span on the named day. Legacy extract_datetime and the oracle are unaffected. test(en): natural-language corpus wave 3 -- clock (62 cases, 4 xfail).
Resolve BC/BCE/AD/CE and before-present year spans via chronologia
resolve_era; numeric deep time ('66 million years ago') via resolve_bp
with a dedicated SCALE slot; named geological periods and their
early/mid/late subdivisions via PERIODS/subdivide with a PERIOD/PART slot.
Add the periods/scales/period_parts vocab maps to the loader/model/matcher
and the English locale data. Scale-words are excluded from the number
fold so they survive to mark deep time.
test(en): natural-language corpus wave 4 -- eras & deep time (50, 3 xfail).
41 hijri/hebrew day and month dates with hand-verified Gregorian equivalents, romanization-variant agreement, and impossible-day guards; Hebrew-new-year idiom and Republican months xfail'd.
41 cases over Nth-week/day scoped ordinals, month/week-of-year, absolute centuries/millennia, nested decade-of-century, and meteorological seasons; fuzzy thirds, spoken/digit decades, and half-years xfail'd pending new constructions.
Build a span from the start of the left sub-parse to the end of the right,
with a guard that a bare 'A to B' never hijacks a clock ('quarter to
five'). Oracle and legacy paths unaffected.
test(en): natural-language corpus wave 8 -- ranges (17, 4 xfail).
Garbage, lone numbers, stranded markers, empty/whitespace, absurdly long input, injection-shaped noise, and embedded-date ambiguity guards; the contract is never-raise and never-fabricate.
Route the untouched NUM surface string and its SI scale unit (million->Ma, billion->Ga, thousand->ka) through chronologia.resolve_bp so the sig-fig rule applies at the spoken unit before value x scale collapses to years. '66 million years ago' is now 1 Ma wide (was 1 year); '66.5 million' 100 ka; '4.5 billion' 100 Ma. Corpus gains deep-time width assertions.
A standalone year -- '2027', 'in 1995', 'the year 2000' -- resolves to a
year-wide span. New GYEAR slot binds only a bare 4-5 digit run so small
integers and digit soup never read as a year. year_ref sits just below
calendar_date in precedence, so era/decade constructions covering the same
digits still win. Enables bare-year range endpoints ('from 2020 to 2023').
'the nineties', 'the 1990s', 'the 90s', 'the 2020s' resolve to a ten-year span. Bare tens use the nearest-past century convention (in 2017 'the twenties' is 1920). early/mid/late slice the decade into thirds via chronologia.subdivide. New decade_word_<tens> vocab and DECADE slot.
'early/mid/late <month>' slices the month into thirds via chronologia subdivide. 'the first/second half of <year|decade|century|millennium>' resolves the calendar half -- the year splits at July 1, longer periods at their midpoint year -- so halves tile cleanly. New 'half' marker vocab.
noon/midnight are now landmark words (LANDMARK slot, minutes-since-midnight
vocab) instead of being folded to an HOUR number, so bare 'noon'/'midnight'
resolve and 'quarter/ten to|past noon|midnight' compose on them. 'to'-side
underflow rolls over ('quarter to midnight' = 23:45). Military 'HHMM hours'
and bare leading-zero 'HHMM' ('0600') read as 24h clocks; a bare 4-digit run
without a leading zero stays a year. Arbitrary-minute 'N to/past H'.
Quantified offsets ('a couple of weeks', 'half an hour', 'a fortnight ago'):
quantifier vocab (a=1, couple=2, half=0.5), sub-day (hour/minute) and
fortnight units, fractional counts, and 'of'/indefinite-article filler.
Compound named days ('the day after/before <named day>') and offset-from-
weekday ('a week from tuesday' = N weeks after the next occurrence). Military
time split into its own construction so 'HHMM hours' beats the scoped-ordinal
reading. Indefinite article kept in a separate voc so the legacy scoped
scanner (which reads marker_article) is untouched.
'from monday to friday' parses bare-weekday endpoints (only inside a range
framing -- a lone weekday still never parses). 'between 3 and 5 pm' lets the
bare left endpoint borrow the right endpoint's trailing meridiem. When the
end lands before the start the right span rolls forward by its own
granularity -- a day for a clock ('9 am to 5 pm', '10 pm to 2 am'), a week
for a weekday -- so both endpoints read on the same cycle.
'the first of the month' resolves the Nth day of the current month (rolling to next month once the day has passed). 'the last decade of the 20th century' wires ordlast into the nested scoped-ordinal order (last unit of the Nth scope).
engine.tokenize merges adjacent tokens that spell a multiword period surface back into a single token (longest phrase first), so 'bronze age'/'iron age' bind one PERIOD slot. Region-default (GB) comes from the vocab entry. Stone age / middle ages stay xfail: no citable single-span entry in chronologia.
'the hebrew/jewish new year N' resolves Rosh Hashanah (1 Tishrei, month 7 in the Nisan-first numbering) of Hebrew year N through the Hebrew calendar JDN hub. French Republican month vocab (vendemiaire..fructidor) wired into the existing reckoned-date family, so '18 brumaire' parses at month level (the small republican year is not yet bound).
Ablative Roman anchor forms (idibus/kalendis/nonis) and ablative month forms (martiis, ianuariis) let 'idibus martiis' resolve the Ides of March; the existing roman-date orders do the reckoning. year_ref gains a 'year NUM SCALE' order so 'the year twelve thousand' = 12000 -- safe because scale words are reserved for deep time only in the 'years ago' framing.
… legacy API Delete the vendored declarative engine and its en/ar/he engine locale (now owned by chronologia) along with the engine-only test suites. Re-export extract_timespan and explain from chronologia so the public surface is unchanged. The legacy extract_datetime / extract_date_xx scanners and their pre-passes are untouched; eras_scan and scoped_scan resolve the moved en locale from chronologia's packaged locale while keeping their own de/es/fr/it/pt phrase sets.
Add nice_span(span, lang): a DateSpan's width is its precision, so the label is chosen from the width -- a day-wide span reads as a date, a month-wide span as a month, a decade as "the 1980s", a century as "the 19th century", symmetrically down to BC eras. For English it is the exact inverse of extract_timespan: every label re-parses to the same span, gated by a round-trip test battery across day/month/year/decade/century/ millennium (past, future and BC year). Widths the reckoning core cannot yet re-parse (weeks; BC decades/centuries/millennia) still get a correct label and xfail the round-trip with the reason recorded. Teach the datetime formatters to accept an AstroDate directly, projecting it to a datetime when it fits one; year-only formatting works even for years no datetime can hold.
The scoped-era extraction constructions re-parse the 'Nth century BC' and 'Nth millennium BC' labels nice_span emits, so both widths move from xfail to the positive gate; the BC decade construct remains a documented gap.
The week-of and BC-decade extraction constructions re-parse the last two labels nice_span emits, so the xfail battery is gone: day, week, month, year, decade, century, millennium -- AD and BC -- all round-trip.
nice_span now emits the exact native calendar phrasing chronologia's ar/he extractors re-parse, closing the round-trip for every width those locales construct: day (21 يوليو 2026 / 20 ביולי 1969), month (يوليو 2026 / יולי 2026), year, year-BC (300 ق.م / 300 לפנה״ס) and 20th-century decades (الثمانينات / שנות השמונים). Hebrew prefixes the month with ב in a full date and leaves it bare in a month-year; BC uses each locale's era marker. Western digits throughout, matching the forms both extraction corpora assert. The he/ar round-trip battery mirrors the English one: a positive gate for every supported width plus strict flip-to-fail xfails for the widths with no native construction (week, century, millennium, BC decade/ century/millennium), each naming the missing construction. Decade words are anchor-relative, so the battery pins a fixed anchor to stay timeless. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One PR for the full modernisation of date extraction (design & 1.0.0 roadmap: #291).
Type system
AstroDateis the base struct: a frozen point in time with unbounded year (astronomical numbering, proleptic Gregorian), fully duck-typed todatetime's public API — attributes,replace(),weekday()/isocalendar(),toordinal(), expanded-yearisoformat()/fromisoformat(),timedeltaarithmetic, comparisons/equality interoperating withdate/datetime(hash-consistent), enforced by a test that walks datetime's public API. Internal time is canonical 24/60/60; AstroDate carries no imprecision fields.DateSpanis the primitive result: frozen half-open[start, end)with AstroDate endpoints. Width is the referential uncertainty ("june 2027" = the month; "44 BC" = the year; "3 pm" = one minute); resolution is derived from width. Explicit ranges are spans with independently-parsed endpoints. Width is never used to encode model/approximation error — that stays documented.extract_timespan(text, lang, anchor) -> (DateSpan, remainder); legacyextract_datetime/extract_datereturnspan.startas realdatetime/date(None out of range) — proven byte-identical against the frozen oracles.Layers
1. Named epochs (absorbed from #277): era registry, Before-Present, the
extract_date_en/extract_time_ensplit gated on a 1112-case frozen oracle, era + scoped extraction with all surface forms as/locale/<lang>/*.vocresources (en/pt/es/fr/de/it), three-way differential harness.2. Declarative engine: language-agnostic core (frozen-dataclass model;
Tokenizer/TemporalNormaliser/ConstructionCompiler/ConstructionMatcher/Resolver/ExplainTrace); constructions compiled from per-language data (lang.jsonfacts +.vocvocabularies); syntheticzzlocale; per-stage tests. Per-language strangler migration on frozen oracles.3. Native arithmetic calendars:
calendars.py— JDN hub; tabular Islamic, Hebrew (molad + dechiyot), proleptic Julian, French Republican (Romme), Badí' — vendored from Reingold–Dershowitz, gold-valued against published tables (Rosh Hashanah 5785 → 2024-10-03; 1 Ramadan 1446 → 2025-03-01), no new runtime deps. Calendar-tagged month vocab (month_<calendar>_<n>.voc); ar Hijri + he Hebrew locale data.4. Unified reckoning model — validated against a 92-system world survey of eras and calendars. Four kinds:
(name, start_JDN)tables (Japanese nengō, Chinese era names, regnal years) — maintained data, not derivable rules.Adopted extensions (all pure arithmetic, from the survey's coverage-per-complexity ranking): era year-start ≠ Jan 1 (Byzantine AM rolls over Sep 1 — an era year straddles two Gregorian years, naturally a DateSpan); epagomenal 13th months (Coptic/Ethiopian/Republican/Haab); day cycles generalising weeks — free-running
(JDN − anchor) mod N(7-day week, sexagenary, Tzolk'in, Pawukon's concurrent cycles) and month-anchored (Republican décades) — bound byweekday_refvocab per cycle; day subdivisions — alternative clock systems as exact parse-time rescalings of the civil day fraction (French decimal time: 1 decimal hour = 2.4h), AstroDate internals staying datetime-canonical. Hub rule: JDN for days and coarser, civil day-fraction for finer.Out of scope, permanently: anything requiring an ephemeris or sighting — East-Asian lunisolar leap-month rules, Hindu drik, observational Islamic, post-2015 Badí', legal Solar Hijri (33-year arithmetic shipped as approximation with validity window). Umm al-Qura qualifies as supportable (bounded deterministic table). Holiday resolution: separate future work.
Replaces #270, #277, #283.
Release: 1.0.0 (breaking)
This PR ships as the 1.0.0 major release. Breaking by declaration:
AstroDatereplaces the 0.x out-of-range representation and duck-typesdatetime(unbounded year; no optional fields, no resolution tag).DateSpanis the primitive of the newextract_timespanAPI; resolution is derived from span width.resolve_erareturns exact calendar-backed dates for anno mundi / French Republican / Bahá'í where 0.x approximated by year counts (values change, correctly — e.g. AM 5786 → 2025-09-23, not 2025-01-01).extract_datetime_xximplementations return lists uniformly.Not breaking:
extract_datetime/extract_datesignatures and returns are unchanged and byte-identical on the 1112-case frozen oracle.The 1.x roadmap (language migrations en+pt+fi first, parity completion, adopted reckoning extensions incl. day cycles / day subdivisions / regnal sequences / Kalends-Ides / eponym tables, new arithmetic calendars, calendar-aware formatters, explicit deferrals) is in the design doc's Release/Roadmap sections in this PR.
BREAKING CHANGE: new AstroDate/DateSpan type system and extract_timespan API; resolve_era returns exact calendar-backed dates; extract_datetime_xx return lists uniformly.