Repository navigation
Measure the CLDR corpus with each locale pack too - #673
Merged
Merged
Conversation
cldr_agreement.py gains a locale view: LOCALE_FILES maps each CLDR file to its language's pack, those renderings are parsed with parser_for(pack) over the shape's base order, and the disagreements are recorded in cldr_agreement_locales.json, ratcheted per pack by test_cldr_agreement.py. The summary prints each pack's all-roles agreement under the default parser and under the pack: JA ja: 7/24 -> 12/24 RU ru: 17/24 -> 17/24 TR_AZ az, tr: 32/48 -> 32/48 ZH yue, yue_Hans, zh, zh_Hant: 22/96 -> 42/96 One --record rewrites both records. JA needs the ja extra: without it the JA ratchet skips visibly (CI's ja-extra job imports namedivider first, so it cannot skip there) and --record keeps the recorded ja entries unchanged. A test fails on a registered pack that is neither mapped nor listed in NO_CLDR, and on a mapped file with no renderings. The default record is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found that an entry whose file is no longer mapped falls in no pack's share of the record: no check read it, and --record's carry-over kept it, since None is not a measured pack. It is now reported as gone, and the carry-over keeps only the entries of mapped packs that were not measured. AGENTS.md records the class beside the filter-test paragraph. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #673 +/- ##
=======================================
Coverage 99.02% 99.02%
=======================================
Files 46 46
Lines 4534 4534
=======================================
Hits 4490 4490
Misses 44 44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #672.
tools/differential/cldr_agreement.pygains a locale view.LOCALE_FILESmaps each CLDR file to its language's pack:zh,zh_Hant,yueandyue_Hansmap toZH.jamaps toJA.rumaps toRU.trandazmap toTR_AZ.Those renderings are parsed with
parser_for(pack), over the sameFAMILY_FIRSTbase where CLDR's order is surnameFirst. Their disagreements are recorded in the newcldr_agreement_locales.json, andtest_cldr_agreement.pyratchets them per pack. The summary now prints each pack's before/after:These figures reproduce the issue's table, except that ZH's default count is 22 where the issue measured 21 on 1f0b374.
How the issue's open questions were settled:
jaextra: there's no env switch. Withoutnamedivider, the JA ratchet is a visible pytest skip. That's safe because CI'sja-extrajob runsimport namedividerbefore pytest, so it can't skip there.--recordrewrites both records. Without the extra, it keeps the recordedjaentries unchanged.LOCALE_FILESnor inNO_CLDR, and on a mapped file with no renderings.cldr_agreement.jsonis byte-identical.Review found that record entries whose file leaves
LOCALE_FILESfell in no pack's check and survived re-records. Fixed in the second commit: such entries are now reported asgone, and the re-record drops them. That commit also adds the lesson to AGENTS.md's filter-test paragraph.Verified:
🤖 Generated with Claude Code