Skip to content

POST Wikidata queries from name_usage.py instead of a GET - #683

Merged
derek73 merged 3 commits into
masterfrom
fix/issue-680-name-usage-post
Oct 14, 2026
Merged

derek73 merged 3 commits into
masterfrom
fix/issue-680-name-usage-post

Conversation

@derek73

@derek73 derek73 commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Closes #680.

tools/name_usage.py sent its SPARQL query as a GET. The query's VALUES block holds one literal per spelling per label language (63 languages). Percent-encoded, a Cyrillic letter takes six bytes, so a seven-letter word like Младший made the URL longer than the endpoint accepts and got HTTP 431. Measured: the largest URL that succeeded was 6,909 bytes (Младши, countries query) and the smallest that failed was 7,198 (Младший, totals query, which is sent first). A four-spelling word such as McDonald written in Cyrillic would also have failed.

The query can't shrink: the comment at WIKIDATA_LANGUAGES explains that exact literals keep it on the label index, while a FILTER times out. So this PR changes the transport. _sparql now POSTs a form-encoded query= body declared charset=utf-8, which has no length limit.

  • Test: test_sparql_sends_the_query_in_the_body_not_the_url captures the Request and asserts POST, a bare endpoint URL, and a body that decodes back to the query. It also checks that the response is read from what urlopen returns. It failed before the fix.
  • Live positive control: non-Latin names return counts through the POST. Иван gives 17,119 given and Петров 964 family, and 김 also gives counts. So the server decodes the body as UTF-8, and the zeros for the seven words the issue names are genuine answers, not failures to match. The control is run in Cyrillic because a misdecoded body would still run and match nothing, which looks like a false zero. An ASCII control (shehu, 180 family) cannot show that.
  • No release-log bullet: the tool isn't shipped.

🤖 Generated with Claude Code

The query's VALUES block repeats every spelling once per label
language, so a seven-letter Cyrillic word (Младший, six bytes a letter
percent-encoded) put the GET's URL past 7 KB and the endpoint answered
HTTP 431. Measured: Младши's longer query is 6,909 bytes and succeeds,
Младший's is 7,198 and failed. A form-encoded POST body has no such
limit. Checked live: the seven words the issue names now answer, and
shehu still reads 180 family as before.

Closes #680

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added the bug label Oct 14, 2026
@derek73 derek73 self-assigned this Oct 14, 2026
@codecov

codecov Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (625bc7e) to head (54c6256).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #683   +/-   ##
=======================================
  Coverage   99.03%   99.03%           
=======================================
  Files          46       46           
  Lines        4541     4543    +2     
=======================================
+ Hits         4497     4499    +2     
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

derek73 and others added 2 commits October 14, 2026 01:06
Review of #683: the body was correct UTF-8, but without a charset its
decoding rested on the server's form default, and a misdecoded query
still runs and matches nothing -- a false zero. Live control: Иван
reads 17,119 given and Петров 964 family through the POST, with and
without the declared charset. The test now also checks the response
is read back from what urlopen returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t's input class

#683's first control for the POST was ASCII (shehu) and could not show
the server decoding a Cyrillic body; review caught it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added this to the 2.5 milestone Oct 14, 2026
@derek73
derek73 merged commit 4410d90 into master Oct 14, 2026
10 checks passed
@derek73
derek73 deleted the fix/issue-680-name-usage-post branch October 14, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tools/name_usage.py fails with HTTP 431 for some Cyrillic words

1 participant