Details

Reviewers

Maniphest Tasks

T1751: Enrich FANTOIR database with Wikidata information

Commits

rDS6580112d2da3: Query Wikidata to enrich FANTOIR file

Summary

Create a fantoir_wikidata table with label and P31 information
to offer rich typography and determine what kind of pseudo-voie
an entry is.

Query Wikidata through the SPARQL end-point and map results.
Select most relevant P31 information.

As some Wikidata entries imports code RIVOLI from BAN, we've two
concurrent formats for DOM/TOM. To allow a foreign key, but also
to identify such Wikidata entries for further improvement, we store
in a separate column the FANTOIR code used by Wikidata.

Ref T1751.

Test Plan

Run it and get a full fantoir_wikidata table

Diff Detail

Repository

rDS Nasqueron Datasources

Lint

No Lint Coverage

Unit

No Test Coverage

Branch

wikidata

Build Status

Buildable 4319
Build 4586: arc lint + arc unit

Event Timeline

dereckson requested review of this revision.Jan 10 2023, 19:53

dereckson created this revision.

Harbormaster completed remote builds in B4312: Diff 6921.Jan 10 2023, 19:53

dereckson added a parent revision: D2729: Import FANTOIR file into PostgreSQL db.Jan 10 2023, 19:53

A lot of code to handle exceptions, mainly because there are
several formats used to represent the FANTOIR code.

Two strategies will be used to find the full code:

Compute it when cle RIVOLI is missing or code direction can be determined.
Query it for departments like 13/59/75/92/94 (*)

(*) Blocked by a search service to query the fantoir_... table to find information.
I'll work on that on another branch, as table can't exist or name can be unknown.

Also, sometimes a code FANTOIR will be missing. For example, I've found one
in a collectivity file not included in the national file on a Wikidata article.

We need code to skip them for now, and add a Wikidata health maintenance report later.

Harbormaster completed remote builds in B4313: Diff 6922.Jan 12 2023, 01:19

dereckson planned changes to this revision.Jan 12 2023, 01:19

Remove test limit

Harbormaster completed remote builds in B4314: Diff 6923.Jan 12 2023, 01:55

Rebased

Harbormaster completed remote builds in B4318: Diff 6931.Jan 12 2023, 05:57

Don't hardcore wikidata table name twice

Harbormaster completed remote builds in B4319: Diff 6932.Jan 12 2023, 05:59

Clean comment and debug code

Harbormaster completed remote builds in B4320: Diff 6933.Jan 12 2023, 06:03

dereckson edited the summary of this revision. (Show Details)Jan 12 2023, 18:58

This version is ready and can correctly update Wikidata. If we can solve the 403 issue when connecting to SPARQL endpoint, it's ready to commit. Meanwhile, to test this code, you can run the query and save the result (in XML format) at tmp/wikidata-query-result.xml

Harbormaster completed remote builds in B4323: Diff 6937.Jan 13 2023, 06:01

dereckson edited the summary of this revision. (Show Details)Jan 13 2023, 07:59

dereckson added a task: T1751: Enrich FANTOIR database with Wikidata information.

dereckson mentioned this in D2735: Generate Wikidate health report for FANTOIR code.Jan 14 2023, 03:01

dereckson added a child revision: D2735: Generate Wikidate health report for FANTOIR code.