Page MenuHomeDevCentral
Files F3751702

D2744.id6962.diff
No OneTemporary
Actions

D2744.id6962.diff
View Options

diff --git a/.arcconfig b/.arcconfig
--- a/.arcconfig
+++ b/.arcconfig
@@ -1,4 +1,4 @@
{
"phabricator.uri": "https://devcentral.nasqueron.org",
- "repository.callsign": "FANTOIR"
+ "repository.callsign": "DS"
}
diff --git a/Cargo.toml b/Cargo.toml
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,34 +1,5 @@
-[package]
-name = "fantoir-datasource"
-version = "0.1.0"
-edition = "2021"
-description = "Generates a Postgres table from FANTOIR raw file"
-authors = [
- "Sébastien Santoro <dereckson@espace-win.org>"
-]
-license = "BSD-2-Clause"
-
-[dependencies]
-lazy_static = "~1.4.0"
-oxrdf = "~0.1.1"
-sparesults = "~0.1.3"
-
-[dependencies.async-scoped]
-version = "~0.7.1"
-features = ["use-tokio"]
-
-[dependencies.clap]
-version = "~4.0.32"
-features = ["derive"]
+[workspace]
-[dependencies.reqwest]
-version = "~0.11.13"
-features = ["gzip", "deflate"]
-
-[dependencies.sqlx]
-version = "~0.6.2"
-features = ["runtime-tokio-native-tls", "postgres", "chrono"]
-
-[dependencies.tokio]
-version = "~1.23.0"
-features = ["full"]
+members = [
+ "fantoir-datasource",
+]
diff --git a/README.md b/README.md
--- a/README.md
+++ b/README.md
@@ -1,51 +1,61 @@
-## How to use?
+# Nasqueron Datasources
-Define your PostgreSQL connection URL in environment:
+This repository offers components to query, extract and enrich data.
-```
-export DATABASE_URL="postgres://fantoir:fantoir@localhost/fantoir"
-```
+Those components are intended to build data pipelines.
-## Development
+## Components
-### Build instructions
+### FANTOIR import tool (fantoir-datasource)
-The PostgreSQL library is required to link against it.
-If not found, you can add the path to the LIB environment variable.
+Import a file from [FANTOIR file][1] into PostgreSQL.
-### Prepare a test database
+Enrich it from other sources like Wikidata.
-Execute the following queries as postgres user:
+More information: [fantoir-datasource README](fantoir-datasource/README.md)
-```
-CREATE ROLE fantoir WITH PASSWORD 'fantoir' LOGIN;
-CREATE DATABASE fantoir OWNER fantoir;
-```
+## Repository structure
-Connected as your database role, enable the pg_trgm extension
-to be able to generate the index for full-text search with trigrams:
+The repository is structured in subdirectories for components.
+It's a **monorepo** of tightly tied components to build our data pipelines.
-```
-CREATE EXTENSION pg_trgm;
-```
+To contribute to one of those components, simply clone this monorepo
+and send a pull request with a branch against like any other repository.
-If the extension doesn't exist, it can be included in a package
-named for example `postgresql-contrib`.
+To install only one component, you can use cargo. For example,
+`cargo install fantoir-datasource` will only install the
+`fantoir-datasource` binary.
-You can then use the code with the default DATABASE_URL documented above.
+To include a component in your own project, just include its name in Cargo.toml,
+crates.io and Cargo supports crates in Git subdirectories and this will only
+download and compile the needed component and its dependencies, ignoring others.
-### Database pitfalls
+There is no plan to export this monorepo in polyrepo/manyrepo as long as
+it contains only Rust code. We'd of course export Composer or npm packages,
+as it's a requirement of their respective packages managers.
-The FANTOIR database uses INSEE department code, they can contain a letter,
-currently only for Corse (2A and 2B).
-That also applies when building the INSEE commune code.
+## License
-If a record is canceled, the cancel date can be omitted.
-The creation date can be omitted too.
+Code is available under BSD-2-Clause license.
-The last line of the FANTOIR database must be ignored.
+Datasets imported by those tools are published under their own respective licenses.
-Wikidata uses the "code FANTOIR", matching the "code RIVOLI"
-documented in FANTOIR file description. This code matches the
-11 first characters of a record.
-See also https://www.wikidata.org/wiki/Property:P3182.
+## Notes
+
+### Interesting links
+
+ * [Documentation on Agora](https://agora.nasqueron.org/Nasqueron_Datasources)
+ * [Project board](https://devcentral.nasqueron.org/project/view/6/) for issues and features requests
+ * [How to contribute]()
+
+### Not to be confused with
+
+ * ***Nasqueron API datasources*** (rAPIS): exposes API for data
+ less easy to parse, see https://api.nasqueron.org/datasources/
+
+ * ***Nasqueron Databases*** (rDB): front-end for datasources and
+ other sources of databases, ie this datasources repository
+ prepares and enriches data than can then be used in https://db.nasqueron.org
+
+
+[1]: <https://data.economie.gouv.fr/explore/dataset/fichier-fantoir-des-voies-et-lieux-dits/information/> "FANTOIR sur data.economie.gouv.fr"
diff --git a/Cargo.toml b/fantoir-datasource/Cargo.toml
copy from Cargo.toml
copy to fantoir-datasource/Cargo.toml
diff --git a/README.md b/fantoir-datasource/README.md
copy from README.md
copy to fantoir-datasource/README.md
diff --git a/src/commands/import.rs b/fantoir-datasource/src/commands/import.rs
rename from src/commands/import.rs
rename to fantoir-datasource/src/commands/import.rs
diff --git a/src/commands/mod.rs b/fantoir-datasource/src/commands/mod.rs
rename from src/commands/mod.rs
rename to fantoir-datasource/src/commands/mod.rs
diff --git a/src/commands/promote/mod.rs b/fantoir-datasource/src/commands/promote/mod.rs
rename from src/commands/promote/mod.rs
rename to fantoir-datasource/src/commands/promote/mod.rs
diff --git a/src/commands/query.rs b/fantoir-datasource/src/commands/query.rs
rename from src/commands/query.rs
rename to fantoir-datasource/src/commands/query.rs
diff --git a/src/commands/wikidata/mod.rs b/fantoir-datasource/src/commands/wikidata/mod.rs
rename from src/commands/wikidata/mod.rs
rename to fantoir-datasource/src/commands/wikidata/mod.rs
diff --git a/src/commands/wikidata/qualification.rs b/fantoir-datasource/src/commands/wikidata/qualification.rs
rename from src/commands/wikidata/qualification.rs
rename to fantoir-datasource/src/commands/wikidata/qualification.rs
diff --git a/src/commands/wikidata/report.rs b/fantoir-datasource/src/commands/wikidata/report.rs
rename from src/commands/wikidata/report.rs
rename to fantoir-datasource/src/commands/wikidata/report.rs
diff --git a/src/db.rs b/fantoir-datasource/src/db.rs
rename from src/db.rs
rename to fantoir-datasource/src/db.rs
diff --git a/src/fantoir.rs b/fantoir-datasource/src/fantoir.rs
rename from src/fantoir.rs
rename to fantoir-datasource/src/fantoir.rs
diff --git a/src/main.rs b/fantoir-datasource/src/main.rs
rename from src/main.rs
rename to fantoir-datasource/src/main.rs
diff --git a/src/queries/wikidata.sparql b/fantoir-datasource/src/queries/wikidata.sparql
rename from src/queries/wikidata.sparql
rename to fantoir-datasource/src/queries/wikidata.sparql
diff --git a/src/schema/fantoir.sql b/fantoir-datasource/src/schema/fantoir.sql
rename from src/schema/fantoir.sql
rename to fantoir-datasource/src/schema/fantoir.sql
diff --git a/src/schema/promote/config.sql b/fantoir-datasource/src/schema/promote/config.sql
rename from src/schema/promote/config.sql
rename to fantoir-datasource/src/schema/promote/config.sql
diff --git a/src/schema/promote/fantoir_view.sql b/fantoir-datasource/src/schema/promote/fantoir_view.sql
rename from src/schema/promote/fantoir_view.sql
rename to fantoir-datasource/src/schema/promote/fantoir_view.sql
diff --git a/src/schema/promote/wikidata.sql b/fantoir-datasource/src/schema/promote/wikidata.sql
rename from src/schema/promote/wikidata.sql
rename to fantoir-datasource/src/schema/promote/wikidata.sql
diff --git a/src/schema/wikidata.sql b/fantoir-datasource/src/schema/wikidata.sql
rename from src/schema/wikidata.sql
rename to fantoir-datasource/src/schema/wikidata.sql
diff --git a/src/services/http_client.rs b/fantoir-datasource/src/services/http_client.rs
rename from src/services/http_client.rs
rename to fantoir-datasource/src/services/http_client.rs
diff --git a/src/services/mod.rs b/fantoir-datasource/src/services/mod.rs
rename from src/services/mod.rs
rename to fantoir-datasource/src/services/mod.rs
diff --git a/src/services/query.rs b/fantoir-datasource/src/services/query.rs
rename from src/services/query.rs
rename to fantoir-datasource/src/services/query.rs
diff --git a/src/services/sparql.rs b/fantoir-datasource/src/services/sparql.rs
rename from src/services/sparql.rs
rename to fantoir-datasource/src/services/sparql.rs

File Metadata

Mime Type
text/plain
Expires
Mon, Nov 18, 11:40 (7 m, 29 s)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
2250560
Default Alt Text
D2744.id6962.diff (8 KB)

Event Timeline

Nasqueron DevCentral · If it had been much bigger the moon would have had a core of ice. · Powered by Phabricator