Page MenuHomeDevCentral
Differential D1205

Deploy FANTOIR for Daeghrefn
ClosedPublic
Actions

Authored by dereckson on Nov 14 2017, 20:43.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Nov 3, 22:55
Unknown Object (File)
Sun, Oct 27, 13:27
Unknown Object (File)
Sat, Oct 26, 08:59
Unknown Object (File)
Thu, Oct 17, 22:31
Unknown Object (File)
Thu, Oct 17, 22:30
Unknown Object (File)
Thu, Oct 17, 22:30
Unknown Object (File)
Thu, Oct 17, 22:30
Unknown Object (File)
Thu, Oct 17, 22:30
View All Files
Subscribers
None

Details

Reviewers
dereckson
Maniphest Tasks
T1309: Deploy Dæghrefn, Wearg and TC2 through Salt
Commits
rOPS5a5814aad433: Deploy FANTOIR for Daeghrefn
Summary

FANTOIR is a database used by Daeghrefn for the .fantoir command.

Ref T1309

Data source:
https://www.data.gouv.fr/en/datasets/fichier-fantoir-des-voies-et-lieux-dits/

Test Plan

Test on Ysul, ensure data directory is populated,
and the streets are well extracted.

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

dereckson created this revision.Nov 14 2017, 20:43
Harbormaster completed remote builds in B1904: Diff 3086.Nov 14 2017, 20:43
dereckson updated this revision to Diff 3087.Nov 14 2017, 20:45

Fix header

Harbormaster completed remote builds in B1905: Diff 3087.Nov 14 2017, 20:45
dereckson added a comment.Nov 14 2017, 20:50

First run

Ysul
$ salt-call --local state.apply roles/viperserv/fantoir
local:                                 
----------                             
          ID: /srv/viperserv/data                                                                                                                              
    Function: file.directory           
      Result: True                                                                                                                                             
     Comment: Directory /srv/viperserv/data updated                            
     Started: 20:31:14.658705          
    Duration: 53.675 ms                
     Changes:                          
              ----------               
              /srv/viperserv/data:     
                  New Dir              
----------                             
          ID: /srv/viperserv/data/dist 
    Function: file.directory           
      Result: True                     
     Comment: Directory /srv/viperserv/data/dist updated                       
     Started: 20:31:14.712841          
    Duration: 8.497 ms                 
     Changes:                          
              ----------               
              /srv/viperserv/data/dist:
                  New Dir              
----------                                                                                                                                                     
          ID: /srv/viperserv/data/dist/fantoir.zip                             
    Function: file.managed                                                                                                                                     
      Result: True                     
     Comment: File /srv/viperserv/data/dist/fantoir.zip updated                
     Started: 20:31:14.721797                                                                                                                                  
    Duration: 6530.063 ms              
     Changes:                                                                  
              ----------                                                       
              diff:                    
                  New file             
              mode:
                  0644
----------
          ID: viperserv_fantoir_archive
    Function: archive.extracted        
        Name: /srv/viperserv/data      
      Result: True                     
     Comment: /srv/viperserv/data/dist/fantoir.zip extracted to /srv/viperserv/data/, due to absence of one or more files/dirs                                 
     Started: 20:31:21.257211          
    Duration: 23414.381 ms             
     Changes:                          
              ----------               
              extracted_files:         
                  - FANTOIR1017        
----------                             
          ID: /srv/viperserv/data/FANTOIR.txt                                  
    Function: file.symlink             
      Result: True                     
     Comment: Created new symlink /srv/viperserv/data/FANTOIR.txt -> /srv/viperserv/data/FANTOIR1017                                                           
     Started: 20:31:44.673497          
    Duration: 33.61 ms                 
     Changes:                          
              ----------               
              new:                     
                  /srv/viperserv/data/FANTOIR.txt                              
----------                             
          ID: viperserv_fantoir_streets
    Function: cmd.run                  
        Name: grep -a "^[0-9]" /srv/viperserv/data/FANTOIR.txt > /srv/viperserv/data/FANTOIR_STREETS.txt                                                       
      Result: True                     
     Comment: Command "grep -a "^[0-9]" /srv/viperserv/data/FANTOIR.txt > /srv/viperserv/data/FANTOIR_STREETS.txt" run                                         
     Started: 20:31:44.711547          
    Duration: 9684.872 ms              
     Changes:                          
              ----------               
              pid:                     
                  21345                
              retcode:                 
                  0                    
              stderr:                  
              stdout:                  

Summary for local                      
------------                           
Succeeded: 6 (changed=6)               
Failed:    0                           
------------                           
Total states run:     6                
Total run time:  39.725 s

Control

Ysul
$ cd /srv/viperserv/data
$ wc *.txt
 7964379 83397597 957122898 FANTOIR_STREETS.txt                                
 7964380 83397600 957122958 FANTOIR.txt                                       
 15928759 166795197 1914245856 total

It seems the code RIVOLI to check isn't the first character of the file, but the character at the 11th position.

See https://www.data.gouv.fr/s/resources/fichier-fantoir-des-voies-et-lieux-dits/community/20150512-103719/Descriptif_FANTOIR.pdf

Second run

Expected output: no-op

Ysul
$ salt-call --local state.apply roles/viperserv/fantoir
local:
----------
          ID: /srv/viperserv/data
    Function: file.directory
      Result: True
     Comment: Directory /srv/viperserv/data is in the correct state
              Directory /srv/viperserv/data updated
     Started: 20:48:55.940437
    Duration: 35.258 ms
     Changes:   
----------
          ID: /srv/viperserv/data/dist
    Function: file.directory
      Result: True
     Comment: Directory /srv/viperserv/data/dist is in the correct state
              Directory /srv/viperserv/data/dist updated
     Started: 20:48:55.976155
    Duration: 4.984 ms
     Changes:   
----------
          ID: /srv/viperserv/data/dist/fantoir.zip
    Function: file.managed
      Result: True
     Comment: File /srv/viperserv/data/dist/fantoir.zip is in the correct state
     Started: 20:48:55.981592
    Duration: 6574.492 ms
     Changes:   
----------
          ID: viperserv_fantoir_archive
    Function: archive.extracted
        Name: /srv/viperserv/data
      Result: True
     Comment: All files in archive are already present
     Started: 20:49:02.560550
    Duration: 10445.015 ms
     Changes:   
----------
          ID: /srv/viperserv/data/FANTOIR.txt
    Function: file.symlink
      Result: True
     Comment: Symlink /srv/viperserv/data/FANTOIR.txt is present and owned by root:wheel
     Started: 20:49:13.007468
    Duration: 234.98 ms
     Changes:   
----------
          ID: viperserv_fantoir_streets
    Function: cmd.run
        Name: grep -a "^[0-9]" /srv/viperserv/data/FANTOIR.txt > /srv/viperserv/data/FANTOIR_STREETS.txt
      Result: True
     Comment: /srv/viperserv/data/FANTOIR_STREETS.txt exists
     Started: 20:49:13.246960
    Duration: 1.837 ms
     Changes:   

Summary for local
------------
Succeeded: 6
Failed:    0
------------
Total states run:     6
Total run time:  17.297 s
dereckson updated this revision to Diff 3088.Nov 14 2017, 20:54

User/group for files operations

dereckson updated this revision to Diff 3089.Nov 14 2017, 21:17

Use 109th character to determine if it's or not a 'voie'.

dereckson updated this revision to Diff 3090.Nov 14 2017, 21:58

Switch to a Python script to extract

Harbormaster completed remote builds in B1908: Diff 3090.Nov 14 2017, 21:58
dereckson planned changes to this revision.Nov 14 2017, 22:00
dereckson edited the summary of this revision. (Show Details)
dereckson edited the test plan for this revision. (Show Details)
dereckson added a task: T1309: Deploy Dæghrefn, Wearg and TC2 through Salt.
dereckson added inline comments.
roles/viperserv/fantoir/files/extract_streets.py
31

stderr

dereckson updated this revision to Diff 3091.Nov 14 2017, 22:04

stderr

Harbormaster completed remote builds in B1909: Diff 3091.Nov 14 2017, 22:04
dereckson accepted this revision.EditedNov 14 2017, 22:05
dereckson marked an inline comment as done.
$ ls -lah /srv/viperserv/data                                                                             (git)-[fantoir] 
total 604886
drwxrwx---  3 viperserv  nasqueron-irc     6B Nov 14 21:55 .
drwxrwx---  7 viperserv  nasqueron-irc     7B Nov 14 20:31 ..
drwxrwx---  2 viperserv  nasqueron-irc     3B Nov 14 20:31 dist
-rw-r--r--  1 viperserv  nasqueron-irc   197M Nov 14 21:55 FANTOIR_STREETS.txt
lrwxr-xr-x  1 viperserv  nasqueron-irc    31B Nov 14 22:05 FANTOIR.txt -> /srv/viperserv/data/FANTOIR1017
-rw-r--r--  1 viperserv  nasqueron-irc   913M Nov 14 20:31 FANTOIR1017
This revision is now accepted and ready to land.Nov 14 2017, 22:05
Closed by commit rOPS5a5814aad433: Deploy FANTOIR for Daeghrefn (authored by dereckson). · Explain WhyNov 14 2017, 22:06
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
pillar/
1 line
viperserv/
12 lines
roles/
viperserv/
fantoir/
files/
34 lines
69 lines
1 line

Diff 3092

View Options

pillar/top.sls

Loading...
View Options

pillar/viperserv/fantoir.sls

Loading...
View Options

roles/viperserv/fantoir/files/extract_streets.py

Loading...
View Options

roles/viperserv/fantoir/init.sls

Loading...
View Options

roles/viperserv/init.sls

Loading...
Nasqueron DevCentral · If it had been much bigger the moon would have had a core of ice. · Powered by Phabricator