Page MenuHomeDevCentral
Feed Advanced Search

Oct 20 2017

dereckson renamed T1272: New Salt deployment hierarchy from New deployment hierarchy to New Salt deployment hierarchy.
Oct 20 2017, 11:18 · Operations sprints (The Dreadnought will produce new officers), Salt, Servers
dereckson created T1272: New Salt deployment hierarchy.
Oct 20 2017, 11:18 · Operations sprints (The Dreadnought will produce new officers), Salt, Servers
dereckson added a project to T644: Setup an etcd K/V store: Operations sprints (The Dreadnought will produce new officers).
Oct 20 2017, 10:53 · Operations sprints (The Dreadnought will produce new officers), Nasqueron Docker deployment squad, Servers
dereckson reopened T644: Setup an etcd K/V store, a subtask of T645: Redeploy Shipyard on Dwellers, as Open.
Oct 20 2017, 10:53 · Product evaluation, Servers, Nasqueron Docker deployment squad
dereckson reopened T644: Setup an etcd K/V store as "Open".

Reopening, as it doesn't run anymore and we lost the container.

Oct 20 2017, 10:53 · Operations sprints (The Dreadnought will produce new officers), Nasqueron Docker deployment squad, Servers
dereckson closed T1269: Update SSHFP records for ysul as Resolved.
Client
$ ssh -4 ysul.nasqueron.org
The authenticity of host 'ysul.nasqueron.org (212.83.187.132)' can't be established.
ECDSA key fingerprint is SHA256:qzQTi0xcuEYHRr1GV4+xqHT6jdIM3iFR8YNK9QIm7e4.
Matching host key fingerprint found in DNS.
ECDSA key fingerprint is MD5:38:38:07:f8:e5:38:82:47:37:ef:9a:f3:bf:a5:b0:7b.
Matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?
Oct 20 2017, 01:19 · DNS, Operations sprints (The Dreadnought will produce new officers), Servers
dereckson closed T1269: Update SSHFP records for ysul, a subtask of T1268: Reprovision Ysul, as Resolved.
Oct 20 2017, 01:19 · Operations sprints (The Dreadnought will produce new officers), Salt, Servers
dereckson added a comment to T1269: Update SSHFP records for ysul.
Ysul
$ ssh-keygen -r ysul.nasqueron.org
ysul.nasqueron.org IN SSHFP 1 1 d1b0a408bbaa22b342464d851c15857746fcd0a7
ysul.nasqueron.org IN SSHFP 1 2 4efb284579508b1ad53000aca96e3104c2c8582fd0aa6ec5a7b767e9f3fa969d
ysul.nasqueron.org IN SSHFP 3 1 085356d6ad1d34731fbdccb137a7faec170ddfe6
ysul.nasqueron.org IN SSHFP 3 2 ab34138b4c5cb8460746bd46578fb1a874fa8dd20cde2151f1834af50226edee
ysul.nasqueron.org IN SSHFP 4 1 4740d85fc1161fea53974049e7df10b28abf5c6d
ysul.nasqueron.org IN SSHFP 4 2 26653b23045c60bbcfea0948227898805d7cc280cf81e4189e87a9b5e1370925
Oct 20 2017, 01:11 · DNS, Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a revision to T1268: Reprovision Ysul: D1070: Provision software to Ysul.
Oct 20 2017, 01:09 · Operations sprints (The Dreadnought will produce new officers), Salt, Servers
dereckson created T1269: Update SSHFP records for ysul.
Oct 20 2017, 00:50 · DNS, Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a project to T1268: Reprovision Ysul: Operations sprints (The Dreadnought will produce new officers).
Oct 20 2017, 00:49 · Operations sprints (The Dreadnought will produce new officers), Salt, Servers
dereckson updated the task description for T1268: Reprovision Ysul.
Oct 20 2017, 00:49 · Operations sprints (The Dreadnought will produce new officers), Salt, Servers
dereckson added a comment to T1247: Bring back Ysul online.

Plan is to reprovision a baremetal server and redirect the Ysul IP there: T1268

Oct 20 2017, 00:48 · Servers
dereckson created T1268: Reprovision Ysul.
Oct 20 2017, 00:48 · Operations sprints (The Dreadnought will produce new officers), Salt, Servers

Oct 19 2017

dereckson added a subtask for T1265: Provision equatower, new Docker engine as a containerized OS: T1266: Upgrade Mastodon to last 1.x version on Equatower.
Oct 19 2017, 20:06 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a comment to T1265: Provision equatower, new Docker engine as a containerized OS.

Equatower has been provisionned as a CentOS Atomic, with git, salt-client and docker-compose added to the base system.

Oct 19 2017, 19:19 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson created T1265: Provision equatower, new Docker engine as a containerized OS.
Oct 19 2017, 19:18 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a revision to T1034: Switch from root@ SSH login to sudo capabilities on every server: D1068: Drop wheel group configuration.
Oct 19 2017, 16:54 · Operations sprints (Consolidate them all), Servers, Eglide
dereckson closed T1056: Rename wheel group in gshadow too, not only on /etc/group as Wontfix.

Actually, we're going to create an 'ops' group instead of fighting the OS.

Oct 19 2017, 16:50 · Salt, Servers
dereckson added a revision to T1034: Switch from root@ SSH login to sudo capabilities on every server: D1067: Create an ops group with full sudo capability.
Oct 19 2017, 16:47 · Operations sprints (Consolidate them all), Servers, Eglide
dereckson claimed T1034: Switch from root@ SSH login to sudo capabilities on every server.
Oct 19 2017, 16:25 · Operations sprints (Consolidate them all), Servers, Eglide
dereckson closed T1247: Bring back Ysul online as Wontfix.

The hypervisor machine is broken, we received a new replacement machine.

Oct 19 2017, 14:35 · Servers

Oct 18 2017

dereckson raised the priority of T1247: Bring back Ysul online from High to Unbreak Now!.

Hypervisor currently unreachable by ping.

Oct 18 2017, 10:41 · Servers

Oct 17 2017

dereckson closed T1259: Don't provision software in core role as Resolved by committing rOPSfcc8044682f4: Don't install software in the core role.
Oct 17 2017, 19:29 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a revision to T1259: Don't provision software in core role: D1064: Don't install software in the core role.
Oct 17 2017, 18:58 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson edited projects for T1259: Don't provision software in core role, added: Operations sprints (The Dreadnought will produce new officers); removed Operations sprints (Operations sprint 1).
Oct 17 2017, 18:56 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a project to T1259: Don't provision software in core role: Operations sprints (Operations sprint 1).
Oct 17 2017, 18:56 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a parent task for T1259: Don't provision software in core role: T1260: Handle containers OS in Salt.
Oct 17 2017, 18:55 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a comment to T1259: Don't provision software in core role.

That means the Let's encrypt unit has to be moved to a shared role folder.

Oct 17 2017, 18:48 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson created T1259: Don't provision software in core role.
Oct 17 2017, 18:39 · Operations sprints (The Dreadnought will produce new officers), Servers

Oct 15 2017

dereckson lowered the priority of T1247: Bring back Ysul online from Unbreak Now! to High.

[ Decreasing priority as it's up for 60 hours. ]

Oct 15 2017, 23:07 · Servers
dereckson renamed T1257: Unit tests for bash script get-containers-list from Unit test for D1050 to Unit tests for bash script get-containers-list.
Oct 15 2017, 21:49 · good-first-issue, Servers
dereckson created T1257: Unit tests for bash script get-containers-list.
Oct 15 2017, 21:48 · good-first-issue, Servers
dereckson closed T1250: get-containers-list --reverse is shifted as Resolved by committing rOPS348e4a75856a: Fix get-containers-list --reverse shift.
Oct 15 2017, 21:44 · Servers, Nasqueron Docker deployment squad
dereckson moved T1067: libcurl.so wants libssl.so.8, but we provide libssl.so.9 on Ysul from Backlog to Analysis / under discussion on the Servers board.

I don't see what we can do about this one.

Oct 15 2017, 12:35 · Servers
dereckson added a comment to T1254: Ensure non-Nasqueron domains DNS don't contain A records to old Dwellers IP.

By the way, as shown at P263, there are still some DNS records too for mail. and lists.nasqueron.org

Oct 15 2017, 12:30 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a project to T1239: FreeBSD port for sphinx-autobuild: freebsd-port-wanted.
Oct 15 2017, 12:27 · freebsd-port-wanted, Servers
dereckson updated the task description for T761: Automate to create subdomains DNS records.
Oct 15 2017, 12:27 · DNS, Servers
dereckson closed T1200: Can't currently build images with Docker compose on Dwellers as Invalid.
Oct 15 2017, 12:26 · Servers
dereckson closed T1195: Install Salt on Dwellers as Resolved.
Oct 15 2017, 12:26 · Servers
dereckson added a project to T1119: Install jid on Ysul: freebsd-port-wanted.
Oct 15 2017, 12:24 · Servers
dereckson moved T1026: Rename deploy account on Ysul and Eglide from Backlog to Ops on the good-first-issue board.
Oct 15 2017, 12:23 · Salt, good-first-issue, Servers
dereckson renamed T1026: Rename deploy account on Ysul and Eglide from Rename deploy account on Ysul to Rename deploy account on Ysul and Eglide.
Oct 15 2017, 12:23 · Salt, good-first-issue, Servers
dereckson added a project to T861: Create a Debian package with extra FIGlet fonts: debian-package-needed.
Oct 15 2017, 12:21 · debian-package-needed, Servers
dereckson added a project to T758: Port for autochmod scripts: freebsd-port-wanted.
Oct 15 2017, 12:20 · freebsd-port-wanted, Servers
dereckson added a comment to T761: Automate to create subdomains DNS records.

@amj you wanted CLI tools automating this like ./new-jail-dns-record awesome-software or just a plain BIND/djbdns/whatever instance?

Oct 15 2017, 12:19 · DNS, Servers
dereckson added a project to T761: Automate to create subdomains DNS records: DNS.
Oct 15 2017, 12:18 · DNS, Servers
dereckson closed T760: Create NAT to avoid to steal port with jails as Wontfix.

Closing former Ysul issues before server decom/migration.

Oct 15 2017, 12:17 · Servers
dereckson closed T852: Investigate matrix.nasqueron.org as Resolved.

Not interest among Nasqueron dwellers those last two years.

Oct 15 2017, 12:09 · Product evaluation, Nasqueron Docker deployment squad, Servers
dereckson created P263 DNS zone for nasqueron.org.
Oct 15 2017, 12:07 · DNS, Servers
dereckson added a comment to T852: Investigate matrix.nasqueron.org.

DNS entry should be removed per T536 close as WONTFIX.

Oct 15 2017, 12:05 · Product evaluation, Nasqueron Docker deployment squad, Servers
dereckson closed T536: Install a demo Matrix server on Dwellers as Wontfix.

Matrix experience at HSBXL seems this software lacks of traction, and can create issues with IRC users with the current state of the gateways.

Oct 15 2017, 12:04 · Nasqueron Docker deployment squad, Servers
dereckson moved T853: Deploy a Let's encrypt certificate to the Mumble server from Backlog to Ops on the good-first-issue board.
Oct 15 2017, 12:02 · TLS certificates, good-first-issue, Mumble, security, Servers
dereckson updated the task description for T1255: Create a temporary API to receive server logs.
Oct 15 2017, 11:38 · Operations sprints (The Dreadnought will produce new officers), Servers, Nasqueron API
dereckson moved T1255: Create a temporary API to receive server logs from Backlog to Working on on the Operations sprints (The Dreadnought will produce new officers) board.
Oct 15 2017, 09:19 · Operations sprints (The Dreadnought will produce new officers), Servers, Nasqueron API
dereckson moved T1215: Provision a new hypervisor from Backlog to Working on on the Operations sprints (The Dreadnought will produce new officers) board.
Oct 15 2017, 09:18 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a project to T1246: A FreeBSD VM can't see the virtual disks on Dreadnought: Operations sprints (The Dreadnought will produce new officers).
Oct 15 2017, 09:14 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a project to T1216: Purchase a new dedicated server for an hypervisor: Operations sprints (The Dreadnought will produce new officers).
Oct 15 2017, 09:14 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson claimed T1215: Provision a new hypervisor.
Oct 15 2017, 09:14 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a project to T1244: Saltize default iptables for Dwellers: Operations sprints (The Dreadnought will produce new officers).
Oct 15 2017, 09:13 · Operations sprints (Consolidate them all), Servers
dereckson added a project to T1254: Ensure non-Nasqueron domains DNS don't contain A records to old Dwellers IP: Operations sprints (The Dreadnought will produce new officers).
Oct 15 2017, 09:13 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a project to T1255: Create a temporary API to receive server logs: Operations sprints (The Dreadnought will produce new officers).
Oct 15 2017, 09:13 · Operations sprints (The Dreadnought will produce new officers), Servers, Nasqueron API
dereckson moved T1237: Upgrade Ysul to FreeBSD 11.1 from Backlog to Analysis / under discussion on the Servers board.

Blocked by Ysul strategy to determine.

Oct 15 2017, 09:02 · Servers
dereckson triaged T1255: Create a temporary API to receive server logs as High priority.
Oct 15 2017, 07:56 · Operations sprints (The Dreadnought will produce new officers), Servers, Nasqueron API
dereckson added a subtask for T1041: Setup infra.nasqueron.org: T1255: Create a temporary API to receive server logs.
Oct 15 2017, 07:24 · Servers

Oct 14 2017

dereckson claimed T1254: Ensure non-Nasqueron domains DNS don't contain A records to old Dwellers IP.
Oct 14 2017, 20:00 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a comment to T1245: Investigate the Stormshear issue.

00:05:29 < philectro> [philectro@ysul ~]$ dd bs=1m count=200 if=/dev/zero of=test
00:05:29 < philectro> 200+0 records in
00:05:29 < philectro> 200+0 records out
00:05:29 < philectro> 209715200 bytes transferred in 13.754970 secs (15246503 bytes/sec)

Oct 14 2017, 00:09 · Servers

Oct 13 2017

dereckson merged T804: Provision a new hypervisor into T1215: Provision a new hypervisor.
Oct 13 2017, 22:46 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson merged task T804: Provision a new hypervisor into T1215: Provision a new hypervisor.
Oct 13 2017, 22:46 · Servers
dereckson added a comment to T1245: Investigate the Stormshear issue.

Hard drive is fine according @ledesillusionniste:

Oct 13 2017, 22:12 · Servers
dereckson closed T1252: Compile statically smartmontools as Invalid.

Not needed, @ledesillusionniste found it at http://www.virten.net/2016/05/determine-tbw-from-ssds-with-s-m-a-r-t-values-in-esxi-smartctl/

Oct 13 2017, 22:11 · Servers
dereckson closed T1252: Compile statically smartmontools, a subtask of T1245: Investigate the Stormshear issue, as Invalid.
Oct 13 2017, 22:11 · Servers
dereckson created P262 smartctl on StormShear.
Oct 13 2017, 22:10 · Servers
dereckson added a comment to T1252: Compile statically smartmontools.

Build process

Oct 13 2017, 21:31 · Servers
dereckson added a comment to T1245: Investigate the Stormshear issue.

Meanwhile, at 10:30 UTC, Ysul froze, hypervisor performance graphs simply showed a stop of activity.

Oct 13 2017, 12:04 · Servers
dereckson added a comment to T1245: Investigate the Stormshear issue.

@ledesillusionniste finds https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146233 about the temperature issue: if we upgrade EXSi, we can get a correct reading.

Oct 13 2017, 12:00 · Servers

Oct 12 2017

dereckson created T1252: Compile statically smartmontools.
Oct 12 2017, 23:23 · Servers
dereckson added a comment to T1245: Investigate the Stormshear issue.
Stormshear
$  esxcli storage core device smart get -d t10.ATA_____HGST_HTS721010A9E630__________________________JR10006PH7JXKE
Parameter                     Value  Threshold  Worst
----------------------------  -----  ---------  -----
Health Status                 OK     N/A        N/A  
Media Wearout Indicator       N/A    N/A        N/A  
Write Error Count             N/A    N/A        N/A  
Read Error Count              100    62         100  
Power-on Hours                32     0          32   
Power Cycle Count             100    0          100  
Reallocated Sector Count      100    5          100  
Raw Read Error Rate           100    62         100  
Drive Temperature             206    0          206  
Driver Rated Max Temperature  N/A    N/A        N/A  
Write Sectors TOT Count       200    0          200  
Read Sectors TOT Count        N/A    N/A        N/A  
Initial Bad Block Count       N/A    N/A        N/A  
Oct 12 2017, 22:54 · Servers
dereckson created T1251: Create an hardware group with sudo permission to run SMART and other hardware checking tools.
Oct 12 2017, 22:06 · Operations sprints (Echoes in the Void), Nasqueron Operations Squad, Servers
dereckson updated subscribers of T1245: Investigate the Stormshear issue.

@ledesillusionniste is checking SMART status, and a dd test to measure I/O.

Oct 12 2017, 22:01 · Servers
dereckson moved T1250: get-containers-list --reverse is shifted from Backlog to Pending review on the Servers board.
Oct 12 2017, 16:38 · Servers, Nasqueron Docker deployment squad
dereckson moved T1250: get-containers-list --reverse is shifted from Backlog to Pending review on the Nasqueron Docker deployment squad board.
Oct 12 2017, 16:38 · Servers, Nasqueron Docker deployment squad
dereckson added a revision to T1250: get-containers-list --reverse is shifted: D1050: Fix get-containers-list --reverse shift.
Oct 12 2017, 16:37 · Servers, Nasqueron Docker deployment squad
dereckson added projects to T1250: get-containers-list --reverse is shifted: Nasqueron Docker deployment squad, Servers.
Oct 12 2017, 16:37 · Servers, Nasqueron Docker deployment squad
dereckson closed T1047: Install oidentd on Eglide as Resolved.
Oct 12 2017, 16:29 · User-Dereckson, Salt, IRC, Eglide, Servers
dereckson added a comment to T1047: Install oidentd on Eglide.

Works now, probably a bug fixed during an upgrade :)

Oct 12 2017, 16:29 · User-Dereckson, Salt, IRC, Eglide, Servers
dereckson updated the task description for T1247: Bring back Ysul online.
Oct 12 2017, 15:42 · Servers
dereckson added subtasks for T1247: Bring back Ysul online: T1246: A FreeBSD VM can't see the virtual disks on Dreadnought, T1245: Investigate the Stormshear issue.
Oct 12 2017, 15:40 · Servers
dereckson added a parent task for T1245: Investigate the Stormshear issue: T1247: Bring back Ysul online.
Oct 12 2017, 15:40 · Servers
dereckson added a parent task for T1246: A FreeBSD VM can't see the virtual disks on Dreadnought: T1247: Bring back Ysul online.
Oct 12 2017, 15:40 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson created T1247: Bring back Ysul online.
Oct 12 2017, 15:39 · Servers
dereckson created T1246: A FreeBSD VM can't see the virtual disks on Dreadnought.
Oct 12 2017, 15:39 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson updated the task description for T1245: Investigate the Stormshear issue.
Oct 12 2017, 15:36 · Servers
dereckson updated the task description for T1245: Investigate the Stormshear issue.
Oct 12 2017, 15:36 · Servers
dereckson created T1245: Investigate the Stormshear issue.
Oct 12 2017, 15:34 · Servers
dereckson added a comment to T1215: Provision a new hypervisor.

So the current IP block is reserved like this:

Oct 12 2017, 15:31 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson added a comment to T1215: Provision a new hypervisor.

Dwellers have been moved here, with our dev infrastructure and Mastodon, but not Jenkins :

Oct 12 2017, 15:29 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson closed T1216: Purchase a new dedicated server for an hypervisor as Resolved.

We've ordered a new hypervisor this 2017-10-12 during the night at OVH, the OP-SAT-1-32, with a first block of 4 IPs, 51.255.124.11/30.

Oct 12 2017, 15:27 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson closed T1216: Purchase a new dedicated server for an hypervisor, a subtask of T1215: Provision a new hypervisor, as Resolved.
Oct 12 2017, 15:27 · Operations sprints (The Dreadnought will produce new officers), Servers
dereckson claimed T1244: Saltize default iptables for Dwellers.
Oct 12 2017, 15:10 · Operations sprints (Consolidate them all), Servers