Like 2020, 2021 was again a "special year" in pandemic terms. How to cope? Like Paul Ramsey writes: I feel like sharing some of my favourite things . Will do in a later post: Bird-watching , Wolf-tracking workshop , 'catching' big game on new wildcam, developed outdoor GPS-game (Veluwe Big Five), some progress on guitar-playing, discovering The Netherlands outback, becoming a Billy Strings fan. These are a few of my favourite things, more later....
TL;DR Main three 2021 highlights:
- Organizing and attending an in-real-life FOSS4G-NL with OSGeo.nl
- Attending, presenting, moderating, online FOSS4G Buenos Aires
- Deep-diving into GitOps .
Below follow 2021-highlights organized by theme and/or project.
1. Fun Home Automation Projects
Some of my 2022 fun resolutions resolved in 2021:
- Installed an HP1000SE PRO Wi-Fi Internet Wireless Weather Station . Using weewx on Raspberry Pi . Publishing a.o. to Weather Underground .
More on these later.
2. New Website
For years my website/blog justobjects.nl ran on Wordpress at the famous, and first (est. 1993), Dutch internet/hosting provider XS4ALL . XS4ALL was sold to KPN (1998), remained an independent brand, but was finally dismantled in 2021.
So in early 2021 a new hosting provider was needed. This was also a good moment to migrate my website to Hugo . With the great effort by @peter-govind (via freelance.com ), we converted the complete site (about 20 years of blogs) to Hugo+Github , using GitHub Workflows to "publish-on-commit". My first piece of GitOps, more to follow, in 2021. Though there are still some fixes to do, very happy with the result. Maintaining is now a breeze.
Gradually GitOps became my main theme in 2021. Discovered that there was actually a name (GitOps) for CI/CD practices I (you) have been doing over the past years. For example, in 2019 I worked on the pygeoapi demo server . GitOps can always be taken further until reaching the phase: "look ma, no hands!". No more "Box Hugging", i.e. no more SSH-ing, manual actions on VMs etc.
"GitOps is a set of practices to manage infrastructure and application configurations using Git, an open source version control system. GitOps works by using Git as a single source of truth for declarative infrastructure and applications."
I often quote here: "The Truth is in Git!". Especially now in times of fake-news, I love speaking those words in presentations. The quest of humanity is over: We Found The Truth, It Is In Git :-). Pun aside, it takes effort to have 100% of your deployment sources and configs in Git. In particular all required credentials (passwords, keys etc). Think of storing credentials in a public GitHub repo, how? Did some amazing discoveries in that area like (combining) GitHub Secrets and Ansible Vault .
I was very lucky that my main three contracts in 2021 involved GitOps. One project sort of followed the other.
The challenge was to provide a testbed environment where third parties could further extend and experiment. Eventually we developed a deployment fully based on GitOps within GitHub as a GitHub Template repo. You can read details in the documentation . The picture below depicts the GitOps workflow:
These results were presented at several occasions in 2021. See the Presentations below.
For GeoCat (again with Paul van Genuchten), exploring and setting out how to further migrate to full GitOps for the GeoCat Live environment.
European Commission - Joint Research Centre
Was already working on a contract with EC-JRC for pan-EU data sharing as a member of the "EU JRC ELISE Pool of Experts". An honour to be (still) working with all these great minds from the international geospatial world!
My focus within the team was/is on data services deployment. My pitch and first deliverable, was called "The Power of Orchestrated Containerisation for Data Services".
The full outcome of the project will be published (publicly) as a JRC Technical Report (JRC127730?) titled: "Emerging approaches for data-driven innovation in Europe". For this report, wrote the chapter "Deployment of an OGC API Features web service using GitOps"
Apart from reporting in text, also provided a full-stack OGC API Features service based on pygeoapi . Using the Geonovum OGC API GitHub Template on a fresh Ubuntu VM provided by JRC, this was a great Proof-of-Concept: took just 2 hours to get the complete stack, including stuff like SSL-certificates, Traefik proxy (love Traefik!), OS-hardening etc, up and running . For now the GitHub repo is at github.com/justb4/ogc-api-jrc/ .
FOSS4G-NL 2021 - IRL
Yes, it happened! In between COVID-Waves a LOC lead by the great Erik Meerburg achieved to organize an in-real-life (IRL) 4th OSGeo.nl FOSS4G-NL conference . Abiding with COVID-regulations at the time. At ITC Enschede on October 19-20 2021. Using 2G access. Zero COVID-cases after.
One day of Workshops followed by a one-day conference with great international keynotes by Amelie A Gagnon from UNESCO-IIEP and Kurt Menke from Septima DK , plus three parallel sessions. We are still preparing the website to publish all presentations.
Looks unreal this group photo, but this was in 2021! Thanks to all who made this event a success!
The Big Geo Show
In 2021 we did six shows, each a Special:
- Episode 14: Newyear with QGIS-NL and OpenStreetMap-NL communities .
- Episode 15: OpenStreetMap Special . Special guest: Ilya Zverev a.k.a. Zverik !
- Episode 16: GIS with your Keyboard
- Episode 17: On Geospatial Databases
- Episode 18: Cartography Special
- Episode 19: Ten Years of OSGeo.nl - Story in Pictures
OSGeo.nl - Ten Year Anniversary
OSGeo.nl was established on Dec 2, 2011. So 10 years, still going strong! On dec 2, 2021 we were in a new wave, so a small online event. Below the epic picture (by Arnulf Christl) taken at the GIN Congres Utrecht 2011 where we got together to start OSGeo.nl.
A special episode of the Big Geo Show presents: Ten Years of OSGeo.nl - Story in Pictures .
5. Geospatial Cloud Services
Moved into providing Geospatial Cloud Services last few years, both as a source of income and to support/strengthen underlying open source projects with which they are developed. Warning: shameless ads below. All below was ongoing in 2021.
- map5.nl is a subscription service for Dutch topographic, historical- and embellished hill-shade and arial maps I started to host in 2015. In 2021 work started for a map5.nl user portal.
- GeoQoS.com is a hosted GeoHealthCheck (GHC) service on a subscription basis. GHC is an uptime and QoS monitor for (OGC) web services. Customers get their own GHC instance. GeoQoS.com saves the burden of self-hosting GHC. In 2021 started work on a GHC API, using FastAPI , and a new GHC-UI.
- geotoko.nl , in short GeoToko, is basically a webshop to download Dutch open geo-datasets. One may ask: Dutch geospatial data like Topography, Addresses and Buildings, is already open and publicly available, mainly via Kadaster-PDOK . So why bother reselling? Explanation follows.
Most Dutch Open geospatial datasets, available from governmental institutions like Kadaster, are provided in a neutral exchange-format. In practice: GML (Application Schema). Most users are not able to directly digest complex GML in their applications (or don't want to spend time on that). For over 10 years we run the Open Source project NLExtract to convert these datasets into formats like PostGIS and CSV that can be directly used in applications. NLExtract itself builds on Stetl , an Open Source geospatial ETL-library in Python. NLExtract/Stetl is used a lot, but still will require users to install/maintain an NLExtract/Stetl installation, download datasets, run the ETL, check results etc. That may also be a bridge too far, if e.g. all a customer needs is a one-time CSV of say, all 10 million addresses in The Netherlands. So GeoToko taps into this niche, providing ready-to-use, often enriched Dutch datasets. Pricing is reasonable, compared to other providers, plus organizations and individuals may get highly reduced pricing when they work on Open data applications like OpenStreetMap.
As for development: the GeoToko webshop was developed with Django, Flask/nginx (Download management), Stripe (backend payments) and CKAN . The latter to provide a product-catalogue, sample data and metadata in general, all via GeoCatalogus.nl .
Further development concerned automating the NLExtract/Stetl ETL-processes. For this a framework called GeoFabriek, "GeoFactory", was developed. This automates the entire chain from checking new dataset versions at Kadaster, through downloading, conversions, packing for download and updating the metadata (in GeoCatalogus.nl).
6. Open Source Contributions
Continuous work as a contributor. Apart from some GitLab Projects, you can find/follow me best on GitHub .
New in 2021:
- FastAPI-Users starter Template repo - adding full user management to FastAPI.
- NLExtract - added ETL for BAG version 2, my first PRs for GDAL!
Ongoing in 2021:
- Stetl - Geospatial ETL in Python, maintaining since 2011.
- GeoHealthCheck - Service Status and QoS Checker for OGC Web Services.
- pygeoapi - a Python server implementation of the OGC API suite of standards - joined this great project in 2019. Also in PSC.
- pygeoapi demo server - provided the (auto-)deployment stack for the pygeoapi demo server (code at GitHub ). Added COVID-19 NL data provider .
- NLExtract - ETL for Dutch geospatial datasets.
- Wegue - geo-webclient framework based on Vue.js with OpenLayers started by Christian Mayer . Joined this great project in 2019. Unfortunately hardly contribs in 2021.
- Heron MC - Web Mapping Client based on GeoExt and OpenLayers. Yes, old tech but still in wide use. In time hope to migrate to Wegue (see above). Migrating Heron to Wegue, see first example below.
More Contributions - Handy Docker Images
To support many of the Cloud services and Open Source projects, I develop and maintain handy Docker Images , also available from my DockerHub . Ongoing in 2021. In progress of moving all to Github Workflows build/push, replacing DockerHub Autobuild (still awaiting DockerHub Open Source waiver...):
- docker-awstats - AWStats in Docker, oldie, but very effective webstats. Deploy multiple instances in single Docker container. Highly configurable, e.g. also for Traefik access logs.
- docker-jmeter - Apache JMeter wrapped in Docker. Over 10 million pulls at end of 2021. And 180 stars, but still zero donations ! C'mon people, not even using JMeter myself...
- docker-mapfish-print - for MapFish Print version 3. MapFish Print allows printing maps as PDFs.
- docker-mapfish-print2 - for MapFish Print version 2 - This version is still used in quite some contexts, at least for Heron and KadViewer .
- docker-rclone - Docker image to perform a rclone sync based on a cron schedule, with healthchecks.io monitoring.
- docker-pgbackup - automated/scheduled PostgreSQL/PostGIS backups for all PostgreSQL-based Docker Containers in its Docker-network. I think it is nifty: just run and forget: your Postgres backups are taken care of. Inspired by: https://github.com/kartoza/docker-pg-backup.
- docker-mapserver - Slim Docker Image for MapServer with Lighttpd FastCGI.
- docker-mapproxy - MapProxy Docker Image adapted, slimmed, from the YAGA Development-Team . Awaiting PR merge .
- docker-mapproxy-mapserver - Docker Image for MapProxy service with built-in MapServer binaries accessed directly (no MapServer service).
- docker-cron - Runs Unix cron, includes docker (compose) client for running remote Docker Images.
7. Conferences - Attended
- Feb 6-7 FOSSDEM 2021 - Virtual
- Sept 27- Oct 22 - FOSS4G Buenos Aires - Virtual
- Oct 19-20 - FOSS4G-NL at ITC Enschede - In Real Life
FOSS4G Buenos Aires needs special mention. As an online conference, apart from amazing content, I was very impressed with the technical and organizational setup. Even the social events were fun to join. Folks are tired of Zoom, Teams, etc but the FOSS4G LOC had introduced tools like Work Adventure and StreamYard (also used by OSgeo.nl in Big Geo Show) to enable more "cozyness". Offcourse nothing can replace a real event, but this came close. The Gala Dinner was really fun!
Jody Garnett words it very well in his Tweet:
Never forget, always in our hearts: Malena Libman (1983-2021). FOSS4G 2021 was also a conference dedicated to, and in memory of, Malena Libman. She was a pillar of the FOSS4G community and a leading organizer of this conference.
Malena was the right and truthful winner of the Sol Katz Award 2021. As the OSGeo.org website states : "Malena Libman is the Sol Katz Award winner 2021. The prize was presented at the FOSS4G closing session and handed over to her friends & family who accepted the award on her behalf since we sadly lost Malena in April 2021 to COVID-19."
8. Talks & Workshops - Provided
Below talks and workshops I provided in 2021. A complete list of presentations is available.
"FOSS4G-NL 2021 - Closing Words" - FOSS4G-NL 2021 - Oktober 20, 2021 - Enschede - [PDF Slides] .
"Zo gemakkelijk kun je geo-webdiensten uitrollen!" - FOSS4G-NL 2021 - Oktober 20, 2021 - Enschede - On GitOps for OSGeo Software - [PDF Slides] .
"Doing Geospatial in Python" - FOSS4G 2021 - September 28, 2021 - Workshop (4h): introduction to performing common GIS/geospatial tasks using Python geospatial tools such as OWSLib, Shapely, Fiona/Rasterio, and common geospatial libraries like GDAL, PROJ, pycsw, as well as other tools from the geopython toolchain. - [HTML Startpage] - [Abstract] .
"Geonovum OGC API Testbed" - DiS Online: OGC en toegankelijke APIs - September 7, 2021 - [Event en PDF Slides] .
"Read BAG (Dutch Buildings & Addresses dataset) with Docker in 5 minutes" - 2021-04-08 - De Grote Geo Show, a live-streaming webshow by OSGeo.nl. Ep. 16 "GIS met je Toetsenbord". Aired on april 8, 2021. Video recording on YouTube .
"OSGeo.nl New Year's Party - Slides OSGeo.nl" - 2021-01-24 - Presentation I gave on behalf of OSGeo.nl on the online joint QGIS-NL, OpenStreetMap-NL and OSGeo.nl New Year's party on January 24, 2021.- [Slideshare]
Not as much fun as in 2019 Bukarest, but providing the Geopython Workshop at FOSS4G Buenos Aires with these great minds is still a 2021-highlight.