Jorge Sanz jsanz

Loading geospatial data into Elasticsearch

Resources:

Note

Asumes a recent Elastic stack (Elasticsearch at least) running on

Building a cartographic web application with pg_tileserv and pg_featureserv

Workshop materials.
Reused the postgis Docker Compose template from previous workshop
Reused also the osgeo/gdal container to use ogr2ogr on the downloaded shapefiles.
Added pg_featureserv and pg_tileserv Docker images passing the .toml configurations as part of the command setting.
Added an nginx service to the compose file for the HTML files.
Played with different HTML pages that load tiles from pg_tileserv and displays them using OpenLayers.
Created a view that returns random points.

Ping sites

This is a CSV with country official websites from all the world, and a simple script that randomly access to five of them every five seconds. The objective of this is to hit as many different country IP's as possible.

The CSV is obtained from this SPARQL query to Wikidata:

SELECT 
  ?idLabel 
 (SAMPLE(?website) as ?web)

This simple scripts scraps the FOSS4G Community Review pages to convert the abstracts into a JSON file that will contain for each abstract:

Page number
Title
Abstract in HTML
Your score if it exists

Requirements: Python 3, [BeautifulSoup][bs], and [Requests][req]

It expects a FOSS4G_ID environment variable that is the variable part of the URL that you get when you sign in for the community review: https://talks.osgeo.org/foss4g-2022/p/voting/talks/{FOSS4G_ID}?.

Simple wrappers around docker and OGR commands to expose ogr2ogr and ogrinfo using the latest builds. You need to be careful to always put absolute paths to your local data so the docker image can access it.

	-- change 18 to whatever the max zoom level you want in your MBTiles

	DELETE FROM images WHERE
	tile_id IN (SELECT tile_id FROM map WHERE zoom_level > 18) AND
	tile_id NOT IN (SELECT tile_id FROM map WHERE zoom_level <= 18);

	DELETE FROM map WHERE zoom_level > 18;

	UPDATE metadata SET value = '18' WHERE name = 'maxzoom';

	"""
	Small script to fix geometries of the first file argument
	using the native QGIS processing algorithm. You may need
	to adjust the path to you installation.
	"""

	import sys
	sys.path.append('/usr/share/qgis/python/plugins')

	from processing.core.Processing import Processing

	# Minimal Docker Compose cluster for ElasticSearch and Kibana 7.4
	# Put this file any folder creating "logs" and "esdata" folders
	# to persist the cluster indices and kibana state

	version: "3"
	services:
	elasticsearch:
	image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
	container_name: elasticsearch
	ports:

	data
	.env

	workshop
	.env