Skip to content

Instantly share code, notes, and snippets.

@alpha-beta-soup

alpha-beta-soup/download-qeii.sh

Last active October 6, 2023 03:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save alpha-beta-soup/26d7116bbc53a47fa3ce9cf7f9cd76f2 to your computer and use it in GitHub Desktop.
Save alpha-beta-soup/26d7116bbc53a47fa3ce9cf7f9cd76f2 to your computer and use it in GitHub Desktop.
Download ZIP
Experiments with a commit time modification in a Kart archive for pre-existing geospatial data
Raw
download-qeii.sh
#!/bin/bash
CLEAR='\033[0m'
RED='\033[0;31m'
function usage() {
if [ -n "$1" ]; then
echo -e "${RED}👉 $1${CLEAR}\n";
fi
echo "Usage: $0 [-u url]"
echo " -u, --url The QEII download URL"
echo ""
echo "Example: $0 --url https://mcusercontent.com/7e7527238b895050d20f82859/files/c8a3ad0e-bf15-31e6-59a2-0482cf0b2481/QEII_OSCovenants_20230929.zip"
exit 1
}
while [[ "$#" -gt 0 ]]; do
case $1 in
-u|--url) URL="$2";shift;shift;;
*) echo "Unknown parameter passed: $1"; exit 1 ;;
esac
shift
done
if [ -z "$URL" ]; then usage "URL is not set"; fi;
TMPF=$(mktemp).zip
BASENAME=./$(basename $URL)
FNAME="${BASENAME%.*}"
curl -iL "$URL" --output $TMPF
unzip -o $TMPF -d $FNAME
rm $TMPF
ogr2ogr -overwrite -nlt MULTIPOLYGON -nln qeii $FNAME/$FNAME.gpkg $FNAME/$FNAME.shp
Raw
make-archive.sh
#!/bin/bash
# Get some downloaded data, in this case, three time-steps of data downloaded from the Queen Elizabeth II National Trust,
# all in the same schema
./download.sh -u https://mcusercontent.com/7e7527238b895050d20f82859/files/0c921999-84b7-0afa-a659-829392844f2e/QEII_OSCovenants_20230328.zip
./download.sh -u https://mcusercontent.com/7e7527238b895050d20f82859/files/c8ee486f-2053-67d0-7b9d-3bfb12a1a5be/QEII_OSCovenants_20230630.zip
./download.sh -u https://mcusercontent.com/7e7527238b895050d20f82859/files/c8a3ad0e-bf15-31e6-59a2-0482cf0b2481/QEII_OSCovenants_20230929.zip
# We want to put these into the same kart repository, and amend the commit date to represent the data's publication date
# These sample times are obtained manually from the response headers of the above links
kart init qeii-kart
cd qeii-kart
kart import ../downloads/QEII_OSCovenants_20230328/QEII_OSCovenants_20230328.gpkg --all-tables --primary-key CovNumber
GIT_COMMITTER_DATE="Tue 28 Mar 03:01:20 2023 GMT" kart commit-files -m "Import from QEII_OSCovenants_20230328.gpkg:qeii to qeii/" --amend
kart import ../downloads/QEII_OSCovenants_20230630/QEII_OSCovenants_20230630.gpkg --all-tables --replace-existing --primary-key CovNumber
GIT_COMMITTER_DATE="Fri 30 Jun 05:04:59 2023 GMT" kart commit-files -m "Import from QEII_OSCovenants_20230630.gpkg:qeii to qeii/" --amend
kart import ../downloads/QEII_OSCovenants_20230929/QEII_OSCovenants_20230929.gpkg --all-tables --replace-existing --primary-key CovNumber
GIT_COMMITTER_DATE="Sun 01 Oct 21:44:12 2023 GMT" kart commit-files -m "Import from QEII_OSCovenants_20230929.gpkg:qeii to qeii/" --amend
@alpha-beta-soup
Copy link
Author

Then you can do things like:

kart checkout `git rev-list -n 1 --first-parent --before="2023-09-30 12:00" main`

Which will checkout the correct version according to the amended commit timestamps, while keeping git authoring time preserved.

@alpha-beta-soup
Copy link
Author

git log --format=fuller

commit 400b1397ba31c0667f36be5d652da2bab1de7d3e (HEAD -> main)
Author:     Richard Law <lawr@landcareresearch.co.nz>
AuthorDate: Fri Oct 6 15:49:03 2023 +1300
Commit:     Richard Law <lawr@landcareresearch.co.nz>
CommitDate: Sun Oct 1 21:44:12 2023 +0000

    Import from QEII_OSCovenants_20230929.gpkg:qeii to qeii/

commit 6091f40dd04dafe31342bc8fb07ba7ae7dd22ecf
Author:     Richard Law <lawr@landcareresearch.co.nz>
AuthorDate: Fri Oct 6 15:48:44 2023 +1300
Commit:     Richard Law <lawr@landcareresearch.co.nz>
CommitDate: Fri Jun 30 05:04:59 2023 +0000

    Import from QEII_OSCovenants_20230630.gpkg:qeii to qeii/

commit 3e5033374fcc7a975e3fb9b314779b6a109df7c3
Author:     Richard Law <lawr@landcareresearch.co.nz>
AuthorDate: Fri Oct 6 15:48:21 2023 +1300
Commit:     Richard Law <lawr@landcareresearch.co.nz>
CommitDate: Tue Mar 28 03:01:20 2023 +0000

    Import from QEII_OSCovenants_20230328.gpkg:qeii to qeii/

@alpha-beta-soup
Copy link
Author

Extract of a diff to demonstrate at least one use of this:

kart diff 6091f40dd04dafe31342bc8fb07ba7ae7dd22ecf

--- qeii:feature:5-13-048
+++ qeii:feature:5-13-048
-                                     geom = MULTIPOLYGON(...)
+                                     geom = MULTIPOLYGON(...)
-                                CreatedBy = Digitised from plan - unverified
+                                CreatedBy = Digitised from plan - verified
-                                  Comment = ␀
+                                  Comment = Digitised 5/09/2005 from survey plan - unverified(QEII_covpolys_09may). Redigitised 15/08/2023 from photodiagram georeferenced to Southland & Central Otago 0.4m Rural Aerial Photos (2015-2017) and primary parcels.
-                               LastUpdate = 2005-09-05
+                               LastUpdate = 2023-08-15
-                                 GeomArea = 14.5395832314
+                                 GeomArea = 14.1706569917
-                               Shape_Leng = 2241.79572801
+                               Shape_Leng = 2248.59132219
-                               Shape_Area = 145395.832314
+                               Shape_Area = 141706.569917

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment