Skip to content

Instantly share code, notes, and snippets.

@walkermatt
Created September 26, 2012 20:09
Show Gist options
  • Save walkermatt/3790269 to your computer and use it in GitHub Desktop.
Save walkermatt/3790269 to your computer and use it in GitHub Desktop.
WFS to PostGIS
# Example of downloading all data from a GeoServer WFS server
# and loading it into a PostGIS database. Use with caution
# as you could put a lot of load on someone's server if they
# host a lot of data which might make them sad.
# In response to: http://underdark.wordpress.com/2012/09/26/wfs-to-postgis-in-3-steps/
BASEURL="http://data.wien.gv.at/daten/geoserver/ows?service=WFS&version=1.1.0"
for LAYERNAME in `wget -qO- $BASEURL"&request=GetCapabilities" | xpath -q -e "//FeatureType/Name/text()"` ; do
PARTS=(${LAYERNAME//:/ })
FILENAME=${PARTS[1]}
wget $BASEURL"&request=GetFeature&typeName="$LAYERNAME"&srsName=EPSG:4326&outputFormat=shape-zip" -O $FILENAME".zip"
mkdir /tmp/$FILENAME
unzip -d /tmp/$FILENAME $FILENAME".zip"
shp2pgsql -s 4326 -I -S -c -W LATIN1 "/tmp/"$FILENAME"/"$FILENAME".shp" $FILENAME | psql postgis
done
@tepepa
Copy link

tepepa commented Sep 15, 2014

What if the WFS service needs for an HTTP Authentication?
Is there any option to manage this case?

Thank you

@ManuelB
Copy link

ManuelB commented Oct 22, 2016

@tepepa you can add username and password to the wget command:

wget $BASEURL"&request=GetFeature&typeName="$LAYERNAME"&srsName=EPSG:4326&outputFormat=shape-zip" -O $FILENAME".zip" --user=username --password=password

http://www.gnu.org/software/wget/manual/wget.html

@waquner
Copy link

waquner commented Jul 8, 2021

I know this gist is almost 10 years old, but as it's on of the first google results for "WFS to PostGIS" it might help:

  • URLs of data.wien.gv.at changed a little bit
  • shp file inside zip has different name than the feature name (added *.shp instead)
  • shp2pgsql start a transaction, but never commits it, -e disabled that

A working version:

BASEURL="https://data.wien.gv.at/daten/geo?version=1.1.0&service=WFS"

for LAYERNAME in `wget -qO- $BASEURL"&request=GetCapabilities" | xpath -q -e "//FeatureType/Name/text()"` ; do 
    PARTS=(${LAYERNAME//:/ })
    FILENAME=${PARTS[1]}
    wget $BASEURL"&request=GetFeature&typeName="$LAYERNAME"&srsName=EPSG:4326&outputFormat=shape-zip" -O $FILENAME".zip"
    mkdir /tmp/$FILENAME
    unzip -d /tmp/$FILENAME $FILENAME".zip"
    shp2pgsql -e -s 4326 -I -S -c -W LATIN1 "/tmp/"$FILENAME"/"*".shp" $FILENAME | psql <database>
    sleep 10 #prevent getting a 403 because of to frequent queries
done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment