Skip to content

Instantly share code, notes, and snippets.

@postfalk

postfalk/README.md

Last active August 22, 2017 22:35
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save postfalk/124f49a421ad5197b5ba6a9f45c8bc18 to your computer and use it in GitHub Desktop.
Save postfalk/124f49a421ad5197b5ba6a9f45c8bc18 to your computer and use it in GitHub Desktop.
Download ZIP
Rivers R example
Raw
README.md

R examples for the Unimpaired flow API

Getting started

  1. Install R from https://cran.r-project.org/

  2. Install git from https://git-scm.com/ or copy code from below in your own project.

git clone https://gist.github.com/124f49a421ad5197b5ba6a9f45c8bc18.git rExample
cd rExample

  1. Install R packages called from the script in question.

  2. Run:

Rscript basic_example.R

1. basic_example.R

Sends a simple request to /api/data/flat/ to demonstrate how to get data for a single stream segment.

2. chunked_example.R

This is a somewhat extreme example of chunked query that will return way more than 20 million records. It is for demonstrating the principle. However, for practical reasons you will reduce the return size further e.g by using projections or do analysis with a subset of the data and aggregate the intermediate results in a map-reduce fashion.

After downloading and concatenating all the hundreds requests you need to dedup the dataset since river segments may touch more than one segment. Looping through small watersheds instead of just squares would be a remedy.

Raw
basic_example.R
library('jsonlite')
library('httr')
# API endpoint
kFlowUrl <- 'https://rivers.codefornature.org/api/data/flat/'
# Obtain token by logging into the https://rivers.codefornature.org/login/
# and then go to https://rivers.codefornature.org/api/user/
# please make sure not to commit a valid token to a GitHub repository
# function retrieving data for a single stream segment
getData <- function(comid) {
query <- list(where=paste('{"stream_segment":', comid, '}'))
res <- GET(
kFlowUrl, query=query)
fromJSON(content(res, 'text'))$results
}
# retrieve all data for comid 4438300
data <- getData(4438300)
data
Raw
chunked_example.R
# Example how to break a giant request down in little chunks
library(jsonlite)
library(httr)
kFlowUrl <- 'https://rivers.codefornature.org/api/data/flat/'
getData <- function(xs, ys, xe, ye) {
geojson <- paste('{"type":"Polygon","coordinates":[[[',
xs, ',', ys, '],[',
xe, ',', ys, '],[',
xe, ',', ye, '],[',
xs, ',', ye, '],[',
xs, ',', ys, ']]]}', sep='')
where <- paste(
'{"geometry":{"$geoIntersects":{"$geometry":', geojson, '}}}', sep='')
query <- list(where=where, page_size=50)
res <- GET(kFlowUrl, query=query)
json <- fromJSON(content(res, 'text', encoding='utf-8'))
data <- json$results
ct <- 1
while(!is.null(json$'next')) {
ct <- ct + 1
print(paste('Requesting page', ct))
res <- GET(json$'next')
json <- fromJSON(content(res, 'text', encoding='utf-8'))
data <- rbind(data, json$results)
}
data
}
alldata <- data.frame()
increment <- .25
for(x in seq(-122, -116-increment, increment)) {
for(y in seq(32, 36-increment, increment)) {
print(paste('Requesting', x, y, x+increment, y+increment))
data <- getData(x, y, x+increment, y+increment)
alldata <- rbind(alldata, data)
print(paste(nrow(alldata), 'records loaded'))
}
}
deduped <- unique(alldata)
print(nrow(deduped))
#data <- getData(-122,33,-115,32)
#data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment