Thomas Thoren ThomasThoren

One or the other of these is the scraper I used for the Philadelphia Court Debt project. I didn't realize until after I promised everyone in the room my code that I had never published it.

Not for nothing, I make no assertion that this is great programming or even remotely reflects best practices in python or scraping. There's plenty here that I would do differently if I did it again today.

I don't even know which of these was the final scraper. I could figure that out but then I'd never get around to throwing it online. So here it is, warts and all.

Himawari-8 animation tutorial

Here’s how to make animations like this one. It requires intermediate Unix command-line knowledge, to install some tools and to debug if they don’t work. You’ll need these utilities:

curl (or you can translate to wget)
convert and montage, part of ImageMagick
ffmpeg, plus whatever codecs
parallel, for iteration that’s nicer than shell for loops or xargs
run everything in zsh for leading 0s in numerical ranges to work

React "Hello World" Examples

Below are a small collection of React examples to get anyone started using React. They progress from simpler to more complex/full featured.

They will hopefully get you over the initial learning curve of the hard parts of React (JSX, props vs. state, lifecycle events, etc).

Usage

You will want to create an index.html file and copy/paste the contents of 1-base.html and then create a scripts.js file and copy/paste the contents of one of the examples into it.

	import sys
	import os
	import json
	import asyncio
	import aiohttp


	# Initialize connection pool
	conn = aiohttp.TCPConnector(limit_per_host=100, limit=0, ttl_dns_cache=300)
	PARALLEL_REQUESTS = 100

	output "SSH Config" {
	value = <<SSHCONFIG
	### START Auto-gen SSH Config for ${random_pet.cluster_id.id} ###

	Host ${random_pet.cluster_id.id}-bastion
	User core
	Hostname ${aws_instance.bastion.public_ip}
	IdentityFile ~/.ssh/${random_pet.cluster_id.id}.key

	Host ${random_pet.cluster_id.id}-agent

	#!/usr/bin/env python
	"""
	Converts Internet Explorer 'capture network traffic' XML to a HAR file.

	Turns out that XML is just a HAR file anyways, but in XML form. So this
	just converts it to JSON, and Bob's your uncle.

	Requires Python 2.7+ and LXML.
	"""
	from __future__ import unicode_literals

	import pandas as pd
	import requests

	API_URL="http://api.censusreporter.org/1.0/data/show/{release}?table_ids={table_ids}&geo_ids={geoids}"
	def get_data(tables=None, geoids=None, release='latest'):
	if geoids is None:
	geoids = ['040\|01000US']
	if tables is None:
	tables = ['B01001']

	# Hello, and welcome to makefile basics.
	#
	# You will learn why `make` is so great, and why, despite its "weird" syntax,
	# it is actually a highly expressive, efficient, and powerful way to build
	# programs.
	#
	# Once you're done here, go to
	# http://www.gnu.org/software/make/manual/make.html
	# to learn SOOOO much more.

	# Based on solution given here: http://geospatialpython.com/2011/11/generating-shapefile-shx-files.html
	# Depends on pyshp

	# Build a new shx index file
	import fnmatch
	import os
	import shapefile

	# List all the shapefiles
	def find(pattern, path):

	-- Found at:
	-- http://www.if-not-true-then-false.com/2009/postgresql-psql-psqlrc-tips-and-tricks/
	-- http://opensourcedbms.com/dbms/psqlrc-psql-startup-file-for-postgres/

	\set QUIET ON

	\pset pager always

	\pset null 'NULL'