Skip to content

Instantly share code, notes, and snippets.

View elliot42's full-sized avatar

Elliot Block elliot42

  • Form Energy
  • Berkeley, CA
View GitHub Profile
import datetime
def period_interval(date, period, n0, n1):
"""Return a 2-tuple of [start_date, end_date],
date: base date from which to calculate forwards/backwards
period: timedelta
n0: int, starting period number
n1: int, ending period number"""
return [date + (period * n0),
The core of what the system is about is:
- Having rough/flexible data strutures
- Performing transformations on those data structures
- Finding the core "must be correct" structures we want to reason about
- Defining and naming those structures with relative precision,
hardening them into definite declarations
- Validating and verifying that input and output conform to those
now-hardened structures.
@elliot42
elliot42 / tmp.txt
Created March 11, 2014 21:15 — forked from anonymous/tmp.txt
Thank you! These are great.
Here's my gut investigation so far:
The multi-item vs. single-item distinction is a really good point. In
things like GMail, it's a multi-item system, there's a breakdown between:
- the items you can select
- a small set of operations you can apply to the items in bulk
rpm -Uvh http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm
yum -y install hadoop-0.20-conf-pseudo hbase hbase-regionserver hbase-master
sudo -u hdfs hdfs namenode -format
for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done
sudo -u hdfs hadoop fs -mkdir /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
sudo -u hdfs hadoop fs -mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred
for x in `cd /etc/init.d ; ls hadoop-0.20-mapreduce-*` ; do sudo service $x start ; done