Skip to content

Instantly share code, notes, and snippets.

@carsonfarmer

carsonfarmer/Database.ipynb

Created May 15, 2020 17:47
Show Gist options
  • Save carsonfarmer/199e43c3272bb52526ff9481be1a6876 to your computer and use it in GitHub Desktop.
Save carsonfarmer/199e43c3272bb52526ff9481be1a6876 to your computer and use it in GitHub Desktop.
Download ZIP
Database.ipynb
Display the source blob
Display the rendered blob
Raw
Database.ipynb
{
"cells": [
{
"cell_type": "code",
"source": [
"const level = require('level')\n",
"{\n",
" // Utility code\n",
" // Hack to get WebSocket into global on Node\n",
" ;(global as any).WebSocket = require('isomorphic-ws')\n",
" \n",
" level.destroy('./threads.db', () => {\n",
" return\n",
" })\n",
"}"
],
"outputs": [],
"execution_count": 1,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": true
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:29.842Z",
"iopub.status.idle": "2020-05-14T22:58:29.917Z",
"shell.execute_reply": "2020-05-14T22:58:29.909Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"# Getting Started with ThreadsDB\n",
"\n",
"ThreadsDB is a serverless peer-to-peer (p2p) database built on [IPFS](https://ipfs.io) and [Libp2p](https://libp2p.io). Together, the Threads *Protocol* and *Database* provide an alternative architecture for data on the web. ThreadsDB aims to help power a new generation of web technologies by combining a novel use of event sourcing, Interplanetary Linked Data (IPLD), and access control to provide a distributed, scalable, and flexible database solution for decentralized applications.\n",
"\n",
"## Developer API\n",
"\n",
"ThreadsDB is designed to be simple enough for any developer to start using. The API will feel familiar to developers who have worked with technologies like MongoDB.\n",
"\n",
"The first three concepts a developer will encounter with ThreadsDB are [Databases](), [Collections](), and [Instances](). The organization is simple. Instances are the individual records you create, update, or delete. Instances are stored in a Collection. Collections have one or many Schemas and can only store Instances that match one of those Schemas. Databases can store many Collections.\n",
"\n",
"### Databases\n",
"\n",
"A Thread-based Database is tied to a single Thread (with associated Thread ID)."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"import { Database, JSONSchema, FilterQuery } from '@textile/threads-database'\n",
"import { ThreadID } from '@textile/threads-id'"
],
"outputs": [],
"execution_count": 2,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:29.924Z",
"iopub.status.idle": "2020-05-14T22:58:31.228Z",
"shell.execute_reply": "2020-05-14T22:58:31.244Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"To start a new, empty database is simple. A new [Level Datastore](https://github.com/ipfs/js-datastore-level) is used as the backing store by default if no datastore is explicitly supplied. See the doc-strings for the `Database.constructor` for further options.\n",
"\n",
"By default, a ThreadsDB will connect with a local `go-threads` daemon. See the [`go-threads` documentation]() for details on getting started with local development. Alternatively, it is possible to connect with a remote daemon by specifying the networking component of the Database (see [Threads Services & The Hub]() for details on connecting to hosted remote services)."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"const db = new Database() // Uses a level datastore by default"
],
"outputs": [],
"execution_count": 3,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.174Z",
"iopub.status.idle": "2020-05-14T22:58:31.295Z",
"shell.execute_reply": "2020-05-14T22:58:31.281Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Next, we simply start the database, and we are ready to take action. Here, we are explicitly providing an 'existing' ThreadID. But default, a random ThreadID will be used. See the doc-strings for `Database.open` for further options."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"const threadID = ThreadID.fromRandom()\n",
"await db.open({ threadID })\n",
"console.log(db.threadID.toString())"
],
"outputs": [
{
"output_type": "stream",
"name": "stderr",
"text": [
"Error: 2\n",
" at /Users/carson/Textile/Dev/Notebooks/node_modules/@textile/threads-network-client/dist/index.js:156:32\n",
" at Array.<anonymous> (/Users/carson/Textile/Dev/Notebooks/node_modules/@improbable-eng/grpc-web/dist/grpc-web-client.js:1:5649)\n",
" at e (/Users/carson/Textile/Dev/Notebooks/node_modules/@improbable-eng/grpc-web/dist/grpc-web-client.js:1:1531)\n",
" at Timeout._onTimeout (/Users/carson/Textile/Dev/Notebooks/node_modules/@improbable-eng/grpc-web/dist/grpc-web-client.js:1:1645)\n",
" at listOnTimeout (internal/timers.js:549:17)\n",
" at processTimers (internal/timers.js:492:7)\n"
]
}
],
"execution_count": 4,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.221Z",
"shell.execute_reply": "2020-05-14T22:58:31.379Z",
"iopub.status.idle": "2020-05-14T22:58:31.332Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"### Collections\n",
"\n",
"To handle different data structures, a Database contains Collections, each of which are defined by a [json-schema.org schema](https://json-schema.org/). These schemas define the 'shape' of Collection Instances. Collections implement a Store with [JSON Patch](https://github.com/Starcounter-Jack/JSON-Patch) semantics by default, but will be made to support other types (CRDT-driven documents for instance) in the future (some of which are already under active development). Ultimately, a Collection is a single document store with a set of APIs to make it feel like a *local database table*.\n",
"\n",
"Collections can be created from an existing Schema."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"// Define a simple person schema\n",
"const schema: JSONSchema = {\n",
" $schema: 'http://json-schema.org/draft-07/schema#',\n",
" title: 'Person',\n",
" type: 'object',\n",
" properties: {\n",
" _id: { type: 'string' },\n",
" name: { type: 'string' },\n",
" age: {\n",
" type: 'number',\n",
" minimum: 0,\n",
" exclusiveMaximum: 100,\n",
" },\n",
" },\n",
"}\n",
"const Person = await db.newCollection('Person', schema)"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.236Z",
"shell.execute_reply": "2020-05-14T22:58:31.384Z",
"iopub.status.idle": "2020-05-14T22:58:31.336Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Or from an existing object/instance."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"const obj = {\n",
" _id: '', // All collections have an _id field\n",
" team: '',\n",
" name: '',\n",
" points: 0,\n",
"}\n",
"const Player = await db.newCollectionFromObject('Player', obj)"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.287Z",
"shell.execute_reply": "2020-05-14T22:58:31.387Z",
"iopub.status.idle": "2020-05-14T22:58:31.340Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"### Instances\n",
"\n",
"Instances are the objects you store in your Collection. Instances are JSON documents with schemas that match those defined in your Collection. Creating and manipulating them is simple."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"const beth = new Player({ _id: '', name: 'beth' }) // Not yet persisted\n",
"await beth.save() // Persist changes to db\n",
"\n",
"// Modify the `beth` instance\n",
"beth.points = 1\n",
"await beth.save() // Save changes\n",
"\n",
"// Modify it again\n",
"beth.team = 'Astronauts'\n",
"beth.points = 2\n",
"\n",
"// Save it from the Collection\n",
"await Player.save(beth)\n",
"\n",
"// Delete it from the Collection\n",
"await Player.delete(beth._id)\n",
"\n",
"// etc!"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.301Z",
"iopub.status.idle": "2020-05-14T22:58:31.345Z",
"shell.execute_reply": "2020-05-14T22:58:31.389Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"### Query\n",
"\n",
"Each Threads implementation supports query and look-up capabilities such as `insert`, `findOne`, `has`, and more. ThreadsDB also supports the [MongoDB query language](https://github.com/kofrasa/mingo). In the JavaScript library, you might write queries like the following."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"await Player.insert(\n",
" { _id: '', points: 11, team: 'Astronauts', name: 'beth'},\n",
" { _id: '', points: 1, team: 'Astronauts', name: 'jim'},\n",
" { _id: '', points: 18, team: 'Astronauts', name: 'issac'},\n",
" { _id: '', points: 7, team: 'Astronauts', name: 'beth'},\n",
")"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.307Z",
"iopub.status.idle": "2020-05-14T22:58:31.350Z",
"shell.execute_reply": "2020-05-14T22:58:31.392Z"
}
}
},
{
"cell_type": "code",
"source": [
"// Setup a query\n",
"const query = {\n",
" $or: [\n",
" { points: { $gt: 10 } },\n",
" { name: 'jim' },\n",
" ]\n",
"}\n",
"\n",
"const all = Player.find(query, { sort: { points: -1 } })"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.312Z",
"iopub.status.idle": "2020-05-14T22:58:31.354Z",
"shell.execute_reply": "2020-05-14T22:58:31.395Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"Queries return `AsyncIterableIterators`, so you can loop over them and take appropriate action."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"import { collect } from 'streaming-iterables'\n",
"\n",
"for (const { key, value } of await collect(all)) {\n",
" console.log(`${key.toString()}: ${value.name}`)\n",
"}"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.316Z",
"iopub.status.idle": "2020-05-14T22:58:31.360Z",
"shell.execute_reply": "2020-05-14T22:58:31.397Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"### Listen\n",
"\n",
"A Database is also an [Event Emitter](https://github.com/EventEmitter2/EventEmitter2), and listeners can subscribe to events using 'wildcard' syntax. The following database manipulations could be observed via the following simple listener."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"const _ = db.on('**', (update: any) => {\n",
" console.log(update)\n",
"})"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.320Z",
"iopub.status.idle": "2020-05-14T22:58:31.364Z",
"shell.execute_reply": "2020-05-14T22:58:31.400Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"## Multi-user Databases\n",
"\n",
"Everything above just looks like a database, so what's a Thread? ThreadsDB combines the storage and management of data (the *Database*) with networking, access control, and replication over IPFS using the Threads *Protocol*. The Threads protocol has been extensively documented in the [whitepaper](https://docsend.com/view/gu3ywqi), but in short, Threads use private-key encryption to manage both security and identity among multiple parties that can access or edit the same Database.\n",
"\n",
"### Access-control\n",
"\n",
"ThreadsDB uses a modular role-based access control system that will allow access control lists (ACLs) to be declared in a wide-variety of ways. ACLs are in active development and you can [follow the development here](https://github.com/textileio/go-threads/issues/295).\n",
"\n",
"### Identity\n",
"\n",
"ThreadsDB allows you to handle user identities (for access control and security/encryption) in the best way for your app and your users. In order to handle *multiple* peers collaborating on a single database, as well as the ability to handle storage *on behalf* of a user, ThreadsDB expects a simple Identity interface for singing and validating database updates.\n",
"\n",
"You can learn more about Identity, Access Control, and other advanced topics, in the [Hub documentation]()."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"{\n",
" // Cleanup after running this notebook\n",
" level.destroy('./threads.db', () => {\n",
" return\n",
" })\n",
"}"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"outputExpanded": false,
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"execution": {
"iopub.status.busy": "2020-05-14T22:58:31.324Z",
"shell.execute_reply": "2020-05-14T22:58:31.403Z",
"iopub.status.idle": "2020-05-14T22:58:31.368Z"
}
}
}
],
"metadata": {
"kernel_info": {
"name": "jslab"
},
"language_info": {
"name": "typescript",
"version": "3.7.2",
"mimetype": "text/typescript",
"file_extension": ".ts",
"codemirror_mode": {
"mode": "typescript",
"name": "javascript",
"typescript": true
}
},
"kernelspec": {
"argv": [
"tslab",
"kernel",
"--config-path={connection_file}",
"--js"
],
"display_name": "JavaScript",
"language": "javascript",
"name": "jslab"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment