Skip to content

Instantly share code, notes, and snippets.

@TheSeamau5
Last active September 23, 2018 00:24
Show Gist options
  • Save TheSeamau5/1dc5597a2e3b7ae5f33e to your computer and use it in GitHub Desktop.
Save TheSeamau5/1dc5597a2e3b7ae5f33e to your computer and use it in GitHub Desktop.
Hacker news requests example
--------------------------
-- CORE LIBRARY IMPORTS --
--------------------------
import Task exposing (Task, ThreadID, andThen, sequence, succeed, spawn)
import Json.Decode exposing (Decoder, list, int, string, (:=), map, object2)
import Signal exposing (Signal, Mailbox, mailbox, send)
import List
---------------------------------
-- THIRD PARTY LIBRARY IMPORTS --
---------------------------------
import Http exposing (Error, get)
import Html exposing (Html, div, ul, li, a, text)
import Html.Attributes exposing (href)
----------------------
-- HELPER FUNCTIONS --
----------------------
-- Convenient for decoding large JSON objects
andMap : Decoder (a -> b) -> Decoder a -> Decoder b
andMap = object2 (<|)
-- Perform a list of tasks in parallel.
-- Analogous to `sequence`
parallel : List (Task x value) -> Task y (List ThreadID)
parallel =
sequence << List.map spawn
-----------
-- MODEL --
-----------
type alias Story =
{ by : String
, id : Int
, score : Int
, time : Int
, title : String
, type' : String
, url : String
}
type alias Model = List Story
initialModel : Model
initialModel = []
----------
-- VIEW --
----------
viewStory : Story -> Html
viewStory story =
li []
[ a [ href story.url ]
[ text story.title ]
]
view : Model -> Html
view stories =
ul []
( List.map viewStory stories )
--------------------------
-- LINKS TO HACKER NEWS --
--------------------------
idsUrl : String
idsUrl =
"https://hacker-news.firebaseio.com/v0/topstories.json"
storyUrl : Int -> String
storyUrl id =
"https://hacker-news.firebaseio.com/v0/item/"++ toString id ++ ".json"
-----------
-- TASKS --
-----------
getIDs : Task Error (List Int)
getIDs =
get (list int) idsUrl
getStory : Int -> Task Error ()
getStory id = get storyDecoder (storyUrl id)
`andThen` \story -> send newStoryMailbox.address (Just story)
mainTask : Task Error (List ThreadID)
mainTask = getIDs
`andThen` \ids -> parallel (List.map getStory ids)
-------------------
-- JSON DECODING --
-------------------
storyDecoder : Decoder Story
storyDecoder = Story
`map` ("by" := string)
`andMap` ("id" := int)
`andMap` ("score" := int)
`andMap` ("time" := int)
`andMap` ("title" := string)
`andMap` ("type" := string)
`andMap` ("url" := string)
---------------
-- MAILBOXES --
---------------
newStoryMailbox : Mailbox Action
newStoryMailbox =
mailbox Nothing
mainTaskMailbox : Mailbox (Task Error (List ThreadID))
mainTaskMailbox =
mailbox mainTask
-----------
-- PORTS --
-----------
port mainTaskPort : Signal (Task Error (List ThreadID))
port mainTaskPort =
mainTaskMailbox.signal
-------------
-- ACTIONS --
-------------
type alias Action = Maybe Story
actions : Signal Action
actions =
newStoryMailbox.signal
------------
-- UPDATE --
------------
update : Action -> Model -> Model
update maybeStory stories = case maybeStory of
Nothing -> stories
Just story -> stories ++ [ story ]
----------
-- MAIN --
----------
main : Signal Html
main =
Signal.map view
( Signal.foldp update initialModel actions )

How to progressively get data and update the view

The idea here is that we want to display stories from Hacker News progressively as they arrive.

Context

Model

First, let's model our application.

Elm applications usually follow the following structure:

initialModel : Model
actions : Signal Action
update : Action -> Model -> Model
view : Model -> Html

main =
  Signal.map view
    (Signal.folp update initialModel actions)

Our Model will consists of a list of stories

type alias Model = List Story

where a Story is just a record containing information about the Hacker News story (modeled after the JSON produced by Hacker News)

type alias Story =
  { by    : String
  , id    : Int
  , score : Int
  , time  : Int
  , title : String
  , type' : String
  , url   : String
  }

The initialModel in this case is pretty trivial, it is simply the empty list:

initialModel : Model
initialModel = []

View

From there we can make a simple view function using elm-html to view the stories.

view : Model -> Html
view stories =
  ul
    []
    ( List.map viewStory stories )

viewStory : Story -> Html
viewStory story =
  li
    []
    [ a [ href story.url ]
        [ text story.title ]
    ]

We simply view our model as a list of links where each link displays the title of the story and links to the url of the story.

Actions

Now we need to model the actions that will update the Model. Since we want to display stories progressively, we want to add each story one by one.

type alias Action = Maybe Story

This action will represent the new story that will come to update the model.

The reason that we model this as a Maybe is because initially, we will start with no story, hence, with Nothing.

Update

So, now that we have our actions, we can update our model.

update : Action -> Model -> Model
update maybeStory stories = case maybeStory of
  Nothing -> stories
  Just story -> stories ++ [ story ]

If we get Nothing, we leave the model untouched. But if we do get a story, we add the story to the list of stories (hence adding a new story to our model).

Now, all that is left is to actually get the new stories from Hacker News.

Asynchrony

We use elm-http to get the JSON from Hacker News.

Goal

We would like a Signal of Actions:

actions : Signal Action

where each new action will get fed into the system. Given that we have all the other pieces of the application, we just need to get these actions and we're done.

Tasks

To do so, we will use tasks. A task describes an eventual computation (if you know JS promises, then tasks are roughly the same).

Examples of tasks :

  • Making an HTTP request
  • Saving files to a database
  • Playing music
  • Requesting a file from a user

These are all examples of stateful computations that you would like the program to perform at sometime. Keep in mind that tasks are merely descriptions of computations. This subtle detail will come into play later.

For now, let's focus on the task of making an HTTP request.

Making an HTTP Request

elm-http provides the get function

get : Json.Decoder a -> String -> Task Http.Error a

get takes in a JSON decoder and a url, and returns a task. This task describes how to send a GET request to the url and then decoding its contents with the decoder.

There are two groups of urls we are interested in getting. First there is the url with a list of all of the story ids. Second, there are the actual stories whose urls are parametrized by story id.

The first url is this:

idsUrl : String
idsUrl =
  "https://hacker-news.firebaseio.com/v0/topstories.json"

This url simply contains a list of integers representing the story ids.

To get the contents we just need to decode a list of integers in JSON, which is done as follows

-- import Json.Decode as Json exposing (list, int)

intListDecoder : Json.Decoder (List Int)
intListDecoder = list int

And now we can create our task to get the ids:

getIDs : Task Http.Error (List Int)
getIDs = get intListDecoder idsUrl

Now that we have the ids, let's construct the individual story urls from the ids

storyUrl : Int -> String
storyUrl id =
  "https://hacker-news.firebaseio.com/v0/item/"++ toString id ++ ".json"

Since we want to get a story out of each of these urls, we need a means to decode a story from JSON.

To, recall, this is the story type:

type alias Story =
  { by    : String
  , id    : Int
  , score : Int
  , time  : Int
  , title : String
  , type' : String
  , url   : String
  }

Thankfully, this type mirrors exactly the json data, so it's not so hard to decode it. A very easy way is as follows:

-- import Json.Decode as Json exposing (int, string, (:=), object2)
-- andMap = object2 (<|)

storyDecoder : Json.Decoder Story
storyDecoder = Story
  `map`    ("by" := string)
  `andMap` ("id" := int)
  `andMap` ("score" := int)
  `andMap` ("time" := int)
  `andMap` ("title" := string)
  `andMap` ("type" := string)
  `andMap` ("url" := string)

This approach to decoding is nice in that it guarantees type safety and reads relatively well. There's almost no need to understand the details to see how it works. Basically, what is happening is that this is an easy way of mapping a function over an arbitrary number of parameters (in this case, 7) where each parameter is itself a decoder.

Now that we have our decoder, let's create our tasks that will get the stories given an id.

getStory : Int -> Task Http.Error Story
getStory id =
  get storyDecoder (storyUrl id)

Combining the two tasks, we can then get our main task:

mainTask : Task Http.Error (List Story)
mainTask = getIDs
  `andThen` \ids -> sequence (List.map getStory ids)

What this task does is :

  1. Get the ids
  2. And then, given the ids, get all the stories in sequence

Where sequence takes a list of tasks and converts it into a single task that returns the result as a list:

sequence : List (Task error value) -> Task error (List value)

The problem of this approach is that this task will get the stories one by one as opposed to in parallel, which is our goal. This means that the page may only show up after all the stories have arrived. This is a problem given that Hacker News provides 500 stories (we could be waiting for a looooong time).

But before we tackle this problem, let's first see how we can actually run this task, because, if you recall, a task is just a description of a computation and not a computation in of itself. It must be performed somehow and then we need to somehow get a Signal of Actions out of it.

Mailboxes

A mailbox is an object with an address and a signal:

type alias Mailbox a =
  { address : Address a
  , signal  : Signal a
  }

This object is special in that you can send values to a given mailbox if you know its address. Sending a value to a mailbox will cause the signal to update with the sent value as the fresh value for the signal.

The easiest way to send a value to a Mailbox is with the send function:

send : Address a -> a -> Task error ()

The send function takes an address and a value to send to the address and returns a task signifying that it has successfully sent the value. This is what allows us to communicate between the world of tasks and the world of signals.

We can then create mailboxes with the mailbox function:

mailbox : a -> Mailbox a

This will take an initial value for the signal inside the mailbox.

So, now we can create a mailbox for our new stories:

newStoryMailbox : Mailbox Action
newStoryMailbox =
  mailbox Nothing

And we could modify our getStory to send the story to this new mailbox:

getStory : Int -> Task Http.Error ()
getStory id = get storyDecoder (storyUrl id)
  `andThen` \story -> send newStoryMailbox.address (Just story)

Note: this will cause mainTask to change type to Task Http.Error (List ())

And now we finally have our actions:

actions : Signal Action
actions =
  newStoryMailbox.signal

But, we're not quite done yet. While this will allow us to send values from tasks to mailboxes and hence extract signals, this is still not enough to actually perform tasks. The way to do so if via ports.

Ports

Ports are Elm's way of communicating with the outside world. This mechanism is used to actually perform task.

To do so, we create a port with the port keyword and define it as something of type: Signal (Task error value)

port taskPort : Signal (Task error value)
port taskPort = ...

Now, how do we get something of type signal of tasks? We have a task (mainTask) and we could create a signal of them if we created a mailbox for the main task

This would lead to:

mainTaskMailbox : Mailbox (Task Http.Error (List ()))
mainTaskMailbox =
  mailbox mainTask

port mainTaskPort : Signal (Task Http.Error (List ()))
port mainTaskPort =
  mainTaskMailbox.signal

By actually having the signal of tasks as port, we explicitly tell Elm that this is a task that should be run. This is very convenient because this means that you can have a single file detailing every single effectful computation performed by the program.

By creating a mailbox with a task, that task will be run as soon as the program starts.

Note: This mailbox is just a regular mailbox. Don't be fooled by the fact that it holds tasks. You can send tasks whenever you want to this mailbox. A good example of this is sending a new task on button click. Each new task will be run as soon as it is sent to the mailbox. Therefore, ports aren't so much tied to individual tasks as much as they are tied to mailboxes. The mailboxes feed the ports. Thus different sources may use the same mailbox.

Main

Now we just need to write our main function:

main : Signal Html
main =
  Signal.map view
    ( Signal.foldp update initialModel actions )

And we have a running program... except that it takes forever to load because we are running each task in sequence... But the good news is that the stories are loading progressively, which is a win.

So, let's see how we can make these task go in parallel

Parallel Tasks

The problem is with the sequence function. It performs a task, then waits for the task to complete, then performs the next task, waits for it to complete, then performs the next task, and so on...

We would like to send them all at once. If somehow there were some parallel equivalent to sequence.

Currently, there isn't any in the core library but it can be easily implemented. All we need is the spawn function:

spawn : Task x value -> Task y ThreadID

spawn tasks a task and then wraps it in its own thread. This thread isolates this task from other tasks and tasks in other threads and thus allows it to run independently.

Note: these threads are not real threads. Javascript is single threaded so there's only one real thread. Elm just can move between threads to work on each progressively.

So, what we want for parallel is to take a list of tasks, and wrap them all in their own threads.

parallel : List (Task x value) -> Task y (List ThreadID)
parallel tasks =
  sequence (List.map spawn tasks)

Here sequence is used to spawn each task one by one. The spawning is what is done in sequence, not the computation. The actual task is now in its own thread.

So, now, we can re-write our mainTask as follows:

mainTask : Task Http.Error (List ThreadID)
mainTask = getIDs
  `andThen` \ids -> parallel (List.map getStory ids)

Note: The types of mainTaskMailbox and mainTaskPort will have to be changed to Mailbox (Task Http.Error (List ThreadID)) and Signal (Task Http.Error (List ThreadID)) respectively

And now, watch as these stories update in parallel. Ah!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment