Full Stack F# – The Long Version (Part 1)

By Kevin Ashton • June 4, 2014
Tags: F#,JSInSA

First off, I apologise for posting this later than I said I would. I didn't realise quite how tired I would be after JSInSA. JSInSA has been and gone, and to give conference attendees (as well as those that couldn’t attend) something to refer back to, here is the long version of my Full Stack F# talk, with a step by step code sample. The demo app can be found here.

Getting Started

As I said in the last article, the first things you will probably want to get started you will probably want to get the following Visual Studio extensions:

 

Having at least the first two will make the process a lot easier to follow, while the third will make creating folders in F# projects a lot easier too.

Lets start the usual way, File > New Project.

If you have installed F# MVC 5 Visual Studio extension, you should be able to navigate to Visual F# > Web > F# ASP.NET MVC 5 and Web API 2. Give your project a name and a location as shown in the screenshot below and then select OK.

You will then be presented with another screen that asks you to choose between MVC 5 and Web API or Web API (like the one shown below). To make your life easier because it sets up the view web.config file for you, I recommend selecting MVC 5 and Web API.

Now that we have the project structure, lets start by defining how we want the final solution to look. For demos like this I like to start with the view and work backwards by defining the behaviour that will populate that view. So if you have looked at the demo app you will see that the html is pretty simple. We are using twitter bootstrap for our CSS so we can basically just split our page into the areas that we want. The first thing we are going to do is clean up the _Layout.cshtml file as it has lots things that aren’t really needed for our purposes. To start with we can remove everything from the body except:

<div class="container body-content">     @RenderBody()
</div>

Moving on to the Index.cshtml in Views/Home, we want to create a space for each of metrics on the dashboard. I quite like using bootstrap’s panels for this. So for me, the Index.html file contains the following:

<div class="row">
    <div class="col-lg-8">
        <div class="panel panel-default">
            <div class="panel-heading">Word Cloud</div>
            <div id="WordCloud" class="panel-body">

            </div>
        </div>
    </div>
    <div class="col-lg-4">
        <div class="panel panel-default">
            <div class="panel-heading">Tweets</div>
            <div class="panel-body" id="TweetsBox">
            </div>
        </div>
    </div>
</div>
<div class="row">
    <div class="col-lg-4">
        <div class="panel panel-default">
            <div class="panel-heading">Most Prolific Tweeter</div>
            <div class="panel-body" id="MostProlific">
            </div>
        </div>
    </div>
    <div class="col-lg-4">
        <div class="panel panel-default">
            <div class="panel-heading">Tweets Per Minute</div>
            <div class="panel-body" id="TweeetsPerMinute">
            </div>
        </div>
    </div>
    <div class="col-lg-4">
        <div class="panel panel-default">
            <div class="panel-heading">Most Mentioned</div>
            <div class="panel-body" id="MostMentioned">
            </div>
        </div>
    </div>
</div>

Now that we have our UI in place, we can get down to the real fun. Lets start by getting SignalR installed and up and running (as we want our dashboards to be real time). First thing we need to do is get the SignalR NuGet package (Microsoft.AspNet.SignalR). Once we have pulled in that in we then need to create an OWIN app startup class that will map SignalR so that we can ensure that it works. For this purpose I create a new file called Startup.fs and add the following lines:

open Owin

type Startup() =
    member this.Configuration(app: Owin.IAppBuilder) =
        app.MapSignalR() |> ignore
        ()

SignalR should now be up and running (you can test it by going to the address of your site /signalr/hubs which should return you some JavaScript. Now we will want to add a few lines to our _Layout.cshtml file to include SignalR. My convention is to keep all my JavaScript references in a div that forms the last item of the body with the Id of Scripts (Just personal preference).

<div id="Scripts">
    <script src="~/Scripts/respond.js"></script>
    <script src="~/Scripts/jquery-2.0.3.js"></script>
    <script src="~/Scripts/bootstrap.js"></script>
    <script src="~/Scripts/jquery.signalR-2.0.3.js"></script>
    <script src="~/SignalR/hubs"></script>
</div>

Getting the Server Side Going

Now that we have the basic references that we want to get the solution up and running, we can start moving on to the server side implementation. We’ll start off by defining the types that we want to be exposing to the dashboard front-end. We know that we want a scaled down version of a Twitter user, a scaled down version of a tweet (including a list of mentions), a type to hold a word and its occurrence frequency, a type to hold a count of tweets by user (which we can reuse to hold mentions per user) and a type to hold tweets per minute. I have defined these as follows:

type TwitterUser = 
    {
        Name: string
        ScreenName: string
        ProfileImageUrl: string
    }

type UserMention = 
    {
        Name: string
        ScreenName: string
    }

type Tweet = 
    {
        Id: string
        Text: string
        Truncated: bool
        CreatedAt: System.DateTime
        Creator: TwitterUser
        Source: string
        UserMentions: UserMention list
    }

type WordCount = 
    {
        text: string
        value: int
    }

type UserTweetCount = 
    {
        ScreenName: string
        TweetCount: int
    }

type TimedTweetCount = 
    {
        Time: string
        TweetCount: int
    }

Now that we have our types defined, lets get the server to start doing something. As this dashboard is providing data from the Twitter stream, we need to connect to the Twitter stream. To simplify this process I wrote a simple Twitter OAuth client in F# and that is what I will use. So we need to add NamelessInteractive.FSharp.OAuth (or your preferred OAuth client library, but for this demo I’ll use my one). To keep our solution loosely coupled we will use Reactive Extensions, so we will also add the Rx-Linq package, and to make RX a little bit easier to work with in F#, we will also add the FSharp.Reactive package. Once we have all of those installed, we can then continue. I will start by defining a System.Reactive.Subjects.Subject for my controlling agent which will fire up all the other agents and start the connection to the Twitter stream; as well as a subject that will be triggered when a line is received from the Twitter stream. . These subjects are defined as follows:

open System.Reactive.Subjects

let StreamLineReceivedSubject = new Subject<string>()
let StartAgentsSubject = new Subject<bool>()

Now we will want to create our ControllingAgent. As mentioned previously, this agent is used to start all the other agents in the solution and then connect to the Twitter stream and stream received tweets. The agent is defined as follows:

namespace JSinSADemo.Agents 

open NamelessInteractive.FSharp.OAuth
open System.IO
open JSinSADemo.Types.Subjects

type Agent<'T> = MailboxProcessor<'T>
    

module ControllingAgent = 
    let BaseFilterStreamUrl = "https://stream.twitter.com/1.1/statuses/filter.json?language=en&track="
    let TrackVariables = "jsinsa"

    let BuildFilteredStreamUrl() = 
        BaseFilterStreamUrl + TrackVariables
    type System.Net.WebRequest with
        member x.GetResponseAsyncWorkflow() =
            Async.FromBeginEnd(x.BeginGetResponse, x.EndGetResponse)

    let MyTwitterCredentials = 
        {
            OAuthCredentials.AccessToken = "Your Twitter Access Token"
            OAuthCredentials.AccessTokenSecret = "Your Twitter Access Token Secret"
            OAuthCredentials.ConsumerKey = "Your Twitter Consumer Key"
            OAuthCredentials.ConsumerSecret = "Your Twitter Consumer Secret"
        }

    let Log message = 
        (System.Diagnostics.Debug.WriteLine(message))

    let StreamTweets() =
        async 
            {
                try
                    let url = BuildFilteredStreamUrl()
                    let request = GenerateOAuthWebRequest url Get MyTwitterCredentials
                    use! response = request.GetResponseAsyncWorkflow()
                    use stream = response.GetResponseStream()
                    use reader = new StreamReader(stream)
                    while not reader.EndOfStream do
                        let line = reader.ReadLine() 
                        StreamLineReceivedSubject.OnNext line
                with
                | _ as e -> 
                    Log e.Message
                    Log e.StackTrace
            }
            |> Async.StartImmediate


    let ControlFunction (inbox: Agent<bool>) =  
        let isStarted = false
        let rec loop started =
            async 
                {
                    let! cmd = inbox.Receive()
                    match cmd with
                    | false -> return! loop started
                    | true ->
                        if started then
                            return! loop started
                        else 
                            // Fire up the other agents
                            StreamTweets()
                            return! loop true
                }
        loop false
    let ControllingAgent = new Agent<bool>(ControlFunction)
    StartAgentsSubject.Subscribe ControllingAgent.Post |> ignore 

Basically what the code above does is create the agent (but not start it yet - we will use the Global.asax.fs to start this agent). The created agent will (when started) wait for a boolean message and if the message received is true then it will fire up all of the other agents and then start streaming the tweets. The reason we do it like this is so that we don't miss any tweets on first connection because our infrastructure is not up and running before we connect to the Twitter stream. Now that we should be able to receive tweets, we can start doing something with them. First thing we will want to do is get up a SignalR hub that will be able to respond to the start the message. This hub is very simple, and defined as follows

type TweetHub() =
    inherit Microsoft.AspNet.SignalR.Hub()
    member this.Start() = 
        StartAgentsSubject.OnNext true

This hub is what will trigger streaming of tweets based on the controlling agent's logic. We now need to make sure that this agent is actually listening on the other side, so should start it up. We can do that by adding the following line to the Application_Start method in our Global.asax.fs:

ControllingAgent.ControllingAgent.Start()

That should be all the infrastructure we need on the server side to at least start streaming tweets. The last thing we need to do is actually process these tweets. To do this we will need an agent that parses the raw tweet JSON into something a little more usable. We will now create this TweetProcessingAgent. To do this we will want to bring in the FSharp.Data NuGet package so that we can take advantage of its type provider goodness. Having pulled in this package, we can now start creating our agent. My processing logic is pretty simple; when we receive a line from the controlling agent, we then just create a strongly typed tweet object by using the JsonProvider from FSharp.Data. This agent is defined as follows:

module TweetParsingAgent = 
    type TweetStructureProvider = FSharp.Data.JsonProvider<"http://namelessinteractive.com/Media/Blog/FullStackFSharpLong/TweetStructure.json">
    let twitterDateFormat = "ddd MMM dd HH:mm:ss zzz yyyy"
    let InvariantCulture = System.Globalization.CultureInfo.InvariantCulture
    let processLine (inbox: Agent<string>) =
        let rec loop() =
            async 
                {
                    let! cmd = inbox.Receive()
                    if (System.String.IsNullOrEmpty(cmd) || cmd.Contains("\"limit\"")) then
                        return! loop()

                    let parsed = TweetStructureProvider.Parse(cmd)
                    // Create a Tweet from the parsed data
                    let tweet = 
                        {
                            Tweet.Id = parsed.IdStr
                            Tweet.CreatedAt = System.DateTime.ParseExact(parsed.CreatedAt,twitterDateFormat, InvariantCulture).AddHours(2.0)
                            Tweet.Creator = 
                                {
                                    TwitterUser.Name = parsed.User.Name
                                    TwitterUser.ScreenName = parsed.User.ScreenName
                                    TwitterUser.ProfileImageUrl = parsed.User.ProfileImageUrl
                                }
                            Tweet.Source = parsed.Source
                            Tweet.Text = parsed.Text
                            Tweet.UserMentions = parsed.Entities.UserMentions |> Seq.map (fun u -> { UserMention.Name = u.Name; UserMention.ScreenName = u.ScreenName }) |> List.ofSeq
                        }
                    TweetReceivedSubject.OnNext tweet
                    MessageReceivedSubject.OnNext (TweetReceived tweet)
                    return! loop()
                }
        loop()
    let TweetParsingAgent= new Agent<string>(processLine)
    StreamLineReceivedSubject.Subscribe TweetParsingAgent.Post |> ignore

The last line subscribes our agent's post event to the StreamLineReceivedSubject that has it's OnNext function called when we receive a line from the Twitter stream. Notice also our parsing of the Twitter date format (I add 2 hours to get the time into my local time, as it seems that Twitter gives us times in GMT). We now have almost enough to start getting our web front-end involved. There is one last step that we will want to perform, and that is getting something to send our tweets to the front end. For this, we are going to use SignalR and Reactive Extensions again. The first thing we will do is define a message type. This message type is a discriminated union that has cases for the different types of messages that we would like to send to the front-end. This union is defined like so:

type Message =
    | TweetReceived of Tweet: Tweet
    | WordCountGenerated of WordCount: WordCount seq
    | UserTweetCountGenerated of TweetCount : UserTweetCount seq
    | TweetTimeCountGenerated of TweetCount: TimedTweetCount seq
    | MostMentionsGenerated of TweetCount: UserTweetCount seq

Now that we have this message type defined, we need something that can handle it. Again, using the magic of Reactive Extensions, we define a Subject of type Message, and then an agent that will process Message objects. We also need to pull in FSharp.Dynamic so that we can use the ? operator as a dynamic operator with SignalR. These are defined as follows:

let MessageReceivedSubject = new Subject<Message>()

module MessagingAgent = 
    open Microsoft.AspNet.SignalR
    open EkonBenefits.FSharp.Dynamic
    let ProcessMessage(inbox: Agent) =
        let hub = GlobalHost.ConnectionManager.GetHubContext()
        let rec loop() =
            async 
                {
                    let! cmd = inbox.Receive()
                    match cmd with
                    | TweetReceived tweet -> 
                        hub.Clients.All?tweetReceived(tweet)
                    | _ -> failwith "No processing specified yet"
                    return! loop()
                }
        loop()
    let MessagingAgent = new Agent<message>(ProcessMessage)
    MessageReceivedSubject.Subscribe MessagingAgent.Post |> ignore

Essentially the agent above just listens for messages on the MessageReceivedSubject, posts them to the MessagingAgent's ProcessMessage function and then uses the ? dynamic operator to send those message to all connected SignalR clients. At this stage, we are finally ready to get some front-end involvement going. To do this, we will need to create a script module and bring in the FunScript nuget package, as well as the FunScript.TypeScript.Binding.jquery and FunScript.TypeScript.Binding.signalr libraries. I created two modules, one is a local bindings module where I can add a little bit of syntactic sugar to the way FunScript handles certain things (I don’t like all of the code that FunScript generates, so I wrote a few things to make it cleaner), as well as a module that contains the core script logic. These modules are shown below (I have them in two separate files, LocalBindings.fs and ScriptModule.fs):

[<AutoOpen>]
module LocalBindings

open JSinSADemo.Types

type TweetHubServer =
    abstract member startAgents : unit -> unit

type FunScript.TypeScript.HubProxy with
    [<FunScript.JSEmitInline("{0}.server"); CompiledName("server")>]
    member this.server with get() : TweetHubServer = failwith "never"

type System.Object with
    [<FunScript.JSEmitInlineAttribute("{0}"); CompiledName("")>]
    member this.AsTweet with get() : Tweet = failwith "never"

    [<FunScript.JSEmitInline("{0}"); CompiledName("")>]
    member this.Ignore with get() : unit = failwith "never"



[<FunScript.JS>]
module ScriptModule

open FunScript
open FunScript.TypeScript


let jq (selector: string) = Globals.Dollar.Invoke(selector)

let DrawTweet (input: obj[]) =
    let tweet = input.AsTweet
    let out = 
            "<div class='list-group-item'>" + 
            "<h4 class='list-group-item-heading'>" +
            "<img src='" + tweet.Creator.ProfileImageUrl + "'/>" + tweet.Creator.Name + " (" + tweet.Creator.ScreenName + ")" +
            "</h4>" +
            "<p class='list-group-item-text'>" + tweet.Text + "</p>" + 
            "</div>"
    jq("#TweetsBox")
        .prepend(out)
        .Ignore

let Main() =
    let hub = Globals.Dollar.connection.hub.createHubProxy("tweetHub")
    hub.on("tweetReceived", fun x -> DrawTweet x).Ignore
    Globals.Dollar.connection.hub.start()._doneOverload2(fun _ -> hub.server.startAgents())

With these two modules in place, there are two things left to do before we have our solution actually doing something. The first is to tell the FunScript compiler to compile the script, and the second is then to reference the compiled script (warning: What I am doing here is getting the script to be compiled on each request. Do not do this in production - rather precompile the script). To get the FunScript compiler to compile our script, in a method on our HomeControlller, we call FunScript.Compiler.Compiler.Compile(<@ ScriptModule.Main() @>), like follows:

type HomeController() =
    inherit Controller()
    member this.Index () = this.View()

    member this.MainScript() =
        FunScript.Compiler.Compiler.Compile(<@ ScriptModule.Main() @>)

After that we need to get our front-end to include this generated script, which is simply a case of adding one line to our _Layout.cshtml:

<script src="~/Home/MainScript"></script>

Provided everything so far has worked, you should be able to hit that url and see your script generated, as well as hit the home page and see tweets flowing in. Here is probably a good time to take a break. In the next post I will show the magic that we do with the local d3 bindings, as well as how to get FunScript and Highcharts working nicely together.