Live Coding On Twitch

2017 Sep14


oing to start doing some live coding on twitch! I've been in the open source space for a long while and it seems fitting that this is the next step. I don't really have a schedule in mind or much else lined out. Other than -
It's probably happening

Check out my channel on twitch. I'll do my best to send out a heads up when I plan on doing something. I'll probably work on Skyring, Tastypie or maybe dust off some stuff from my megadoomer collection of things.

I'm Twitchin'!

Read More
filed under:  twitch streaming live coding

Exactly Once Execution In A Distributed System

2017 Sep05


kyring is a distributed system for managing timers, or delayed execution similar to setTimeout in javascript. The difference being that it is handled in a reliable and fault tolerant way. setTimeout in javascript is transient. If the running application is restarted or crashes, any pending timers are lost forever. The one guarantee that skyring provides is that a timer will execute after the specified delay, and that it only executes once. Exactly once is an interesting challenge in distributed systems, and Skyring makes use of a number of mechanisms at the node level to achieve this. From a high level, this is what the behavior on individual nodes looks like.

Skyring Node Behavior

Shared Nothing

Skyring follows the shared nothing mantra similar

Read More

Custom Tansports For Skyring

2017 May30


kyring is a distributed system for managing timers. When a timer lapses, a message is delivered to destination that you have defined. How that message is delivered is configurable. Out of the box, Skyring comes with an HTTP transport, and there is an official package enabling tcp delivery of messages with connection pooling. They are pretty easy to write, and you can use any of the tools you are currently used to using.

STDOUT Transport

To illustrate the process, we're going to make a simple transport handler to write the data to stdout. Basically, speaking a transport is just a node.js module that exports a named function

Module [ˈmäjo͞ol] -n., --noun

any of a number of distinct

Read More
filed under:  zmq skyring timers node.js

Getting Started With Skyring Distributed Timers

2017 May02


he very idea of distributed timers is complex. Conceptually is full of race conditions and edge cases. Skyring for Node.js boils the problems space down to a simple to use library and API for building scalable service that need to perform time sensitive, actions. That is a mouthful - Think An email gateway, a web-hook service, auto-dialers for telephony systems. Or in the most practical sense, anytime you might need functionality like setTImeout but needs to survive restarts / crashes; Or are using a language that doesn't support non-blocking timers. Skyring fills that gap, and it is easy to use. We can get something going in less that 20 lines of code.

To start, we just install the skyring

Read More
filed under:  skyring timers node

Build JSON API Responses With Postgres CTEs

2017 Apr30


agination is a recurring problem that developers have to deal with when implementing data access layers for APIs. It can be particularly tricky with the more traditional RDMS like MySQL or Postgresql. For example, let's say we had an API endpoint that allowed consumers to search a data base of moves. We could search by title, director, starring actors, etc. Our data base has millions of movies, and we know we don't want to return all all the potential matches for every search request. We only want to return the top 25 or so records and indicate in the response that there are more results to query for:

  meta: {
    total: 12000
  , limit: 25
  , next: <URL TO NEXT PAGE&

Read More
filed under:  sql postgres node.js

Distributed Timers With Node.js and Skyring

2016 Dec28


orking with timers a distributed system is a really nasty problem that pops up more often than most people would like. Something as simple an useful as setTimeout / clearTimeout becomes brittle, unreliable and a bottle neck in today's stateless scaleable server mindset. Basically, I need to be able to set a timer somewhere in a cluster with out knowing or caring about what server. And reversely, I need to be able to cancel that timer without having to know where in the cluster that timer lives. But before we can start to understand possible solutions, let's dive into a use case to understand the problem and why existing solutions aren't suitable replacements.


Un-send an email - A simple

Read More

Distributed Timers with Node.js, Dgram and multicast

2016 Sep30


ver the years I have had a handful of times where I have had the need for Multicasting. It usually comes about in service or node ( application ) discovery. Or if you are lucky enough, the dreaded distributed setTimeout / clearTimeout. Every time find a need for it, I spend hours in the documentation trying to remember how to use it, remembering the Node.js Docs for dgram - which are virtually void of any useful explanation or examples. Finally resorting to finding googling around for the 1 or two examples of multi casting out there and hack something together. It shouldn't like that. It is actually really easy to do with Node.js. Let's give it a shot.

Let's say

Read More
filed under:  udp multicast dgram networking node

Summer of Sockets Part 5: Node, Nanomsg and Websockets

2016 Sep22


eroMQ has its fair share of quirks and oddities. It manages everything in a global state, requires things be manually grouped into Contexts, allocates a thread per context (making it not thread-save) , transports are baked into the library, and so on. It can be a bit clunky to work with at times. As a result, one of the original developers on the ZMQ project, Martin Sustrik, started a new project that evolved into a complete re-write / re-realization of the ZMQ project, called nanomsg.

Nanomsg aims to resolve many of the underlying short comings of the zeromq library, but remain compliant with the ZMTP spec. It provides many of the messaging patterns ( which are refereed to as scalability patterns ) as

Read More

Node Style Woes - Domains and Promises

2016 Sep12


omains have been the red-headed step child of error handling in Node.js It is a library that has been deprecated since v0.12 and has been awaiting a suitable replacement ever since ( we are at v6.5 at the time of writing ). Until one has been implemented by the Node Core team, it is still de-facto way to deal with error propagation. As Node.js supports more and more ES6 features, I have been upgrading my open source projects where it seem appropriate. In my command line tool package, seeli, I was doing some updates and came across some exceptionally odd behavior around ES6 Promises and implicit Domain binding. In a nutshell - It's broke.

ˈməNGkē paCH -v,

Read More
filed under:  class es6 promises domain node

Custom Merge Tools Using GIT

2016 Sep09

IT has been my SCM tool of choice for white a number of years. Along with it, VIM has always been my intermediate editor for doing commit messages, viewing diffs and resolving merge conflicts. Recently, I have made the switch to Neovim as modern replacement for VIM. The nice thing with vim is that it came with a number of handy aliases, such as vimdiff. Which was unfortunate, because I really liked VIM's diff support for GIT. I wanted to replicate that, but that would require some manual work on my part to get the two to play nice. Luckily, it isn't really that hard - turns out you can define any custom diff / merge tool for GIT. Here

Read More
filed under:  git SCM merge