Shelloid is an open source IoT-ready real-time big data web application platform built using Node.js and Clojure that contains integrated stream query capability. Shelloid allows developers to easily integrate events from IoT sources such as the Nest thermostat by annotating functions with simple stream query specifications.
The following code illustrates this point
/**
@stream.sdl select: ambient_temperature_f, name_long;
from: nest.thermostat;
where: ambient_temperature_f >= %threshold%
@stream.init highTemperatureInit
@stream.auth highTemperatureAuth
@stream.edgeTriggered
*/
exports.highTemperature = function(data, streamInst, user){
console.log("highTemperature Data: ", data, " for user id ", user);
}
exports.highTemperatureAuth = function(req, streamInst, user, done){
done(true);
}
exports.highTemperatureInit = function(streamDef, done){
var instance = streamDef.newInstance({threshold:80},
"global");
done();
}
In the above code snippets doc-style comments are used to succinctly specify annotations supported by Shelloid. If you prefer not to use comments for such a purpose, a direct style is also supported:
exports.highTemperature =
{
annotations:{
stream:
{
sdl:"select: ambient_temperature_f, name_long;" +
"from: nest.thermostat;" +
"where: ambient_temperature_f >= %threshold%",
init: highTemperatureInit,
edgeTriggered: true,
auth: highTemperatureAuth
}
},
fn: highTemperature
}
function highTemperature(data, streamInst, user){
console.log("highTemperature Data: ", data, " for user id ", user);
for(var k in streamInst.pendingRequests){
var pending = streamInst.pendingRequests[k];
pending.res.send(data);
}
}
function highTemperatureInit(streamDef, done){
var instance = streamDef.newInstance({threshold:80},
"global");
done();
}
function highTemperatureAuth(req, streamInst, user, done){
done(true);
}
In the above code snippet the “highTemperature” function defines a real-time IoT data stream owing to its associated stream annotations. The “stream.sdl” (sdl stands for stream definition language) defines the stream characteristics in a syntax remindful of the SQL. The “stream.init” specifies an initialization function, which in this case, creates a global stream instance. Shelloid will process the stream query in the background invoke the “highTemperature” function when the query condition holds true.
Shelloid supports special edge-triggered stream queries (specified with edgeTriggered annotation). An edge triggered stream handler is invoked if and only if there is a change in the query result from the previous query.
Each stream has an associated URL, similar to the way URLs are associated with controllers. An HTTP request to this URL essentially supplies the caller with the stream query result. The HTTP response to stream URLs contains a ‘tag’ field which can be provided back as query parameter. Whenever there is change in the query result the tag maintained by the server changes. If tag field is specified then the request will return only if server’s tag changes from the specified tag. If provided tag is same as the server’s data tag for the stream instance, Shelloid will hold the request (long polling style) until the stream event fires (support for websocket-based streaming is in progress).
The stream.auth annotation is used for specifying custom authorization function for stream URL requests (note that basic authorization can be achieved using standard authorization modules, e.g., Shelloid has a built-in RBAC module).
Shelloid stack combines two cutting-edge technologies to achieve scalable real-time data processing - Node.js (for real-time I/O) and Clojure (for real-time stream computations). Real-time events from raw data can be generating using simple, declarative SQL-like stream queries.
With its stream query annotations, Shelloid enables web applications to easily generate and process real-time events from IoT sources such as the Nest thermostat, Fitbit/Jawbone fitness bands, and many more. Our vision is to simplify the development of secure and robust IoT-enabled web applications and web services, improving programmer productivity and enabling quick time-to-market.
Shelloid takes care of the infrastructure logic and lets you focus on your business logic, leading to quick time to market for your business-critical applications. Shelloid is open sourced under LGPL license, allowing you to run your commercial closed source applications on the top of it.
Shelloid attempts to bring out the best of two cool modern programming platforms: Node.js and Clojure. Node.js is great for web and real-time. Clojure is great for concurrent computations. So, why not let the Node.js handle the web requests and real-time messaging and use Clojure when there is a heavy computation at hand? Shelloid does just that.
Shelloid is essentially a Node.js web application server with integrated Clojure-based compute service which runs as a separate process. Shelloid takes care of all the integration details. While the Clojure compute service is executing a heavy computation, Node.js event loop is freed up to process other requests.
The following code snippet illustrates how the integration works. In the code, we define a compute service named add in Clojure. From Node.js this service is invoked by calling sh.ccs.add function (sh stands for shelloid, ccs for Clojure compute service). This results in parameters being passed to the CCS process and the Clojure add service function being executed. The result of the Clojure function is passed back to Node.js and the callback is invoked with err message if any and the result value. After passing off the computation to the Clojure, Node.js event loop is freed up to execute other requests.
Node.js
sh.ccs.add(100, 200, function(err, r){
console.log("Result: " + r);
});
Clojure
(service add [a b]
(+ a b)
)
Clojure compute service (CCS) requires Leiningen to be installed somewhere in the system path.
Please Note: More documentation is on the way. Please bear with us for couple of weeks!
Shelloid has an open, extensible architecture. While open sourcing is great for the users, Shelloid goes beyond merely being open source. Shelloid has a open architecture created from ground up, allowing you to easily write extensions that can modify the behaviour of the engine to suit your needs. Many of Shelloid’s built-in features are themselves built as extensions, leading to a relatively small and robust core.
Our vision is to simplify the development of secure and robust web applications and services, improving programmer productivity and enabling quick time-to-market. Shelloid takes care of the infrastructure logic and lets you focus on your business logic, leading to quick time to market for your business-critical applications. Shelloid is open sourced under LGPL license, allowing you to run your commercial closed source applications on the top of it.