Wednesday , October 28 2020

Deepstream 5.0: Resurrected using MIT license, Hacker News

Features :

  • New License
  • Singular HTTP Service
  • SSL Support reintroduced
  • Better Config file validation
  • JSON LoggerNGINX Helper

  • Combined authentication handler
  • Embedded dependencies
  • Builtin HTTP Monitoring
  • Storage authentication endpoint
  • Guess whats back, official clustering support!

Backwards compatibility

  • Custom authentication plugins now have to use the async / await API
  • Custom permission handlers need to slightly tweak the function arguments
  • Deployment configuration has to be (simplified) to respect the single HTTP / Websocket port

Upgrade guide

You can see the upgrade guide for backwards compatibilityhere



This part deserves its own post and I can’t try to do it justice here. The summary is that all deepstream components will be released using the MIT license, starting with the heavily worked on Server and Client in V5. What exactly does this mean?

  • Clustering is open source (with redis support out of the box on docker and binaries)
  • Monitoring endpoint is open source

Single HTTP Service

After working with multiple clients on deploying deepstream we realized one large annoyance was how our plugins previously ran in isolation. What this meant was that we would run our HTTP service under port 8080, deepstream on 6020, and all the other services on different ports (like monitoring and such). This has a massive limitation of having to constantly update the deployment config to respect the ports, as well as generally just not supporting more then one on some load balancers.

The other issue was that we had multiple different network stacks running in deepstream with code duplicated in all those endpoints.

In V4 (and prior) we had config that looked like this (ignoring a bunch of configuration options and extra endpoints):

connectionEndpoints:  -(type):WS-websocket     options:                 port:6020                 host:                  healthCheckPath:/ health-check    -(type):node-http     options:             port:8080             host:              healthCheckPath:/ health-check    -(type):UWS-websocket     options:                 port:6020                 host:                  healthCheckPath:/ health-check                   sslKey:(null)          sslCert:(null)          sslDHParams:(null)          sslPassphrase:(null)

What this means is every single endpoint was responsible for setting up its own HTTP / Websocket Server, health-checks, SSL certification and so forth.

Now lets take a look at V5:

httpServer:  type:default   options:         healthCheckPath:/ health-check                         allowAllOrigins:(true)           Origins:      -''         Headers:      -  (user)  -agent connectionEndpoints:  -(type):WS-binary     options:      urlPath:/ Deepstream       ...  -(type):WS-text     options:      urlPath:/ Deepstream-v3       ...-(type):WS-json     options:      urlPath:/ Deepstream-json       ...  -(type):http     options:        ...

Different right? The idea here is the entire connection stack can now share a global HTTP server via some nifty HTTP helpers:

thisservices.httpService.registerPostPathPrefix(prefix,(body,meta,Response)=>{  Response(error,responseData)})thisservices.httpService.registerGetPathPrefix(prefix,(meta,response)=>{  Response(error,responseData)})thisservices.httpService.registerWebsocketEndpoint(urlPath,createWSSocketWrapper,{  onConnection:(socketWrapper)=>{},  onSocketClose:(socketWrapper)=>{}}thisservices.httpService.sendWebsocketMessage(socket,message,  (isBinary) )thisservices.httpService.getSocketWrappersForUserId(userId)

The best part about this is that you can change the httpServer to uws to have the entire server run using it

httpServer:  type:uws   options:         healthCheckPath:/ health-check

SSL Reintroduced

SSL was removed from the V4 server because of the multiple ports issue. We recommend (and still do!) That our customers deploy SSL in production via SSL termination, such as that provided with Nginx and all modern Load balancers.

However, sometimes we don’t want to justify having to run an entire separate process (clouds ain’t cheap) just to have a demo application running. Because of that, and the introduction of our newfileLoadandfileconfig macros we can now enable SSL by just doing the following:

httpServer:  type:default   options:         healthCheckPath:/ health-check                         allowAllOrigins:(true)           Origins:      -''         Headers:      -  (user)  -agent          SSL:      key:fileLoad (ssl / key.pem)       cert:fileLoad (ssl / cert.pem)     


type:UWSoptions:     healthCheckPath:/ health-check      Headers:    -  (user)  -agent      SSL:    key:file (ssl / key.pem)     cert:file (ssl / cert.pem)      

Better Config file validation

Remember when you would stare at the screen with despair because you would get an error like thisInvalid configuration: data should NOT have additional propertiesand have no idea why or where its from?

V5 now uses some smarter error handling, allowing you to get errors that make a bit more sense.

There was an error validating your configuration: 1) Property listens is not expected to be here 2) / logger / type should be equal to one of the allowed values: default, Did you mean default?

JSON Logger

You can now use a JSON logger for deepstream (useful for streaming data to ELK stack and so forth) by setting the name of the logger to pino

NGINX Helper

You can output nginx config for deepstream (automatically generated from config) by running

Not the most useful command for some, but here by popular demand!

Combined authentication handler

The Authentication mechanism has changed in order to support multiple strategies at the same time.

In V4 you could only set one auth endpoint:

When in V5 you now are required to set an array:

What this means is when a user logs in we can check multiple different authentication mechanisms before saying a user was not found. Some useful scenarios are when you want to keep deepstream services authentication local to the server, and only forward front-end clients to a login server. Another extremely useful setup is to first check if a user is actually available, and if not give them an anonymous view (which can be extremely restricted when it comes to permissions).

This also means that individual authentication handlers moved from this API:

isValidUser(connectionData,authData,callback){  callback(isValid,{    username:'string',    clientData:{},    serverData:{}  })}

To this one:

asyncisValid User(connectionData:any,authData:any){     return{    isValid:true,    ID:'string',    clientData:{},    userData:{}  }        return(null)}

Embedded Dependencies

The docker image and binaries now have all the official plugins pre-bundled. This has been done to remove the wholedeepstream installCLI. It was really painful trying to create a binary for each individual plugin and operating system (for native dependencies). Since plugins are barely changed any fixes will be released in the following deepstream release. You can still select specific versions, but it’s unlikely you would need to.

Builtin HTTP Monitoring

Added a HTTP monitoring API which provides data in a format that can easily be consumed by a data visualization / metric tool. A guide will be released to show how this can be done using the ELK stack!

OpenSource clustering support

This deserves a blog post for itself one day talking about the difficulties of trying to have an entirely opensource server while also being able to run a company around it. The final outcome however is starting from V5 clustering is officially back and supported, with a redis clustering mechanism that can be used via node@ deepstream / clusternode-redisand is shipped with deepstream binary / docker images. Happy Clustering!

Brave Browser
Read More

About admin

Leave a Reply

Your email address will not be published. Required fields are marked *