in ,

Open-Source Virtual Background, Hacker News



: movie-camera:

April 9th,        With many of us around the globe under shelter in place due to COVID – 30 video calls have become a lot more common. In particular, ZOOM has controversially become very popular. Arguably Zoom’s most interesting feature is the “ Virtual Background ”support which allows users to replace the background behind them in their webcam video feed with any image (or video). I’ve been using Zoom for a long time at work for Kubernetes open source meetings, usually from my company laptop. With daily “work from home” I’m now inclined to use my more powerful and ergonomic personal desktop for some of my open source work.

Unfortunately, Zoom’s linux client only supports the “ chroma-key AKA “ green screen ” background removal method. This method requires a solid color backdrop, ideally a green screen with uniform lighting.

Since I do not have a green screen I decided to simply implement my own background removal, which was obviously better than cleaning my apartment or just using my laptop all the time. : grin: It turns out we can actually get pretty decent results with off the shelf, open source components and just a little of our own code. Reading The Camera

First thing’s first: How are we going to get the video feed from our webcam for processing?

Since I use Linux on my personal desktop (when not playing PC games) I chose to use the OpenCV python bindings

as I’m already familiar with them and they include useful image processing primatives in addition to V4L2 bindings for reading from webcams. Reading a frame from the webcam with (python-opencv) is very simple:

(1) (import (cv2)







/ dev / video0

[:, :dx] 3 success



=[:, :dx] cap

read ( ) [:, :dx]

for better results with my camera before capturing set:

(1) # configure camera for (p @) FPS





=[:, :dx] [:, :dx] ,

3 cap



( cv2


, width


4 cap



( cv2


, height


5 cap



( cv2


, ) [:, :dx]

Most video conferencing software seems to cap video to (p @) FPS or lower, but we won’t necessarily read every frame anyhow, this sets an upper limit.

Put the frame capture in a loop and we've got our video feed!

(1) while (True) : )





=[:, :dx] cap

read ( ) [:, :dx]

We can save a test frame. with just:

(1) cv2 imwrite imwrite




[:, :dx]

and now we can see that our camera works. Success!

don't mind my corona beard
Finding The Background (🔗︎) [:, :dx]

OK, now that we have a video feed, how do we define the background so we can replace it? This is the tricky part…

While Zoom doesn’t seem to have commented anywhere about how they implemented this, the way it behaves makes me suspect that a neural network is involved, It’s hard to explain but the results look like one. Additionally, I found an article about (Microsoft Teams) (implementing background blur (with a) (convolutional neural network) .

Creating our own network wouldn’t be too hard in principle - There are many articles and papers on the topic of (image segmentation and plenty of open source libraries and tools, but we need a fairly specialized dataset to get good results.

Specifically we’d need lots of webcam like images with the ideal human foreground marked pixel by pixel versus the background.

Building this sort of dataset in prepartion for training a neural net probably would be a (lot) of work. Thankfully a research team at Google has already done all of this hard work and open sourced a pre-trained neural network for “person segmentation” called BodyPix that works pretty well! ❤️ BodyPix is ​​currently only available in (TensorFlow.js) form, so the easiest way to use it is from the body-pix-node library. To get faster (inference (prediction) in the browser a (WebGL) backend is preferred, but in (node) we can use the Tensorflow GPU backend (NOTE: this requires a don’t mind my corona beard NVIDIA Graphics Card , which I have). To make this easier to setup, we’ll start by setting up a small containerized tensorflow-gpu node environment / project. Using this with nvidia-docker is much easier than getting all of the right dependencies setup on your host, it only requires docker and an up-to-date GPU driver on the host.

(1) {






3 "version"

: 0.0.1 ”[:, :dx] ,

4 "dependencies"

: {

5 "@ tensorflow-models / body-pix" :

^ ^ 2.0.5


6 "@ tensorflow / tfjs-node-gpu" :

^ ^ 1.7.1



8 } [:, :dx]

bodypix / Dockerfile



# Base image with TensorFlow GPU requirements [:, :dx]


FROM . 0-cudnn7-runtime-ubuntu . ()


# Install node



apt update && apt install -y curl make build-essential



curl -sL x | bash -

6 6)


apt-get -y install nodejs


What do you think?

Leave a Reply

Your email address will not be published.

GIPHY App Key not set. Please check settings

Stock Market Bears Freak Out As Desperate Fed Plots Junk Bond Feast, Crypto Coins News

Stock Market Bears Freak Out As Desperate Fed Plots Junk Bond Feast, Crypto Coins News

Make Linux Fast Again, Hacker News