Open-Source Virtual Background, Hacker News

 for better results with my camera before capturing set:  
  (1)  # configure camera for (p @) FPS  
 2 
  height  
, 
 width 

=[:, :dx] [:, :dx] , 
  3   cap  
. 
 set 
  ( cv2  

 CAP_PROP_FRAME_WIDTH 
,  width 
) 
 4   cap  
. 
 set 
  ( cv2  
 CAP_PROP_FRAME_HEIGHT 
,  height 
 ) 
 5   cap  
. 
 set 
  ( cv2  
 CAP_PROP_FPS 
,   )  [:, :dx] 
 Most video conferencing software seems to cap video to (p @) FPS or lower, but we won’t necessarily read every frame anyhow, this sets an upper limit.  
 Put the frame capture in a loop and we've got our video feed! 
 
  (1)  while  (True) : ) 
 2 
  success  
, 
 frame 

=[:, :dx] cap 

 read  ( )  [:, :dx]  
 We can save a test frame. with just:  
  (1)  cv2  imwrite  imwrite 
 ) 
  
 test.jpg 

 ,   
 [:, :dx]  
 and now we can see that our camera works. Success!  
 don't mind my corona beard          
Finding The Background (🔗︎) [:, :dx] 
 OK, now that we have a video feed, how do we define the background so we can replace it? This is the tricky part…  
 While Zoom doesn’t seem to have commented anywhere about how they implemented this, the way it behaves makes me suspect that a neural network is involved, It’s hard to explain but the results look like one. Additionally, I found an article about (Microsoft Teams) (implementing background blur (with a) (convolutional neural network) . 
 Creating our own network wouldn’t be too hard in principle - There are many articles and papers on the topic of (image segmentation  and plenty of open source libraries and tools, but we need a fairly specialized dataset to get good results.  

 Specifically we’d need lots of webcam like images with the ideal human foreground marked pixel by pixel versus the background.  
 Building this sort of dataset in prepartion for training a neural net probably would be a (lot) of work. Thankfully a research team at Google has already done all of this hard work and open sourced a pre-trained neural network for “person segmentation” called BodyPix  that works pretty well! ❤️   BodyPix is currently only available in (TensorFlow.js) form, so the easiest way to use it is from the body-pix-node  library.   To get faster (inference  (prediction) in the browser a (WebGL) backend is preferred, but in (node) we can use the  Tensorflow GPU backend  (NOTE: this requires a  NVIDIA  Graphics Card , which I have).   To make this easier to setup, we’ll start by setting up a small containerized tensorflow-gpu node environment / project. Using this with nvidia-docker  is much easier than getting all of the right dependencies setup on your host, it only requires docker and an up-to-date GPU driver on the host.  

  (1)  { 
 2 
  "name"  
: 
 “bodypix” 
, 
  3   "version"  
:  0.0.1 ”[:, :dx] , 

 4   "dependencies"  
:  { 
 5   "@ tensorflow-models / body-pix"  :  
 ^ ^ 2.0.5 
, 
 6   "@ tensorflow / tfjs-node-gpu"  :  
 ^ ^ 1.7.1 
 7 
  }  
 8   }  [:, :dx] 

 bodypix / Dockerfile 
 Dockerfile 

 
 (1) 
 # Base image with TensorFlow GPU requirements [:, :dx]   
 (2)  
 FROM  nvcr.io/nvidia/cuda: . 0-cudnn7-runtime-ubuntu . () 
  
 3  
 # Install node  
  
 4  
 RUN 
 apt update  &&  apt install -y curl make build-essential 









  
 5  
 && 
 curl -sL https://deb.nodesource.com/setup_20. x  |  bash -  
 6 6)  
 && 
 apt-get -y install nodejs   
 7