Producing real-time Video with Node.js and FFmpeg

Recently I spent a lot of time with child processes and decided to write a simple tutorial about that, and if you intend to use child processes you better know more about streams.

A stream is an abstract interface for working with streaming data in Node.js

We are interested in stdin and stdout streams in this tutorial. Both are streams in Node.js while stdin is a writable stream and stdout is a readable stream.

What we are going to make?

We will make a simple Node.js app that generates an mp4 video from screenshots taken by our program, directly from the buffer without saving those images on disk.

FFmpeg

A complete, cross-platform solution to record, convert and stream audio and video. learn more

FFmpeg obviously has many features but we are only interested in one right now.

Let’s Start Coding

I used the screenshot-desktop module to capture screenshots of my desktop and return a buffer, I used that package to generate images and this is not an essential part of this tutorial instead you can read those images from disk as well.

I use the FFmpeg standalone executable and not any Node API wrapper. The following FFmpeg code is used to produce a slideshow from images provided to stdin.

ffmpeg.exe -framerate 1 -f image2pipe -i - output.mp4

Let's assume we have 5 images in our ./img folder and we want to generate video from these while each frame has a 1-second duration. Since the code above taking inputs from stdin we can use:

cat ./img/* | ffmpeg.exe -framerate 1 -f image2pipe -i - output.mp4

We used cat to read images as raw data and piped to FFmpeg using the | operator. This is actually what we want to accomplish in Node.js in the end.

Creating the Child Process

We used child_process’s spawn method to spawn a new child. The first parameter is the program which is executed by the child and the second one is the argument array. This Node.js code is identical with the FFmpeg we wrote above, the only difference is the framerate parameter because we want our video to have 10fps.

Now we have our FFmpeg process running and expecting images on it’s stdin. As I said above I’ll generate screenshots to use as an image input to FFmpeg. I wrote this code that takes screenshots and sends those image buffers to child process -which running ffmpeg- we spawned.

screenshot() function called every 100ms and it returns the image buffer, child.stdin.write() allow us to write data directly to the child’s stdin buffer. In this case, we are sending the image data we just obtained by taking screenshot.

This interval will be executed 50 times, after that _interval being terminated and we call child.stdin.end(). By calling child.stdin.end() we notify the FFmpeg child process about we finished the proceeding data. FFmpeg is waiting for the end signal, if we don’t send that the child process will hang and won’t produce our output file.

We took screenshots 50 times and the video generated in the and will be 5 seconds length since our fps was set to 10.

Conclusion

The main purpose of this tutorial is about piping and buffers. This is not the best way of recording the screen and I experienced very high CPU usage with higher FPSs and I don’t recommend this approach.

If there any mistakes or unclear parts do not hesitate to contact me by twitter.

References:
https://nodejs.org/api/child_process.html
https://nodejs.org/en/knowledge/child-processes/how-to-spawn-a-child-process/
https://trac.ffmpeg.org/wiki/Slideshow