Apply in live streaming and WebRTC

  • By ossrs
  • Last update: Aug 8, 2023
  • Comments: 0


This is a demonstration for applying k2-fsa/sherpa-ncnn to live streaming and WebRTC by SRS.

It's not bound to SRS, so you're able to use other media servers, or use microphone or device as audio source of FFmpeg, for example, use FFmpeg to pull IPC/microphone/RTMP/RTSP/UDP stream and converting to text.

The workflow is like this, the original picture is here

+------------+                          +------------------------+
| Microphone +---------->---------------+ sherpa-ncnn-microphone +--+
+-----+------+                          +------------------------+  |
      |                                                             +--> Text
      |   +---------+     +------------+    +--------------------+  |
      +->-+   OBS   +-->--+  SRS Cloud +-->-+ sherpa-ncnn-ffmpeg +--+
          +---------+     +-----+------+    +--------------------+ 

Usage: OBS

First, run in docker:

docker run --rm -it -p 1935:1935 -p 1985:1985 -p 8080:8080 ossrs/k2:1

Note: Please use if in China.

Then, download and publish by OBS:

  • Server: rtmp://localhost/live/
  • Stream Key: livestream

Or, publish microphone by ffmpeg for macOS:

ffmpeg -f avfoundation -i ":0" -acodec aac -ab 64k -ar 44100 -ac 2 \
  -f flv rtmp://localhost/live/livestream

Note: Use ffmpeg -f avfoundation -list_devices true -i "" to list microphones.

Usage: URL

If you got a FFmpeg URL, you can set the env SHERPA_NCNN_INPUT_URL:

docker run --rm --env SHERPA_NCNN_INPUT_URL=url ossrs/k2:1

Please note that the url can be any URL that supported by FFmpeg, for example:

  • RTMP: rtmp://
  • HLS: htto://
  • HTTP-FLV: htto://

Or any other format supported by FFmpeg.

Usage: WebRTC

First, get your ip by ifconfig, for example

ifconfig en0 |grep 'inet '
#	inet netmask 0xffffff00 broadcast

Next, start docker with IP setting:

docker run --rm -it -p 1935:1935 -p 1985:1985 -p 8080:8080 -p 8000:8000/udp \
    --env CANDIDATE="" ossrs/k2:1

Note: Please use if in China.

Then open the web page:


Click the button and it starts to work like magic.