Wednesday, February 1, 2012

gst-plugins-cl - OpenCL plugins for gstreamer, written in Vala.

Some of you may know OpenCL - the open computing language. This language allows us to write kernels in a C99 dialect, which can be run on data - in parallel.
A common use case for such a language - and processing concept - is the classical convolution - which was my motivation. Sure, this can also be realized using the frequency domain, but - hey - I feel fine in the spatial one.

Anyhow, there are bindings for OpenCL - which is vendor (NVidia, AMD, Intel, F/LOSS, ...) and hardware (GPU vs. CPU - AMD, Intel, IBM Cell, ...) independent for many languages, including MatLab, python, C, C++  and so on ...
In my case I needed a way to apply some filter function (or [OpenCL] kernel) to a video stream from some FireWire camera.
The experienced reader might directly associate FireWire with gstreamer our multimedia framework of choice under linux.
I had the option to Gather the data from a gstreamer pipeline reading the FireWire camera or create a gstreamer element providing, as of wrapping, the OpenCL functionality. The mentioned plugin does the latter, it provides a couple of elements to apply some OpenCL kernel to the data of a gstreamer pipeline.

You need vala and some gstreamer sources to build this project:
$ sudo yum install gstreamer-devel vala
.. might be a good start.
You also need some OpenCL implementation, as of today pocl can be used, but it's just intensively tested with the Nvidia implementation of it's GPU SDK and with Intel OCL SDK.
Building the plugin is quite simple
$ ./
$ ./configure
$ make

Testing should also be quite easy:
$ export GST_PLUGIN_PATH=src/.libs/
$ gst-launch audiotestsrc ! clkernel ! autoaudiosink
This doesn't do much, as the default kernel just passes the data - but at least you can test if the OpenCL is available. There are currently three plugins:
  • clkernel - An element to manipulate any buffer
  • clkernel2d - An element to manipulate a 2D buffer, like an gray image/video
  • clvideofilter - An element to manipulate an RGBA image/video
The OpenCL kernel can reside in an external file. You can pass the filename of that file and the kernel to use to the element:
$ gst-launch audiotestsrc \
  ! clkernel kernel-name=somekernel \
  ! autoaudiosink
Have a look at the sources or and files (they are plaintext files, the kernel are build on the fly - heard of JIT?) in the src directory to see what kind of signature the kernel has to provide. (This should be documented more thoroughly).
If you've got more than one platform which is OpenCL capable you can use the platform-idx property to specify a specific platform.

This plugins should help to offload some functionality to one or more devices (like GPU, CPU and accelerator, like the CELL) and you don't need to care about the details.
This can now be used to do things like color conversion, all sorts of filtering, mixing and what else can be done in parallel.
Thos plugins are just a start, it would be nice to be able to provide more sinks which then can be used as an input to the kernels.



  1. hi, interesting project:)
    I work generally on debian's based Ubuntu and have problems getting Intel OCL SDk installed there, do You know how to do it?

    1. Sorry, I've got no idea if or how to install Intel's SDK on debian based distors. But I'd guess that it might be already packaged.

  2. Hi,
    I have not much idea about Gstreamer but it seems to be a way of using GPU HW decoding units in gstreamer with multiple apis like VDPAU (nv) XBVA (ati) and VAAPI (intel).. would be good know if a video source plugin using this apis can interoperate efficiently with your OpenCL plugin (read no CPU-GPU transfers) basically the idea would be to use in your kernels image objects from OpenGL textures (so CL/GL interop) which would be from video frames decoded using this HW video APIs (using VDPAU/XvBA/VAAPI GL interop)..
    Are you interested in trying to optimize your sample for this usage..
    (VAAPI seems the most reasonable to use since there a XVBA VDPAU wrappers)..

    1. You are absolutely right, it would be a lot more efficient to pass the OpenCL buffer/mem to a subsequent element. In such a case we could save us the expensive GPU/CPU copies.
      I've already noted this in the README - it's somewhat a longterm goal. Patches are welcome :)

  3. ok I've managed to install it on Ubuntu but have problems with install Your package, ubuntu has valac and gstreamer-dev packages I don't know if they're similar with 'gstreamer-devel and vala'
    after cloning git (I'm in "gst-plugins-cl" folder) when I hit ./ there's no file found

    1. Just ask this question in your favourite Ubuntu or Debian IRC channel - I suppose you need some distro specififc help.