Build a streaming data source backend plugin

Introduction

Grafana supports a wide range of data sources, including Prometheus, MySQL, and even Datadog. In previous tutorials we have shown how to extend Grafana capabilities to query custom data sources by building a backend datasource plugin. In this tutorial we take a step further and add streaming capabilities to the backend datasource plugin. Streaming allows plugins to push data to Grafana panels as soon as data appears (without periodic polling from UI side).

For more information about backend plugins, refer to the documentation on Backend plugins.

In this tutorial, you’ll:

• Extend a backend plugin with streaming capabilities

Set up your environment

Before you can get started building plugins, you need to set up your environment for plugin development.

To discover plugins, Grafana scans a plugin directory, the location of which depends on your operating system.

1. Create a directory called grafana-plugins in your preferred workspace.

2. Find the plugins property in the Grafana configuration file and set the plugins property to the path of your grafana-plugins directory. Refer to the Grafana configuration documentation for more information.

   [paths]
   plugins = "/path/to/grafana-plugins"
   

3. Restart Grafana if it’s already running, to load the new configuration.

Alternative method: Docker

If you don’t want to install Grafana on your local machine, you can use Docker.

To set up Grafana for plugin development using Docker, run the following command:

docker run -d -p 3000:3000 -v "$(pwd)"/grafana-plugins:/var/lib/grafana/plugins --name=grafana grafana/grafana:7.0.0

Since Grafana only loads plugins on start-up, you need to restart the container whenever you add or remove a plugin.

docker restart grafana

Create a new plugin

To build a backend for your data source plugin, Grafana requires a binary that it can execute when it loads the plugin during start-up. In this guide, we will build a binary using the Grafana plugin SDK for Go.

The easiest way to get started is to use the Grafana create-plugin tool. Navigate to the plugin folder that you configured in step 1 and type:

npx @grafana/create-plugin

Follow the steps and select datasource as your plugin type and answer yes when prompted to create a backend for your plugin.

cd my-plugin

Install frontend dependencies and build frontend parts of the plugin to dist directory:

yarn install
yarn build

Run the following to update Grafana plugin SDK for Go dependency to the latest minor version:

go get -u github.com/grafana/grafana-plugin-sdk-go
go mod tidy

Build backend plugin binaries for Linux, Windows and Darwin to dist directory:

mage -v

Now, let’s verify that the plugin you’ve built can be used in Grafana when creating a new data source:

1. Restart your Grafana instance.
2. Open Grafana in your web browser.
3. Navigate via the side-menu to Configuration -> Data Sources.
4. Click Add data source.
5. Find your newly created plugin and select it.
6. Enter a name and then click Save & Test (ignore any errors reported for now).

You now have a new data source instance of your plugin that is ready to use in a dashboard. To confirm, follow these steps:

1. Navigate via the side-menu to Create -> Dashboard.
2. Click Add new panel.
3. In the query tab, select the data source you just created.
4. A line graph is rendered with one series consisting of two data points.
5. Save the dashboard.

Troubleshooting

Grafana doesn’t load my plugin

By default, Grafana requires backend plugins to be signed. To load unsigned backend plugins, you need to configure Grafana to allow unsigned plugins. For more information, refer to Plugin signature verification.

Anatomy of a backend plugin

Add streaming capabilities

What we want to achieve here is to issue a query to load initial data from a datasource plugin and then switching to data streaming mode where the plugin will push data frames to Grafana time-series panel.

In short – implementing a streaming plugin means implementing a backend.StreamHandler interface which contains SubscribeStream, RunStream, and PublishStream methods.

SubscribeStream is a method where the plugin has a chance to authorize user subscription requests to a channel. Users on the frontend side subscribe to different channels to consume real-time data.

When returning a data.Frame with initial data we can return a special field Channel to let the frontend know that we are going to stream data frames after initial data load. When the frontend receives a frame with a Channel set it automatically issues a subscription request to that channel.

Channel is a string identifier of topic to which clients can subscribe in Grafana Live. See a documentation of Grafana Live for details about channel structure.

As said in docs in Grafana Live channel consists of 3 parts delimited by /:

• Scope
• Namespace
• Path

For datasource plugin channels Grafana uses ds scope. Namespace in the case of datasource channels is a datasource unique ID (UID) which is issued by Grafana at the moment of datasource creation. The path is a custom string that plugin authors free to choose themselves (just make sure it consists of allowed symbols). I.e. datasource channel looks like ds/<DATASOURCE_UID>/<CUSTOM_PATH>.

So to let the frontend know that we are going to stream data we set a Channel field into frame metadata inside QueryData implementation. In our tutorial it’s a ds/<DATASOURCE_UID>/stream. The frontend will issue a subscription request to this channel.

Inside SubscribeStream implementation we check whether a user allowed to subscribe on a channel path. If yes – we return an OK status code to tell Grafana user can join a channel:

status := backend.SubscribeStreamStatusPermissionDenied
if req.Path == "stream" {
    // Allow subscribing only on expected path.
    status = backend.SubscribeStreamStatusOK
}
return &backend.SubscribeStreamResponse{
    Status: status,
}, nil

As soon as the first subscriber joins a channel Grafana opens a unidirectional stream to consume streaming frames from a plugin. To handle this and to push data towards clients we implement a RunStream method which provides a way to push JSON data into a channel. So we can push data frame like this (error handling skipped):

// Send frame to stream including both frame schema and data frame parts.
_ = sender.SendFrame(frame, data.IncludeAll)

Open example datasource query editor and make sure With Streaming toggle is on. After doing this you should see data displayed and then periodically updated by streaming frames coming periodically from RunStream method.

The important thing to note is that Grafana opens a unidirectional stream only once per channel upon the first subscriber joined. Every other subscription request will be still authorized by SubscribeStream method but the new RunStream won’t be issued. I.e. you can have many active subscribers but only one running stream. At this moment this guarantee works for a single Grafana instance, we are planning to support this for highly-available Grafana setup (many Grafana instances behind load-balancer) in future releases.

The stream will be automatically closed as soon as all subscriber users left.

For the tutorial use case, we only need to properly implement SubscribeStream and RunStream - we don’t need to handle publications to a channel from users. But we still need to write PublishStream method to fully implement backend.StreamHandler interface. Inside PublishStream we just do not allow any publications from users since we are pushing data from a backend:

return &backend.PublishStreamResponse{
    Status: backend.PublishStreamStatusPermissionDenied,
}, nil

Summary