Plugins, Extensible Functions, and Services Center

This document describes the Extensions, Plugins, JavaScript UDFs, External Services, File Storage, and Backup Migration Center in rekuiper.

---

Overview

The Extensions and Services Center provides a unified administration console for edge extensibility. It allows operators to install native and WebAssembly (Wasm) plugins, write and test JavaScript user-defined functions (UDFs/UDAFs), discover RPC methods on external microservices, upload configuration files, and export or import complete system backups.

The system requires no external dependencies. The entire management hub is built into the single binary edge distribution of rekuiper.

---

Architecture Flow

The diagram below shows the interaction between the five extensibility sub-modules, the edge SQL engine, and external services:

---

1. Native and WebAssembly (Wasm) Plugins

The Plugins panel (#ext-panel-plugins) manages executable binaries and modules loaded by the engine.

Plugin Types

The system classifies plugins into five categories:

1. Source Plugins (source): Custom protocol adapters that ingest streaming data into the engine.
2. Sink Plugins (sink): Custom publishers that deliver processed messages to proprietary backends.
3. Function Plugins (function): Native compiled algorithms callable directly inside SQL statements.
4. Wasm Plugins (wasm): WebAssembly bytecode modules running in a secure, memory-isolated runtime.
5. Portable Plugins (portable): Independent system processes communicating over local IPC channels.

Management Procedures

1. View Installed Plugins:

   • Open the web console at http://<host>:9081/ui.
   • Select the Extensions & Services tab.
   • Choose the 🔌 Native & Wasm Plugins sub-tab.
   • Filter plugins by category: All Plugins, Sources, Sinks, Functions, or Wasm.

2. Register a Plugin:

   • Click the + Install Plugin button.
   • Select the plugin type.
   • Enter the unique plugin name.
   • Provide the file URI or artifact location.
   • Click Install Plugin to initiate verification and loading.

3. Inspect Plugin Metadata:

   • Click the Inspect button on any installed plugin card.
   • Review file size, target architecture, export symbols, and memory status.

---

2. JavaScript UDFs and Extensible Functions Studio

The JavaScript Studio (#ext-panel-udfs) enables inline script creation and immediate verification.

Capabilities

• Zero-Compile Authoring: Write custom transformation and aggregation logic without setting up a Rust or C build chain.
• Embedded Engine: Scripts run inside the embedded QuickJS engine for sub-millisecond execution latency.
• Client-Side Test Sandbox: Evaluate scripts with sample JSON parameters in the browser prior to saving to the edge server.
• 221+ Built-in Function Palette: Search and copy standard SQL functions across 14 categories (Math, String, Aggregate, Conversion, Time, JSON, Array, Bitwise, Hashing, System, IIoT, EdgeX, Dynamic, State).

Authoring and Testing Workflow

1. Click + New JS Function.
2. Select an authoring template:
   • Celsius to Fahrenheit: Linear scalar transformation.
   • JSON Field Filter: Dynamic object filtering.
   • Running Mean: Stateful rolling accumulator.
3. Specify the Function Name (e.g. celsius_to_fahrenheit).
4. Write or customize the JavaScript function in the code editor:

   function celsius_to_fahrenheit(c) {
     if (c === null || c === undefined) return null;
     return (c * 9.0 / 5.0) + 32.0;
   }

5. Enter test inputs into the Test Input Arguments box:

   [25.0]

6. Click ⚡ Run Test in Sandbox.
7. Confirm that the status badge indicates ✓ Sandbox Passed with output 77 and verified execution latency (e.g. 12 µs).
8. Click Save JS Function to deploy to the edge engine.

Using in SQL Rules

Call the function directly inside streaming SQL queries:

SELECT
  device_id,
  temperature AS celsius,
  celsius_to_fahrenheit(temperature) AS fahrenheit
FROM demo_stream
WHERE celsius_to_fahrenheit(temperature) > 100.0;

---

3. External Services and RPC Discovery Hub

The External Services panel (#ext-panel-services) connects rekuiper to external microservices over gRPC or REST.

Key Features

• Service Registration: Connect external containers, cloud endpoints, or on-premise RPC servers.
• Automated Method Discovery: Query service reflection to enumerate callable RPC methods.
• Dynamic SQL Integration: Call remote methods directly within SQL projections using the service_name.method_name(args...) syntax.

Registration Procedure

1. Click + Register Service.
2. Fill in the connection properties:
   • Service Name: Unique identifier (e.g. defect_detector_rpc).
   • Protocol: gRPC or REST.
   • Endpoint URL: Network address (e.g. http://127.0.0.1:50051).
   • Timeout (ms): RPC call deadline (default: 5000).
3. Click Register Service.
4. The system validates connectivity and lists all exported functions in the Service Functions grid.

---

4. File Storage and Artifacts Hub

The File Storage panel (#ext-panel-uploads) manages auxiliary edge files without external shared storage.

Supported File Types

• TLS / SSL Certificates: Client certificates (.pem, .crt, .key) for mutual TLS.
• Protobuf Specifications: .proto definitions for binary message decoding.
• Machine Learning Models: .onnx and .tflite model files for inference rules.
• Binary Codecs: Compiled dynamic libraries and custom register lookup tables.

Upload and Usage

1. Click Upload File.
2. Select a local file or drop it into the designated drop zone.
3. Click Upload File.
4. Use the Copy Path button to copy the absolute edge filesystem path (e.g. data/uploads/sensor_contract.proto) for use in stream configurations and sink rules.

---

5. Data Backup, Snapshot Export, and Migration Hub

The Backup and Migration panel (#ext-panel-backup) protects edge configurations against hardware failure and simplifies fleet provisioning.

Backup Options

• JSON Full Snapshot (/data/export): Produces a single JSON document containing all streams, tables, rules, plugins, JavaScript UDFs, schemas, and configurations.
• GitOps YAML Bundle (/v2/data/export): Produces human-readable declarative YAML manifests for version-controlled GitOps workflows.

Restore and Migration Procedure

1. Navigate to the 💾 Data Backup & Migration sub-tab.
2. Under Restore & Migrate Configuration, select a backup file (.json or .yaml).
3. Select the merge strategy:
   • Safe Merge (Default): Adds new definitions and updates modified entities without deleting unreferenced streams.
   • Full Overwrite: Replaces existing configurations with the contents of the backup snapshot.
4. Click Apply Restore Snapshot.
5. The console shows a real-time summary of restored entities:
   • Streams restored: N
   • Rules restored: N
   • UDFs restored: N
   • Services registered: N

---

REST API Reference

The Extensions and Services Center is fully accessible through standard REST endpoints:

Method	Endpoint	Description
GET	/plugins/sources	List installed native source plugins
GET	/plugins/sinks	List installed native sink plugins
GET	/plugins/functions	List installed native function plugins
GET	/plugins/wasm	List registered WebAssembly modules
POST	/plugins/<type>	Register a new plugin of specified type
DELETE	/plugins/<type>/<name>	Remove an installed plugin
GET	/udf/javascript	List all saved JavaScript UDFs and UDAFs
POST	/udf/javascript	Create or update a JavaScript function
DELETE	/udf/javascript/<id>	Delete a JavaScript function
GET	/metadata/functions	Retrieve the complete palette of 221+ built-in functions
GET	/services	List registered external microservices
POST	/services	Register a new external microservice
GET	/services/functions	Discover exported RPC functions
GET	/config/uploads	List uploaded files and certificates
POST	/config/uploads	Upload a new file artifact
GET	/data/export	Download full system JSON snapshot
POST	/data/import	Upload and apply system configuration snapshot