Metadata Property Forms and Shared Connection Pool Hub

This document describes the Shared Connection Pool Hub, Configuration Keys (ConfKeys) Manager, and dynamic PropertyForm architecture in rekuiper.

---

1. Overview

The web console provides a centralized connection management architecture. This architecture lets operators manage connector credentials, test preflight connectivity, and inspect pipeline dependencies before making changes.

       +-------------------------------------------------------+
       |             Shared Connection Pool Hub                |
       |  - Pooled socket endpoints (MQTT, Kafka, HTTP, Modbus)|
       |  - Real-time reference count safety checks            |
       +-------------------------------------------------------+
                                   |
         +-------------------------+-------------------------+
         |                                                   |
         v                                                   v
+------------------+                               +------------------+
| Dynamic Property |                               | ConfKeys Manager |
| Form Component   |                               | Hub              |
| - Strongly typed |                               | - Reusable key   |
|   controls       |                               |   profiles       |
| - Eye password   |                               | - Master-detail  |
|   reveal         |                               |   browser        |
| - Dual mode JSON |                               | - Preflight test |
|   sync           |                               +------------------+
+------------------+                                         |
         |                                                   |
         +-------------------------+-------------------------+
                                   |
                                   v
             +-------------------------------------------+
             |           Egress Sinks & Streams          |
             |  - Ingestion Streams & Lookup Tables      |
             |  - Heterogeneous Rule Sinks (MQTT, REST)  |
             +-------------------------------------------+

---

2. Shared Connection Pool Architecture

The following vector graphic displays the relationship between connection pools, configuration keys, pipelines, and backend storage:

---

3. Dynamic PropertyForm Component

The renderPropertyForm function creates user interface forms dynamically from connector metadata schemas.

3.1 Metadata Resolution

The component resolves properties through the following procedure:

1. Requests /metadata/sources/{type}, /metadata/sinks/{type}, or /metadata/connections/{type}.
2. If the API returns property schemas, the component uses those schemas.
3. If the API returns empty definitions or 404, the component uses the built-in fallback schema catalog.

3.2 Supported Form Controls

Control Type	Trigger Condition	Rendered Element	Behavior
Password	control === 'password' or name matches secret, password, token	<input type="password"> with eye button	Click 👁 to show plain text. Click 🙈 to hide plain text.
Select	values array exists or control === 'select'	<select> dropdown	Presents allowed values from schema.
Switch	type === 'bool' or control === 'switch'	Toggle slider	Toggles true or false.
Number	type === 'int' or type === 'float'	<input type="number">	Converts input string to numeric value.
Textarea	control === 'textarea' or type === 'json'	<textarea>	Supports multi-line input and formatted JSON strings.
Text	Default string scalar	<input type="text">	Captures standard string configuration.

3.3 Dual-Mode Raw JSON Synchronization

Operators can switch between Visual Form and Raw JSON at any time:

• Edits in the visual form immediately update the raw JSON buffer.
• Edits in the raw JSON editor validate on input. Valid JSON objects immediately update the form fields.
• Syntax errors show an alert box and prevent invalid state submission.

---

4. Shared Connection Pool Hub

The Connection Pool Hub manages reusable network connections.

4.1 Connection Table Fields

• Connection ID: Unique identifier for the shared connection pool.
• Type & Target: Connector protocol (MQTT, HTTP, Kafka, etc.) and the destination host or URL.
• Active Pipeline References: Live count of streams, tables, and rules that use this connection ID.
• Configuration Preview: Clean summary string of non-secret configuration parameters.
• Actions:
  • Test ⚡: Executes a preflight network connectivity check.
  • Edit ✏: Opens the connection modal with current values.
  • Delete 🗑: Performs reference-count safety validation before deletion.

4.2 Safe Deletion Procedure

Before deleting any shared connection, the system verifies pipeline dependencies:

1. Scans all active streams for matching CONF_KEY or connectionSelector.
2. Scans all active tables for matching connection options.
3. Scans all rule sink definitions for matching connector targets.
4. If references exist, the console displays a warning dialog that lists every dependent rule and stream. The user must confirm before the DELETE /connections/{id} request is sent.

---

5. Connector Configuration Keys (ConfKeys) Manager

Configuration keys (confKeys) let operators define named profiles for connectors without hardcoding credentials in every pipeline.

5.1 Endpoints Used

• List keys: GET /metadata/sources/{type}/confKeys
• Read key configuration: GET /metadata/sources/{type}/confKeys/{keyName}
• Save key configuration: PUT /metadata/sources/{type}/confKeys/{keyName}
• Delete key: DELETE /metadata/sources/{type}/confKeys/{keyName}

5.2 Preflight Validation

Click ⚡ Test Preflight to validate credentials and server reachability before saving. The preflight check confirms that:

• Required host, URL, and port parameters are present.
• URL formats match protocol specifications.
• Credentials and TLS certificates match file paths on disk.

---

6. Designer Integration

The PropertyForm component is integrated directly into the visual designers:

• Rule Designer (Step 2 - Sinks): Click ⚙ Advanced on any sink card to configure protocol parameters, TLS certificates, and MQTT QoS.
• Stream & Table Designer: Click ⚙ Advanced Properties next to the ConfKey selector to configure connector-specific polling intervals, headers, and socket settings.