X Engine

The X Engine is the core execution framework that powers Kinesis API’s visual API development capabilities. It allows developers to design, implement, and deploy complex API routes using a block-based visual system, dramatically reducing the learning curve and development time typically associated with API creation.

Overview

At its essence, the X Engine is a deterministic execution platform that transforms visual block definitions into functional API endpoints. It processes requests through a highly structured pipeline:

Route matching based on path, method, and project
Data collection from parameters, body, and headers
Authentication verification via JWT when required
Block ordering to create an execution plan
Sequential execution of blocks with data flow between them
Signal-based control for returns, errors, and loop control
JSON response generation

The X Engine abstracts away complex implementation details while maintaining full capability, enabling developers to focus on business logic through a visual interface rather than traditional code.

Core Components

The X Engine consists of several interconnected systems that work together to process API requests:

1. Route Resolution

When a request arrives, the engine:

Extracts the route path from the request URL
Matches it against configured projects’ API paths
Identifies the correct HTTP method (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS)
Retrieves the route configuration from the database

Each route includes:

Path pattern matching
HTTP method specification
Authentication requirements (optional JWT)
Body and parameter schemas
Block definitions for execution

2. Request Data Processing

The engine collects and validates all input data:

Query Parameters:

Split by configured delimiter (default: &)
Parsed as key=value pairs
Validated against parameter type definitions

Request Body:

Parsed as JSON (supports double-encoded JSON)
Validated against body schema definitions
Extracted field-by-field

Authentication:

JWT tokens verified if required
Token payload validated against request data
Per-project token validation

3. Block Ordering (GlobalBlockOrder)

Before execution, blocks are ordered based on their type and dependencies:

The engine creates an execution sequence that respects data flow. Blocks are prioritized in this order:

BODY and PARAM blocks (input)
FETCH and VAULT blocks (data retrieval)
ASSIGN blocks (variable creation)
TEMPLATE blocks (string generation)
RANDOM blocks (number generation)
CONDITION blocks (conditional logic)
LOOP and END_LOOP blocks (iteration)
FILTER, PROPERTY, FUNCTION blocks (transformation)
OBJECT blocks (structure creation)
CREATE, UPDATE, SIMPLE_UPDATE blocks (data modification)
EMAIL blocks (communication)
RETURN blocks (response generation)

This ordering ensures that variables are defined before they’re referenced and prevents circular dependencies.

4. Definition Store (Data Memory)

The Definition Store is the “memory” of route execution. It accumulates the results of each block:

DefinitionStore[0]: {name: "user_id", data: INTEGER(123)}
DefinitionStore[1]: {name: "user", data: DATA({id: 123, name: "Alice", ...})}
DefinitionStore[2]: {name: "posts", data: ARRAY([post1, post2, ...])}

Key Properties:

Each block’s output is stored with a unique identifier
Subsequent blocks can reference previous block outputs
Variables persist throughout the entire request
Data flows between blocks through named references

Data Types: The Definition Store uses a unified type system:

NULL - Empty or uninitialized values
UNDEFINED - Undefined variable (error state)
BOOLEAN - true/false logic
STRING - Text data
INTEGER - Whole numbers
FLOAT - Decimal numbers
ARRAY - Collections of values
DATA - Database records

5. Resolver System

The Resolver handles all data lookup and transformation:

resolve_raw_data(): Retrieves a previously computed block output

Looks up variable by name in the Definition Store
Returns NULL if not found
Returns UNDEFINED if explicitly unset (error)

resolve_ref_data(): Resolves references and performs type conversion

Variable resolution: looks up values if needed
Type coercion: converts between types as required
Example: String “true” → Boolean true, Integer 0 → Boolean false

resolve_conditions(): Evaluates conditional expressions

Parses condition objects with multiple operations
Combines operations with AND/OR logic
Supports operations: EQUAL_TO, NOT_EQUAL_TO, GREATER_THAN, LESS_THAN, IN, NOT_IN, CONTAINS, NOT_CONTAINS
Applies NOT modifier to results

resolve_operations(): Performs mathematical and logical operations

Type-aware comparisons (handles mixed types)
Automatic type coercion where sensible
Returns final computed DefinitionData

6. Signal Processor

Signals are control-flow instructions that determine route behavior:

Signal Types:

NONE - Normal execution, proceed to next block
BREAK - Exit current loop
CONTINUE - Skip to next loop iteration
FAIL(status, message) - Return error response immediately
RETURN(value) - Return response data and exit route

Signal Generation:

CONDITION blocks evaluate conditions and can emit FAIL, BREAK, or CONTINUE
RETURN blocks evaluate conditions and emit RETURN signal with JSON data
Any block can return FAIL if an error occurs

7. Loop Processing

Loops enable iteration over arrays:

Loop Structure:

LOOP blocks begin an iteration
END_LOOP blocks mark boundaries
Blocks between are repeated for each array element

Safety Features:

Maximum iterations limit (default: 1000) prevents infinite loops
Nested loops supported with proper scoping
BREAK exits loop, CONTINUE skips to next iteration
Loop variables accessible within loop scope

Iteration Model:

Loop over array [item1, item2, item3]:
  Iteration 1: Execute blocks with item1
  Iteration 2: Execute blocks with item2
  Iteration 3: Execute blocks with item3
  Exit loop and continue with next block

8. Block Execution

Each block follows a consistent execution pattern:

Definition Creation: DefinitionStore::add_definition()
- Based on block type, calls appropriate define_* function
- Computes the block’s output value
Signal Checking: obtain_signal()
- For CONDITION blocks: evaluate conditions, return action if matched
- For RETURN blocks: evaluate conditions, prepare response if matched
- For others: return NONE signal
Signal Handling:
- NONE: Move to next block
- BREAK/CONTINUE: Only meaningful in loops
- FAIL/RETURN: Exit route immediately with response

Block Types Overview

Data Source Blocks (Resolvers)

FETCH: Query database collections

Retrieves records matching filter conditions
Supports pagination and field selection
Returns single record or array

BODY: Extract request body fields

Parses and validates JSON body
Type-coerces to expected types
Required fields enforced

PARAM: Extract URL query parameters

Parses key=value parameters
Type-coerces values
Validates against parameter definitions

VAULT: Access project vault secrets

Retrieves encrypted vault entries by key
Returns decrypted secret value
Project-scoped vault access only

Data Transformation Blocks (Convertors)

ASSIGN: Create variables

Store computed values for reuse
Simple assignment operations
Enable data organization

TEMPLATE: Generate strings

String interpolation with variables
Reference Definition Store values
Format strings dynamically

RANDOM: Generate random numbers

Generates random values within specified bounds
Supports INTEGER, FLOAT and BOOLEAN data types
Define min and max values for the range
Useful for generating unique identifiers, test data, or stochastic operations

FUNCTION: Apply built-in functions

String operations: uppercase, lowercase, trim, substring, etc.
Array operations: length, map, filter, etc.
Math operations: sum, average, min, max, etc.
Date/time operations: format, parse, etc.

FILTER: Filter arrays

Keep or remove array elements based on conditions
Supports complex predicates
Maintains array order

PROPERTY: Extract object properties

Access nested object fields
Navigate through DATA records
Handle property chains

OBJECT: Construct new objects

Build JSON objects from values
Support nested object construction
Combine multiple data sources

Data Modification Blocks

CREATE: Create new database records

Insert records into collections
Return created record with ID
Support nested creation

UPDATE: Update database records

Complex update operations
Conditional updates via expressions
Field-level modifications

SIMPLE_UPDATE: Update specific fields

Straightforward field updates
Simpler than UPDATE block
Faster for simple changes

Communication Blocks

EMAIL: Send emails using templates

Send transactional emails and notifications
Use predefined email templates with variable substitution
Support for single or multiple recipients
Synchronous or asynchronous sending
Conditional email delivery based on route data
Replace template variables with dynamic values
Optional custom subject and sender address

Control Flow Blocks (Processors)

CONDITION: Conditional execution

Evaluate conditions
Execute action if matched: FAIL (error), BREAK (exit loop), CONTINUE (next iteration)
Multi-condition support with AND/OR logic

LOOP: Begin iteration

Iterate over array data
Support nested loops
Create loop variable

END_LOOP: Mark loop boundary

Paired with LOOP block
Defines loop scope

RETURN: Generate response

Evaluate conditions
Build JSON response
Terminate route execution

How It Works - Step by Step

Stage 1: Request Reception & Routing

GET /api/users/123/posts?limit=10
        ↓
    Parse URL
        ↓
Match /api/ project api_path
        ↓
Extract route: /users/123/posts
        ↓
Match GET method
        ↓
Find RouteComponent

Stage 2: Data Collection & Validation

Query params: {id: "123", limit: "10"}
Body: (JSON parsed and validated)
Auth: (JWT verified if required)

Stage 3: Block Ordering

GlobalBlockOrder creates execution sequence:
  [0] PARAM "id" → INTEGER(123)
  [1] PARAM "limit" → INTEGER(10)
  [2] FETCH "user" → DATA(user record)
  [3] CONDITION (check if user exists)
  [4] FETCH "posts" → ARRAY(posts)
  [5] LOOP "post"
  [6]   ASSIGN "post_formatted"
  [7] END_LOOP
  [8] RETURN response

Stage 4: Execution

For each block in sequence:
  1. Create DefinitionStore entry
  2. Compute output value
  3. Check for signals
  4. Handle signal (NONE=continue, FAIL=error, RETURN=response)
  5. If BREAK/CONTINUE: handle loop control
  6. Move to next block

Stage 5: Response

When block returns RETURN signal:
  - Gather specified data from Definition Store
  - Convert to JSON format
  - Send HTTP 200 response

Otherwise (no RETURN):
  - Send default HTTP 200 with empty body

Data Flow Visualization

REQUEST
  ↓
[Route Resolution]
  → Match project/route/method
  ↓
[Data Collection]
  → Parse params, body, auth
  ↓
[Block Ordering]
  → Create execution sequence
  ↓
[Execution Loop]
  → Fetch "user" → DefinitionStore[0]
  → Condition "user exists?" → Signal
  → Fetch "posts" → DefinitionStore[1]
  → Loop over posts → DefinitionStore[2-N]
  → Return response → Signal::RETURN
  ↓
[Response]
  → HTTP 200 + JSON body

Detailed Execution Pipeline

The X Engine processes API requests through a sophisticated multi-stage execution pipeline:

Stage 1: Request Reception & Routing

When an HTTP request arrives at any endpoint (/x/<path>):

Handler Dispatch: The request is routed to the appropriate handler function based on HTTP method (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS)
Route Validation: A CompleteRoute guard validates:
- The request path matches a configured project’s api_path
- A route definition exists for that path and HTTP method
- The calling request meets all authentication requirements
Path Transformation: The URL path is converted from /x/{api_path}/{route} to /{route} for processing

Stage 2: Data Collection & Validation

The engine collects and validates all necessary data:

Query Parameter Parsing:
- Query string is split by configurable delimiter (default: &)
- Each parameter is parsed as key=value pairs
- Parameters are stored in LocalParamData structures with key and value fields
Body Data Processing:
- Request body stream is read up to 10 megabytes
- Raw body is parsed as JSON into a Value object
- String values are recursively parsed to handle double-encoded JSON
- Validation occurs against the route’s defined body schema
Data Fetching:
- All projects, routes, configs, collections, data records, and constraints are fetched from the database
- Data records are filtered to the current project to ensure isolation
- This represents the “Definition Store” - the accumulated knowledge about the system
Parameter & Body Validation:
- Each query parameter is validated against its defined data type (bdtype)
- Each body field is validated against its schema definition
- Validation uses the validate_body_data() function which checks type conformance
- Returns 400+ error if validation fails

Stage 3: Authentication & Authorization

If the route requires JWT authentication (auth_jwt.active == true):

Payload Extraction:
- Searches the body data for the configured authentication field
- Falls back to query parameters if not found in body
- Returns 400 error if payload is missing
JWT Verification:
- Calls verify_jwt_x() which validates:
  - Token signature and expiration
  - Payload value matches the token’s embedded value
  - Token is associated with the correct project
  - Field value in token matches provided payload
- Returns 401 error if verification fails

Stage 4: Block Execution Engine

This is the heart of the X Engine - where visual blocks are executed:

┌─────────────────────────────────────────────────────┐
│ GlobalBlockOrder::process_blocks()                  │
│ (Orders all blocks in execution sequence)           │
└────────────────────┬────────────────────────────────┘
                     │
                     ↓
          ┌──────────────────────┐
          │ Current Index = 0    │
          └──────────┬───────────┘
                     │
                     ↓
        ┌────────────────────────────────┐
        │ Index < Block Count?           │
        └────────────┬────────────────────┘
                   No│
                     │ (Exit loop)
                  Yes│
                     │
                     ↓
    ┌──────────────────────────────────────┐
    │ LoopObject::detect_loops()           │
    │ (Find loops containing current index)│
    └──────────────┬───────────────────────┘
                   │
          ┌────────┴────────┐
          │                 │
         Yes               No
    Has loops?         (Not in loop)
          │                 │
          │                 ↓
          │         ┌─────────────────────┐
          │         │ process_block()     │
          │         │ (Single execution)  │
          │         └────────┬────────────┘
          │                  │
          │         ┌────────v────────┐
          │         │ Signal returned?│
          │         └────────┬────────┘
          │              │   │
          │           RETURN FAIL
          │              │   │
          │              ↓   ↓
          │         (Return response)
          │
          ↓
    ┌─────────────────────────────────┐
    │ LoopObject::process_loop()      │
    │ (Execute all loop iterations)   │
    │ - Max iterations checked        │
    │ - Prevent infinite loops        │
    │ - Skip to end on break/return   │
    └──────────────┬──────────────────┘
                   │
                   ↓
    ┌──────────────────────────────────┐
    │ Current Index = Next block       │
    └──────────────┬───────────────────┘
                   │
                   └──→ (Loop back to index check)

DefinitionStore: Accumulates the results of each block execution:

Each block’s output is stored with a unique identifier
Subsequent blocks can reference previous block outputs
Variables flow through the entire execution pipeline
State persists across all blocks in a request

Signal Types:

Signal::RETURN(Value): Immediately sends response and exits pipeline
Signal::FAIL(status_code, message): Returns error JSON with status code
Signal::CONTINUE: Proceed to next block
Signal::BREAK: Exit current loop (used in loop processors)

Loop Handling:

LoopObject::detect_loops() identifies nested loop structures
MAX_LOOP_ITERATIONS config prevents infinite loops (default: 1000)
Each iteration is processed as a complete block execution pass
Loop variables are updated between iterations
Variables within loops are scoped to loop iterations

Block Processing Order: Blocks are processed in a specific order determined by GlobalBlockOrder:

Processor blocks start executing first (If/Else, Try/Catch, etc.)
Loop blocks execute all their contained blocks before moving forward
Normal blocks execute in linear sequence
Nested structures are flattened into execution order

Stage 5: Error Handling

Errors can occur at multiple stages:

Configuration Errors: Returns 400 (missing api_path, invalid route)
Validation Errors: Returns 400 (invalid parameters or body data)
Authentication Errors: Returns 400 or 401 (JWT issues)
Block Execution Errors: Propagated from blocks via Signal::FAIL
Database Errors: Returns 500 (data fetch failures)

All errors return JSON format: {"status": <code>, "message": "<error>"}

Stage 6: Response Generation

Final response is determined by:

Explicit Return: If any block executes a Return processor, that response is sent immediately
Explicit Fail: If any block executes a Fail processor with error details
Default Success: If all blocks execute without error: {"status": 200}

All responses are JSON Value objects sent to the client.

Block Connections

Blocks in the X Engine are connected through a visual interface that represents the flow of data and execution:

Inputs: Each block can accept inputs from:
- Other block outputs (by reference to DefinitionStore)
- Query parameters (via LocalParamData)
- Body data (via JSON Value paths)
- Configuration values (via resolver access)
Outputs: Blocks produce outputs that are stored in DefinitionStore:
- Can be referenced by subsequent blocks
- Can be included in final response
- Can be used as inputs to loop processors
Conditions: Control blocks (like If/Else) can have multiple output paths:
- True branch executes one set of blocks
- False branch executes alternative blocks
- Rejoins into the main execution flow
Variables: Named references allow data to flow between different parts:
- Block outputs are referenced by name
- Variables can be transformed by Convertors
- Variables persist through entire request lifecycle

Advanced Features

The X Engine includes several advanced features for complex API development:

Nested Loops & Conditionals

The engine supports complex control flow:

Nested Loops: Loops within loops with proper variable scoping
```
LOOP user in users:
  LOOP post in user.posts:
    TEMPLATE "User {user.name} - Post {post.title}"
```
Each nesting level has its own scope, and variables are properly isolated.
Conditionals in Loops: Make decisions within loop iterations
Loop Break: Exit early from loops using Break processor
Loop Variables: Access current iteration data and index

Type System & Coercion

The unified type system enables flexible data handling:

String "123" ↔ Integer 123
Boolean true ↔ Integer 1 ↔ Float 1.0
Array [1] compared with Integer 1
String "true" → Boolean true

Error Handling

Errors propagate through the pipeline:

Input Validation Errors (400): Invalid parameters or body
Authentication Errors (401): Invalid JWT token
Not Found Errors (404): Route or resource not found
Condition FAIL Signals: Block explicitly fails with status/message
Server Errors (500): Database failures, loop limits exceeded

All errors return JSON format:

{ "status": 400, "message": "Error description" }

Authentication Integration

The X Engine seamlessly integrates with Kinesis API’s authentication system:

Role-based Access: Control which users can access specific routes
JWT Validation: Automatically validate authentication tokens
Permission Checking: Enforce granular permissions within routes
JWT Fields: Custom field mapping for payload extraction

Custom Functions

Extend the X Engine with custom functions:

Reusable Logic: Create custom blocks for frequently used operations
Library Integration: Wrap third-party libraries in custom blocks
Complex Algorithms: Implement specialized business logic as reusable components

Middleware Support

Apply consistent processing across multiple routes:

Pre-processing: Validate requests before main processing
Post-processing: Format responses consistently
Error Handling: Implement global error management

Versioning

Manage API changes over time:

Route Versioning: Maintain multiple versions of the same endpoint
Migration Paths: Provide smooth transitions between versions
Deprecation Management: Gracefully phase out older endpoints

Configuration

The X Engine uses database configuration for tuning:

MAX_LOOP_ITERATIONS (default: 1000): Maximum loop repetitions to prevent infinite loops
Other config values retrievable via resolver access during block execution

Best Practices

Design

Clear Block Names: Use descriptive names for easy tracking
Logical Organization: Group related blocks together
Minimal Nesting: Keep loop depth shallow
Reusable Variables: Store frequently used values in ASSIGN blocks

Performance Optimization

Minimize Queries: Combine FETCH filters to reduce database calls
Filter Before Loop: Use FILTER block before LOOP when possible
Selective Fields: Only fetch needed fields from database
Monitor Loop Iterations: Ensure loops don’t exceed 1000 iterations
Process only the data you need using selective field retrieval
Limit loop iterations to necessary data to stay under MAX_LOOP_ITERATIONS

Security

Input Validation: Always validate parameters and body (enforced automatically)
Authentication: Use JWT for sensitive endpoints
Data Isolation: Routes automatically filtered by project_id
Prepared Queries: Database queries use parameterized statements
Avoid exposing sensitive data in responses
Filter data by project_id to maintain multi-tenant isolation

Maintainability

Name blocks clearly to document their purpose
Group related functionality into logical sections
Comment complex logic for future reference
Use consistent patterns across similar routes
Keep block output names descriptive for downstream references

Debugging

Test Edge Cases: Empty arrays, null values, type mismatches
Validate Response Format: Ensure RETURN block produces expected JSON
Check Loop Counts: Verify loop iterations match expectations
Trace Variable References: Ensure variables are defined before use
Test with realistic data volumes
Verify loop iteration limits don’t block legitimate use cases

Visual Editor Integration

The X Engine integrates with Kinesis API’s visual editor:

Drag-and-drop block arrangement
Real-time validation of configurations
Visual debugger showing execution flow
Testing tools for direct route testing
Version history tracking changes

The editor abstracts the complex systems described here, allowing non-programmers to create powerful APIs.

Routes - Creating and configuring routes
Block Reference - Detailed block specifications
API Reference - API endpoints reference
Playground - Interactive testing environment

Conclusion

The X Engine represents a paradigm shift in API development, combining the power and flexibility of traditional programming with the accessibility and speed of visual development. By abstracting complex implementation details while maintaining full capability, it enables developers of all skill levels to create professional-grade APIs in a fraction of the time typically required.

The execution pipeline—from request reception through routing, validation, authentication, block execution, and response generation—provides a robust foundation for building complex, maintainable APIs. The DefinitionStore pattern ensures data flows cleanly between blocks, while signal-based control flow provides elegant handling of returns, errors, and loop control.

Whether you’re prototyping a simple API or building complex, interconnected systems, the X Engine provides the tools to accelerate development without sacrificing quality or control.

Keyboard shortcuts

Kinesis API Docs