Kinesis DB

Kinesis DB is a custom-built, ACID-compliant embedded database system written entirely in Rust. It forms the core data storage and management component of the Kinesis API platform, eliminating dependencies on external database systems while providing significant performance advantages.

Key Features

Multiple Storage Engines

Kinesis DB offers multiple storage engines to match your application's specific needs:

In-Memory Engine: Ultra-fast, volatile storage ideal for temporary data, caching, or situations where persistence isn't required. Provides maximum performance but data is lost on shutdown.
On-Disk Engine: Durable storage with ACID guarantees for critical data. Ensures data survives system restarts and power failures through persistent storage and write-ahead logging.
Hybrid Engine: Combines the speed of in-memory operations with the durability of on-disk storage. Uses intelligent caching with persistent backing for balanced performance, making it the recommended default choice for most applications.

Schema Management

Kinesis DB provides robust schema management capabilities:

Flexible Schema Definition: Create and modify schemas at runtime, allowing your data model to evolve with your application's needs.
Strong Type System: Supports various data types with strict validation to ensure data integrity.
Comprehensive Constraints:
- Required fields (non-null constraints)
- Unique constraints to prevent duplicate values
- Default values for fields
- Pattern matching for string fields using regular expressions
- Min/max value constraints for numeric fields
- Custom validation rules
Schema Versioning: Track and manage schema changes over time.

Transaction Support

Kinesis DB is designed with full ACID compliance:

Atomicity: All operations within a transaction either complete fully or have no effect.
Consistency: Transactions bring the database from one valid state to another, maintaining all defined constraints.
Isolation: Multiple isolation levels to control how concurrent transactions interact:
- ReadUncommitted: Lowest isolation, allows dirty reads
- ReadCommitted: Prevents dirty reads
- RepeatableRead: Prevents non-repeatable reads
- Serializable: Highest isolation, ensures transactions execute as if they were sequential
Durability: Once a transaction is committed, its changes are permanent, even in the event of system failure.
Deadlock Detection and Prevention: Automatic detection and resolution of transaction deadlocks.
Write-Ahead Logging (WAL): Ensures durability and supports crash recovery.

Performance Optimizations

Kinesis DB incorporates several performance optimizations:

Efficient Buffer Pool Management: Minimizes disk I/O by caching frequently accessed data in memory.
Configurable Caching Strategies: Tune caching behavior to match your workload characteristics.
Automatic Blob Storage: Large string values are automatically managed for efficient storage and retrieval.
Asynchronous I/O Operations: Non-blocking I/O to maximize throughput.
Indexing: Supports various indexing strategies to speed up queries.
Query Optimization: Intelligent query planning and execution.

Query Interface

Kinesis DB provides an intuitive query interface:

SQL-Inspired Command Syntax: Familiar syntax for developers with SQL experience.
CRUD Operations: Comprehensive support for Create, Read, Update, and Delete operations.
Data Search Capabilities:
- Equality matching
- Range queries
- Pattern matching with regular expressions
- Full-text search capabilities
Multiple Output Formats:
- Standard output
- JSON formatting for API responses
- Table format for human-readable outputs

Configuration

Kinesis DB can be configured through environment variables:

Variable	Description	Possible Values	Default
`DB_NAME`	Name of the database (affects file names)	Any valid filename	`main_db`
`DB_STORAGE_ENGINE`	Select storage engine	`memory`, `disk`, `hybrid`	`hybrid`
`DB_ISOLATION_LEVEL`	Default transaction isolation level	`read_uncommitted`, `read_committed`, `repeatable_read`, `serializable`	`serializable`
`DB_BUFFER_POOL_SIZE`	Configure the buffer pool size	Any positive integer	`100`
`DB_AUTO_COMPACT`	Enable/disable automatic database compaction	`true`, `false`	`true`
`DB_RESTORE_POLICY`	Control how transactions are recovered after a crash	`discard`, `recover_pending`, `recover_all`	`recover_pending`

Best Practices

Performance Optimization

Choose the appropriate storage engine for your use case
Tune the buffer pool size based on your memory availability and working set size
Use indexes for frequently queried fields
Batch related operations in transactions
Consider denormalizing data for read-heavy workloads

Data Integrity

Use constraints to enforce business rules at the database level
Always use transactions for related operations
Implement proper error handling for database operations
Regularly back up your data

Schema Design

Design schemas with future growth in mind
Use appropriate data types for each field
Consider query patterns when designing your schema
Use meaningful field names and conventions

Data Management - How to manage data in Kinesis API
Structures - Working with data structures
Backups - Backing up your Kinesis DB data

Current Schema

Kinesis API Database Schema

The above diagram illustrates the current database schema used in Kinesis API. The schema represents the relationships between core components of the system, including users, configs, projects, collections, structures and data objects. This schema is implemented directly in Kinesis DB, leveraging the type system and constraints described earlier to ensure data integrity across all operations.

Conclusion

Kinesis DB provides a powerful, embedded database solution that combines performance, reliability, and ease of use. By integrating directly with Kinesis API, it eliminates the need for external database dependencies while providing all the features expected of a modern database system.

Keyboard shortcuts

Kinesis API Docs