Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Kinesis DB

Kinesis DB is a custom-built, ACID-compliant embedded database system written entirely in Rust. It forms the core data storage and management component of the Kinesis API platform, eliminating dependencies on external database systems while providing significant performance advantages.

Key Features

Multiple Storage Engines

Kinesis DB offers multiple storage engines to match your application's specific needs:

  • In-Memory Engine: Ultra-fast, volatile storage ideal for temporary data, caching, or situations where persistence isn't required. Provides maximum performance but data is lost on shutdown.

  • On-Disk Engine: Durable storage with ACID guarantees for critical data. Ensures data survives system restarts and power failures through persistent storage and write-ahead logging.

  • Hybrid Engine: Combines the speed of in-memory operations with the durability of on-disk storage. Uses intelligent caching with persistent backing for balanced performance, making it the recommended default choice for most applications.

Schema Management

Kinesis DB provides robust schema management capabilities:

  • Flexible Schema Definition: Create and modify schemas at runtime, allowing your data model to evolve with your application's needs.

  • Strong Type System: Supports various data types with strict validation to ensure data integrity.

  • Comprehensive Constraints:

    • Required fields (non-null constraints)
    • Unique constraints to prevent duplicate values
    • Default values for fields
    • Pattern matching for string fields using regular expressions
    • Min/max value constraints for numeric fields
    • Custom validation rules
  • Schema Versioning: Track and manage schema changes over time.

Transaction Support

Kinesis DB is designed with full ACID compliance:

  • Atomicity: All operations within a transaction either complete fully or have no effect.

  • Consistency: Transactions bring the database from one valid state to another, maintaining all defined constraints.

  • Isolation: Multiple isolation levels to control how concurrent transactions interact:

    • ReadUncommitted: Lowest isolation, allows dirty reads
    • ReadCommitted: Prevents dirty reads
    • RepeatableRead: Prevents non-repeatable reads
    • Serializable: Highest isolation, ensures transactions execute as if they were sequential
  • Durability: Once a transaction is committed, its changes are permanent, even in the event of system failure.

  • Deadlock Detection and Prevention: Automatic detection and resolution of transaction deadlocks.

  • Write-Ahead Logging (WAL): Ensures durability and supports crash recovery.

Performance Optimizations

Kinesis DB incorporates several performance optimizations:

  • Efficient Buffer Pool Management: Minimizes disk I/O by caching frequently accessed data in memory.

  • Configurable Caching Strategies: Tune caching behavior to match your workload characteristics.

  • Automatic Blob Storage: Large string values are automatically managed for efficient storage and retrieval.

  • Asynchronous I/O Operations: Non-blocking I/O to maximize throughput.

  • Indexing: Supports various indexing strategies to speed up queries.

  • Query Optimization: Intelligent query planning and execution.

Query Interface

Kinesis DB provides an intuitive query interface:

  • SQL-Inspired Command Syntax: Familiar syntax for developers with SQL experience.

  • CRUD Operations: Comprehensive support for Create, Read, Update, and Delete operations.

  • Data Search Capabilities:

    • Equality matching
    • Range queries
    • Pattern matching with regular expressions
    • Full-text search capabilities
  • Multiple Output Formats:

    • Standard output
    • JSON formatting for API responses
    • Table format for human-readable outputs

Configuration

Kinesis DB can be configured through environment variables:

VariableDescriptionPossible ValuesDefault
DB_NAMEName of the database (affects file names)Any valid filenamemain_db
DB_STORAGE_ENGINESelect storage enginememory, disk, hybridhybrid
DB_ISOLATION_LEVELDefault transaction isolation levelread_uncommitted, read_committed, repeatable_read, serializableserializable
DB_BUFFER_POOL_SIZEConfigure the buffer pool sizeAny positive integer100
DB_AUTO_COMPACTEnable/disable automatic database compactiontrue, falsetrue
DB_RESTORE_POLICYControl how transactions are recovered after a crashdiscard, recover_pending, recover_allrecover_pending

Best Practices

Performance Optimization

  • Choose the appropriate storage engine for your use case
  • Tune the buffer pool size based on your memory availability and working set size
  • Use indexes for frequently queried fields
  • Batch related operations in transactions
  • Consider denormalizing data for read-heavy workloads

Data Integrity

  • Use constraints to enforce business rules at the database level
  • Always use transactions for related operations
  • Implement proper error handling for database operations
  • Regularly back up your data

Schema Design

  • Design schemas with future growth in mind
  • Use appropriate data types for each field
  • Consider query patterns when designing your schema
  • Use meaningful field names and conventions

Current Schema

Kinesis API Database Schema

The above diagram illustrates the current database schema used in Kinesis API. The schema represents the relationships between core components of the system, including users, configs, projects, collections, structures and data objects. This schema is implemented directly in Kinesis DB, leveraging the type system and constraints described earlier to ensure data integrity across all operations.

Conclusion

Kinesis DB provides a powerful, embedded database solution that combines performance, reliability, and ease of use. By integrating directly with Kinesis API, it eliminates the need for external database dependencies while providing all the features expected of a modern database system.