GitHub - danielecr/hot-fixture-tool: HFit - Hot Fixture Tool. A solution for retrieve data in secure mode for development

Hot Fixture Tool (hfit)

Hot Fixture Tool is compound of two pieces:

server (hfitd) installed on server with access hot db and fs data
client (hfit) is a CLI interface to the server

HFit Architecture

graph TB
subgraph "Production Environment"
DB1[(Database 1)]
DB2[(Database 2)]
FS1[File System 1]
FS2[File System 2]
end

subgraph "Hot Fixture Tool Server"
HFITD[hfitd Daemon]
API[REST API]
AUTH[Authentication]
REDIS[(Redis Cache)]
end

subgraph "Development Environment"
CLI[hfit CLI]
TESTS[Integration Tests]
LOCAL[(Local Data and DB)]
end

DB1 --> HFITD
DB2 --> HFITD
FS1 --> HFITD
FS2 --> HFITD

HFITD --> API
API --> AUTH
HFITD --> REDIS

CLI --> API
CLI --> LOCAL
LOCAL --> TESTS
...

Hot Fixture Tool (hfit)

| Donate | ko-fi

Hot Fixture Tool is compound of two pieces:

server (hfitd) installed on server with access hot db and fs data
client (hfit) is a CLI interface to the server

HFit Architecture

graph TB
subgraph "Production Environment"
DB1[(Database 1)]
DB2[(Database 2)]
FS1[File System 1]
FS2[File System 2]
end

subgraph "Hot Fixture Tool Server"
HFITD[hfitd Daemon]
API[REST API]
AUTH[Authentication]
REDIS[(Redis Cache)]
end

subgraph "Development Environment"
CLI[hfit CLI]
TESTS[Integration Tests]
LOCAL[(Local Data and DB)]
end

DB1 --> HFITD
DB2 --> HFITD
FS1 --> HFITD
FS2 --> HFITD

HFITD --> API
API --> AUTH
HFITD --> REDIS

CLI --> API
CLI --> LOCAL
LOCAL --> TESTS

Redis cache is used only for metadata (package export definition) and user data (public key for access challenge)

Ratio and use cases

Developer workflow

Developer workflow:

a developer need to work on code, code that read and modify data
developer in general has no access to data
developer need to adopt strategy to test data read and data write, by mocking library, and mocking requests
using mocks, the service code is not exactly the code the developer tested.
real integration tests are often done just by CI tools, and often are hard to define.

Thanks to hfit and hfitd, developer can:

define one or more hfit/pkg-tmpl-xxx.yaml
automate the download of real server resource (testing server or production server)
automate the import into local machine
implements effective integration tests
use integration tests to fix bug on hot real data
keep the .yaml definitions files in the shared repo, since those does not contains real data

Thanks to hfitd, administrator can warrant that only a restrict number of developer has access to data for reading. Also hfit command can be integrated into the CI pipeline to automate the import, and the integration tests running.

Motivations

Development of DataBase centric service involve unit tests and integration tests definition. Unit tests are used to keep the code clean and to lower cyclomatic and other code metrics. Integration tests are used to check the effective correctnes of the service.

This service and tool targets the integration test definition.

By using hfit, a developer can:

define resources as DDL, db rows, and file to be ready to execute integration test locally
run the integration tests against a real set of data
retrieve hot data for hot fixing of bug a bug
automate the retrievement of hot data to execute all integration tests and avoid regressions

Most of the time, running a dbms locally is not resource consuming as it is the repeated cycle of “blinded development”.

The most common case is the deal with legacy application that need to be ported, refactored, adapted.

An application become “legacy” after 5 or 6 year of development, so this tool is mostly targeted at successful projects with business logic, that one want to keep alive and current, want to update to newer execution engine, compiler or interpreter, want to change the protocol exposed, etc.

Usage

create a folder on your service repo: mkdir hfit-data; cd hfit-data
keep your hot-export-package.yaml definition in files stored in the repo.
run hfit to download data based on hot-export-package.yaml definition, into a target folder
for each folder:
run import tool to read data from downloaded resource.
execute integration tests
cleanup (if needed)

A progressive workflow might fit better the integration test job. For this use this policy:

define a base-export-package.yaml
define as many hot-export-package.yaml as there are hot test cases.

Example of folder content:

└── hfit-data
├── .gitignore
├── METADATA
│   ├── package.json
│   ├── replacements.json
│   └── timestamps.json
├── base-export
│   ├── db1.table1.create.sql
│   ├── db1.table2.create.sql
│   └── db1.table3.create.sql
├── base-export.yaml
├── use-case123
│   ├── db1.table3.data.sql
│   └── file-import-123.txt
└── use-case123.yaml

Importing data

To use the hfit CLI to import data, use the local-i sub-command

hfit local-i create my-docker.yaml extract-from base-export.yaml use-case123.yaml

Open a text editor and fill the required data for (write) connection to your local dbms and filesystem provider.

Then run

hfit local-i run-on my-docker.yaml base-export.yaml --no-download

With --no-download option it imports data and just use the existing data.

Use a .gitignore in hfit/ folder like this:

*
!.gitignore
!*.yaml

to avoid the checkout of hot data.

As helper there is a convenient command:

hfit prepare

This creates hfit-data/ folder and populates with .gitignore :

└── hfit-data
└── .gitignore

Definition

Service API:

list DBS
list tables
export table definition
export table data + where filter (with streaming NDJSON support)
list accessible folder
list files with filter (stream LDJSON)

🚀 Streaming Database Rows (New Feature)

Database table rows now support streaming NDJSON format for optimal performance with large datasets:

Traditional JSON Response:

curl -H 'Authorization: Bearer $JWT_TOKEN' \
'http://localhost:8080/db/mysql/mydb/table/users/rows'

Streaming NDJSON Response (Optimized):

curl -H 'Authorization: Bearer $JWT_TOKEN' \
-H 'Accept: application/x-json-stream' \
'http://localhost:8080/db/mysql/mydb/table/users/rows'

Advanced Filtering with SQL injection protection:

# WHERE clause filtering
curl -H 'Accept: application/x-json-stream' \
'http://localhost:8080/db/mysql/mydb/table/users/rows?filterpart=WHERE age > 25'

# Complex queries with ORDER BY and LIMIT
curl -H 'Accept: application/x-json-stream' \
'http://localhost:8080/db/postgres/mydb/table/orders/rows?filterpart=WHERE status = "active" ORDER BY created_at DESC LIMIT 100'

Key Benefits:

O(1) Memory Usage: Constant memory regardless of result size
Real-time Streaming: First rows appear immediately
SQL Injection Protection: Safe filterpart parameter validation
Multi-DBMS Support: Works with MySQL, PostgreSQL, and more
Enterprise Scalability: Handles millions of rows efficiently

Client:

interactively run all API provided by service
define JSON for export package with parameters
modify export JSON
execute export package.

Features:

service ACL per user
user auth list
user authentication by key-pair, challenge, and jwt
storage by redis
admin user- public key
simple admin interface based on vanillaJs

Export package format (WIP)

The command export-package accepts a .yaml file as a template definition for exporting hot data. The format is:

hfitVersion: 1
templateName: usecase_data
projectName: project_name # typically the repository or project name
packageName: basedata_$1
prepare:
- setVar: dataid
from: input
source: $1
- setVar: usrId
from: hot-data
hdata:
type: dbquery
dbms: dbms_mysql1
query: "SELECT usrId FROM dbname.datatable WHERE dataid=${dataid} ORDER BY utime LIMIT 1"
- setVar: fBaseName
from: hot-data
hdata:
type: volume
volume: vol1
glob: "*_{dataid}_{usrId}_*.txt"
# take the first in mtime desc order:
sort: "mtime|desc"
# extract the first number of filename as fBaseName value
regex_replace: "/([0-9]+).*/$1/"
exports:
dbcreate.sql:
type: dbcreate
data:
dbms: dbms_mysql1
tablelist:
- dbname1
- dbname2
tablegroup1.create.sql:
type: table-create
data:
dbms: dbms_mysql1
tablelist:
- dbname1.table1
- dbname2.table2
- dbname1.tablex
option: <dropcreate|ifnotexists>
tabledata1.data.sql:
type: table-data
data:
dbms: dbms_mysql1
table: dbname1.table1
filter: WHERE key<34 AND key>12
target-filename.txt:
type: file
data:
volume: datavol1
path: relative/path/to/file

NOTES:

order of retrievement is not important
order of prepare is relevant

Once you defined this file, the common usage is to to store it in hfit-data/ folder of the repo. For this reason:

hfit repo-prepare

Does create an hfit-data/ folder on the root of your git repo and write there some package template example.

hfit CLI should be used to create and populate this .yaml file.

Create the .yaml file:

./hfit pkg create base-data-package.yaml basedata

Execute package download and unpack into target folder (the target folder is named as the package name)

./hfit pkg downpack base-data-package.yaml

Manipulate by adding resource (this check if resource exists):

./hfit pkg add base-data-package.yaml

Removing a resource:

./hfit pkg rm base-data-package.yaml

Edit (option/filter/tablename)

Plan

First release has no ACL control, all users read access to all resources

Quick Start with Docker

The fastest way to get started is using Docker:

# Start the complete stack (PostgreSQL + Redis + hfitd)
make demo

# Or manually:
docker-compose up -d
make adduser EMAIL=admin@yourdomain.com KEY=your_public_key.pem

Visit http://localhost:8080 and see DOCKER.md for complete deployment guide.

Looking for Contributors & Feedback

This project is source-available - you can inspect the code, contribute improvements, and use it for personal/educational purposes. I’m actively seeking:

Contributors to help improve the codebase
Feedback from users who find this tool useful
Sponsorship or commercial partnerships
Real-world use cases and feature requests

If you find this project valuable or want to use it commercially, please reach out! I’m open to discussing licensing, sponsorship, or collaboration opportunities.

License

This project is licensed under a custom source-available license. You may:

Inspect and study the source code
Contribute improvements and bug fixes
Use for personal, non-commercial purposes

You may NOT:

Use commercially without permission
Distribute without permission

See LICENSE for full details. For commercial use, please contact me to discuss licensing options.

Contributing

Contributions are welcome! Please feel free to:

Report bugs and issues
Suggest new features
Submit pull requests
Provide feedback and suggestions

By contributing, you help make this tool better while allowing me to explore sustainable development models.

submodules

doc (public)
sellpoint
premium

Hot Fixture Tool (hfit)

HFit Architecture

Hot Fixture Tool (hfit)

HFit Architecture

Ratio and use cases

Developer workflow

Motivations

Usage

Definition

🚀 Streaming Database Rows (New Feature)

Export package format (WIP)

Plan

Quick Start with Docker

Looking for Contributors & Feedback

License

Contributing

submodules

Similar Posts