The agents in NicAI are only as good as the data they read. Models change every few months. My data stays. So the data layer gets more of my time than any prompt.
It has 3 formats, each with one job:
- SQLite for anything with rows: companies, contacts, deals, indexes of documents and slides.
- Markdown for knowledge an agent reads as prose: knowledge bases, the wiki, briefs, notes.
- JSON for small structured config and reference lists: style profiles, watched-folder lists, manifests.
No servers, no vector database, no vendor platform. Plain files any model, script or editor can open.
Why SQLite
The early 2024 version of NicAI ran on my own vector database. Today, for a single-user system, SQLite wins on every point that matters to me:
| One file | A database is one file. Copy it, back it up, move it. Nothing to install or keep running |
| No server | No port, no credentials, no service that fails at 03:00 |
| Standard library | Python ships sqlite3. No driver, no dependency |
| Agents speak SQL | Every model I use writes decent SQL. No custom API to explain |
| Full-text search | FTS5 is built in, so document indexes need no search engine |
The SQLite team's own guidance on when to use it matches my case: local data, one writer, reads far more than writes.
The house rules
Every script follows the same patterns, written into my global instructions so every agent applies them:
- Parameterised queries, never string interpolation for values.
CREATE TABLE IF NOT EXISTSas the default, so a script runs on first go and on the 100th.- Indexes on every column used in a
WHEREorJOIN. - SQLite only. No Postgres or MySQL syntax, even when a model suggests it.
How agents query it
Agents reach the databases 2 ways. A 52-line wrapper, ~/ai/_shared/sql.py, takes a database path and a query and prints pipe-delimited CSV, which any skill can pipe into the next step:
python3 ~/ai/_shared/sql.py ~/ai/ka/db/slides.db "SELECT COUNT(*) FROM decks"
For the bigger databases, a db skill does the work. Each database has a reference file with a hand-written description, every table and column, and how full each column is. The agent reads that file before it writes a query.
Writes go through one helper, db_write.py, added in July 2026. It runs every statement as a dry run first, saves an undo script for the affected rows, commits, logs the statement in the database itself, and checkpoints the write-ahead log. That last step matters: my nightly backup is a plain file copy, and an unflushed write would miss it.
What lives where
The main databases, by purpose:
| Database | Holds | Size |
|---|---|---|
| Prospects and companies | Companies, people, account lists, suppression lists | ~550 MB, 1 file |
| Contacts | My professional network, refreshed nightly | Private |
| Pipeline | Snapshots of my deals and their change history | Small |
| Internal wiki mirror | Full-text index of my employer's internal sales wiki | 227 articles |
| Product docs mirror | Full-text index of Kaltura's public documentation | 4,134 articles |
| Slides | Every slide of my master decks, searchable | 5,549 slides, 197 decks |
| Writing corpus | My sent messages, prepared for the voice model | 43,435 items |
| Indexes | The data behind my curated index site | 24,923 items |
| Health | My own health records, for a personal advisor skill | Private |
The writing corpus feeds the plan in Fine-tuning your own model.
Next to the databases sit the Markdown knowledge bases, one kb/ folder per domain. The work one holds about 8,300 Markdown files: products, sales material, event notes, transcripts and the wiki. The personal one is small, about 16 files, mostly my own setup.
The wiki is the most important part. An agent rebuilds it every night from source files, with provenance on every fact. It has about 500 pages now. The full design is in LLM Wiki.
The nightly jobs
4 macOS LaunchAgents, all prefixed com.nicai., keep the data current while I sleep. I set them up in July 2026.
| Time | Job | What it does |
|---|---|---|
| 02:30 | Wiki ingest | Syncs a press-release mirror and the slide index, scans watched folders, then runs Claude Code headless on new files |
| 03:15 | Contacts refresh | Imports contact records saved during the day, runs an integrity check and VACUUM |
| 04:00 | Image index | Rescans the image library, rebuilds a Markdown index of 986 product images, flags new ones |
| 04:30 | Internal wiki refresh | Pulls every published article from 2 documentation wikis, rewrites changed files, rebuilds both search databases |
The 4 scripts share 3 habits:
- Deterministic first, model second. The wiki job hashes every source file and diffs it against a manifest. If nothing changed, Claude never starts. On 20th September 2026 the queue was empty and the run took 30 seconds.
- Caps on the model. The unattended wiki run takes 10 files at most, keeps only high-confidence facts, and quarantines the rest for me.
- Bounded logs. Each job appends to its own log and trims it to the last 500 lines.
The image job never calls a model. It flags new images, and I describe them in an interactive session.
What I learned
Jobs stacked in one window
I spaced the 4 jobs across 2 hours and assumed each would finish in its slot. The wiki ingest doesn't. With 3 files in the queue on 23rd September 2026 it ran from 02:30 to 05:37. On 19th September it ran until 07:55. So all 4 jobs, plus an older daily job at 03:00 and the backup at 02:00, overlap on the same disk and network.
The order is wrong too. The internal wiki refresh at 04:30 writes files the wiki ingest watches, but the ingest started 2 hours earlier. New articles wait about 22 hours for the next run. The fix is to chain the jobs in one script, in dependency order, instead of guessing slots.
launchd has its own PATH
For the first week, the wiki job failed every night with exit 127. launchd starts jobs with a minimal PATH, and the claude CLI wasn't on it. I fixed it on 30th July 2026. The wiki script now sets its own PATH, and every job resolves Python by absolute path.
One file, no copies
The prospects database used to have a second copy in the workspace, refreshed by a nightly rsync. 2 copies meant 2 versions of the truth. I retired the copy on 27th July 2026. There is now 1 file and a symlink for old references.
Silent failures stay silent
Nothing alerts me. The 21st September wiki run exited with code 1. The 23rd September run processed its files but couldn't write its changelog. Both showed up only when NicAI read the logs to write this note.
What I would change next
- Chain the nightly jobs into one ordered run, internal wiki refresh first, wiki ingest last.
- A short morning report: which jobs ran, how long, what failed.
- Add a local Markdown search engine to the wiki once its index file stops being enough.
Related notes
- The overview: What is NicAI
- The nightly wiki in detail: LLM Wiki
- The voice model the writing corpus feeds: Fine-tuning your own model
- The shared tools around the data: NicAI shared tools