# Datagripe — the whole site Generated 2026-09-24 from https://datagripe.com. Every page, in reading order. Individual pages are at the same paths with `.md`; `/llms.txt` is the index. --- # Datagripe > A database IDE for PostgreSQL, MySQL, SQLite and Redis that reads your > queries and your schema back to you and says what is wrong with them. > Free and open source, MIT. This is the landing page as Markdown. It is written rather than converted: what the page has that this does not is a particle animation making fun of particle animations, and a transcript of that would help nobody. Everything load-bearing is below. ## What it is Datagripe is a database IDE. It has the parts you expect — an editor with tabs and splits, query execution with streamed results, a data grid you can edit, an object browser, four engines — and one part you do not: it reads your SQL as you write it and tells you what is wrong with it, before you press run. It runs three ways, all the same server: ```bash bunx @datagripe/cli personal # your machine. no account, no config docker run -p 3001:3001 -v datagripe:/data ghcr.io/datagripe/datagripe helm install datagripe oci://ghcr.io/datagripe/charts/datagripe ``` There is also a desktop build for Linux, macOS and Windows, and the web app installs as a PWA. ## The complaining part Eighteen rules, and none of them are about formatting. A rule earns its place by knowing something the query text does not say, or by being about damage rather than tidiness — style filler is a formatter's job and the fastest way to get the whole thing switched off. Three properties matter more than the list: - **A rule that cannot tell says nothing.** No hedging, no "this might be slow, maybe". If the schema is not cached, the rule that needs it stays silent rather than guessing. - **Findings are analysis; wording is presentation.** The same finding is re-worded at four levels — `notice`, `warning`, `fatal`, `panic` — and the level is applied when the sentence is shown, so turning the dial never re-runs the analysis and cannot change what was found. - **Everything is dismissable**, for that occurrence, that object, or the whole project, and the app always tells you it is hiding something. `panic` is a joke and is opt-in; `fatal` swears; `notice` is dry and still critical. See [gripe levels](/docs/gripe-levels.md). ## The rest of it - **Editor** — movable tabs and splits, one model per document, drafts and layout recovered across reloads, Markdown documents beside SQL. - **Query execution** — run the selection, the statement at the cursor, or the whole file. Streamed results, cancellation that works on a query that is stuck, CSV and JSON out. - **Table view** — sort, filter, page, edit cells, insert and delete rows, with a panel for the JSON column that is unreadable in a grid. - **Object view** — columns, indexes, constraints, triggers, grants, statistics and DDL from one catalog call. Column edits show the SQL before running it. - **Four engines, honestly** — each adapter declares what it can do, so SQLite's narrow `ALTER` is reflected in the interface rather than failing at runtime. - **Multiplayer** — workspace files, presence, follow mode, and queries other people ran, each under their own identity with an audit trail. - **Domains, access reports, git datasources, MCP** — see [what it can do](/docs/features.md). ## For agents Every page on this site is also Markdown at the same path with `.md` instead of the trailing slash. `/llms.txt` indexes all of them; `/llms-full.txt` is all of them in one request. Datagripe itself speaks MCP. A project can expose an endpoint an agent connects to; it is off until an owner turns it on, read-only until they say otherwise, and every call lands in the query history under the token's name. ## Where to go next - [Getting started](/docs/getting-started.md) — running it, and the first five minutes after that. - [All 18 rules](/rules.md) — what it complains about. - [Gripe levels](/docs/gripe-levels.md) — the four-level dial. - [What it can do](/docs/features.md) — the whole surface. - [Deploying](/docs/deploy.md) — Docker, compose, Kubernetes, Helm. - [Configuration](/docs/configuration.md) — every environment variable, a page per decision. - [Roadmap](/roadmap.md) — the gripes it has about itself. - [All documentation](/docs.md) ## Licence and boundary MIT. An independent parody: not affiliated with, endorsed by, or connected to any other software vendor. --- # Documentation DataGripe is a database IDE for PostgreSQL, MySQL, SQLite and Redis that reads your queries and your schema back to you and says what is wrong with them. It runs as a web app you host, a desktop app you install, or one command on your own machine. Every page is in the sidebar, and in the menu on the bar above on a narrow screen. This page is for working out which one you want. ## I want to try it [Getting started](/docs/getting-started/) is about twenty seconds and needs nothing installed. After that, [the rules](/rules/) are what the whole thing is for — the 18 things it complains about, and none of them are formatting. The two shortcuts nobody finds on their own are on the [keyboard page](/docs/keyboard/), and they are worth thirty seconds: `Ctrl/Cmd+Enter` runs the statement the caret is in without selecting it, and `Ctrl+Alt+L` reformats. ## I want to run it for other people Start at [deploying](/docs/deploy/), which is the four shapes and how to pick one, then the page for whichever you picked. Two settings decide whether a deployment works at all and neither has a default that can be right everywhere — `WEB_ORIGIN` and `CONNECTION_ENCRYPTION_KEY`. Both are on [configuration](/docs/configuration/), which is also where the rest of the environment lives, a page per decision: [accounts and sign-in](/docs/authentication/) — including [Google](/docs/authentication/#google-sign-in) — [database and storage](/docs/database/), [datasources](/docs/datasources/), [limits](/docs/limits/), [files, git and commands](/docs/files-and-git/) and [MCP](/docs/mcp/). [Security](/docs/security/) is what to change before it faces anybody. ## I want to know why it is like this [Why it complains](/docs/why-it-complains/) is the one sentence the whole gripes feature follows from, and what it costs. From there: [gripe levels](/docs/gripe-levels/) for the four-level dial, [dismissing gripes](/docs/dismissing-gripes/) for turning one off, and [writing a rule](/docs/writing-a-rule/) for adding one. The [FAQ](/docs/faq/) has the questions that come back, including the ones with an answer nobody likes, and the [glossary](/docs/glossary/) has the words this project uses in a particular way. ## I want to know what is missing The [roadmap](/roadmap/) is the gripes DataGripe has about itself, in the same voice it uses on your schema — accepted, unscheduled, unfiled, and the ones deliberately declined, which stay on the page because the reason something was refused is the useful part. [Release notes](/docs/release-notes/) are what has already shipped. ## I am an agent Every page here is also Markdown: append `.md` to the path instead of the trailing slash, so this page is [`/docs.md`](/docs.md) and the rules are [`/rules.md`](/rules.md). No scraping and no JavaScript required. - [`/llms.txt`](/llms.txt) — the whole site indexed, a sentence each. - [`/llms-full.txt`](/llms-full.txt) — every page in one request. - [`/sitemap.xml`](/sitemap.xml) DataGripe itself speaks MCP: a project can expose an endpoint an agent connects to, off until an owner turns it on and read-only until they say otherwise, with every call in the query history under the token's name. ## Something is wrong The app tells you what it refused and why, and the two settings that break a deployment most often are described where they are set. If a gripe is wrong, that is a bug worth [reporting](https://github.com/datagripe/datagripe/issues) — a wrong gripe costs more trust than a missing one. ## I am building it, not using it The [specifications](/specs/) are the other half of this site: twenty documents, one per subsystem, recording what each does, what it deliberately does not, and which alternatives were rejected and why. They are engineering documents and read like it. The rest of the repository's [`docs/`](https://github.com/datagripe/datagripe/tree/main/docs) holds the ADRs and the brand system, and [`roadmap.md`](https://github.com/datagripe/datagripe/blob/main/roadmap.md) is where the roadmap page is rendered from. --- # What it can do Nearly everything here has a keyboard path and a menu; some of it has neither a button nor a banner, which is what this page is for. The [keyboard reference](/docs/keyboard/) is the short version. ## Gripes DataGripe reads your SQL back to you and says what is wrong with it. None of the rules are about formatting — a rule earns its place by knowing something the query text does not say, or by being about damage rather than tidiness. [The rules page](/rules/) is the whole catalogue, rendered from the code, so it is never a count somebody forgot to update. - **Statements**: a join with no condition, `NOT IN` against a subquery that can yield `NULL`, a write with no `WHERE`, an inequality against a nullable column. - **Schema**: a table with no primary key, a view built on `SELECT *`, a duplicate index, a non-concurrent index build, a `SECURITY DEFINER` routine with no `search_path`, and a routine declared volatile that only reads. - **Grants**: a routine PUBLIC can execute, an untrusted role that can write, row security switched on with no policy behind it, a view that reads its base tables as its owner. These stay silent until a role is marked untrusted — guessing which role is the anonymous one would be worse than saying nothing. **The volume is yours.** The same finding is re-worded at four levels — notice, warning, fatal and panic — and the wording is chosen when it is shown, so turning the dial never re-runs the analysis. `panic` is a joke and is opt-in; `fatal` swears. **Anything you disagree with is dismissable** for that one occurrence, that table, or the whole project — and the app always tells you it is hiding something rather than quietly showing you less. ## Projects A project holds datasources, shared files, domains, members and roles. The switcher is the prompt in the header — `>Datagripe:_` — and *New project…* is the last item in it. - **Rename** it in project settings, if your role can manage the project. - **Delete** it from the danger zone of the same panel: reveal, type the project's name, execute. It takes everything DataGripe holds about the project — its datasources and their stored credentials, its shared files, its domains and tags, its dismissed gripes, its members, its roles and its MCP tokens — in one cascade, and it cannot be undone. - **Nothing on the host's disk is touched.** A repository datasource is a clone somebody else also has, an exported domain is a directory under version control, and a datasource path points at work that was never DataGripe's. Deleting the project is DataGripe forgetting them. - **Nobody is left without a project.** Deleting your only one is refused, and so is deleting a shared project somebody else has no alternative to — remove them from it first. An account with no project cannot open the application. ## The interface - **One Files section** in the sidebar: the datasource's own directories, the project's shared files and this browser's scratchpads, as three kinds of root in one tree. Each has its own `new`, and every section — Files, Repository, Online, MCP Server — starts collapsed and stays where it is in the list whether it is open or shut. - **A section header carries the facts you would otherwise open it for**: the branch, the number of changed files and a refresh on Repository; how many people are here on Online; whether an agent can connect, and with what, on MCP Server. - **`refresh` on a repository fetches first**, so *0 behind* is about the remote as it is now rather than the last time somebody pulled. The counts sit on the buttons that change them — `push 2`, `pull 0`. - **One avatar in the header** opens the account menu: the project you are in and your role, the two settings panels, and the way out. Set a name in Account settings and the button wears it. - **The version is bottom left**, as one number — and as *Update Available* when a newer release exists, when the page has fallen behind the server, or when a downloaded build is waiting. Clicking it tells you the one thing that applies an update *here*, which is different in every shape, and in Kubernetes offers the button that does it. - **Scale.** Account settings has a slider for how big the interface is, from 80% to 180%. Every size in the application is a multiple of it, so the tree, the tabs, the editor and the results move together. It is stored in the browser rather than the account — the reason to turn it up is usually the screen in front of you. ## Editor and documents - Movable tabs, horizontal and vertical splits, and one editor model per document so a split never desynchronises from itself. - Drafts and layout survive a reload, stored in the browser. - **Markdown documents** render beside SQL — a runbook lives in the project with the queries it describes, and a SQL fence inside it runs and formats like SQL. - **You name the file**, in the row it will occupy, prefilled with `query-N.sql` and with the extension left out of the selection so typing keeps it. The extension is the whole rule: `.md` is a runbook, `.sql` is a query, anything else opens as plain text rather than being refused. Renaming `notes.sql` to `notes.md` switches it there and then — no reload. - **A name that is taken counts up** — `notes.md`, `notes-1.md` — rather than being refused. Rename it afterwards if that is not what you meant. - **Revert to last save** is in the right-click menu of any file with unsaved changes: the counterpart to the dot that says it has them. - **Reformatting** on `Ctrl+Alt+L` for SQL, JSON and Markdown. ## Running queries - Run the selection, the statement at the cursor, or the whole document. You do not have to select a statement to run it. - Results stream in batches, and cancellation goes down a control path that is not the one being blocked — so cancel works on a query that is stuck, which is the only time it matters. - Row, byte, timeout and concurrency limits are enforced by the server, not asked of the client. - CSV and JSON out. ## Browsing and editing - **Explorer** with lazy schema, table and column introspection and a short-lived cache. - **Table view**: sort, filter, page, edit cells, insert and delete rows, with a side panel for the JSON column that is unreadable in a grid. - **Select cells** in either grid — click, Shift-click, drag, or Shift+arrows in the table view for a rectangle; Ctrl/Cmd-click to add or remove individual cells anywhere across rows and columns. With more than one cell selected, the bottom of the panel says what they add up to: `sum`, `avg`, `min`, `max` for numbers, `distinct` for everything else, and how many of them are null. `Ctrl/Cmd+C` copies selected cells as tab-separated rows. Disjoint selections copy in row/column order, omitting gaps. Shift-click returns to a rectangle from the original anchor; a plain click starts over. - **The total is of what is highlighted**, not of the table behind it. The bar counts the cells first for exactly that reason: a `sum` that quietly meant the whole table would be a lie told in small type. - **Object view**: columns, arguments, indexes, constraints, triggers, grants, statistics and DDL from one catalog call. Column edits show you the SQL before they run it. - **Four engines, honestly.** Each adapter declares what it can actually do, so SQLite's narrow `ALTER` is reflected in the interface rather than failing at runtime. ## Organising a database - **Domains** group objects by what they are for — billing, identity, the reporting tables nobody owns — because `information_schema` cannot tell you which tables are billing. Domains export to a directory or a git repository, and import somewhere else. Sync includes each table's or view's triggers in its SQL file, after the object definition and before grants. The DDL tab shows the same definitions. PostgreSQL trigger functions remain separate routines: tag them too if you want their bodies in the export. - **Access report** answers "who can read this" across a schema or a domain, and refuses above a cell count rather than timing out. - **Datasource paths** let a datasource point at a directory on the host that the sidebar then browses. ## Working with other people - Workspace files, presence, and follow mode. - Queries other people ran, each under their own identity, with an audit trail. - **Roles you define**: a role is a name and a set of capabilities — run queries, tag domains, run the sync, manage the MCP server, sixteen of them — edited as a matrix and checked on every action rather than at the door. Owner, editor and viewer are there in every project as the ranks they always were. ## Git Off by default, and enabled per deployment. - **Git datasources**: clone a repository and treat its SQL and documentation as part of the project. - **Commit controls** on the domain export, so a schema snapshot lands in version control as a reviewable change. - **Repo commands**: a repository's own `.datagripe/run.yaml` can declare commands DataGripe may run. This has a separate switch from the rest of git on purpose — every other git feature reads and writes files, and this one executes a program somebody else wrote. Nothing runs until a person approves the list, and any change to it needs a fresh approval. ## MCP The switch is in the sidebar's **MCP Server** header, and the section wears a green frame while the server is running, because whether something outside the app can read a project is not something you should have to open a panel to check. Click the MCP Server title to open its management tab for mode, tokens and independent domain-management, sync and Git opt-ins. Agents can manage domains and object assignments, preview or write sync snapshots, and inspect Git status, commit and push. Writes require read/write mode and editor access. See [MCP](/docs/mcp/) for the gates and commit scope. Every project can expose an endpoint an AI agent connects to, so the agent reads the project's own documentation and queries its datasources under the same rules a person gets. Off per project until an owner turns it on, and read-only until they say otherwise — a read-only project cannot change a row no matter what it is asked to run. An MCP query is an execution like any other: it appears in history, with the token's name beside it. ## Getting in - **No accounts at all** for the personal and desktop shapes. - **Accounts** for a shared deployment, with a [security key](/docs/authentication/) as an alternative to a password — sign-in is usernameless, so there is no email to type and none to leak by asking about. - Signup closes after the first account unless you say otherwise. --- # 18 rules Rendered from `packages/gripes`. Each line is the finding at `warning`, the default of four attitude levels; the technical content is the same at every level and only the register changes. | Rule | Severity | Needs | Engines | At warning | | --- | --- | --- | --- | --- | | `delete.no-where` | blocker | statement | all engines | Delete with no where clause. That is the table, not the row. | | `grant.default-privileges-untrusted` | blocker | access | postgres | Default privileges grant every new {objects} to {role}. Nothing yet exists. | | `grant.definer-public-reach` | blocker | access | postgres | Security definer, and {role} can call it. It runs as {owner}, not them. | | `grant.public-execute` | blocker | access | postgres | PUBLIC can execute {routine}, so {role} can call it. That is the default. | | `grant.untrusted-write` | blocker | access | postgres | {role} can {privileges} {relation}. Anonymous callers, writing. | | `join.no-condition` | blocker | statement | all engines | This join has no condition. I'll allow it. I won't forget it. | | `routine.definer-no-search-path` | blocker | object | all engines | Security definer, no search_path. Anyone who can add a schema owns this. | | `update.no-where` | blocker | statement | all engines | Update with no where clause. Every row gets this, not just yours. | | `column.nullable-inequality` | warning | statement, schema | all engines | {column} can be null, and null is not <> anything. Those rows vanish here. | | `grant.rls-no-policy` | warning | access | postgres | RLS on {relation}, zero policies. Every row is denied, silently. | | `grant.untrusted-read-no-rls` | warning | access | postgres | {role} reads every row of {relation}. Row-level security is off. | | `grant.view-owner-bypass` | warning | access | postgres | {view} is not security_invoker. {role} reads {relation} as {owner}. | | `index.not-concurrent` | warning | statement | postgres | No concurrently. Writes to this table queue up until the index is built. | | `subquery.not-in` | warning | statement | all engines | not in against a subquery. One null in there and you get zero rows, silently. | | `table.no-primary-key` | warning | object | all engines | No primary key on {table}. Every row is indistinguishable from its twin. | | `view.select-star` | warning | statement | all engines | The star in this view is expanded once, at creation. New columns never appear. | | `index.duplicate` | style | object | all engines | {index} is already covered by {covering}. It costs writes and buys nothing. | | `routine.volatile-but-readonly` | style | object | all engines | {routine} only reads, yet claims volatile. The planner takes you at your word. | ## Not built yet - `index.missing` — Nothing knows which indexes a relation has. The completion catalog does not carry them and the object describe result is not wired to the runner. - `execution.truncated` — It cannot tell you that it stopped reading. Needs the server-side runner, which is being written. - `statement.over-ddl` — A view created with `select *` is a finding the catalogue already has, and it fires in a query document but never on the view itself, which is the one place anyone would look. - `plan.seq-scan` — Nothing reads an execution plan, so a sequential scan over forty million rows reads exactly like one over forty. - `select.unqualified-star` — `select * from t limit 100` is the single most common legitimate query there is. Griping at it is precisely the style filler this catalogue exists to avoid. See also: [gripe levels](/docs/gripe-levels.md), [dismissing gripes](/docs/dismissing-gripes.md), [writing a rule](/docs/writing-a-rule.md). --- # Gripes about Datagripe It has opinions about your schema. These are the ones it has about itself. Rendered from `roadmap.md` in the repository. Nothing here is a promise, and declined items stay on the list because the reason something was refused is the useful part. ## About itself - **[being written]** `engine · server-side-gripes` — Gripes stop at the query text. Nothing yet complains about how a query actually turned out, only about how it was written. - **[being written]** `config · settings-server-side` — Project class and attitude live in your browser's localStorage. Two machines, two personalities, and a per-project setting the server has never heard of. - **[accepted]** `object-view · danger-zone-inert` — The danger zone states real consequences and then does nothing. Truncate and drop are decoration until project class leaves the browser. - **[accepted]** `brand · placeholder-icon` — The app icon is a scowling cylinder drawn by hand. The mascot is real; the flat version that can animate in-app is not drawn. - **[accepted]** `ui · browser-chrome-in-an-app` — Right-click anywhere it has nothing to say and the browser's own menu arrives, offering to reload the frame and save the image. Drag past the edge of a tree and the whole page selects, prose and all. Both are the moment it stops feeling like an application. - **[accepted]** `ui · one-of-each-control` — Inputs, selects and now buttons are one component, but only the sidebar goes through the button: the forms, the status bar and the object view still hand-roll theirs, and toggles and segmented controls have no component at all. Every variation is a fix that has to be made twice, and eventually is made once. - **[accepted]** `rules · catalogue-too-thin` — Eighteen rules, and seven of them are about grants. A rules page wants about thirty before it stops looking like a sample. - **[accepted]** `docs · counts-in-prose` — The website no longer states a count it cannot derive. The specs still do, in prose, and prose is not checked by anything. - **[unscheduled]** `engine · explain-aware-gripes` — Nothing reads an execution plan. The worst queries in the database still look completely fine in text. - **[unscheduled]** `rules · migration-lint` — Migrations are not linted. The most dangerous SQL anyone writes gets the least attention from the thing that exists to complain about SQL. - **[unscheduled]** `schema · diff-view` — Two databases cannot be compared. You are diffing schemas in a text editor like it is 2009. - **[unscheduled]** `adapters · mssql` — Four engines, and none of them is the one your employer makes you use. - **[unscheduled]** `adapters · sqlite-table-rebuild` — SQLite cannot change a column's type, nullability or default here, because that needs the twelve-step table rebuild and nobody has written it. - **[unscheduled]** `object-view · structure-editing` — Columns can be edited. Indexes, constraints and triggers cannot. The preview-then-apply shape is proven; the rest is just dialect SQL nobody has typed. - **[unscheduled]** `domains · migration-generation` — The domain export is a structure snapshot, not an ordered rebuild. It will not tell you how to get from the old schema to this one. - **[unscheduled]** `execution · arrow-transport` — Results travel as JSON batches. Arrow would be faster and nobody has profiled the JSON to prove it is the problem. - **[unscheduled]** `connections · ssh-tunnels` — No SSH tunnels, no cloud IAM auth, no network agent. If the database is not reachable from the server, it is not reachable. - **[unfiled]** `idea · result-shape-gripes` — It never comments on what came back. Four million rows dropped into a grid is a gripe, and it is silent about it. - **[unfiled]** `idea · silent-mode` — There is no way to make it shut up entirely except switching it off, which is not the same thing and loses the dismissals. - **[unfiled]** `idea · ci-mode` — Gripes only exist inside the app. They could exist in a pull request, which is where somebody else would have to read them. - **[unfiled]** `git · sidebar-pull-ff-only` — The sidebar's pull is fast-forward only. On a server's clone holding an unpushed commit, a person in the browser is as stuck as the agent was before MCP learned to rebase, and "go to a terminal" means a machine they may not be able to reach. - **[unfiled]** `brand · light-theme` — Dark theme only. That is an open item, not a stylistic position, and saying so is cheaper than pretending. - **[declined]** `rules · formatting` — No formatting rules, ever. Style filler is a formatter's job and the fastest way to get the whole thing switched off. - **[declined]** `privacy · usage-analytics` — No telemetry, ever. It answers nothing but this machine, and that is the entire point of it. - **[declined]** `editor · ai-query-generation` — It will not write your query. A tool whose thesis is that it reads SQL back to you critically cannot also be the thing that wrote it. - **[declined]** `product · query-health-score` — No grade, no "query health 62%". A number invites gaming and averages away the one blocker that mattered. - **[declined]** `product · visual-schema-design` — No ERD canvas, no DBA workflows. Drawing boxes is not the job and doing it badly is worse than not doing it. ## Rules not built yet - **[unscheduled]** `index.missing` — Nothing knows which indexes a relation has. The completion catalog does not carry them and the object describe result is not wired to the runner. - **[unscheduled]** `execution.truncated` — It cannot tell you that it stopped reading. Needs the server-side runner, which is being written. - **[unscheduled]** `statement.over-ddl` — A view created with `select *` is a finding the catalogue already has, and it fires in a query document but never on the view itself, which is the one place anyone would look. - **[unscheduled]** `plan.seq-scan` — Nothing reads an execution plan, so a sequential scan over forty million rows reads exactly like one over forty. - **[declined]** `select.unqualified-star` — `select * from t limit 100` is the single most common legitimate query there is. Griping at it is precisely the style filler this catalogue exists to avoid. See also: [rules that exist](/rules.md), [release notes](/docs/release-notes.md). --- # Gripe levels The attitude level decides how a finding is phrased. It does not decide what was found. Turning the dial re-words what is already on screen and never re-runs the analysis, which is why it is instant and why it cannot change the answer. The four levels are PostgreSQL severities, on purpose. ## The same finding, four ways One rule — `join.no-condition` — at each level. The technical content is identical in all four; only the register moves. | Level | | | --- | --- | | `notice` | This join has no condition. That is a cross product. | | `warning` | This join has no condition. I'll allow it. I won't forget it. | | `fatal` | No join condition. You've asked for every row times every row. Absolute state of this. | | `panic` | NO JOIN CONDITION. Every row. Times every row. I want you to sit and think about what that number is. | `notice` is dry and still critical — it is the level for a shared screen, and it is guaranteed profanity-free by a test. `warning` is the default. `fatal` swears. `panic` is a joke and is written as one. ## Severity is not attitude These are two independent axes and conflating them is the failure mode the whole design is arranged against. - **Severity** is how bad the finding is: `blocker`, `warning`, `style`. It comes from the rule and never moves. - **Attitude** is how rudely it is phrased. It comes from your setting and never changes what was found. A `style` finding rendered at `fatal` swears about something trivial, which is exactly the waste to avoid — a gripe that swears at a redundant index has spent the currency for nothing. The engine does not couple the two axes; the message catalogue is where that judgment is made, per rule, by whoever wrote the four strings. ## Setting it Resolution order, first hit wins: 1. **A session override**, set as a real statement in the editor: ```sql set datagripe.attitude = 'notice'; ``` It is a statement rather than a setting so it can be dropped in before a screen share without opening a settings dialog. It lives for the session. 2. **The project setting.** 3. **`warning`**, if neither of the above says otherwise. ## The three rules about panic All three exist so that the joke stays a joke: - **It can never be the default.** - **It cannot be set for an organisation** — only by the person reading it, for their own session or their own project. Nobody gets to make `panic` happen to somebody else. - **It resets on upgrade.** The app version is stored alongside the setting; if the version has changed and the level is `panic`, it drops to `warning`. A joke that cannot be escaped is not a joke. ## What is behind the dial A finding carries facts, not sentences — `{ rows: 41203882, column: "status" }`, never `"41M rows and no index on status"`. That is the whole reason attitude is a presentation layer rather than four parallel analyses: | Stage | Produces | Depends on attitude | | --- | --- | --- | | Evaluate | a finding: rule id, severity, location, facts | no | | Render | one sentence | yes | Two consequences worth naming. The server never sends prose, because it would have to know the reader's attitude and attitude is a per-reader setting. And the rule id in the footer is the same string at every level, so a gripe you screenshot at `panic` and a gripe you dismiss at `notice` are provably the same gripe. ## A caveat, and it is current Attitude and project class live in your browser's `localStorage` today. A per-project attitude the server has never heard of is really a per-browser attitude: two machines, two personalities. Moving both into the workspace model is one migration and it is being written — it is on the [roadmap](/roadmap/) as `config · settings-server-side`, and it is the same thing blocking the object view's danger zone. ## Related - [Why it complains](/docs/why-it-complains/) — the reasoning underneath. - [The rules](/rules/) — each one at `warning`. - [Dismissing gripes](/docs/dismissing-gripes/) — turning one off. --- # Adapters DataGripe supports PostgreSQL, MySQL, SQLite and Redis. They are not equally capable, and the interface says so rather than offering you an action that fails at runtime. Every adapter declares a set of capability flags. The UI and the request dispatcher gate on those flags and never on the adapter's name, which is the mechanism that makes the honesty structural rather than a matter of remembering. ## What each one can do The table below is generated from `ADAPTER_CAPABILITIES` in the repository — the same constant the application itself gates on, so this page cannot describe an engine the app does not. | | PostgreSQL | MySQL | SQLite | Redis | | --- | --- | --- | --- | --- | | SQL dialect | postgres | mysql | sqlite | none | | Explorer | sql | sql | sql | keyspace | | Execution | cursor | buffered | buffered | — | | Cancellation | yes | yes | — | — | | Table view | readwrite | readwrite | readwrite | — | | Column changes | all seven | all seven | add, rename, drop | — | | Access report | yes | — | — | — | | Default port | 5432 | 3306 | — | 6379 | | `database` means | Database | Database | File path | DB index | `—` means the adapter does not have the capability at all, and the corresponding control is absent rather than disabled with a tooltip. ## Reading the flags **Explorer.** `sql` gives the schema → table → column tree. `keyspace` gives Redis's database → prefix → key browser, split on the `:` delimiter, with a truncation marker where a level exceeded its scan bound. **Execution.** `cursor` is a server-side cursor with batched fetches, so a large result streams and the first rows arrive immediately. `buffered` fetches the whole result and truncates it to `QUERY_MAX_ROWS` and `QUERY_MAX_BYTES`. Where execution is absent, `execution.start` is rejected server-side *and* the run buttons are disabled client-side — both, because a client-side check is a convenience and a server-side one is the rule. **Cancellation.** True means cancelling interrupts the running statement on the server, down an administrative control path that is not the one being blocked. That is the only reason cancel works on a query that is stuck, which is the only time it matters. SQLite has no driver-level interrupt; local files make a runaway query cheap to bound by caps instead. **Table view.** `readwrite` browses rows and writes single-row edits back in one transaction. Where it is absent, double-clicking an object opens a tab that says so rather than rendering an empty grid. **Column changes.** A list rather than a boolean, and SQLite is the reason. Its `ALTER TABLE` does add, rename and drop and nothing else, so the columns tab enables per operation and says why the rest are off. A type, nullability or default change needs the twelve-step table rebuild, which is on the [roadmap](/roadmap/). **Access report.** PostgreSQL only. MySQL's grant model is different enough — host-qualified grantees, no roles before 8.0 — to be a separate design rather than a port, and SQLite has no grants at all. ## Per-engine notes ### PostgreSQL The reference implementation, and the only one with everything. Cursor streaming, `KILL`-equivalent cancellation from a second connection, reconstructed DDL, routines and sequences in the object view, the access report, and the `grant.*` and `routine.*` rules. It is also the only engine where `CREATE INDEX CONCURRENTLY` exists, which is why `index.not-concurrent` is gated on the dialect: on MySQL and SQLite that advice is not merely unhelpful, it is a syntax error. ### MySQL Introspection through `information_schema`, system schemas excluded. Timeouts are enforced primarily by `SET SESSION max_execution_time`, with a client-side watchdog as the deterministic backstop; cancellation is `KILL QUERY` from a second connection. One driver quirk is worth knowing because it shaped the design: Bun's MySQL driver resolves an interrupted statement *cleanly* — `SLEEP` returns 1, no error, empty `SHOW WARNINGS`. Outcomes are therefore tracked out of band rather than inferred from the driver's response, or a cancelled query would report success. `allowPublicKeyRetrieval` is set only when TLS is disabled, which is the `caching_sha2_password`-over-plain-TCP case on a trusted local link. TLS modes never set it. ### SQLite The `database` field is a **server-side file path**, not a database name, and host, port, username and TLS are all absent from its dialog. That means the file has to be reachable from wherever the server runs — which is your own machine in the personal and desktop shapes, and probably not what you want in a hosted deployment. Schemas come from `PRAGMA database_list` (main plus attached), objects from `sqlite_master`, columns from `PRAGMA table_info`. `readOnly` opens the file read-only at the driver level rather than filtering statements. ### Redis A distinct capability rather than a limited SQL engine. The browser is read-only: `PING` to test, a `SCAN`-bounded keyspace tree at 10,000 keys per level with a truncation marker, and value fetches for strings, hashes, lists, sets and sorted sets with a TTL and a 100-entry cap. There is no query execution, so the run controls are absent. Whether Redis earns command execution later is an open question, and it would be a new capability flag rather than pretending it speaks SQL. ## Adding one An adapter is an implementation and one entry in `ADAPTER_CAPABILITIES`. There is no third place to update, and nothing in the UI branches on the adapter's name — which is what lets a new engine appear with its own honest set of limits instead of a set of runtime surprises. SQL Server is the one most often asked for. It is on the [roadmap](/roadmap/) as `unscheduled`, which means nobody is working on it. ## Related - [What it can do](/docs/features/) — the surface all four share. - [The rules](/rules/) — which rules are dialect-gated. - [The adapters spec](/specs/adapters/) — the capability model, the driver quirks, and the open questions. --- # Keyboard shortcuts `Ctrl` on Linux and Windows, `Cmd` on macOS — DataGripe accepts either, everywhere, so a keyboard that travels between machines does not have to be relearned. ## The two people miss These are the shortcuts worth reading this page for. Neither is written on a button, and both are the ones a DataGrip or IntelliJ user reaches for without thinking. | | | | --- | --- | | `Ctrl/Cmd` `Enter` | **Run.** The selection if there is one, otherwise the statement the caret is in — you do not have to select a statement to run it. | | `Ctrl` `Alt` `L` | **Reformat.** The whole document, or just the selection if there is one. IntelliJ's binding, and DataGrip's, on every platform. | `Ctrl+Alt+L` formats SQL, JSON and Markdown. Inside a Markdown document it reformats only the fenced block the caret is in, and leaves the prose alone. ## Editor | | | | --- | --- | | `Ctrl/Cmd` `Enter` | Run the selection, else the statement at the cursor | | `Ctrl/Cmd` `Shift` `Enter` | Run the whole document | | `Ctrl/Cmd` `S` | Save the document | | `Ctrl` `Alt` `L` | Reformat the document, or the selection | Run and save are window-level and routed to the editor that last had text focus, so they work from the results panel and the sidebar too — not only from inside the editor. **Middle-click and drag** makes a columnar, multi-line selection, the way IntelliJ does. Monaco can do it and normally gives the middle button to Linux's primary clipboard instead; a browser tab has no primary clipboard to paste from, so DataGripe takes the gesture back. Everything else is Monaco's own, unmodified: `Ctrl/Cmd+F` to find, `Ctrl/Cmd+D` for the next occurrence, `Alt+Up`/`Alt+Down` to move a line, `F1` for the command palette, and multi-cursor on `Alt+Click`. ## Cell selection in table and result views | Gesture | Action | | --- | --- | | Click | Start a new selection with one cell | | Shift-click | Select the rectangle from the original cell to this one | | Drag | Select a rectangle across rows and columns | | Ctrl/Cmd-click | Add or remove any individual cell, keeping the others selected | | Shift+arrow | Extend the rectangle in table view | | Ctrl/Cmd+C | Copy selected cells as tab-separated rows | With more than one cell selected, the bottom bar shows cell count and `sum`, `avg`, `min`, `max` for numeric values; nonnumeric selections show `distinct`, and nulls are counted separately. Numeric strings count as numbers. Only highlighted cells on the current page contribute. Shift-click returns a disjoint selection to a rectangle from its original anchor, including when Ctrl/Cmd is held too. Disjoint copies contain only selected cells in row/column order, with gaps and empty rows omitted. In results, Escape clears the selection. ## Table view A focused cell behaves like a spreadsheet cell. It is a button, not a text selection, so the browser's own copy has nothing to act on — these shortcuts are what make it work. | | | | --- | --- | | `Enter` or `F2` | Edit the focused cell | | `Ctrl/Cmd` `C` | Copy selected cells, or the focused cell’s value | | `Ctrl/Cmd` `V` | Paste into the cell | | `Ctrl/Cmd` `Backspace` | Set the cell to `NULL` | | `Enter` | Commit the edit | | `Escape` | Abandon the edit | `NULL` has its own gesture on purpose: an empty string and a `NULL` are different values, and a grid that cannot tell you which one you just wrote is a grid you cannot trust. ## Everywhere | | | | --- | --- | | `Escape` | Close the open menu, popover or editor | | `Ctrl/Cmd` `Click` | Add an object to a multi-selection in the explorer | | `←` `→` | Resize the sidebar, when its divider has focus | Tabs, splits and panels are all reachable by keyboard: the layout is Dockview's, and every control in it is a real focusable element rather than a div that listens for clicks. --- # Comparison This page names no products. That is partly the [parody boundary](https://github.com/datagripe/datagripe/blob/main/docs/brand/brand-system.md) — DataGripe is an independent parody and does not trade on anyone else's name — and partly that a page comparing feature lists with four named competitors is wrong within two releases of any of them. What does not rot is the shape of the tools. There are about five kinds, and DataGripe is one of them. ## The desktop database IDE The heavyweight one, usually commercial, usually a JVM underneath. Deep introspection, a real editor, refactoring, diagrams, a decade of polish. **Use it if** you want schema comparison, migration generation, visual schema design, or support you can escalate to. DataGripe has none of those and the first two are on the [roadmap](/roadmap/) as `unscheduled`, which means nobody is working on them. **DataGripe instead if** you want the editor and the grid without the installer, a tool your whole team opens at a URL, and something that tells you the delete has no `WHERE` clause before you run it. Some bindings here are deliberately familiar — `Ctrl+Alt+L` reformats, `Ctrl/Cmd+Enter` runs the statement the caret is in. That is muscle memory, not imitation, and breaking it would help nobody. ## The web admin panel Ships with the database or beside it. Browser-based, a form for everything, runs on the server. **Use it if** you want a tool that has been maintained for twenty years, that your DBA already trusts, and that nobody has to be persuaded to install. **DataGripe instead if** you want a real editor with tabs and splits rather than a text area, streamed results with cancellation that works on a stuck query, and more than one engine behind the same interface. ## The `psql` prompt The terminal, and the one that is always right. **Use it if** you are on a server, in a pipeline, or doing anything a script will do again tomorrow. Nothing in this project replaces it and nothing should try. **DataGripe alongside it**, mostly. A grid you can sort and edit, an object view that fills eight tabs from one catalog call, and a linter that reads the query as you type are worth a browser tab. The commit you push is still going through the terminal. ## The SQL linter in CI A formatter with rules, usually run on a directory of `.sql` files. **Use it if** you want consistent formatting enforced on a pull request. DataGripe deliberately has **no formatting rules at all** — style filler is a formatter's job and the fastest way to get a linter switched off — so these two do not overlap and both are worth having. **DataGripe instead if** you want findings that know something the query text does not say: that the column is nullable, that the index is redundant, that the definer routine has no pinned `search_path`. A linter working from the text alone cannot reach any of those. The reverse is also true today: DataGripe's gripes live inside the app and not in your pull request. That is on the roadmap as `idea · ci-mode`, and it is honestly labelled an idea. ## The notebook Cells, results, charts, and a document you share. **Use it if** the output is an analysis somebody reads. DataGripe has Markdown documents beside SQL and a SQL fence inside one runs and formats like SQL — but it has no charts and no output cells, and it is not pretending to be a notebook. ## What only this one does Everything above is a real tool with real reasons. The honest summary of what DataGripe adds is one paragraph: It reads your SQL and your schema as you type and tells you what is wrong with them, in a register you choose, from a catalogue of rules where every rule either knows something the query text does not say or is about damage rather than tidiness — and where a rule that cannot tell is required to say nothing. That is the feature. The rest of it is a competent database IDE built so that the feature has somewhere to live. Whether that is worth switching for is genuinely your call. It runs in about twenty seconds with nothing to install, which is a cheap way to find out. ## Related - [Getting started](/docs/getting-started/) — twenty seconds. - [The rules](/rules/) — the whole catalogue. - [Roadmap](/roadmap/) — including what it deliberately will not do. --- # Roles and permissions A project's members hold **roles**, and a role is a name and a set of capabilities. Owner, editor and viewer are in every project and cannot be removed — what they may do is yours to change — and you can add your own beside them. Project settings → **Roles** is the matrix: capabilities down, roles across, tick what each may do. Each member row above it picks the role that member holds. ## Reading is not on the list Being a member is being able to read: browse the schema, read the query history, watch what other people run, open the access report. There is no capability for it and no way to take it away. That is a limit worth being honest about. If somebody must not see the data, they need a different project or a database role of their own — hiding a table behind a checkbox, in an application whose whole purpose is handing you a SQL prompt, would be a promise DataGripe cannot keep. ## What you can grant | | | | --- | --- | | **Run queries** | Run and cancel statements. | | **Edit data** | Change, insert and delete rows in the table view. | | **Change structure** | Alter columns, with the SQL shown first. | | **Edit files** | The project's shared files, and files opened from a datasource path. | | **Dismiss gripes** | Turn a finding off for an occurrence, an object or the project. | | **Manage datasources** | Add, edit and remove them, and their per-project settings. | | **Manage domains** | Create domains and tag objects into them. | | **Mark roles untrusted** | Which database roles the grant rules treat as untrusted. Reading the report needs nothing. | | **Commit and pull** | Stage, commit, fetch and pull in a repository datasource. Fetching and pulling read the remote; neither writes to it. | | **Push, and add repositories** | Reaches the remote. | | **Export and import domains** | Writes the host's filesystem. | | **Approve and run repo commands** | The only feature that runs a program DataGripe did not write. | | **Manage the MCP server** | Turn the project's endpoint on, and mint or revoke its tokens. | | **Manage members and roles** | Add and remove people, and edit this matrix. | | **Rename or delete the project** | Deleting removes everything DataGripe holds about it. Files on the host are left where they are. | | **Restart the server** | Where the deployment is [supervised](/docs/updates/). Interrupts everybody. | The three built-ins start as what they always were: a viewer holds none of these, an editor holds the first nine, an owner holds all sixteen. An upgrade changes nothing about what anybody can already do. ## Four rules that will stop you - **Somebody has to keep "manage members and roles".** Moving the last person who has it into a role that does not is refused — otherwise the project can never be given its roles back. - **A role somebody holds cannot be deleted.** Move them first. A role deleted out from under a member would demote them silently. - **The built-ins cannot be renamed or removed.** Their capabilities are yours; the names are fixed so "owner" means the same thing in every deployment. - **Nobody can be left without a project.** Deleting one is refused when somebody in it has no other — including you. An account with no project cannot open the application at all, so this rule is the difference between "a shorter list" and somebody locked out by a button they did not press. ## Changes take effect immediately Take a capability away and the people holding it lose it now, in the tab they already have open — the interface stops offering what the server would refuse. No reconnect, no reload. ## Related - [Security](/docs/security/) — how each action is checked, and the rest of the model. - [permissions](/specs/permissions/) — the design, and what was deliberately not built. --- # Configuration DataGripe is configured by environment variables, everywhere: the CLI, the container, compose and the Helm chart all read the same names. [`.env.example`](https://github.com/datagripe/datagripe/blob/main/.env.example) in the repository is the same list with the same commentary, in the file you copy to `.env`. Nothing here is a secret kept in a database or a settings screen. A deployment's behaviour is decided by its environment, which is the one place a container, a systemd unit and a Helm chart all agree on. ## The two that matter most | | | | --- | --- | | `WEB_ORIGIN` | The exact origin browsers use — scheme, host, and port if it is not the default. Both the HTTP routes and the WebSocket upgrade compare against it, and everything in the app runs over that socket. A mismatch is an app that loads and then does nothing. | | `CONNECTION_ENCRYPTION_KEY` | Encrypts datasource passwords at rest. Not recoverable: losing it does not sign people out, it orphans every stored password. At least 32 characters — `openssl rand -base64 32`. | ## Where it listens | | Default | | | --- | --- | --- | | `PORT` | `3001` | | | `HOST` | `0.0.0.0` | The interface to bind. `datagripe personal` sets `127.0.0.1`, because that shape has no accounts. | | `NODE_ENV` | `development` | `production` adds `Secure` to the session cookie. Correct behind TLS and **wrong in front of plain http**, where the browser will not send the cookie back and sign-in will not stick. | | `WEB_STATIC_DIR` | — | A directory of built web assets to serve. Set by the CLI and the image; unset while developing against Vite. | ## The rest, by what it decides - **[Accounts and sign-in](/docs/authentication/)** — whether there are accounts at all, which of the three ways in exist, security keys, and Google. - **[Database and storage](/docs/database/)** — embedded or external PostgreSQL, where the data directory is, and the secrets. - **[Datasources](/docs/datasources/)** — the predefined connections file, and which hosts DataGripe may connect out to. - **[Limits](/docs/limits/)** — the ceilings a client cannot ask past. - **[Files, git and commands](/docs/files-and-git/)** — everything that touches the host's disk, each with its own switch. - **[Updates](/docs/updates/)** — the version check, and the one shape that can restart itself into a new image. - **[MCP](/docs/mcp/)** — the endpoint an AI agent connects to. ## Older names Three variables were renamed when the feature they belonged to grew past the domain export. The old name is still read, so nothing breaks on upgrade, and the new one wins when both are set. | Old | Now | | --- | --- | | `DOMAIN_EXPORT_ROOTS` | `HOST_FS_ROOTS` | | `DOMAIN_EXPORT_GIT` | `GIT_ENABLED` | | `DOMAIN_GIT_TIMEOUT_MS` | `GIT_TIMEOUT_MS` | ## Nothing here drifts Every variable the server reads is on one of these pages and in `.env.example`, and the site build fails rather than publishes when one is missing from either — or when a page names one the server no longer reads. A variable that is internal, experimental or "you will never need this" is still documented, with that said in words: an undocumented variable is not a warning, it is a search through the source. --- # Accounts and sign-in Two decisions, in this order: whether there are accounts, and which ways in exist if there are. ## Whether there are accounts | | Default | | | --- | --- | --- | | `AUTH_DISABLED` | derived | `true` means there is no sign-in at all: the session is implicit and everybody who reaches the port is the same person. Off in external mode, on in embedded — set it to override either. | | `ALLOW_SIGNUP` | `false` | The first account may always sign up, because a server nobody can get into is not a server. This governs the ones after it. | | `SESSION_SECRET` | — | Signs session cookies. Required in external mode; regenerating it signs everyone out and does nothing worse. | `AUTH_DISABLED=true` is the personal and desktop shape — one person, one machine, nothing to log into. It pairs with `HOST=127.0.0.1`: a server with no accounts has no business answering the network it happens to be plugged into. Turning accounts off does **not** turn the roles off inside a project; it means there is one person to hold them. ## Which ways in exist Three methods — password, security key, Google — each independently switchable, so a deployment can be locked to one of them. Off means **absent**: the route answers 404 and the sign-in screen does not draw a form that would refuse. Turning off the last way in fails at startup rather than at the login screen, where you would find out by being locked out. | | Default | | | --- | --- | --- | | `PASSWORD_AUTH_DISABLED` | `false` | Turns off email and password entirely: no login form, no password signup. What a deployment sets when every account is supposed to arrive through Google or a security key. | | `PASSKEY_AUTH_DISABLED` | `false` | The same, for security keys. | Google is the third and is configured rather than switched: setting its OAuth client is what turns it on. None of the three is a second factor on top of another. A security key and a Google identity are each an *alternative* to a password, and an account may hold several ways in at once — the server refuses to remove the last one. ## Security keys A FIDO2 key — a YubiKey, or a passkey your device or phone holds — works for both signup and sign-in, and sign-in is usernameless: insert the key and touch it, with no email to type and none to leak. Nothing to switch on; these only tune where credentials are scoped. | | Default | | | --- | --- | --- | | `WEBAUTHN_RP_ID` | `WEB_ORIGIN`'s host | The registrable domain a credential is bound to. Set it to a parent domain when the app answers on several subdomains and one key should work across all of them. It must be that host or a parent of it. | | `WEBAUTHN_RP_NAME` | `DataGripe` | The name the key shows while asking for a touch. | | `WEBAUTHN_EXTRA_ORIGINS` | — | Comma-separated. `WEB_ORIGIN` is always accepted; this is for the second hostname a proxy answers on, or the desktop shell's own origin. | WebAuthn needs a secure context — https, or localhost. On plain http anywhere else the browser refuses the ceremony, and the app says so rather than failing silently. ## Google sign-in Off until you configure an OAuth client. `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` **together** switch it on; setting exactly one is refused at startup rather than quietly ignored. | | Default | | | --- | --- | --- | | `GOOGLE_CLIENT_ID` | — | The OAuth 2.0 client id, from Google Cloud console → APIs & Services → Credentials → *Create credentials* → *OAuth client ID*, application type **Web application**. | | `GOOGLE_CLIENT_SECRET` | — | Its secret. The authorization code is exchanged on a backchannel from the server, so this never reaches a browser. | | `GOOGLE_REDIRECT_URI` | `WEB_ORIGIN` + `/api/auth/google/callback` | Must match an *Authorised redirect URI* on that client exactly. The default is right whenever the browser reaches the API on the origin it loaded the app from; set it when it does not. | | `GOOGLE_ALLOWED_DOMAINS` | — | Comma-separated Google Workspace domains allowed to sign in, matched against the `hd` claim and falling back to the email's domain. | Setting it up is four steps: 1. Create the OAuth client as above, and put `https://your-host/api/auth/google/callback` in its authorised redirect URIs — the same value `GOOGLE_REDIRECT_URI` would have. 2. Set `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET`. 3. Set `GOOGLE_ALLOWED_DOMAINS` to your Workspace domain. 4. Restart. The sign-in screen grows a Google button; nothing else changes, and existing accounts keep their passwords and keys. **Set `GOOGLE_ALLOWED_DOMAINS` if signup is open.** Empty means any Google account, and with `ALLOW_SIGNUP=true` that means anyone on the internet can make themselves an account on your deployment. Three things worth knowing about how it behaves: - **Identities are matched on Google's `sub`, never on the email.** People rename their address; the subject does not move. - **A first sign-in links to the existing account owning that address** when Google reports the address verified, and otherwise creates one under the same bootstrap / `ALLOW_SIGNUP` rule as any other signup. An unverified address is refused outright. - **Failures come back as a redirect** to `WEB_ORIGIN` with an `auth_error` parameter — a navigation cannot return anything else. The sign-in screen shows it and scrubs it from the address bar. The scope asked for is `openid email`: the address is all DataGripe stores. There is no refresh token and no Google API call after sign-in. ## See also - [Security](/docs/security/) — the session model, CSRF and the rest of the hardening. - [auth-and-hardening](/specs/auth-and-hardening/) — the design, and what was deliberately not built. --- # Database and storage DataGripe keeps its own state — accounts, projects, documents, domains, history — in PostgreSQL. That is separate from the datasources you connect to, which it never writes into unless you ask it to. One variable decides the shape. | | Default | | | --- | --- | --- | | `APP_DATABASE_URL` | — | A PostgreSQL for DataGripe's own state. Setting it selects **external** mode: accounts on, and the app applies pending migrations at startup. | | `DATABASE_MODE` | derived | `embedded` or `external`, forced. Without it, external when `APP_DATABASE_URL` is set and embedded otherwise. Set it to run embedded while a stray `APP_DATABASE_URL` is still in the environment. | ## Embedded mode No database to install: the server starts and manages a PostgreSQL cluster of its own, migrates it at startup, and generates the secrets once beside the data directory. There is nothing to configure at all. | | Default | | | --- | --- | --- | | `EMBEDDED_PG_DATA_DIR` | `./data/pg` | Where the cluster lives. | | `EMBEDDED_PG_PORT` | `0` | The port it listens on. `0` picks a free one at startup, which is what you want unless something else has to reach it. | | `DATAGRIPE_DATA_DIR` | per-OS | Read by the packaged launcher and the container image, not from a checkout: it puts the cluster, the git checkouts and the generated secrets under one path. `~/.local/share/datagripe` by default, `/data` in the image. | ## External mode A PostgreSQL you run. Both secrets are required, because there is no data directory the server owns to generate them into. | | Default | | | --- | --- | --- | | `CONNECTION_ENCRYPTION_KEY` | — | Encrypts datasource passwords at rest. Losing it orphans every stored password — see [Configuration](/docs/configuration/). | | `SESSION_SECRET` | — | Signs session cookies. Regenerating it signs everyone out and does nothing worse. | | `MIGRATIONS_DIR` | the checkout's | Directory of migration `.sql` files. A packaged build ships them beside the bundled server and points here; from a checkout you should not need to set it. | The app checks `schema_migrations` on every startup, in embedded and external database modes, and applies missing files before accepting HTTP or WebSocket connections. Each migration and its history row commit together. A PostgreSQL advisory lock serializes concurrent app starts and manual runners; a failed migration rolls back and stops startup. The app database account must have permission to apply the schema changes. See [upgrading](/docs/upgrading/). --- # Datasources Most datasources are added in the app and stored encrypted. Two things about them are the deployment's decision rather than a user's. ## Predefined connections | | Default | | | --- | --- | --- | | `CONNECTIONS_FILE` | `./connections.json` | Read-only connections declared in a file, with secrets resolved from the environment at boot and held only in memory. They never touch the application database. | A predefined connection is how a deployment hands people a datasource without handing them its password: the file names an environment variable, the server reads it at boot, and the value is never stored and never sent to a browser. A connection whose variable is unset is shown as unavailable, in those words, rather than failing when somebody runs a query against it. ## Reaching other machines Datasource connections are outbound connections made by the server on a user's behalf, which is the shape of an SSRF. Private ranges, loopback and cloud metadata endpoints are blocked by default. | | Default | | | --- | --- | --- | | `TARGET_HOST_ALLOWLIST` | — | Comma-separated hosts allowed despite the private-range block. This is how you reach the database on the same private network as DataGripe. | | `SSRF_DISABLED` | `false` | Turns the policy off entirely. Trusted networks only: the server will then connect to any host, including loopback and cloud metadata endpoints. | Development against a database on your own machine needs `TARGET_HOST_ALLOWLIST=localhost,127.0.0.1,::1`, which is what `.env.example` sets. Prefer naming the hosts to switching the policy off. ## Refresh and domains The database is the source of truth for object membership. Datasource refresh reloads grouped domains and removes tags for objects absent from fresh database catalogs, including old routine signatures. Domain definitions remain, even when empty; a replacement signature is a new object and starts untagged. Sync performs the same reconciliation without requiring a refresh first, and deletes the old objects' exported files. Preview shows those deletions without changing tags or files. A failed catalog read aborts reconciliation instead of treating the failure as an empty database. --- # Limits Every limit here is enforced by the server. A client can ask for more and will be refused or clamped — the browser is not trusted to police itself, and neither is an agent over MCP. | | Default | | | --- | --- | --- | | `QUERY_TIMEOUT_MS` | `30000` | Per statement, applied by the database as well as by DataGripe, so a query that outlives the connection still stops. | | `QUERY_MAX_ROWS` | `10000` | Rows returned to the grid. Beyond it the result is marked truncated rather than silently short. | | `QUERY_MAX_BYTES` | `25000000` | Serialized result size, which is the limit that actually protects a browser tab. | | `MAX_CONCURRENT_QUERIES_PER_USER` | `3` | Queries in flight per person. A fourth waits rather than adding load to a database that is already struggling. | | `ACCESS_REPORT_MAX_CELLS` | `250000` | Above it the access report asks for a filter instead of timing out. | | `DOMAIN_EXPORT_MAX_DATA_ROWS` | `10000` | Per table, for a domain exported with its data. Exceeding it fails that table and reports it, rather than writing a file that looks complete. | MCP has its own, much lower, row and byte caps: the consumer there is a context window rather than a grid. They are on [MCP](/docs/mcp/). --- # Files, git and commands All off or restricted by default, and each is a separate decision. The ordering here is deliberate: every switch below grants strictly more than the one above it. ## The host filesystem | | Default | | | --- | --- | --- | | `HOST_FS_DISABLED` | `false` | Turns off **every** host-filesystem feature: domain export and import, datasource paths, git datasources. This is what a hosted, multi-tenant deployment sets — there, the person pressing the button does not own the disk. | | `HOST_FS_ROOTS` | — | Optional colon-separated allowlist of absolute directories. Empty means no allowlist, because the directory is already named per datasource and making people configure the same thing twice bought nothing. Re-resolved with `realpath` on every access, so a symlink swapped in later is caught. | ## Git | | Default | | | --- | --- | --- | | `GIT_ENABLED` | `false` | Every git feature: the domain export's commit controls and git datasources. Off means absent, not disabled-with-a-tooltip. Git runs with argv, never a shell, and with `GIT_TERMINAL_PROMPT=0` so a missing credential errors instead of hanging. | | `GIT_REPOS_DIR` | `/repos` | One directory per datasource. DataGripe deletes a directory only when it created it. | | `GIT_TIMEOUT_MS` | `60000` | Every git invocation is killed at this. | | `GIT_CLONE_TIMEOUT_MS` | `600000` | Clone gets its own budget: a big repository is not a hung one. | ## Repository commands | | Default | | | --- | --- | --- | | `REPO_COMMANDS_ENABLED` | `false` | Lets a repository's `.datagripe/run.yaml` declare commands DataGripe can run. | | `REPO_COMMAND_DEFAULT_TIMEOUT_MS` | `120000` | Used when a command declares no `timeoutSeconds` of its own. | | `REPO_COMMAND_TIMEOUT_MS` | `600000` | The hard ceiling per run, whatever the command asked for. | `REPO_COMMANDS_ENABLED` has **its own switch on purpose**. Every other git feature reads and writes files; this one executes a program somebody else wrote, arriving over the network on `git pull`. A deployment that wants git datasources does not thereby want arbitrary execution, and the two decisions should not share a checkbox. Even on, nothing runs until a person approves the command list, and any change to that list needs a fresh approval. The approval *is* the security boundary; there is no sandbox. --- # Updates Bottom left of the window is the version: `● 0.0.8` when there is nothing to do, and `● Update Available` when there is. Click it for both version numbers, what to do about them, and the way to do it. Three different things put you behind and it says the same two words for all of them, because your next move is the same: a newer release exists, the page has fallen behind the server, or a downloaded build is waiting to be applied. For the last two the popup has a **refresh**, and it is worth knowing that this is not the browser's refresh. DataGripe is a progressive web app: a service worker answers from what it has already stored, so reloading the page re-renders the build you are trying to leave. The button asks for the new one, waits for it to install, and hands over — which is also what happens on its own after a restart. **The check asks once, when you open a project.** No interval, no background thread, no telemetry — and the server holds the answer for ten minutes, so a team opening the same project is one request between them. Close the tab and nothing asks again. A failed check says so. Reporting "up to date" when the request timed out is how a person misses a security release, so a check that could not reach the release feed says that instead. | | Default | | | --- | --- | --- | | `UPDATE_CHECK_DISABLED` | `false` | Removes the check and the button both. What an airgapped deployment sets: an offer that cannot work is worse than no offer. | ## Applying one What the menu suggests depends on how this deployment runs, because the answer is completely different per shape and the person reading it is often not the person who deployed it. | Shape | What the menu says | | --- | --- | | Kubernetes | Upgrade the image or chart, then restart. The app checks migrations before serving requests. | | Desktop | Nothing to do: the app updates itself and offers the new version. | | CLI (`bunx @datagripe/cli`) | Stop it and run it again with `@latest`. | | Container | Pull the image and restart the container. | | A checkout | Pull the repository and restart. | Every one of them also links to [upgrading](/docs/upgrading/), which has the commands and, more importantly, the migration story per shape. | | Default | | | --- | --- | --- | | `DATAGRIPE_SHAPE` | detected | Which of `desktop`, `cli` or `container` this is. The desktop shell, the CLI launcher and the image each set it for you, and Kubernetes is detected from `KUBERNETES_SERVICE_HOST` and wins over it. Setting it by hand only changes which upgrade advice you are shown. | ## Restarting the server The **restart server** button ends the app process. Kubernetes restarts the container inside the same pod and may pull a new image, but it does not rerun completed init containers or Helm hooks. The app itself checks `schema_migrations` and applies pending files at startup in both database modes. It waits for concurrent runners and refuses to serve if a migration fails. See [Upgrading](/docs/upgrading/). | | Default | | | --- | --- | --- | | `RESTART_TO_UPDATE` | true in Kubernetes | Whether ending the process brings a new one back. Set it for a compose stack or a systemd unit that restarts, and the button appears there too; set it `false` in a pod where you would rather nobody had it. | The button is there whether or not the update check found anything: a deployment that builds its own image from a moving tag has updates the feed has never heard of. A moving tag still requires the new image's migrations; the app startup check applies them before serving. Four things about that button: - **It is offered only where something will start DataGripe again.** Off by default everywhere but Kubernetes, because a container started without a restart policy that exits is a DataGripe nobody is running. The action is refused by the server as well as hidden by the app. - **Owners only, behind a confirm.** It disconnects everyone in the deployment for a few seconds. Unsaved editor content is in the browser, not in the process, so it survives. - **It goes out the way a `SIGTERM` does** — the embedded PostgreSQL cluster included, which is what stops the next process finding a stale lock file. - **It is in the audit log** as `app.restart`, with who pressed it. The page then waits for the server to answer again and reloads, because a restart that pulled a new image also replaced the web bundle. ## The account menu The avatar in the top right is the other half: the project you are in and your role, your address, both settings panels and the way out. No versions — those are a property of the deployment, not of you, which is why they are at the other end of the window. It comes from Gravatar, by hash: your address never leaves the browser, only a SHA-256 of it does, and nothing is requested until you open the menu for the first time. An address with no Gravatar gets no generated image — the request asks for a 404 — so what you see is your initials. That is also what you see offline, on an airgapped deployment, or behind a content-security policy that blocks it. Set a **name** in Account settings and the button shows that instead of the initials. It is stored in this browser: it does not rename your account, and other members still see your address. --- # MCP Every project can expose an endpoint an AI agent connects to, so the agent reads the project's own documentation and queries its datasources under the same rules a person gets. The opt-in is **per project** and off until an owner turns it on in the sidebar, read-only until they say otherwise. That is why the deployment switch defaults to on, unlike git and repository commands: a second default-off gate would only mean editing `.env` to try it on your own machine. | | Default | | | --- | --- | --- | | `MCP_ENABLED` | `true` | The deployment's kill switch. Off means the route and the panel are both absent. | | `MCP_PUBLIC_URL` | `WEB_ORIGIN` | What the panel tells people to point their client at. The endpoint is the same process on the same origin as the app, so the address you already gave browsers is the right one — including behind a proxy. Set this only when MCP answers on a different hostname. | | `MCP_MAX_ROWS` | `200` | Much lower than the grid's, because the consumer is a context window. A call asking for more is clamped, not refused. | | `MCP_MAX_BYTES` | `1000000` | Serialized bytes for one tool call's rows. | | `MCP_READ_MAX_BYTES` | `65536` | Per call, reading a file or a resource. The reply says how to continue rather than silently ending. | | `MCP_INSTRUCTIONS_MAX_BYTES` | `16384` | Cap on the briefing handed to every client at `initialize`. | The sidebar section says what is true of it without being opened: a grey **no tokens** while nothing can connect, green **read only** once something can, and violet **read/write** when what connects can commit. Three words, three colours, and the words are there because colour on its own is not a label anybody can rely on. An MCP query is an execution like any other: it appears in history under the token's name, with the same timeouts and the same concurrency limit a person has. A read-only project cannot change a row no matter what an agent is asked to run — and a datasource marked read-only stays read-only however the project's mode is set. Click **MCP Server** in the sidebar to open its management tab. The sidebar keeps the on/off switch and read/write status; the tab holds mode, functionality switches, tokens, endpoint and client configuration. You can configure it and create or revoke tokens while it is off. A failed status read keeps the MCP Server launcher visible for owners and shows **status unknown**, with **retry**. Open the tab to see the error and retry loading settings. Unknown does not mean off. The header is hidden only when the deployment explicitly reports MCP disabled or the account cannot manage MCP. If settings report a pending migration, follow [Upgrading](/docs/upgrading/) and retry after applying it. Domain management, sync and Git are separate opt-ins, initially off: - **Domains:** list, create, edit and delete domains; assign or unassign objects; set descriptions, ordering, hidden and include-data settings. Hidden domains are excluded from sync. - **Sync:** preview or write a datasource snapshot to its configured export directory. Preview is the default. Sync writes files and does not touch Git; commit and push them with the Git tools. - **Git:** inspect status, commit, pull and push as separate tool calls. Status fetches first, so *behind* is about the remote as it is now. Commit stages named paths then commits the whole staged index, including files staged earlier. Inspect status first. Deployment Git must also be enabled, and the datasource must have a repository. - **Pull and push keep a shared repository moving.** When other people push to the same branch — hand-written migrations next to DataGripe's export, say — pull fast-forwards, or rebases the clone's unpushed commits onto the remote. Push does the same first, so a remote that moved on other files does not reject it. Neither ever forces or makes a merge commit. A real conflict, uncommitted changes to tracked files when a rebase is needed, or incoming changes to a file with unsaved edits in DataGripe refuse with nothing changed and nothing pushed; that one needs a person with a terminal. The sidebar's own **pull** stays fast-forward only. Domain changes, sync writes, commits, pulls and pushes additionally require read/write mode and editor access. Reads and sync previews work in read-only mode once their functionality is enabled. The account must also hold the corresponding domain, sync or Git capability; custom role restrictions still apply. Agents should reuse the same idempotency key when retrying a completed mutation. The design, the tool list and the token model are in [the MCP spec](/specs/mcp/). --- # Getting started ## On your own machine ```bash bunx @datagripe/cli personal # or: npx @datagripe/cli personal ``` Then open . That is the whole setup. DataGripe starts its own PostgreSQL for its own state, migrates it, and opens without a login. It listens on loopback only — a DataGripe with no accounts has no business answering the network it is plugged into. Your work lives in `~/.local/share/datagripe` on Linux, `~/Library/Application Support/DataGripe` on macOS, and `%LOCALAPPDATA%\DataGripe` on Windows. It is still there next time, wherever you run the command from. The `npx` version works with no Bun on the machine. DataGripe's server runs on [Bun](https://bun.sh), so it fetches one. ``` -p, --port Port to listen on (default 3001) --data-dir Where the database and its secrets live -h, --help Every option ``` **Why `personal` and not just `bunx @datagripe/cli`?** Because it pins that shape rather than defaulting to it. Most developers have an `APP_DATABASE_URL` exported in their shell; without the word, DataGripe takes it as an instruction to run a shared deployment and stops, asking for two secrets nobody meant to need. Asking for the personal one should not depend on what else is in your environment. ## As a desktop app [Download it](https://github.com/datagripe/datagripe/releases/latest) for Linux, macOS or Windows. The same application in a frameless window, with its own database underneath and no login. The web app also installs as a PWA if you would rather it lived in the browser. ## For other people See [deploying](/docs/deploy/). The short version is one container: ```bash docker run -p 3001:3001 -v datagripe:/data ghcr.io/datagripe/datagripe ``` ## The first five minutes **Add a datasource.** The connection dialog tests before it saves. Passwords are encrypted at rest with a key the deployment holds, and never come back to the browser. **Run something.** `Ctrl/Cmd+Enter` runs the selection, or the statement the caret is in if there is no selection — you do not have to select a statement to run it. `Ctrl/Cmd+Shift+Enter` runs the whole document. **Reformat with `Ctrl+Alt+L`.** IntelliJ's binding, and DataGrip's. It is not written on a button anywhere, which is the single most common thing people fail to find. **Read the gripes.** They appear in the gutter as you type, before you press run. If one is wrong, dismiss it — for that occurrence, that table, or the whole project — and [tell us](https://github.com/datagripe/datagripe/issues), because a wrong gripe costs more trust than a missing one. Then: [everything it can do](/docs/features/), and [every shortcut](/docs/keyboard/). --- # Deploying Four shapes, smallest first. All of them run the same server, take the same [configuration](/docs/configuration/), and are built from the same release. | | State lives in | Accounts | | | --- | --- | --- | --- | | `bunx @datagripe/cli personal` | an embedded cluster under the data dir | off | [Getting started](/docs/getting-started/) | | `docker run … -v datagripe:/data` | the same, on a volume | off | [Docker](/docs/docker/) | | compose | a PostgreSQL container beside it | on | [Compose](/docs/compose/) | | Kubernetes, Helm | a StatefulSet or a managed database | on | [Kubernetes](/docs/kubernetes/) | Picking one is mostly a question of who else uses it. One person: the CLI or the desktop app. A team on one machine: compose. A team on a cluster: Helm. ## Read this first Two settings decide whether a deployment works, and neither has a default that can be right everywhere. Everything else has a sensible default and these two cannot. **`WEB_ORIGIN` must be exactly the URL browsers use** — scheme, host, and port if it is not the default. Both the HTTP routes and the WebSocket upgrade compare against it, and everything in the app runs over that socket. A mismatch is not a degradation, it is an app that loads and then does nothing. It is the first thing to check, every time. **`CONNECTION_ENCRYPTION_KEY` is not recoverable.** Datasource passwords in the application database are encrypted with it. A database backup without the key is a backup of connections nobody can open. Store it somewhere other than beside the backup. There is a third, narrower one worth knowing before it costs you an afternoon: `NODE_ENV=production` adds `Secure` to the session cookie, which is correct behind TLS and **wrong in front of plain HTTP**, where the browser will not send the cookie back and sign-in will not stick. ## The quickest real one ```bash docker run -p 3001:3001 -v datagripe:/data ghcr.io/datagripe/datagripe ``` Its own PostgreSQL under `/data`, no accounts, nothing to configure. The volume is not optional — without it a restart loses the database and the generated key its stored passwords are encrypted with. [More](/docs/docker/). ## Turning accounts on Accounts come on when DataGripe stops running its own database. Setting `APP_DATABASE_URL` selects external mode, which turns accounts on and stops the server migrating itself; the first person to open it creates the first account, and after that signup is closed until you say otherwise. That is what [compose](/docs/compose/) and [Kubernetes](/docs/kubernetes/) both are, with the database supplied differently. ## Migrations The app applies pending migrations at startup in every deployment, before serving requests. The image also has a migration-only entry point for operators who want to apply changes before rollout: ```bash docker run --rm -e APP_DATABASE_URL=… -e CONNECTION_ENCRYPTION_KEY=… \ -e SESSION_SECRET=… ghcr.io/datagripe/datagripe migrate ``` Compose runs it as a one-shot service, Helm as a pre-install hook, the plain manifests as an init container. See [upgrading](/docs/upgrading/). ## Related - [Configuration](/docs/configuration/) — every environment variable, a page per decision. - [Security](/docs/security/) — what to switch off before it faces anybody. - [Upgrading](/docs/upgrading/) — backups, migrations, key rotation. - [`docs/operations.md`](https://github.com/datagripe/datagripe/blob/main/docs/operations.md) — backups, the audit log, and the full production checklist. --- # Docker The smallest shared deployment is one container and one volume. ```bash docker run -p 3001:3001 -v datagripe:/data ghcr.io/datagripe/datagripe ``` Zero-config DataGripe: it starts its own PostgreSQL under `/data`, migrates it, generates its own secrets, and runs with no accounts. Open . ## The volume is not optional Without `-v`, a restart loses two things: the database, and the generated key the datasource passwords in it were encrypted with. The second is the one that hurts, because it is not recoverable — see [configuration](/docs/configuration/). Anything under `/data` is state. Back it up as a unit, or run an external PostgreSQL and back that up instead. ## Any other port means saying so `WEB_ORIGIN` is the exact origin browsers use, and both the HTTP routes and the WebSocket upgrade compare against it. Publishing on a different port without setting it is the single most common broken deployment: the page loads, and then nothing in it works, because the socket everything runs over was refused. ```bash docker run -p 8080:3001 -e WEB_ORIGIN=http://localhost:8080 \ -v datagripe:/data ghcr.io/datagripe/datagripe ``` The same applies behind a proxy or a hostname: ```bash docker run -p 3001:3001 -e WEB_ORIGIN=https://datagripe.example.com \ -e NODE_ENV=production -v datagripe:/data ghcr.io/datagripe/datagripe ``` `NODE_ENV=production` adds `Secure` to the session cookie. That is correct behind TLS and **wrong in front of plain HTTP**, where the browser will not send the cookie back and sign-in will not stick. ## Turning accounts on The zero-config shape has no accounts, which is right for one person and wrong for a team. Accounts come on when you give it a database of its own: ```bash docker run -p 3001:3001 \ -e APP_DATABASE_URL=postgres://user:password@db:5432/datagripe \ -e CONNECTION_ENCRYPTION_KEY="$(openssl rand -base64 32)" \ -e SESSION_SECRET="$(openssl rand -base64 32)" \ -e WEB_ORIGIN=https://datagripe.example.com \ -e NODE_ENV=production \ ghcr.io/datagripe/datagripe ``` Setting `APP_DATABASE_URL` selects external mode, which turns accounts on and stops the server migrating itself. The first person to open it creates the first account; after that signup is closed unless `ALLOW_SIGNUP=true`. Generate those two secrets once and keep them. Regenerating `SESSION_SECRET` only signs everyone out; regenerating `CONNECTION_ENCRYPTION_KEY` orphans every stored datasource password. ## Migrations The app applies pending migrations at startup in every deployment, before serving requests. The image also has a migration-only entry point for operators who want to apply changes before rollout: ```bash docker run --rm \ -e APP_DATABASE_URL=… -e CONNECTION_ENCRYPTION_KEY=… -e SESSION_SECRET=… \ ghcr.io/datagripe/datagripe migrate ``` Running it separately is optional. Applied files are skipped, and concurrent runners are serialized by a database advisory lock. [Compose](/docs/compose/) runs this as a one-shot service, Helm as a pre-install hook, and the plain manifests as an init container — see [upgrading](/docs/upgrading/). ## Images and tags The image is `ghcr.io/datagripe/datagripe`. A version tag is a specific release and is what a deployment should pin; `latest` follows the newest one, which is fine for trying it and not for a service other people depend on. ## Related - [Compose](/docs/compose/) — the same thing with a PostgreSQL beside it. - [Kubernetes](/docs/kubernetes/) — Helm and plain manifests. - [Configuration](/docs/configuration/) — every variable, a page per decision. - [Security](/docs/security/) — what to switch off before it faces anyone. --- # Compose Compose is the shape for a team on one machine: DataGripe, a PostgreSQL it owns, accounts switched on, and a migration that runs before the app starts. ```bash curl -O https://raw.githubusercontent.com/datagripe/datagripe/main/deploy/compose.yaml curl -o .env https://raw.githubusercontent.com/datagripe/datagripe/main/deploy/.env.example $EDITOR .env docker compose up -d ``` DataGripe on . The first person to open it creates the first account; after that signup is closed unless `ALLOW_SIGNUP` says otherwise. The compose file is written to work on its own, downloaded next to a `.env` and nothing else. There is no `build:` in it. > Not to be confused with the `compose.yaml` at the repository root, > which starts a PostgreSQL for the test suite and nothing else. ## The three secrets `.env.example` generates all three with the right shape. If you would rather write them yourself, the shapes matter: | | | | --- | --- | | `POSTGRES_PASSWORD` | **Hex, not base64.** It is interpolated into a connection URL, so a `/` or a `+` ends the password early and the app cannot connect. `openssl rand -hex 32`. | | `CONNECTION_ENCRYPTION_KEY` | Encrypts datasource passwords at rest. Not recoverable — losing it orphans every stored password. At least 32 characters. | | `SESSION_SECRET` | Signs session cookies. Regenerating it only signs everyone out. | That first row is a real failure and not a theoretical one. It looks exactly like a wrong password, which is the wrong place to go looking. ## Ports and origins `WEB_ORIGIN` must be the exact origin browsers use, and the published port has to agree with it. They are set in the same file for that reason; change them together. ```bash WEB_ORIGIN=http://localhost:3000 # matches the published port ``` Behind a proxy that terminates TLS: ```bash WEB_ORIGIN=https://datagripe.example.com NODE_ENV=production ``` `NODE_ENV=production` adds `Secure` to the session cookie. Set it only once `WEB_ORIGIN` is `https://` — on plain HTTP it does not harden the deployment, it breaks sign-in, because the browser will not send the cookie back. ## The migration service The stack runs the image's `migrate` entry point as a one-shot service before the app, so an upgrade applies its migrations automatically on `docker compose up -d`. The app also checks and applies pending migrations at startup before serving requests — see [upgrading](/docs/upgrading/). ## The database is not published Deliberately. DataGripe is the only thing that needs it and reaches it on the compose network. If you want to point `psql` at it, do that through `docker compose exec` rather than opening a port. ## Backups The durable state is the PostgreSQL volume: accounts, sessions, workspaces, datasource metadata and encrypted secrets, documents, layouts, query history. Target databases are never backed up by DataGripe. ```bash docker compose exec -T postgres \ pg_dump --format=custom -U datagripe datagripe > datagripe-$(date +%Y%m%d).dump ``` Store `CONNECTION_ENCRYPTION_KEY` with the same care as the dump, and **not in the same place**. A backup without the key restores everything except access to any datasource. ## Related - [Docker](/docs/docker/) — one container, no compose. - [Kubernetes](/docs/kubernetes/) — Helm and plain manifests. - [Upgrading](/docs/upgrading/) — what to run, in what order. - [Configuration](/docs/configuration/) — every variable, a page per decision. --- # Kubernetes Two shapes, the same deployment. The chart if you want it parameterised, the manifests if you would rather read a deployment than configure one. ## Helm ```bash helm install datagripe oci://ghcr.io/datagripe/charts/datagripe \ --namespace datagripe --create-namespace \ --set webOrigin=https://datagripe.example.com \ --set ingress.enabled=true \ --set ingress.hosts[0].host=datagripe.example.com ``` The chart is an OCI artifact published beside the image, so there is no `helm repo add` and no `index.yaml` to go stale. Its version is DataGripe's version: `--version 0.0.16` is the chart that installs `0.0.16`. ### Upgrades and restarts The chart's bundled PostgreSQL StatefulSet is an **external app database** from DataGripe's perspective. Its schema needs the same migration step as a separately managed PostgreSQL service. The app checks and applies missing migrations on every startup, before serving requests. This also runs when Kubernetes restarts only the app container in an existing pod. Concurrent starts share a database advisory lock, and a failed migration prevents the app from becoming ready. `helm upgrade` may pre-apply migrations through its hook, and the plain manifests retain a migration init container. Both are compatible with the startup check: already-recorded files are skipped. Pin the migration and server images to the same release. For an older server, such as 0.0.13, that has already started without its migrations, apply them from the running image and reload the browser: ```bash kubectl exec -n datagripe deployment/datagripe -- \ bun run /app/bin/datagripe.mjs migrate ``` ### A managed database Better operated than one you now own: ```bash --set postgresql.enabled=false \ --set database.url=postgres://user:password@db.example.com:5432/datagripe ``` ### No database at all DataGripe runs the PostgreSQL it needs inside the pod. One workload, one volume, strictly one replica: ```bash --set database.mode=embedded \ --set postgresql.enabled=false \ --set persistence.enabled=true ``` ### Secrets The encryption key and the session secret are generated on first install and kept across upgrades. Their Secret is marked `helm.sh/resource-policy: keep`, so an accidental `helm uninstall` does not take them with it. Back them up anyway, or set them yourself: ```bash --set secrets.existingSecret=datagripe-secrets ``` Losing `CONNECTION_ENCRYPTION_KEY` does not sign anyone out. It orphans every datasource password in the database, permanently. ## Plain manifests [`deploy/k8s`](https://github.com/datagripe/datagripe/tree/main/deploy/k8s) is the same deployment written to be read top to bottom rather than parameterised: a Deployment with the migration as an init container, a PostgreSQL StatefulSet, a Service, and an Ingress commented out until you have a hostname. ```bash $EDITOR deploy/k8s/secret.yaml # the secrets $EDITOR deploy/k8s/configmap.yaml # WEB_ORIGIN kubectl apply -k deploy/k8s ``` ## One replica The default is one replica. Migration runners serialize through a database advisory lock, so concurrent starts do not apply a file twice. Embedded database mode and a shared local data volume still require one writer. Nothing else in DataGripe requires a single replica. Sessions live in the database, not in the pod. ## Ingress and websockets This is the setting that breaks a Kubernetes deployment in a way nothing else explains. DataGripe holds **one WebSocket open per browser tab**, for as long as the tab is open. ingress-nginx's default 60-second read timeout closes every one of them a minute after it opens. The app reconnects, so it looks like an intermittent glitch rather than a configuration error — but a query in flight does not survive it. The chart sets these already. If you are writing your own ingress, set them too: ``` nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" nginx.ingress.kubernetes.io/proxy-body-size: 32m ``` Other ingress controllers have their own equivalent. The requirement is the same: a long-lived upgraded connection, and a body size that fits an import. ## `WEB_ORIGIN` The exact origin browsers use — which behind an ingress is the ingress hostname and scheme, not the Service and not the pod. Both the HTTP routes and the WebSocket upgrade compare against it, so a mismatch is an app that loads and then does nothing at all. Set `NODE_ENV=production` alongside it once that origin is `https://`. ## Related - [Upgrading](/docs/upgrading/) — the migration hook, and what to do when it fails. - [Security](/docs/security/) — what to switch off for a shared deployment. - [Configuration](/docs/configuration/) — every variable, a page per decision. --- # Upgrading An upgrade is: new image, migrations applied, server ready. The app checks and applies pending migrations on every startup, before serving requests, whether its PostgreSQL is embedded or external. While the version is `0.0.x`, nothing in the application is promised to stay put. Read the [release notes](/docs/release-notes/) before upgrading a deployment other people use. ## The short version, per shape The account menu — the avatar in the top right — shows the version you are running and checks for a newer one when you press the button. What it tells you to do about it is one of these. | You run | Upgrading is | | --- | --- | | The desktop app | **Nothing.** It updates itself: it offers the new version and installs it when you accept. | | `bunx @datagripe/cli personal` | `bunx @datagripe/cli@latest personal`. The data directory is untouched. | | One container | `docker pull … && docker restart datagripe`. It migrates itself on start. | | Compose | `docker compose pull && docker compose up -d`. The migration service runs first. | | Kubernetes or Helm | Upgrade the image or chart, then restart. The app checks migrations before serving; existing hooks and init containers may still pre-apply them. Image tags and pull policy determine which version starts. | | A checkout | `git pull && bun install && bun run db:migrate`. | The rest of this page is what those commands are doing, and the two things worth doing before any of them. ## Back up first Two things make an upgrade unrecoverable rather than annoying, and both are cheap to avoid. **The database.** The durable state is the application PostgreSQL: accounts, sessions, workspaces, datasource metadata and encrypted secrets, documents, layouts, query history. ```bash pg_dump --format=custom --file=datagripe-$(date +%Y%m%d-%H%M).dump "$APP_DATABASE_URL" ``` **`CONNECTION_ENCRYPTION_KEY`.** The datasource passwords in that dump are encrypted with it. A backup without the key restores everything except the ability to open any datasource. Store the key with the same care as the backup and **not in the same place**. ## Embedded shapes The personal CLI, the desktop app and the zero-config container migrate themselves on start. Upgrading is replacing the binary or pulling the image. The **desktop app** is the one shape that needs nothing from you: it checks its own release feed, offers the new version, and replaces itself when you accept. Declining asks again next time it starts. ```bash bunx @datagripe/cli@latest personal docker pull ghcr.io/datagripe/datagripe && docker restart datagripe ``` The data directory is untouched. It is the thing worth backing up. ## Shared shapes The app checks `schema_migrations` on every startup, in embedded and external database modes, and applies missing files before accepting HTTP or WebSocket connections. Each migration and its history row commit together. A PostgreSQL advisory lock serializes concurrent app starts and manual runners; a failed migration rolls back and stops startup. The app database account must have permission to apply the schema changes. The image also has a migration-only entry point for an explicit pre-rollout step: ```bash docker run --rm \ -e APP_DATABASE_URL=… -e CONNECTION_ENCRYPTION_KEY=… -e SESSION_SECRET=… \ ghcr.io/datagripe/datagripe migrate ``` | Shape | Runs the migration as | | --- | --- | | [Compose](/docs/compose/) | a one-shot service, before the app | | [Helm](/docs/kubernetes/) | a `pre-install,pre-upgrade` hook Job | | [Plain manifests](/docs/kubernetes/) | an init container | | By hand | `bunx @datagripe/cli migrate` | So in practice: `docker compose pull && docker compose up -d`, or `helm upgrade datagripe oci://ghcr.io/datagripe/charts/datagripe`, and the migration is already in the path. Already-recorded migrations are skipped. Concurrent runners wait on the same advisory lock and recheck history before executing each file. ### Restarting into a pulled image With `imagePullPolicy: Always` on a tag that moved, an app-container restart may pull the new image. Completed init containers do not rerun, but the app itself checks migration history before serving. The **restart server** button therefore includes the schema check in both database modes. A pinned image still needs its tag updated to get a new release. The button requires a supervisor that will restart the process. Kubernetes enables it by default; other supervised shapes can set `RESTART_TO_UPDATE=true` — see [updates](/docs/updates/). ## MCP missing after a 0.0.13 upgrade Version 0.0.13 silently hid the MCP sidebar header when its status request failed. One cause is a shared app database missing migration 0027: the status query attempted to read the new functionality columns. Apply the pending migrations using the upgraded image or CLI, then reload the app. This does not revoke tokens or reset the project's MCP setting. The corrected client keeps the launcher visible on status failures and shows **status unknown** with a retry action. Its management tab reports missing migrations explicitly. If migration 0027 is already applied, check the server log for the failed `mcp.status` action; a missing header alone does not prove the server is off. ## Pin the version The chart's version is the app's version, and the image's tag should be too. `latest` is fine for trying it and wrong for a service other people depend on — an upgrade should be a thing you did, not a thing that happened. ```bash helm upgrade datagripe oci://ghcr.io/datagripe/charts/datagripe --version 0.0.16 ``` ## Restoring ```bash createdb datagripe_restore pg_restore --dbname=datagripe_restore datagripe-YYYYMMDD-HHMM.dump APP_DATABASE_URL=postgres://…/datagripe_restore bun run db:migrate ``` Run the migration after every restore: it is idempotent, and it brings an older dump up to the current schema before the server starts. Sessions survive a restore. They only become invalid if you also rotate `SESSION_SECRET`, and the two can coexist safely. Worth doing quarterly into a scratch database rather than discovering the procedure during an incident. Compare `select count(*) from users` and `select count(*) from connections` between the two databases. ## Rotating the encryption key Add the new key as version 2 in the keyring, restart, and re-save connections opportunistically. Old key versions keep decrypting, so there is no flag day and no window where a datasource cannot be opened. ## Related - [Release notes](/docs/release-notes/) — what changed. - [Configuration](/docs/configuration/) — every variable, a page per decision. - [Updates](/docs/updates/) — the version check and the restart button. - [`docs/operations.md`](https://github.com/datagripe/datagripe/blob/main/docs/operations.md) — backups, the audit log, and the full production checklist. --- # Security DataGripe holds credentials for other people's databases. That is the threat model, and most of what follows comes from taking it seriously rather than from a checklist. This page is what an operator needs. The full design is in the [authentication and hardening spec](/specs/auth-and-hardening/), and the production checklist is in [`docs/operations.md`](https://github.com/datagripe/datagripe/blob/main/docs/operations.md). ## Before it faces anyone Three settings, in order of how often getting them wrong causes a real problem. **`WEB_ORIGIN`** — the exact origin browsers use. Every HTTP route and the WebSocket upgrade validate `Origin` against it. This is a security control, not only a convenience: it is what stops another origin driving the socket. **`NODE_ENV=production`** — adds `Secure` to the session cookie. Correct behind TLS and **wrong in front of plain HTTP**, where the browser will not send the cookie back at all and sign-in silently fails. **`ALLOW_SIGNUP=false`** — the default. The first account may always sign up; this governs every one after it. Leaving signup open on a deployment reachable from anywhere means anyone can create an account on it. ## Accounts and sign-in Cookie sessions, stored in the database. Sessions survive a restart and a restore; rotating `SESSION_SECRET` signs everyone out and nothing else. A **FIDO2 security key** is an alternative to the password for both signup and sign-in, and sign-in is **usernameless** — insert the key and touch it. There is no email to type, which means there is no email to leak by asking whether one exists. Nothing needs switching on; `WEBAUTHN_RP_ID` only matters when the app answers on several subdomains and one key should work across all of them. **Google sign-in** is the third way in, off until a deployment configures an OAuth client. Identities are matched on Google's `sub` rather than the email address, an unverified address is refused, and `GOOGLE_ALLOWED_DOMAINS` restricts it to your Workspace domains — set it if `ALLOW_SIGNUP` is true, because empty means any Google account on the internet. The only claim stored is the address. Each of the three is independently switchable, and none is a second factor on top of another: a key and a Google identity are alternatives to a password, not additions to one. Turning off the last way in is refused at startup. All of it is on [accounts and sign-in](/docs/authentication/). State-changing HTTP routes require a CSRF token matching the session's. ## Authorization is per action Socket authentication is not object authorization. The upgrade binds the socket to a user, a session, a workspace and the **capabilities** their role holds — and the dispatcher checks the capability an action needs on **every single message**, not once at the door. Roles are a project's own: a name and a set of capabilities, edited as a matrix in project settings ([roles](/docs/roles/)). Owner, editor and viewer exist in every project with the capabilities those three ranks always had, so nothing about an upgraded deployment changes. | | viewer | editor | owner | | --- | :-: | :-: | :-: | | Browse schema, read history, watch executions | yes | yes | yes | | Run queries, edit data and structure, manage datasources | – | yes | yes | | Push, sync, repo commands, MCP, members, restart | – | – | yes | Reading is not a capability: being a member is being able to read. A project where somebody must not see the data wants a second datasource, not a checkbox. Somebody must always be able to manage members — the modern form of "owners cannot remove the last owner" — and taking a capability away reaches open sessions immediately rather than on their next reconnect. ## Datasource credentials Encrypted at rest with `CONNECTION_ENCRYPTION_KEY` (AES-GCM, versioned keys) and **never returned to the browser** — not masked, not write-only-in-the-form, absent. The server holds them. Consequences worth planning for: - Losing the key does not sign anyone out. It orphans every stored password, permanently. Back it up separately from the database. - Rotating it is additive: add the new version to the keyring, restart, re-save connections opportunistically. Old versions keep decrypting, so there is no flag day. - **Predefined connections** (`connections.json`) resolve their secrets from the environment at boot and are held only in memory. They never touch the application database, which is the right shape when the secret belongs to your secret manager rather than to DataGripe. ## Reaching other machines Before opening any target connection, the hostname is resolved and **every** returned address is classified. Blocked by default: loopback, RFC1918, link-local — including `169.254.169.254`, the cloud metadata endpoint — unique local, multicast, unspecified and reserved ranges. `TARGET_HOST_ALLOWLIST` overrides per hostname, exact or `*.suffix`. `SSRF_DISABLED=true` turns the policy off entirely. On a hosted or multi-tenant deployment this is the difference between a database client and an open proxy into your network, because the server will then connect to any host the user names, including loopback and metadata endpoints. **One accepted limitation, stated plainly.** The driver resolves hostnames itself at connect time, so a DNS answer can change between the check and the connection — classic rebinding. Closing it fully needs a connect-time address hook the driver does not expose. A deployment that needs that guarantee must enforce egress at the network layer. ## The host filesystem, and git All off or restricted by default, and each is a separate decision. | | Default | | | --- | --- | --- | | `HOST_FS_DISABLED` | `false` | Set it to `true` for anything hosted or multi-tenant. It turns off **every** host-filesystem feature — domain export and import, datasource paths, git datasources — because there the person pressing the button does not own the disk. | | `HOST_FS_ROOTS` | — | Optional allowlist of absolute directories, re-resolved with `realpath` on every access, so a symlink swapped in later is caught. | | `GIT_ENABLED` | `false` | Off means absent, not disabled-with-a-tooltip. Git runs with argv and never a shell, with `GIT_TERMINAL_PROMPT=0` so a missing credential errors instead of hanging. | | `REPO_COMMANDS_ENABLED` | `false` | See below. | **Repo commands deserve their own paragraph.** They let a repository's `.datagripe/run.yaml` declare commands DataGripe may run. It has a separate switch from the rest of git on purpose: every other git feature reads and writes files, and this one **executes a program somebody else wrote, arriving over the network on `git pull`**. Even switched on, nothing runs until a person approves the list, and any change to that list needs a fresh approval. The approval *is* the security boundary. There is no sandbox, and the documentation says so rather than implying one. ## MCP `MCP_ENABLED` defaults to `true` and is the deployment's kill switch — off here means the route and the panel are both absent. It is on by default because the opt-in that matters is per project and defaults to off: an owner has to turn it on, and it is read-only until they say otherwise. A read-only project cannot change a row no matter what it is asked to run. An MCP call is an execution like any other. It appears in the query history with the token's name beside it. `MCP_MAX_ROWS` defaults to 200, much lower than the grid's, because the consumer is a context window. ## Limits Server-enforced, so a client cannot ask past them: query timeout, max rows, max bytes, concurrent queries per user, access-report cell count, domain-export row count. All on [limits](/docs/limits/). Rate limits are in-memory token buckets — 5 logins per minute per email, 30 per minute per IP, 30 executions per minute per user, and others. Exhaustion returns `RATE_LIMITED`, not a slow response. ## What reaches the internet Three things, and only three. Datasource connections you configured. The MCP endpoint, when a project's owner turns it on. And two about the app itself: the **update check**, which asks the release feed when a project is opened and at most once every ten minutes per server — no interval, no background thread — and the avatar, which is a hash of your address sent to Gravatar the first time the account menu is opened. Both are described on [updates](/docs/updates/), and the first is removed outright by `UPDATE_CHECK_DISABLED`. ## Restarting from inside the app In Kubernetes, where a Deployment is guaranteed to start it again, a workspace **owner** can restart the server from the account menu — with `imagePullPolicy: Always` it can pull a new image, but does not rerun migration init containers or Helm hooks. The app itself applies pending migrations before serving requests, in both database modes. It is off by default in every other shape and refused by the server there, not just hidden, because a process nothing will restart must not be able to stop itself. `RESTART_TO_UPDATE` decides it either way, and every press is audited. ## Audit log Structured JSON on stdout, tagged `"msg":"audit"`: signup, login success and failure, logout, security-key registration and removal, datasource create/update/delete, execution start and cancel, workspace membership changes, domain operations, access role changes, `app.restart`, and `ssrf.blocked`. Never passwords, never secrets, never result values. Pipe it to your log stack and alert on two things: bursts of `auth.login.failure`, and any `ssrf.blocked` at all — the second means somebody pointed the server at an address the policy refused. ## Reporting something Open an [issue](https://github.com/datagripe/datagripe/issues). For anything you would rather not file in public, say so in the issue without the details and a private channel will be arranged. ## Related - [Configuration](/docs/configuration/) — every variable, a page per decision. - [Upgrading](/docs/upgrading/) — backups and key rotation. - [Kubernetes](/docs/kubernetes/), [Compose](/docs/compose/) — the shapes with accounts on. ## MCP functionality The MCP Server tab has separate, default-off switches for domain management, datasource sync and Git. Enabling the server or changing its query mode does not enable these switches. Domain mutations, sync writes, commits and pushes also require read/write mode and an editor-level token account, with the corresponding `domain.manage`, `sync.run`, `git.commit` or `git.push` capability. Custom role restrictions are checked on each authenticated call. Existing host filesystem restrictions and the deployment Git switch still apply. See [MCP](/docs/mcp/) for tool behavior and commit scope. --- # Why it complains Every decision in the gripes engine follows from a single sentence: > A gripe that is merely rude is noise and users disable it within a > week. A gripe that is correct and rude is a feature they screenshot > into the team channel. So correctness outranks coverage, and silence outranks a guess. The rest of this page is what that costs. ## A rule that cannot tell says nothing There is a hedged sentence the brand specification carries as a counter-example, and it is worth reading because it is exactly what most tools of this kind produce: > Hmm, this query might be a little slow, maybe? It is rejected for being hedged into uselessness. That is not a style note, it is a hard property of the engine: a rule that needs the schema and cannot see it returns nothing. It does not downgrade to a guess and it does not append "possibly". This has a visible consequence. The client's view of your schema is the completion catalogue, which fetches a table's columns on demand — so a schema rule's first look usually knows nothing, and correctly says nothing. Asking for the columns is the side effect. When they arrive the catalogue notifies, the analysis re-runs, and the finding appears a beat late. Late is a cost worth paying. Wrong is not. The same rule is why a `null` from the schema may never be read as `false`. A rule treating "not fetched yet" as "NOT NULL" would go silent on a real problem; one treating it as "nullable" would invent one. ## No rule is about formatting sqlfluff exists, and it is better at that than this will ever be. A rule earns its place in DataGripe by knowing something about *this* database — the row count, the missing index, the column that is nullable in practice — or by being about damage rather than tidiness. A finding derivable from the query text alone, with no reference to the schema or the cost, is usually a lint rule and belongs to a formatter. Shipping those is the fastest way to get the whole feature switched off, and a feature that is switched off catches nothing. There is a concrete casualty. `select.unqualified-star` was specified and is not going to ship: in a SQL client, `select * from t limit 100` is the single most common legitimate query there is. The case with real teeth — a star frozen into a view definition at creation — shipped as `view.select-star`. The ad-hoc case did not. ## The wording is fixed, and written by a person Four strings per rule. No LLM, no template shuffling, no synonym rotation. The same rule firing twice uses the same wording; variants to seem clever are explicitly out. What this buys is that a gripe is *auditable*. Its footer carries `severity · rule id · line`, and the rule id is the same string at every attitude level and for every reader. You can grep it, dismiss it by it, and file a bug against it. It also means the strings are copy, reviewed like copy. Mechanical constraints are enforced by a test — length caps, a barred-term list, no profanity at `notice`, and every placeholder resolving — but the judgment of whether a sentence is *good* is a person's, every time. ## It criticises the query, never you The complaint is always about the statement, the schema or the grant. It is never about the competence of whoever wrote it. That line is the difference between a tool with a personality and a tool that is unpleasant to use, and it is a hard rule rather than a preference. Three more constraints hold it in place: - **A gripe never blocks anything.** It does not prevent a run, gate a save, or fail a build. It is an opinion held loudly. - **There is no score.** No grade, no "query health 62%". A number invites gaming and averages away the one blocker that mattered. - **Everything is dismissable**, at every attitude level, and the app always tells you it is hiding something. See [dismissing gripes](/docs/dismissing-gripes/). ## Where correctness actually bit The discipline above is cheap to write down. These are the places it cost something, each of which is a rule that was wrong before its test caught it: - `delete.no-where` reads the statement's *main verb*, looking through a leading `WITH`. Without that, `create trigger t after delete on x` reads as an unqualified delete — and it would have fired on every trigger in the database. - `view.select-star` distinguishes a projection star from multiplication by what surrounds it. Depth alone is not enough: the `*` in `select qty * price` sits at the same depth as a real one. - `index.not-concurrent` is gated on the *dialect*, not the adapter id. `CONCURRENTLY` is not merely unhelpful on MySQL and SQLite, it is a syntax error, so the advice would have been actively wrong. - `routine.volatile-but-readonly` reads the routine *body*, not its definition. Reading the whole definition finds `CREATE` in every routine, so the rule never fired at all — which is how it was first written. - Two bugs underneath the rules had to be fixed before any of them could be trusted, and both were silent. The tokenizer dropped `!`, so `a != 1` tokenized identically to `a = 1` and a rule about equality would have fired on its exact opposite. And `SqlDialect` was structurally `string`, so every caller type-checked while a hardcoded `"postgres"` was passed for every connection. That list is the actual argument for the discipline. Every one of those would have produced a confident, specific, wrong gripe — and a wrong gripe costs more trust than a missing one. If you find one, [report it](https://github.com/datagripe/datagripe/issues). That is a bug, and it is treated as one. ## Related - [The rules](/rules/) — all of them, rendered from the catalogue. - [Gripe levels](/docs/gripe-levels/) — the four-level dial. - [Writing a rule](/docs/writing-a-rule/) — what adding one costs. --- # Writing a rule A rule is a pure function that declares what it needs. The runner gives it exactly that and nothing else, which is what lets the same catalogue run in the browser for the inputs already in memory and on the server for the inputs that need a connection. Adding one means four things, and there is no way to do three of them. ## 1. The rule ```ts interface Rule { /** `.`, lower-kebab. Stable, and a public contract. */ id: string; severity: "blocker" | "warning" | "style"; /** What it needs to decide. The runner only calls rules whose * inputs it can supply. */ inputs: RuleInput[]; /** Returns nothing when it cannot tell. Never a hedge. */ evaluate(context: GripeContext): Finding[]; } ``` The inputs are `statement`, `schema`, `object`, `access`, `execution` and `plan`. A rule declaring `["statement", "schema"]` is never called without both, so it never has to check. A finding carries **facts, not sentences**: ```ts { ruleId: "column.nullable-inequality", severity: "warning", at: documentLocation(statement, start, end), facts: { column: "status" } } ``` `{ rows: 41203882 }`, never `"41M rows"`. This is what makes attitude a presentation layer rather than four parallel analyses — see [gripe levels](/docs/gripe-levels/). ## 2. The entry One line in `packages/gripes/src/catalogue.ts`. That file is the catalogue, and the [rules page](/rules/) on this site is rendered from it — so a rule that is in the code is on the website, and a rule that is not is not. ## 3. Four strings One per attitude level, in `messages.ts`, with no fallback. A test enforces the mechanical half: - all four present, none missing and none inherited; - length caps, so a gripe fits where gripes go; - the barred-term list at every level; - `notice` is profanity-free; - every `{placeholder}` resolves against the facts the rule emits; - the id matches `.`. What no test can check is whether the sentence is any good. These are copy and they are reviewed like copy. The calibration to write against is `join.no-condition`, which is quoted verbatim from the brand specification and is deliberately not paraphrased or improved. A `style` finding never swears. Spending the currency on a redundant index is the exact failure the brand spec warns about. ## 4. Three fixtures One that fires. One that does not. And the one that earns its place: > **a case that looks like the finding and is not.** That third class is not a formality. It is where every false positive in this catalogue would have come from, and each existing rule carries the one that caught it: - `delete.no-where` — `create trigger t after delete on x`, which reads as an unqualified delete unless you read the statement's main verb through a leading `WITH`. Without it the rule fires on every trigger in the database. - `view.select-star` — `select qty * price`, where the `*` sits at exactly the same depth as a real projection star. Depth alone is not enough; the rule has to look at what surrounds it. - `index.duplicate` — a **unique** index whose columns prefix a wider one. It enforces something the wider index does not, so dropping it changes behaviour rather than saving writes. Never flag it. - `routine.volatile-but-readonly` — a `plpgsql` body, which can write through dynamic SQL that no amount of reading will reveal. The rule declines to judge it. - `subquery.not-in` — `NOT EXISTS`, which is the fix, and a written-out list, where a null is visible to whoever reads it. Neither fires. ## The id is a contract A rule id appears in the gripe footer, in dismissal rows, and in whatever anyone greps their logs for. **Renaming one silently un-dismisses it for every user.** So ids are stable, and a rule that changes meaning gets a new id rather than new behaviour under the old one. This is the same discipline the engine applies to the wording, for the same reason: the parts users build habits on do not move. ## Where it runs The split is by input, not by preference: - **The browser** runs rules whose inputs are already in memory — `statement`, the client's `SchemaInput`, and `object` in the object view, where the describe result is the whole input. - **The server** runs rules needing a connection. `execution` and `plan` rules wait on the server-side runner, which is [being written](/roadmap/). The client's `SchemaInput` is honest about the cache behind it: `isNullable` is real, and `rowsFor` and `indexLeadsWith` always answer `null` because the completion catalogue carries neither. They are declared rather than omitted so a rule needing them compiles and stays quiet. ## Related - [Why it complains](/docs/why-it-complains/) — what a rule has to earn. - [The rules](/rules/) — the current catalogue. - [Roadmap](/roadmap/) — rules written down and not built. - [The gripes engine spec](/specs/gripes/) — the full design, including the parts that are not built. --- # Dismissing gripes Every gripe is dismissable, at every attitude level, always. A tool that argues with you about something you have already decided is a tool you switch off, and a switched-off tool catches nothing. ## Three scopes The labels say what they mean rather than what they are called internally: | What you click | Scope | Keyed on | | --- | --- | --- | | **not here** | occurrence | rule id + this statement | | **not in this file** (or **not on this object**) | target | rule id + document or object | | **never in this project** | project | rule id | Dismissing the same thing twice is a no-op rather than an error. ## It is never silent This is the part that matters more than the scopes. A dismissal that quietly made the tool show you less would be a way to make it lie. - The gripes panel keeps a **count of what is hidden**, with a **show all again** beside it. - A dismissed finding leaves the gutter and the annotation rail as well as the panel. Silencing the panel while a squiggle continued to argue with it would be worse than not dismissing at all. - Findings stay in the store either way, so restoring them costs no re-analysis. ## Why an occurrence is not a line number An occurrence dismissal keys on a hash of the statement's normalized text, not on its position in the document. Offsets move the moment anything above them is typed, so an offset-keyed dismissal would evaporate on the next keystroke. Hashing the statement gets the behaviour right in both directions, which is the actual argument for it: - Editing an unrelated statement — or running the formatter over the whole file — **keeps** the dismissal. Nothing about your statement changed. - Editing *this* statement **drops** it. The finding may no longer hold once the text changed, and re-deciding is cheap while a stale suppression is not. ## Where a dismissal lives Server-side, per workspace, so it survives a browser and a machine. It is currently **team-wide**: a schema finding is a team fact, and having four people each dismiss the same missing primary key is four people doing the same work. That is still an open question, so `dismissed_by` is recorded on every row — which is enough to make dismissals personal later without losing the history of who decided what. One wrinkle worth knowing: a local scratchpad's dismissal references a document id the server does not otherwise know about. It is harmless, and it is why a dismissal made in a scratchpad follows that scratchpad rather than appearing against a shared file. ## Before you dismiss it If the gripe is wrong — not unwanted, *wrong* — that is a bug and it is worth more than a dismissal. [Report it](https://github.com/datagripe/datagripe/issues). A wrong gripe costs more trust than a missing one, and the catalogue has a test class for exactly this: every rule carries a fixture that *looks* like the finding and is not, because that is where a false positive comes from. If it is right but you do not care about it in this project, dismiss it at project scope and move on. That is what the scope is for. ## Related - [Why it complains](/docs/why-it-complains/) — correctness over coverage. - [Gripe levels](/docs/gripe-levels/) — turning the volume down instead. - [The rules](/rules/) — what each id means. --- # FAQ ## Is it actually rude? Yes, at the levels you ask for. `warning` is the default and is dry with one beat of personality. `fatal` swears. `panic` shouts and is a joke. `notice` is profanity-free and stays that way — there is a test for it, because `notice` is the level for a shared screen. It is rude about the query. It is never rude about you. See [gripe levels](/docs/gripe-levels/). ## Will it ever be wrong? It should not be, and when it is, that is a bug worth [reporting](https://github.com/datagripe/datagripe/issues) rather than dismissing. A wrong gripe costs more trust than a missing one, so a rule that cannot tell is required to say nothing rather than guess. That choice is why it is sometimes quiet about something you can see. ## Can I turn it off? You can dismiss any finding for one occurrence, one file or object, or the whole project, and the app tells you what it is hiding rather than quietly showing you less. There is no global mute short of not using the feature — that is on the [roadmap](/roadmap/) as `idea · silent-mode` and honestly labelled as an idea. ## Does it send my queries anywhere? No. No telemetry, no analytics, no crash reporting, no phoning home. It answers nothing but the machine it is running on, and that is a [declined](/roadmap/) item rather than an unimplemented one. The personal shape binds to loopback only, because a DataGripe with no accounts has no business answering the network it is plugged into. ## Will it write my query for me? No, and it is not going to. A tool whose whole thesis is that it reads SQL back to you critically cannot also be the thing that wrote it. That one is on the roadmap under `declined`. ## Does it need an account? Not for the personal and desktop shapes — there is no login screen at all. A shared deployment has accounts, with a security key as an alternative to a password; sign-in is usernameless, so there is no email to type and none to leak by asking about. Signup closes after the first account unless you say otherwise. ## Which databases? PostgreSQL, MySQL, SQLite and Redis. Each adapter declares what it can actually do, so the interface reflects SQLite's narrow `ALTER` rather than offering an edit that fails at runtime. The exact matrix is on [adapters](/docs/adapters/). SQL Server is on the roadmap as `unscheduled`, which means nobody is working on it. ## What does 0.0.x mean? It works and it is tested. Nothing in it is promised to stay put. The holes are not hidden: they are on the [roadmap](/roadmap/), written in the same voice the tool uses on your schema, and the declined ones stay there too. ## Is this a DataGrip clone? It is an independent parody, not affiliated with, endorsed by, or connected to any other software vendor. Some bindings are deliberately familiar — `Ctrl+Alt+L` reformats because that is what a decade of muscle memory expects — but the thing it does that nothing else does is the complaining, and that is not a clone of anything. ## Can an agent use it? Two ways. This website is Markdown at every path — append `.md`, or read `/llms.txt` — so documentation needs no scraping. And DataGripe speaks MCP: a project can expose an endpoint an agent connects to, off until an owner turns it on and read-only until they say otherwise, with every call in the query history under the token's name. ## Why is it dark only? Because a light theme has not been drawn, which is an open item rather than a position. It is on the [roadmap](/roadmap/) as `brand · light-theme`, and saying so is cheaper than pretending it was deliberate. ## Something is broken and I cannot tell why Two settings break more deployments than everything else combined, and both are on [configuration](/docs/configuration/): - **`WEB_ORIGIN` must be exactly the origin browsers use.** A mismatch is not a degradation, it is an app that loads and then does nothing, because everything runs over a WebSocket that checks it. - **`NODE_ENV=production` in front of plain HTTP** adds `Secure` to the session cookie, the browser refuses to send it back, and sign-in will not stick. If it is neither, the app says what it refused and why — and [issues](https://github.com/datagripe/datagripe/issues) are open. --- # Glossary Most of these are ordinary words doing a specific job. Where DataGripe uses a term differently from the rest of the industry, that is said. ## Gripes **Gripe** — one finding, rendered. What you see in the gutter or the panel. **Finding** — the analysis behind a gripe: a rule id, a severity, a location, and a set of facts. Carries no prose. A finding is produced once; the sentence is chosen when it is shown. **Rule** — a pure function that declares what inputs it needs and returns findings, or nothing. `.`, lower-kebab: `join.no-condition`. The id is a public contract and does not get renamed. **Severity** — how bad the finding is. `blocker`, `warning`, `style`. Comes from the rule. **Attitude** — how rudely the finding is phrased. `notice`, `warning`, `fatal`, `panic`. Comes from your setting. Independent of severity, and [deliberately so](/docs/gripe-levels/). **Facts** — the named values a rule emits for the wording to interpolate. `{ column: "status" }`, never a pre-formatted sentence. **Dismissal** — hiding a finding at occurrence, target or project scope. Never silent; the panel always says what is hidden. See [dismissing gripes](/docs/dismissing-gripes/). **Barred terms** — the list of words the gripe wording will not use at any level, enforced by a test. It grows with the wording rather than gating it. ## The workspace **Workspace** — a project. Everything is scoped to one: documents, datasources, domains, members, dismissals. Switching workspaces rebinds the socket and rescopes the whole app. **Datasource** — a configured database connection. Its password is encrypted at rest with a key the deployment holds and never comes back to the browser. **Document** — a file in the editor. A **scratchpad** is local to your browser (IndexedDB); a **workspace file** is shared with the workspace and broadcast as it changes. **Predefined connection** — a read-only datasource declared in `connections.json`, with secrets resolved from the environment at boot and held only in memory. Never touches the application database. **Project class** — production, staging, local, analytics. Colour is the safety mechanism: production magenta is the only permanent magenta in the interface, so peripheral vision learns it. ## Engines **Adapter** — the implementation for one engine: PostgreSQL, MySQL, SQLite, Redis. **Capability** — what an adapter declares it can actually do. The interface gates on capabilities and never on adapter ids, which is what makes SQLite's limits visible in the UI instead of arriving as a runtime error. The matrix is on [adapters](/docs/adapters/). **Dialect** — the SQL an adapter speaks, for the tokenizer and the statement splitter. Declared separately from the adapter id because Redis is an adapter with no dialect at all. **Introspection** — reading the catalog. `sql` gives the schema/table/column tree; `keyspace` gives Redis's key browser. ## Surfaces **Object view** — the tabbed structure of one object: columns, arguments, indexes, constraints, triggers, grants, statistics, DDL. One catalog call fills all of them. **Danger zone** — the part of the object view that states what truncate and drop would do. It currently does not execute, on purpose, and is on the [roadmap](/roadmap/). **Table view** — the editable grid over a table's rows. Sort, filter, page, edit cells, insert and delete rows. **Annotation rail** — the marks beside the scrollbar showing where findings are in the whole document, not just the visible window. Capped at forty, worst first, because a rail with two hundred marks is a gradient rather than a map. **Domain** — a named group of objects, by what they are for: billing, identity, the reporting tables nobody owns. `information_schema` cannot tell you which tables are billing; a domain can, and it exports to a directory or a git repository. **Access report** — who can read or write what, across a schema or a domain. Everything in it is *effective* access, with `has_*_privilege` already resolved through PUBLIC, inheritance and superuser. **Untrusted role** — a role you have marked as one, typically the anonymous role behind an API. The `grant.*` rules stay silent until at least one role is marked, because guessing which one it is would be worse than saying nothing. **Repo commands** — commands a repository's own `.datagripe/run.yaml` declares. Has its own switch, separate from the rest of git, because every other git feature reads and writes files and this one executes a program somebody else wrote. The approval is the security boundary; there is no sandbox. ## Deployment **Embedded mode** — DataGripe runs its own PostgreSQL for its own state, generates its own secrets, and has no accounts. The personal and desktop shapes, and the zero-config container. **External mode** — you supply `APP_DATABASE_URL`. Accounts on, the app applies pending migrations at startup. Selected by setting that variable, or forced with `DATABASE_MODE`. **`WEB_ORIGIN`** — the exact origin browsers use. Both the HTTP routes and the WebSocket upgrade compare against it. The single most common reason a deployment loads and then does nothing. **`CONNECTION_ENCRYPTION_KEY`** — encrypts datasource passwords at rest. Not recoverable. Losing it does not sign anyone out; it orphans every stored password. --- # Release notes Every release, newest first. This is `CHANGELOG.md` from the repository, rendered — there is no second copy of it to fall behind. ## 0.0.16 — 2026-09-24 ### Fixed - A repository datasource shared with other pushers no longer wedges its clone. MCP `git_push` fetches and brings the branch up to date before pushing — fast-forward, or a rebase of the clone's unpushed commits onto the remote — so a remote that moved on other files (hand-written migrations next to the export) no longer rejects the push as non-fast-forward. A real conflict aborts the rebase and refuses with git's output, leaving the clone and the remote exactly as they were. Never a merge commit, never a force. - MCP `git_status` fetches first, so `behind` reflects the remote rather than the last fetch. A failed fetch is returned as `fetchError` next to the local status. ### Added - MCP `git_pull`: fetch, then fast-forward or rebase unpushed commits onto the upstream, with the same refusals as `git_push` and the sidebar pull's unsaved-edits guard. Open tabs are told when it moves `HEAD`. The sidebar's own pull is unchanged and stays fast-forward only. ## 0.0.15 — 2026-09-18 ### Fixed - Datasource refresh repopulates grouped domains and removes tags for dropped objects and obsolete routine signatures. Sync reconciles against fresh database catalogs and deletes their old exported files; previews leave tags unchanged. - MCP status errors no longer remove the sidebar launcher. Owners can open the management tab, see the failure and retry. Only an explicit deployment disablement hides the header; unknown status is never shown as off. - The MCP header reads only the original settings columns, so a deployment missing migration 0027 still shows its configured mode and token count. The settings tab explains that pending migrations must be applied. - App startup now checks migration history and applies missing files before serving requests in both database modes. A transaction advisory lock prevents concurrent replicas or manual runners applying a file twice; failed migrations roll back and stop startup. - Updated restart advice and deployment/configuration documentation, published MCP/updates specs and operator guides for automatic startup migrations and MCP status recovery. Existing migration jobs remain valid. ## 0.0.14 — 2026-09-17 ### Fixed - Table and view sync now includes attached trigger definitions in each object’s SQL file and DDL tab, across PostgreSQL, MySQL/MariaDB and SQLite. PostgreSQL trigger enable modes are preserved. ### Added - Login header artwork and its original source image under brand assets. ## 0.0.13 — 2026-09-17 ### Added - MCP domain management, datasource sync preview/apply, and Git status, commit and push tools, with separate project opt-ins. Mutations require read/write mode and editor access. Migration 0027 leaves new controls off; external PostgreSQL deployments must run `bun run db:migrate` before upgrade. - MCP management now opens in a dock tab from its sidebar title. The sidebar keeps the on/off switch and status; the tab manages modes, functionality, tokens and client configuration even while off. - Updated the MCP, security and features site pages, published MCP/workspace specs, roadmap and contributor map for these tools and the title-click gesture. ## 0.0.12 — 2026-09-16 ### Added - Ctrl/Cmd-click toggles individual cells across rows and columns in table and result views. Statistics and copy use only selected cells; Shift-click and dragging still select rectangles. - Documented selection gestures, statistics and disjoint copying in the keyboard reference, features page and table-view spec. Task completion now explicitly checks documentation; release step 1 audits docs against changes since the previous release. ## 0.0.11 — 2026-09-16 ### Added - **A project can be deleted.** It could always be renamed; the way to get rid of one was a database session. Project settings now has a danger zone, gated the way every destructive action in the application is — reveal, type the project's name, execute — and it takes everything DataGripe holds about the project with it: its datasources and their stored credentials, its shared files, its domains and tags, its dismissed gripes, its members, its roles, its MCP tokens. One statement, one cascade, no list in the code to forget a table from. **Nothing on disk is touched.** A repository datasource is a clone somebody else also has; an exported domain is a directory under version control; a datasource path points at work that was never DataGripe's. Deleting a project is DataGripe forgetting it, not a `rm -rf` of somebody's repository. One rule stops it, and it is the same rule in two shapes: **nobody may be left without a project.** Deleting your only one is refused, and so is deleting a shared project that somebody else has no alternative to — remove them from it first. An account with no project cannot open a socket at all, so the alternative to this rule is somebody locked out of the application by a button they did not press. Everybody with the project open is moved out of it when it goes, including the other people in it, and the browser drops what it had cached for it — the dock layout, the shared files, the class. Deleting needs **rename or delete the project**, the capability that already gated renaming. - **You name a file when you make one.** `new` opens a row to type in, prefilled with `query-N.sql` and with the extension left out of the selection, so typing replaces the name and keeps the suffix. The suffix is the whole rule and always has been — `.md` is a runbook, `.sql` is a query — and it now has a third answer: **any other extension opens as plain text** instead of being called SQL, so a `.csv` from a datasource path is a file rather than a syntax error with a thousand findings in it. Renaming works the same way, in the row, and takes effect where you are: renaming `notes.sql` to `notes.md` turns the open tab into a rendered runbook without a reload. It was `window.prompt` before — the browser's dialog in the middle of an application with its own — and the language only followed on the next page load. **A name that is taken counts up rather than being refused**: `notes.md`, `notes-1.md`, `notes-2.md`. Typing a name somebody used is far more often "another one of these" than a mistake, and a dialog saying *that name is taken* is how people end up with `notes2.md`. - **Revert to last save**, in the right-click menu of any file with unsaved changes. The sidebar has always been able to say a document is dirty; it could not offer to undo it, and the way back was closing the tab and answering a confirm about discarding the whole file. - **Cells select as a block, in both grids.** Press one, shift-press another, or drag across them — and in the table view, shift+arrows, because its cells are focusable and the results panel's are not. `Ctrl+C` puts the block on the clipboard as tab-separated rows, which is what a spreadsheet reads; a single cell still copies the full, pretty-printed value the way it always did. Along the bottom, what the selection adds up to: `sum`, `avg`, `min` and `max` for numbers, `distinct` for everything else — `sum 0` over a column of names is a number that means nothing — and `null` when there are any. A string that is *exactly* a number counts as one, because `numeric` and `bigint` arrive as strings from every driver here and a total over a money column is the main case, not an edge one. **It is the cells, not the table.** The bar says `cells 12` before it says anything else: this is arithmetic on what is highlighted, not a second query, and a `sum` that quietly meant the whole table behind a page of 200 rows would be worse than no sum at all. - **The repository section got its space back.** The branch, the number of changed files and `refresh` moved into the section header, where they are legible whether or not the section is open — *which branch am I on* is a question people ask without wanting the file list. What is left inside is one line (`3 changed · 2 staged`) and the buttons, and the ahead/behind counts now ride on the presses that change them: `push 2`, `pull 0`, instead of a pair of arrows above them. **Refresh fetches.** The counts come from `rev-list` against the remote-tracking ref, which is a memory of the last fetch — so the old refresh could only ever repeat what the last pull had learned, and a button saying *0 behind* when there are four commits waiting is worse than no button. It is still a press: nothing fetches on a timer. - **The MCP section says what it is without being opened.** A pill beside the switch: grey *no tokens* while nothing can connect, green *read only* once something can, violet *read/write* when what connects can commit. Inside, the panel lost the paragraph at the top — the pill says it in two words — and gained the two headings that were missing, so it reads *tokens* then *endpoint* with the client config across the bottom. - **One button in the sidebar.** `commit…`, `push`, `create`, `revoke`, `copy client config` and the rest were six hand-written buttons with five different paddings; they are one `Button` with three tones now, so the next improvement to any of them is made once. The forms and the status bar still have their own — that is on the roadmap rather than claimed here. ### Fixed - **Every save of a shared file said the server was ahead of you.** The `document.changed` broadcast went to every socket in the project *including the one that saved*, and it left before the response to that save did — so the saver was told a newer revision existed while their own document was still dirty, which is exactly what a conflict looks like. The banner then offered to take the server's version, which was their own save, to see their own save. A broadcast now skips the socket that caused it. Other tabs of the same session are not that socket and are still told. - **Renaming a project was impossible from the interface.** The name field disabled itself unless the name already differed from the project's, and typing in it was the only way to make it differ — so it was disabled forever, and the only rename anybody could do was over the socket. Whether the field can be typed in is about the role and nothing else, which is what it now asks. ## 0.0.10 — 2026-09-16 ### Added - **Roles are yours now.** A role is a name and a set of capabilities — sixteen of them, from "run queries" to "manage the MCP server" to "restart the server" — edited as a matrix in project settings, and a project can add its own beside the three that ship with it. The question people actually have is narrow and does not nest: *this* person should be able to expose the project over MCP, *that* one runs the sync, support tags domains and touches nothing. A rank could not express any of it. Owner, editor and viewer are still there, seeded with exactly the capabilities those ranks always had, so **an upgrade changes nothing about what anybody can do** — there is a test asserting precisely that. Every action names at most one capability, the dispatcher checks it on every message where the rank check used to be, and taking a capability away reaches open sessions immediately rather than on their next reconnect. Reading stays off the list. Being a member is being able to read, and hiding a table behind a checkbox in an application that hands out SQL prompts would be a promise it cannot keep — that is written down in [the spec](/specs/permissions/) rather than left to be discovered. Two rules will stop you, and both are the old "cannot remove the last owner" in a new shape: somebody must keep *manage members and roles*, and a role somebody still holds cannot be deleted. ### Fixed - **A server that upgraded under an open tab left it there.** Restarting into a new image worked; the page then reloaded, the service worker answered from the build it already had, and the version popup said "this page is running an older build than the server" — at somebody who was refreshing. A reload is not a way out of a stale bundle, and the only one that worked was a hard reload nobody should need to know about. Everything that knows the page is behind now asks for the new service worker, waits for it to install, and hands over to it, falling back to an ordinary reload only where there is no worker to hand over to. The restart does it once the server answers again, the popup's refresh does it on demand, and a version mismatch now asks for a new worker the moment it is noticed rather than on the hourly timer — a server on a different version is proof that a new bundle exists. ## 0.0.9 — 2026-09-15 ### Changed - **The version moved to the bottom left, and says one number.** `● 0.0.8` when there is nothing to do; `● Update Available` when there is. Three different things put a deployment behind — a newer release, a page that has fallen behind its server, a downloaded build waiting to be applied — and it says the same two words for all of them, because the reader's next move is the same: open it. Inside are both versions, the refresh, the restart button where it applies, and the link to the upgrade page. It took the place of the datasource, its namespace and the project class, all three of which the sidebar's breadcrumb and the prompt already say a foot up the same screen. A status bar that repeats the chrome is a status bar nobody reads. The check now runs **once when a project opens** rather than only when somebody presses a button. That reverses a decision made one release ago, and the status bar is why: a badge that lights up only after you press something is a badge nobody sees, and "am I behind" is exactly the question a person does not know to ask. There is still no interval and no background thread, the server holds the answer for ten minutes so a team is one request between them, and `UPDATE_CHECK_DISABLED` removes the whole thing. - **The account menu is about you, not the build.** Project, role, address, both settings panels, the way out — the versions have gone to the other end of the window. - **The chrome scales with the type it carries.** The tab strip was a fixed 29px, so at any scale above 1 it cropped its own labels; it is now that height times the scale. The header is the same story in an installed window, where it was pinned to the height the operating system reserves for the titlebar — a floor now rather than a ceiling. - **The avatar button is the full height of the bar** and wears the ordinary button radius rather than a circle: this is a square-cornered interface, and one circle in it read as an import from another product. ### Added - **A name.** Account settings takes one, and the header button shows it beside the avatar instead of the initials taken from your address. It is stored in this browser like the scale, because it is how this browser addresses you — it does not rename the account, and other members still see the address. ## 0.0.8 — 2026-09-15 ### Added - **One avatar where four controls were.** The role, the settings cog, the signed-in address and the log-out button have become a single Gravatar in the header, and a menu behind it: the project and your role in it, Project settings, Account settings, Log out — and what version this is. The avatar is a hash. Your address never leaves the browser, only a SHA-256 of it does, nothing is requested until you open the menu, and an address with no Gravatar gets initials on a coloured disc rather than a generated face — which is also what you get offline or on an airgapped deployment. - **Versions, and what to do about them.** The menu shows the app's version and the server's, separately, because they come apart: a service worker that has downloaded a build but not applied it leaves a browser running the old bundle against a new server, and the menu now says so instead of leaving it to be diagnosed. **Check for updates** is a button and nothing else. No timer, no startup call, no telemetry — it is the only thing in DataGripe that reaches the internet on its own behalf, and only when pressed. `UPDATE_CHECK_DISABLED` removes it. A check that failed says so: "up to date" on a timed-out request is how somebody misses a security release. What it tells you to do is per shape, because the answer is completely different in each and the person reading it is often not the person who deployed it. The desktop app updates itself; the CLI wants `@latest`; a container wants a pull; a checkout wants `git pull`. The server detects which it is, and the menu names the one that applies. - **In Kubernetes, the menu can just do it.** With `imagePullPolicy: Always` a restart *is* the upgrade, so a workspace owner gets a **restart to apply** button. It is offered only where something is guaranteed to start DataGripe again — Kubernetes by default, and anything that sets `RESTART_TO_UPDATE` — and refused by the server elsewhere rather than merely hidden, because a process nothing will restart must not be able to stop itself. It goes out the way a `SIGTERM` does, embedded PostgreSQL included, it is audited as `app.restart`, and the page waits for the server to come back and reloads onto whatever it is now serving. - **How big the interface is, is yours.** Account settings has a scale slider, 80% to 180%. Every type size in the application is that one number times a brand token, so the tree, the tabs, the status bar, the gripes and the editor all move together — Monaco included, live, with its undo history intact. It is stored in the browser rather than the account, because the reason to turn it up is the screen in front of you and the same account is also open on a phone, and it is applied before the first paint so nothing renders at one size and jumps to another. - **The sidebar's MCP switch is in its header.** The section is titled MCP Server, the toggle sits on its title bar, and the whole section wears a green frame while the server is running — collapsed as well as open. Whether something outside the app can read your project is not a fact that should need a panel opened to see. It costs one new `mcp.status` read, three fields off the settings row; the panel's full state, which walks every datasource path to count files, still waits until somebody opens it. - **The Online section counts who is there** — grey at nought, green when it is not. - **The specifications are published.** `docs/spec` — eighteen documents recording what each subsystem does, what it deliberately does not, and which alternatives were rejected and why — is now `/specs/` on datagripe.com, with a Markdown twin for each and a **Specs** entry in the top nav beside Docs. Pages that used to send you to GitHub for the design now link to the page. Specs cross-reference each other as backticked paths, eighty-eight times across the set, which is a link inside a repository and a dead end on a website; the build turns every one into a real link — to the spec's page where it names a spec, and to GitHub where it does not. Each page carries the spec's own status and phase as a strip, and a note saying it is an engineering document, because publishing a spec does not make it user documentation and a reader should not have to infer that from the tone halfway down. The build now insists every spec has a heading, a status, a phase and a goal, and fails on one that does not; the goal's first sentence is the page's description and its `llms.txt` entry. `docs/adr` is deliberately not published — that is a separate decision and it has not been made. ### Changed - **Configuration is a section of the documentation, not a page.** Fifty environment variables in one wall of tables is a page people search rather than read, and the question somebody arrives with — "how do I turn on Google sign-in" — is now a heading on a short page about accounts and sign-in. Alongside it: database and storage, datasources, limits, files/git/commands, and MCP. **Google sign-in was documented nowhere on the site**, despite being one of the three ways into a deployment. It now has the four steps, the OAuth client type, what the redirect URI must match, and the warning that matters: an empty `GOOGLE_ALLOWED_DOMAINS` with signup open means anyone on the internet can make themselves an account. Fourteen other variables were undocumented too — the two sign-in kill switches, the Google four, both repository-command timeouts, the MCP briefing cap, `EMBEDDED_PG_PORT`, `MIGRATIONS_DIR`, and the three pre-rename names. Thirteen were missing from `.env.example`. All are there now, and **the site build fails when the next one is not**: it reads `envSchema` out of `apps/server/src/config.ts` and checks both directions, so a variable added without documentation, or a page describing a name that was renamed, stops the build rather than shipping. - **Every file the editor can open is in one Files section.** The datasource's own directories, the project's shared files and this browser's scratchpads are now three kinds of root in one tree, each with its own `new` on the row it creates in — and `new` opens that root, because a file appearing in a folder you cannot see is indistinguishable from nothing happening. They were three to five separate sections whose number changed with the datasource. The sidebar's sections are now Files, Repository (when the datasource has one), Online and MCP Server, in that order, and they keep it: every section starts collapsed, and a collapsed one stays where it is in the list instead of docking at the bottom. Opening one used to re-order the sidebar around it, so the shape you learned was never the shape you were looking at. ### Fixed - **An installed window hid the header behind the window controls.** With `window-controls-overlay` the header only padded its left edge, so on Windows and Linux — where the controls are on the right — the account menu and the log-out button sat underneath them, unreachable. It now pads whatever the browser says is reserved on *both* sides, gives the activity bar back the 4px it was overhanging the titlebar strip by, and ellipsises a long address rather than pushing the buttons off the edge. Installed on a phone, the shell now insets for the cutout and the home indicator too. - **The MCP panel gave out an address that only worked on the server.** It printed `http://localhost:$PORT` unless `MCP_PUBLIC_URL` was set, which is the port the process listens on and not the address anybody reaches it at. It now defaults to `WEB_ORIGIN` — the same origin the app is served from, which is where the endpoint actually is — and in development Vite proxies `/mcp` so the printed address answers there too. `MCP_PUBLIC_URL` stays, for a deployment that answers MCP on a different hostname. - **Every page but the landing one was flush against the edge of a phone.** Sections carry `.shell` for their horizontal padding, and `.sec`'s `padding: 64px 0 0` shorthand won on source order and set it to zero. The hero escaped because it sets its own. The documentation layout had the opposite problem and a worse one: it paid 24px of shell padding *and* 26px of cell padding, 13% of a 390px screen gone on each side, and it opened with twenty-one navigation links above the title. Below the two-rail breakpoint the body now comes first, the rails follow it, and the lattice borders come off — a hairline down the edge of the viewport is an artefact, not structure. - **The wordmark is the application's prompt lockup.** `>Datagripe` in the mono face, the chevron in magenta, `Data` in ink and `gripe` in green — the same three colours, from the same `.dg-prompt__*` classes in `tokens.css`, that the project switcher wears in the app. It replaces an icon and a two-colour `datagripe`, and it is set larger than everything else on the row rather than level with it: on a toolbar the prompt is one control among many, but on a website it is the only thing identifying the site. There is no `:project_` tail, because that half of the prompt carries project identity and a website has no project. - **The header was 233px narrower than the page.** It used the 960px shell while the section bar, the content and the footer all used the 1440px one, so at full width the wordmark started a long way in from everything under it. All four now agree. - **Narrow screens get the navigation back.** Below the two-rail breakpoint the documentation and spec rails now fold into the sticky bar under the header: the page's own headings in a row that scrolls sideways, and a **Docs** or **Specs** dropdown pinned to its right holding the whole rail. The dropdown does not move with the headings, so what gets cut off on a page with twenty of them is a heading, never the way out of the page. Both render from one list, so the menu and the sidebar cannot disagree about what exists. It is a `details` element and opens with no JavaScript; the script only adds closing it on an outside click or Escape. - **The documentation index stopped repeating the navigation.** It enumerated all twenty-one pages, which the sidebar, the new dropdown and the footer all already do — four copies on one screen. It now routes by what you are trying to do rather than restating the directory. - **Documentation and spec pages have top padding**, and clear the section bar instead of starting under it. - **The site no longer states the version by hand.** Two pages said `--version 0.0.6` on the day 0.0.7 shipped. The Helm examples are substituted from `package.json` like every other count on the site. ## 0.0.7 — 2026-09-14 ### Added - **Google sign-in, and a switch for every other method.** Setting `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` puts "Continue with Google" on the sign-in screen: OIDC authorization code with PKCE, ending in the same session cookie a password or a security key issues. Identities are matched on Google's `sub` rather than the address, so renaming it at Google does not strand the account, and a first sign-in links to the local account owning a verified address before it considers creating one. `GOOGLE_ALLOWED_DOMAINS` restricts sign-in to your Workspace domains. `PASSWORD_AUTH_DISABLED` and `PASSKEY_AUTH_DISABLED` turn off the other two, which is how a deployment locks itself to Google, or to keys, or to any combination. Off means the routes are absent and the screen does not offer them. Turning off the last way in fails at startup rather than at the login screen. ### Changed - **datagripe.com is rebuilt, and four of its pages are no longer written.** The site said "eleven rules" in three places on a day the catalogue held eighteen, which is the argument for all of this: the rules page now renders from `packages/gripes`, the roadmap page from `roadmap.md`, the release notes from this file, and the adapter capability table from `ADAPTER_CAPABILITIES`. The landing page's counts are substituted at build time. None of it can state a number a human has to remember to change. Twenty-three pages in three groups, where a page's `group` is both its sidebar heading and its footer column — one field with two uses rather than two fields that disagree. The build fails rather than publishes on a broken internal link, a nav entry pointing at nothing, an ungrouped page, an unparseable roadmap line, a duplicate gripe slug, an unsubstituted placeholder, or a rule listed as planned that has already shipped. Visually it is a hairline lattice, a sticky section bar and a three-column documentation layout, with every colour a `var()` from `tokens.css` so the site and the application cannot drift apart on colour or type. - **Every page is also a Markdown file.** Drop the trailing slash, add `.md`: `/docs/faq/` is also `/docs/faq.md`, and the landing page is `/index.md`. Plus `/llms.txt` indexing the whole site with a sentence per page, `/llms-full.txt` for a client that would rather make one request than twenty-three, `/sitemap.xml` and `/robots.txt`. Half the readers of a tool like this arrive as an agent, and an agent should not have to render a page to read it. - **The roadmap is one list now, in the product's own voice.** The unscheduled/parking-lot section is replaced by "Gripes about Datagripe" — twenty-four complaints it has about itself, at five statuses, including the declined ones, which stay on the list because the reason something was refused is more useful than its absence. - **`AGENTS.md`**, which exists mainly to say one thing: a change to behaviour is a change to datagripe.com in the same commit, and here is the table of what to check for whatever you touched. - **The landing page animates, once.** A particle canvas whose readout counts what is actually there and reports its insights and its relevance at zero. It is a joke about the genre and it suspends a real brand rule, so `docs/brand/brand-system.md` "Motion" now argues for it by name rather than being quietly contradicted. It stops drifting under `prefers-reduced-motion`; nothing else on the site moves. ## 0.0.6 — 2026-09-14 ### Added - **DataGripe runs from one command, anywhere.** `bunx @datagripe/cli` — or `npx @datagripe/cli` — is the whole application: the server, the web app it serves, and a PostgreSQL it starts for itself, with no install, no configuration and no account to create. There is a container image too, `ghcr.io/datagripe/datagripe`, which does the same thing with a volume mounted at `/data`. The npm package and the image are the same build. `bun run build:dist` stages a checkout-free distribution — the bundled server, its migrations, the built web app, a launcher — and both are made of that directory, so there is no deployment where the code differs. The distribution reproduces the checkout's directory layout on purpose. `config.ts` derives the repository root from its own location and resolves every relative path against it, so putting the bundle where the source was makes that root the distribution root; the alternative was a package whose defaults for migrations, `.env` and `connections.json` pointed three directories above wherever npm happened to unpack it. The launcher is written for plain node rather than Bun, because that is what `npx` hands it to. It finds a Bun to run the server with — the one you have, or one it installs as an optional dependency — and otherwise stays out of the way. - **`deploy/` for the deployments that are not one person's laptop.** A compose stack with its own PostgreSQL, accounts switched on and the migration as a one-shot service; plain Kubernetes manifests meant to be read top to bottom; and a Helm chart with the shapes a cluster actually comes in — a managed database, the bundled StatefulSet, or the embedded cluster on a volume — plus secrets it generates once and then keeps, since rotating `CONNECTION_ENCRYPTION_KEY` does not sign people out but orphans every datasource password in the database. `WEB_ORIGIN` is the one setting with no default that can be right, and every one of these says so: DataGripe compares it against the browser's `Origin` on every WebSocket upgrade, and everything in the app runs over that socket, so a mismatch is not a degradation but an app that loads and then does nothing. - **`bunx @datagripe/cli personal`** — the shape somebody actually wants when they are trying DataGripe on their own machine: its own database, no accounts, and loopback only, because a DataGripe with no accounts has no business answering the network it is plugged into. `HOST` is the new setting under that; it still defaults to every interface, which is what a container and a shared deployment need. `personal` pins the shape rather than defaulting to it, and that is the reason for the word. Most developers have an `APP_DATABASE_URL` exported in their shell; without `personal` it is taken as an instruction to run a shared deployment, and DataGripe stops asking for two secrets nobody meant to need. Asking for the personal one should not depend on what else is in your environment. - **`datagripe migrate`** as a second entry point, for the deployments that do not migrate themselves. The compose stack runs it before the app starts, the Helm chart as a pre-install hook, the plain manifests as an init container. - **datagripe.com is a documentation site now, not a page.** Six pages rendered from markdown in `site/content/`, with a [keyboard reference](https://datagripe.com/docs/keyboard/) that exists because two of the most useful bindings are written on no button: `Ctrl/Cmd+Enter` runs the statement the caret is in without selecting it, and `Ctrl+Alt+L` reformats. Also [what it can do](https://datagripe.com/docs/features/), which is the first complete list of the surface in one place, a [deployment guide](https://datagripe.com/docs/deploy/) and a [configuration reference](https://datagripe.com/docs/configuration/). The build is one file, `scripts/site/build.ts`, and not a static-site generator: there is no theme to override and the markup it emits is the markup that was already written by hand, which is what keeps the brand intact rather than reskinned. It **fails on a broken internal link**, which is a docs site's characteristic rot and cheaper to catch in CI than from a reader. A release rebuilds it, so the documentation describes the version that just shipped rather than the one before it. - **The Helm chart is a direct link.** It is published as an OCI artifact beside the image it runs, so there is no `helm repo add` and no `index.yaml` to go stale: ```bash helm install datagripe oci://ghcr.io/datagripe/charts/datagripe \ --set webOrigin=https://datagripe.example.com ``` Its version is DataGripe's version — the chart has no lifecycle of its own, and one of the two silently lagging is how people end up installing last month's manifests against this month's image. - **Sign in with a security key instead of a password.** A FIDO2 key — a YubiKey, or a passkey your laptop or phone holds — can create an account and sign into it, and one account may register as many keys as it likes. The account settings tab (click your email in the header) lists them, names them, and adds more. Sign-in is usernameless: no email, no password, insert the key and touch it. That means credentials are registered as discoverable, which costs a slot on a hardware key — a YubiKey 5 holds around 25 — and it buys a sign-in screen with nothing to type and no email addresses to leak by asking about them. The key always asks for its PIN. For an account created with a key and no password there is nothing behind it, so possession of the key alone must not be enough. For the same reason the server refuses to remove an account's last credential. A password account is unaffected, and adding a key to one does not take the password away — this is another way in, not a second factor. Deployment knobs are `WEBAUTHN_RP_ID`, `WEBAUTHN_RP_NAME` and `WEBAUTHN_EXTRA_ORIGINS`; all three have working defaults derived from `WEB_ORIGIN`. See [docs/spec/auth-and-hardening.md](docs/spec/auth-and-hardening.md). ### Changed - **DataGripe lives at `github.com/datagripe/datagripe`.** The image is `ghcr.io/datagripe/datagripe` and the chart is under the same organisation. GitHub redirects the old URLs, so existing clones and installed desktop apps — which ask the releases URL for updates — carry on working. The packages stay in one repository. The chart's `appVersion`, the image tag and the npm version are the same number bumped in one commit and proven by one CI run; split across repositories, the failure mode is a chart that installs an image it was never tested against, and it is silent. See [docs/adr/0003-one-distribution-three-registries.md](docs/adr/0003-one-distribution-three-registries.md). - **Nothing publishes with a long-lived credential.** npm publishes through trusted publishing — the release workflow exchanges its OIDC token for a short-lived npm one, and provenance comes with it rather than being a flag — and GHCR through the job's own `GITHUB_TOKEN`. There is no `NPM_TOKEN` in the repository to leak or rotate. [docs/releasing.md](docs/releasing.md) has the one-time setup. ### Fixed - **A bundled server no longer runs the migration CLI on the way up.** `migrate.ts` guarded it with `import.meta.main`, which is true for every module in a single-file bundle rather than only the entry one — so the packaged server applied migrations against `APP_DATABASE_URL` before starting, and in embedded mode threw the error that explains the CLI is for external databases. It is its own module now. The desktop app's bundled server had the same defect. - **A bundled server no longer reads `connections.json` from outside itself.** The default path counted `..` from the module's own location, which bundling flattens, so the distribution resolved it four directories up and could pick up a file belonging to something else entirely. Both it and the migrations directory now go through `resolveRepoPath`, so there is one definition of where the root is. ## 0.0.5 — 2026-09-12 ### Added - **A datasource can carry `search_path`.** PostgreSQL runtime parameters are sent in the startup packet, so an unqualified name resolves in the schemas the datasource names instead of every query having to say so. `application_name` comes with the same mechanism, and shows up in `pg_stat_activity`. A pasted connection string carrying either now keeps it rather than reporting that it could not. An allowlist rather than free-form name/value pairs, because an unrecognised parameter in the startup packet is a connect-time FATAL rather than a warning — a typed-in name would be a datasource that cannot connect at all, reporting a parameter instead of the field it came from. `statement_timeout`, `client_encoding` and `default_transaction_read_only` are refused by name with the reason. They travel into `.datagripe/config.yaml` on export, so a teammate who clones the repository resolves names the same way. They are not secrets, and the form says so. ## 0.0.4 — 2026-09-10 ### Fixed - **The desktop app can reach its own update manifest.** Every check since the updater shipped failed with `unable to get local issuer certificate`: the bundled runtime does not find the system's certificate authorities on its own, while the system Bun on the same machine fetches the same URL fine. It is pointed at them now, before anything reaches the network — the server inherits it too, and needs it for the same reason. 0.0.2 and 0.0.3 cannot update themselves; 0.0.4 has to be installed by hand, and updates work from there. - **The update dialog no longer says "0.0.3 is available" to someone running 0.0.3.** Updates are compared by build hash, so two builds can share a version; when they do, the dialog names the builds instead. ## 0.0.3 — 2026-09-10 ### Added - **Paste a connection string to create a datasource.** A box above the engine picker on the new-datasource form reads a provider's URL — scheme, host, port, database, user, password, `sslmode` — and fills the fields in for you to check. Parsed in the browser; the string itself is never sent anywhere. Anything it cannot honour is named underneath with the reason, so nothing is dropped in silence — `channel_binding` has no option in this driver, and runtime parameters like `application_name` and `search_path` have nowhere to live until a datasource can carry them. - **`verify-ca` joins the TLS modes**, so a pasted `sslmode` has somewhere to land. libpq's `allow` and `prefer` deliberately do not: measured against a non-TLS PostgreSQL, both hang until the connection timeout because the driver has no negotiated fallback, so a pasted one is raised to `require` and you are told. - **A setting for the blank window on some Linux GPUs.** `disableDmabufRenderer` in `settings.json` beside the data directory, for when WebKit cannot allocate a DMABUF buffer and the app opens as an empty rectangle. Previously fixable only by launching from a terminal with an environment variable, which the desktop icon cannot do. - **[docs/moving-a-datasource.md](docs/moving-a-datasource.md)** — how to export a datasource into git and import it on another machine, and where the password lives instead of in the repository. ### Fixed - **Quitting the desktop app no longer leaves its database running.** Closing the window quits Electrobun natively, without running a Bun exit handler, so the shell never signalled the server it had spawned — the server outlived the app, the embedded PostgreSQL kept its lock on the data directory, and the next launch could not start its own and never opened. The shell now stops the server from `before-quit`, which every quit path passes through. - **A cluster left behind by a crash no longer bricks the app.** The server adopts a postmaster already serving its data directory instead of failing to start beside it, and stops it on the way out — including from an exit handler, which is the only thing that runs when shutdown is cut short. ## 0.0.2 — 2026-09-09 ### Fixed - **The packaged desktop app now runs.** It expected a monorepo checkout beside the install and died at startup looking for `apps/server/src/index.ts`. `apps/desktop/scripts/bundle-server.ts` now stages the backend — the server bundled with `bun build`, its migrations, the built web app and the PostgreSQL binaries — into the bundle, and the shell runs that when there is no checkout to run from. Build packaged desktop apps with `bun run build` in `apps/desktop`; `electrobun build` on its own skips the staging step. - **The launcher no longer shows a broken icon.** The generated `.desktop` entry had no `Icon=` line, because Electrobun's `build.linux.icon` was never set and no icon shipped with the app. - **The desktop app checks for its own updates.** It asks 10 seconds after launch and every six hours after, offers the new version in a dialog, and on acceptance downloads it, stops the server and restarts into it. Electrobun had shipped the updater and the build had been writing an update manifest all along; nothing pointed at a URL, nothing published the manifest, and nothing ever asked. `DATAGRIPE_DISABLE_UPDATES=true` turns it off. ### Changed - **The shipped brand assets live in `brand/`.** The app icon and the painted mascot set had been sitting in `apps/web/public/`, hand-copied into `site/`. `brand/app-icon/icon.svg` is now the drawing everything else comes from: `bun run brand:render` rasterises it into the sizes the platforms ask for, `bun run sync:brand` copies the results into `apps/` and `site/`, and CI fails if a copy drifts. The desktop, the PWA and the landing page now show the same mark, and the placeholder cylinder favicon is retired. ## 0.0.1 — 2026-09-07 First tagged release. Version zero in the honest sense: it works, it is tested, and nothing in it is promised to stay put. ### What is in it A web-based database IDE — Bun, React 19, TypeScript. - **Editor workspace.** Movable tabs and splits (Dockview), one Monaco model per document, drafts and layout recovered from IndexedDB across reloads. - **Connections and explorer.** Encrypted connection storage (AES-GCM, versioned keys), predefined connections from config, lazy schema/table/column introspection. - **Query execution.** Run a selection, the statement at the cursor, or a whole document; streamed bounded results, reliable cancellation, server-enforced row/byte/timeout/concurrency limits, CSV and JSON export. - **Adapters.** PostgreSQL, MySQL, SQLite, and Redis, each declaring what it can actually do through `ADAPTER_CAPABILITIES` — the UI gates on capability flags, never on an adapter id. - **Table view.** Sort, filter, page, edit cells, insert and delete rows, a value panel for large and JSON cells, and transpose. - **Object view.** Seven structure tabs from one catalog call, per-engine DDL, an editable columns tab (add, rename, retype, nullability, default, comment, drop) with `dryRun` so the SQL is reviewed before it runs. - **Multiplayer.** Shared workspace documents, presence, follow mode, shared execution visibility under your own identity, and an audit trail. - **Projects.** Workspace create and switch, local scratchpads separate from shared files, a per-project default connection. - **Gripes.** Eleven static-analysis rules over your SQL and your schema, at four attitude levels, dismissible per occurrence, per target, or per project. Findings are analysis and wording is presentation, so changing the attitude re-renders and never re-analyses. - **Shipping.** Installable PWA, and an Electrobun desktop shell that runs its own embedded PostgreSQL. ### What is not in it - The object view's **danger zone** states consequences but does not execute: truncate and drop are deliberately unimplemented. - Gripes run in the client only. The **server-side runner** on the execution path is designed and not built, so there are no rules about how a query actually turned out. - **Project class and attitude live in `localStorage`**, marked as a mock. Both need to move server-side before they can be trusted, which is what the danger zone is waiting on. - Nothing analyses the text on the **DDL tab**, so a view created with `select *` is flagged in a query file but not on the view itself. ### Trying it `bun install && bun run dev` starts an embedded PostgreSQL and opens without a login. `scripts/demo/` seeds a project whose files and objects trip every gripe rule, alongside the cases that look like findings and are deliberately silent. --- # Specifications 20 specifications, rendered from `docs/spec` in the repository. Each records what a subsystem does, what it deliberately does not, and which alternatives were rejected and why. > Engineering documents, not user documentation. For using DataGripe, > read [the documentation](/docs.md). ## Current - [Access report](/specs/access-report.md) — Answer one question for a whole datasource at once: what can each role actually do? Not what was granted — what is reachable, once PUBLIC, role inheritance, ownership, schema USAGE, SECURITY DEFINER and view ownership have all been resolved. *(phase 13)* - [Database adapters](/specs/adapters.md) — MySQL, SQLite, and Redis sit beside PostgreSQL behind the adapter boundary. *(phase 5)* - [Authentication, authorization, and hardening](/specs/auth-and-hardening.md) — Every request is authenticated and workspace-authorized; credentials and targets are protected by session, CSRF, SSRF, and rate-limit controls that match the self-hosted deployment model. *(phase 4)* - [Connection sources](/specs/connection-sources.md) — Connections are managed like DataGrip data sources: first-class, named, testable objects the user adds and organizes — not a deployment detail. *(phase 2)* - [Connections and schema explorer](/specs/connections.md) — A user can save a PostgreSQL connection with an encrypted password, test it, and lazily browse schemas, tables, views, and columns — without the password ever leaving server memory or appearing in any client payload. *(phase 2)* - [Datasource paths](/specs/datasource-paths.md) — A database does not arrive out of nowhere. *(phase 12)* - [Domains](/specs/domains.md) — A schema is a flat pile of objects; a database is not. *(phase 12)* - [Editor workspace](/specs/editor-workspace.md) — An IDE-grade editor shell in the browser: many SQL documents open at once, tabs that move between groups, horizontal/vertical splits, and no lost changes across tab switches, splits, reloads, or browser crashes. *(phase 1)* - [Git datasources](/specs/git-datasources.md) — docs/spec/datasource-paths.md got DataGripe as far as seeing the checkout: you name a directory, it becomes a sidebar section, files open and save back. *(phase 14)* - [Markdown documents](/specs/markdown-documents.md) — docs/spec/datasource-paths.md shipped with a wart it named out loud: a .md opened out of a datasource path gets SQL highlighting, "and the fix is a language field on the document rather than anything here". *(phase 14)* - [MCP server](/specs/mcp.md) — Every project speaks MCP, so an agent working on the project's code can ask the project's own database questions — and read the project's own explanation of it. *(phase 15)* - [Multiplayer](/specs/multiplayer.md) — DataGripe is multiplayer. *(phase 6 (post-MVP))* - [Object view](/specs/object-view.md) — The ⊞ affordance, or any structure entry in the tree's context menu, opens one object's structure as an editor tab. *(phase 9)* - [Roles and permissions](/specs/permissions.md) — Three ranks answered one question — can this person edit — and the questions people actually have are narrower and do not nest. *(phase 20)* - [Query execution](/specs/query-execution.md) — Run SQL from the editor — a selection, the statement at the cursor, or a whole document — with server-enforced limits, reliable cancellation, live result streaming into a data grid, and an auditable history record for every execution. *(phase 3)* - [Repository commands](/specs/repo-commands.md) — A datasource that comes from a repository can bring its own directories, its own domains and its own sync target. *(phase 14)* - [Table view](/specs/table-view.md) — Double-clicking a table or view in the tree opens its rows as an editor tab and lets you work with them: browse, sort, filter, page, export, inspect and edit a large value, edit a cell, insert a row, delete a row. *(phase 8)* - [Versions, updates and the account menu](/specs/updates.md) — Somebody running DataGripe should be able to answer three questions without leaving it: what am I running, is there anything newer, and what do I do about it. *(phase 19)* - [Workspaces](/specs/workspaces.md) — The workspace IS the project: a named container with its own members, connections, shared files, history, and default target connection. *(phase 7)* ## Draft - [Gripes engine](/specs/gripes.md) — DataGripe notices things about your query and your schema, and says so — accurately, specifically, and rudely. *(phase 11 (planned))* --- # Spec — Access report **Status:** current **Phase:** 13 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/access-report.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** nothing (extends [`docs/spec/object-view.md`](/specs/object-view.md) "grants", [`docs/spec/gripes.md`](/specs/gripes.md), [`docs/spec/domains.md`](/specs/domains.md)) ## Goal Answer one question for a whole datasource at once: **what can each role actually do?** Not what was granted — what is reachable, once `PUBLIC`, role inheritance, ownership, schema `USAGE`, `SECURITY DEFINER` and view ownership have all been resolved. The question is urgent for PostgREST deployments, where the database's grant graph *is* the API surface. A function nobody remembers writing is an endpoint; a table with `SELECT` to `anon` and no row-level security is a public data dump. There is no application tier in between to catch it. ## The gap this closes DataGripe's object-view grants tab and the hand-written `permissions_matrix.sh` scripts it replaces both read `information_schema.role_table_grants`. That view lists **direct grants only**, and it is worse than incomplete: - A `GRANT … TO PUBLIC` appears once, as grantee `PUBLIC`. Scan the column for `anon` and you see nothing, while `anon` has the privilege. - A grant to a role that `anon` is a member of appears against the parent role's name. Same blind spot. - The owner's implicit privileges never appear at all, because an ACL of `NULL` means "defaults apply", not "no access". - It is filtered to *currently enabled roles*, so what it shows depends on who DataGripe connected as. And the functions half of the problem is worse than the tables half: **PostgreSQL grants `EXECUTE` on every new function to `PUBLIC` by default.** A `proacl` of `NULL` is not "no grants", it is "owner has everything and so does everybody else". Under PostgREST that means every function you create is callable by `anon` from the internet until somebody remembers to `REVOKE EXECUTE … FROM PUBLIC`. A report built on direct grants shows that function's row as empty. A report you consult before adding a role has to be right about this or it is worse than no report, because it will be believed. ## Non-goals - Not grant editing. The report reads; `GRANT`/`REVOKE` stay in the editor. (Emitting the statements to fix a finding is on the list — see "Open questions".) - Not a PostgREST configuration reader. DataGripe does not parse `postgrest.conf` or query `pgrst` settings; the roles that matter are named by the user, once, per datasource. - Not RLS policy authoring or simulation. The report shows that a policy exists and what its expression is; it does not evaluate it against a hypothetical row. - PostgreSQL only in the first phase. MySQL grants are a different model and SQLite has none; both report the tab unsupported, the way every other engine-specific surface already does. ## Design ### Effective, direct, and why they differ Every cell in the matrix carries two facts: what a role can do, and where that came from. | Source | Computed from | | --- | --- | | Effective | `has_table_privilege(role, oid, priv)`, `has_function_privilege`, `has_schema_privilege`, `has_column_privilege` | | Direct | The object's ACL — `relacl`, `proacl`, `nspacl` — with the role named explicitly | | via PUBLIC | The ACL has a `PUBLIC` entry (`=` with no grantee) for the privilege | | via role | `pg_has_role(role, grantee, 'USAGE')` for some ACL grantee, with the membership chain named | | owner | The role owns the object and the ACL is silent | | superuser | `rolsuper`, or the privilege is unconditional | The `has_*_privilege` family is the authority: it resolves `PUBLIC`, inheritance and superuser in the server, which is the only place that logic is correct. The ACL parse exists to *explain* the answer, never to produce it. Where the two disagree, effective wins and the difference is the interesting part of the report. A cell is rendered as its effective privilege letters, with a marker when nothing direct accounts for it: ``` | Object | admin | manager | client | anon | RLS | |-----------------------|-------|---------|--------|-------|-----| | public.agg | RCUD | R | R | – | yes | | public.casinos | RCUD | R | R | R° | no | | public.login(text,…) | X | X | X | X° | – | ° reachable without a direct grant — hover for the path ``` Hovering or focusing a `°` cell names the path: `via PUBLIC`, `via member of authenticated`, `owner`. That string is the whole product. "anon can read this" is a fact; "anon can read this because somebody granted it to PUBLIC three years ago" is an action. ### Schema USAGE gates everything A privilege on a table the role cannot reach is not access. The matrix carries a `USAGE` row per schema at the top of that schema's group, and a cell whose schema `USAGE` is missing renders its letters struck through with the effective answer of "no". Reporting `R` for a table in a schema the role cannot enter is the single easiest way to make a security report wrong in the reassuring direction — so it is inverted here: the report never claims access the role does not have, and never hides access it does. ### What else the matrix carries Beside the privilege columns, per relation: - **RLS** — `relrowsecurity` / `relforcerowsecurity` as `no`, `yes`, or `forced`, plus the policy count. - **RLS enabled, zero policies** is called out explicitly. It denies every row to every non-owner, which is usually not what was intended and which fails as an empty result set rather than as an error. - **View execution** — a view without `security_invoker=true` runs as its owner, so it reads base tables with the owner's privileges and the owner's RLS exemptions. A view granted to `anon` over a table `anon` cannot read is a deliberate pattern *and* a common accident; the report marks it and lets a person decide which. - **`SECURITY DEFINER`** — routines are marked, together with whether `proconfig` pins a `search_path`. The gripes engine already complains about the unpinned case per object (`definer-no-search-path`); the report is where you see all of them at once. - **Column grants** — a relation with column-level ACLs shows a `col` marker, because an object-level cell necessarily rounds them off. Expanding the row shows the per-column matrix. ### Default privileges `ALTER DEFAULT PRIVILEGES` decides what the *next* object gets, which is the difference between an audit you do once and an audit you do forever. A separate panel lists `pg_default_acl` entries as "future tables created by `api` in `public` → `SELECT` to `anon`". An entry that grants anything to a role marked untrusted is a finding in its own right, before any object exists to point at. ### Which roles are columns The role set is per datasource, chosen once and stored, because DataGripe cannot know which of forty roles matter. - Candidates are `pg_roles` minus `pg_%` system roles. The picker shows each with `LOGIN`, `SUPERUSER`, `BYPASSRLS`, `NOINHERIT` and its memberships, since all five change how a cell should be read. - A role may be marked **untrusted** — the PostgREST anonymous role. Roles named `anon`, `web_anon`, `anonymous` or `unauthenticated` are *suggested*, never assumed: guessing wrong either floods the report with findings or silences the one role that mattered. - A role may be marked **authenticator**. When one is, the report adds every role that role can `SET ROLE` to, transitively via `pg_auth_members`, because under PostgREST those are exactly the identities a request can arrive as. This set is shown as a small graph above the matrix — for most people it is the first time they have seen it drawn. - `rolsuper` roles render their whole row as `all` in Ink dim rather than as letters. A superuser column of solid `RCUD` teaches the eye to ignore solid rows, which is the opposite of what the report is for. Stored in `datasource_roles` (migration 0011) keyed by `(workspace_id, connection_ref, role_name)` with the two flags and a `shown` boolean. ### Findings, not opinions in the margin The report's judgements are gripes, not a second parallel notion of "warning" ([`docs/spec/gripes.md`](/specs/gripes.md), "Findings are analysis; wording is presentation"). They run in the existing engine, key off the object, get dismissed at occurrence / target / project scope like everything else, and appear in the gripes panel as well as inline in the report. Proposed rule ids — the wording, as always, is the brand pass's: | Rule id | Fires when | | --- | --- | | `grant.public-execute` | A routine is executable by `PUBLIC`, in a schema an untrusted role has `USAGE` on | | `grant.definer-public-reach` | `SECURITY DEFINER` **and** reachable by an untrusted role — the escalation pair, and strictly worse than either alone | | `grant.untrusted-write` | An untrusted role has `INSERT`, `UPDATE` or `DELETE` on a relation | | `grant.untrusted-read-no-rls` | An untrusted role can `SELECT` a table with RLS off | | `grant.rls-no-policy` | RLS on, zero policies — denies everything, silently | | `grant.view-owner-bypass` | A view without `security_invoker` is readable by a role that cannot read its base relation | | `grant.default-privileges-untrusted` | A `pg_default_acl` entry grants to an untrusted role | A proposed eighth, `grant.public-schema-usage`, was **dropped**: "a schema holding objects it has no business reaching" is a judgement DataGripe cannot make, and a rule that cannot tell must say nothing. Schema `USAGE` still gates every cell, so an unreachable object simply reports no access — which is the honest version of the same fact. `grant.public-execute` deserves the loudest severity the catalogue allows, because it is the *default state* of a new function rather than something anybody chose. Every other rule here describes a decision somebody made; this one describes a decision nobody made. Each rule needs its "looks like the finding and is not" fixture, per the gripes testing rule: a function deliberately granted to `PUBLIC` and marked as such, a table with RLS off that no untrusted role can reach, a `SECURITY DEFINER` routine with a pinned `search_path` and no untrusted grant. ### Where it lives A dock tab, `access: `, opened from the breadcrumb overflow or from the sync tab. Not a modal: it is a document you read, scroll, filter and keep open beside a query while you fix things. Clicking an object name opens its object view on the grants tab. The role picker lives at the top of the same tab — flags, memberships, and the three marks — because choosing the columns is part of reading the report rather than a setting somewhere else. - **Filter by domain.** With domains tagged ([`docs/spec/domains.md`](/specs/domains.md)) the report groups by domain, so "what can `anon` reach in `auth`" is one click. Untagged objects group last, as everywhere else. - A **differences only** toggle hides every row where nothing is reachable without a direct grant. On a mature database that toggle is the report. - Filtering by schema is available on the wire (`schemas`) but the tab does not expose a control for it yet; the domain filter and the cell cap cover the cases that made it necessary. - Sorting is stable and the default order is `(schema, name)`, matching the export. ### One query, bounded The whole matrix is one read-only query in one transaction with a statement timeout, like `object.describe`. It is a cross join of candidate objects against selected roles against privilege types, which is cheap per cell and countable in advance: the request refuses above `ACCESS_REPORT_MAX_CELLS` (default 250,000) and asks for a schema or domain filter rather than sitting there. The count is shown before the run when it is close. `access.report` is a `viewer`-role WebSocket action — reading who can do what is not a mutation, and hiding it from viewers would mean the people most likely to notice a mistake cannot look. ### Export The report is part of the domain dump, replacing the two markdown files `permissions_matrix.sh` produces: ``` / access/ matrix.md # roles × objects, effective, with ° markers rls-policies.md # every policy, expression included default-acl.md # ALTER DEFAULT PRIVILEGES entries findings.md # the gripes above, current at export time ``` Written under the same determinism rules as everything else ([`docs/spec/domains.md`](/specs/domains.md), "Determinism is the requirement") — which specifically means **no `Generated: ` line in the body**. The existing scripts write one, so every run dirties both files and the diff that would have told you `anon` gained `UPDATE` last Tuesday is buried under a date change. The timestamp belongs in the run history and in `pull.log`, not in the artifact being diffed. `findings.md` is the one people will read in review, so it is ordered by severity then object, and it names dismissed findings in a separate trailing section rather than omitting them. A dismissal is a decision the repository should record. ### Grants in the per-object DDL Related, and specified here because it is the same catalog read: the per-object `.sql` files in the domain export carry their grants ([`docs/spec/domains.md`](/specs/domains.md), "Export"). A `CREATE FUNCTION` with no grant block does not tell a reviewer whether the function is public, and under PostgREST that is the only thing they needed to know. - Emitted as canonical `GRANT ON TO ;` lines, sorted by `(grantee, privilege)`. - **Grants to `PUBLIC` are emitted explicitly**, including the implicit default `EXECUTE` on routines. Writing out `GRANT EXECUTE ON FUNCTION public.login(text, text) TO PUBLIC;` when nobody typed it is the point: the default becomes visible, and it shows up in the diff the day a new function is added. - When there is genuinely nothing beyond the owner, the file says so in a comment. Absence and silence are different, per the object view's "Three kinds of nothing". ## Testing - A table granted only to `PUBLIC`: every non-superuser role's cell reports effective access with a `via PUBLIC` path, and the naive `role_table_grants` query the current tab uses is asserted to *miss* it — the regression test for this whole spec. - A function with `proacl IS NULL`: `anon` shows `X`, `via PUBLIC`, and `grant.public-execute` fires. - `REVOKE EXECUTE … FROM PUBLIC` on that function: the cell empties and the finding clears. - A role inheriting through two levels of membership: the path names both hops, in order. - `NOINHERIT` membership: no effective privilege, and the report does not claim one. (This is the case a hand-written `pg_has_role` query usually gets wrong in the unsafe direction.) - Schema `USAGE` revoked: table cells strike through and report no access, and no finding fires for an unreachable object. - A view over a table the grantee cannot read, with and without `security_invoker=true`: `grant.view-owner-bypass` fires only for the first. - RLS enabled with zero policies is distinguished from RLS disabled. - Column-level grants: the object cell shows `col` and the expanded row matches `has_column_privilege` per column. - Export determinism: two consecutive exports of `access/` are byte-identical, and no file contains a date. - Cell cap: a request over the cap is refused with the count and a named error, not truncated. ## Open questions - Should the report emit the `REVOKE`/`GRANT` statements that fix a finding? It is a small step from knowing to a reviewed script, and it reuses the `object.alter` dry-run shape exactly. It is also the first time DataGripe would propose a security change, which deserves its own decision rather than arriving as a convenience. - Diffing two reports — "what changed since last Tuesday's export" — is the question people actually ask. Today `git diff` on `matrix.md` answers it, which may be enough, and is an argument for keeping the markdown human-first. - MySQL. The model is different enough (`mysql.user`, `mysql.db`, host-qualified grantees, no roles before 8.0) that it is a separate design rather than a port. - Whether the anonymous-role marking should be inferable from a PostgREST `pre-request`/`db-anon-role` setting when one is discoverable in `pg_settings`. It would remove the one manual step, at the cost of a wrong guess being invisible. ## What is not built - **Rendering the per-column grant expansion** in the tab. The data is in the report payload and in the wire type; the row does not expand yet, so `col` currently means "look at this one by hand". - **A schema filter control** in the tab (the wire field exists). - **MySQL.** `accessReport` is false for every other adapter, and the tab reports itself unsupported rather than guessing. --- # Spec — Database adapters **Status:** current **Phase:** 5 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/adapters.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** nothing (extends [`docs/initial_idea.md`](https://github.com/datagripe/datagripe/blob/main/docs/initial_idea.md) §6; [`docs/spec/query-execution.md`](/specs/query-execution.md) for execution mechanics) ## Goal MySQL, SQLite, and Redis sit beside PostgreSQL behind the adapter boundary. Adapters report honest capability flags; the UI and dispatcher gate on those flags and never branch on adapter ids. ## Non-goals - Redis command execution (the browser is read-only: SCAN + value fetch). - MySQL server-side cursor streaming (does not exist for arbitrary SELECTs; buffered fetch with caps instead). - SQLite cancellation (no driver-level interrupt; local files make runaway queries cheap to bound by caps). - MySQL TLS `verify-full` with custom CAs (same limitation as postgres). ## Capability model `ADAPTER_CAPABILITIES` (`packages/contracts/src/adapters.ts`) is the single source of truth, shared by server and client: | | postgres | mysql | sqlite | redis | | --- | --- | --- | --- | --- | | introspection | sql | sql | sql | keyspace | | execution | cursor | buffered | buffered | – | | cancellation | ✓ | ✓ | – | – | | tableData | readwrite | readwrite | readwrite | – | | object view | ✓ | ✓ | ✓ | – | | routines / sequences | both | routines only | neither | – | | columnChanges | all 7 | all 7 | add, rename, drop | none | | default port | 5432 | 3306 | – | 6379 | | dialog fields | full | full | file path, read-only | host, port, db index, password, tls, read-only | - `introspection: "sql"` → schema/tables/views/columns tree. - `introspection: "keyspace"` → db → prefix → key tree over the `:` delimiter; leaf clicks fetch values via `redis.get`. - `execution: "cursor"` → server-side cursor with FETCH batching. - `execution: "buffered"` → full result fetch, truncated to `QUERY_MAX_ROWS` / `QUERY_MAX_BYTES`. - `execution: null` → `execution.start` is rejected server-side and the Run buttons disable client-side. `cancellation: false` hides Cancel. - `tableData: "readwrite"` → the table view browses rows and writes single-row edits back ([`docs/spec/table-view.md`](/specs/table-view.md)). - `tableData: null` → double-clicking an object has no grid to open; the tab says so instead of rendering an empty one. - The object view ([`docs/spec/object-view.md`](/specs/object-view.md)) is gated on `introspection: "sql"` rather than a flag of its own — an object's structure is exactly what SQL introspection describes. Individual tabs an engine cannot answer are reported `unsupported` per object, so SQLite says it has no permission system instead of showing an empty grants tab. - `columnChanges` lists the column DDL an engine can express. It is a list rather than a boolean because SQLite's `ALTER TABLE` does add, rename and drop and nothing else — the columns tab enables per operation and says why the rest are off. ## Per-adapter behavior ### MySQL - Introspection via `information_schema` (system schemas excluded). - Execution: buffered. Timeout is enforced primarily by `SET SESSION max_execution_time`; a client-side watchdog (KILL at timeout + elapsed check) is the deterministic backstop. - Cancellation: `KILL QUERY ` from a second connection — the same administrative-control-path shape as postgres. - **Driver quirk (verified):** Bun's MySQL driver resolves interrupted statements *cleanly* (`SLEEP` returns 1, no error, empty `SHOW WARNINGS`). Outcomes are therefore tracked out-of-band: `cancelIssued` flag for kills, watchdog for timeouts. - `allowPublicKeyRetrieval: true` only when `tlsMode = "disable"` (caching_sha2_password over plain TCP — acceptable on trusted local links; TLS modes never set it). ### SQLite - `database` carries the **server-side file path**; host/port/username/ TLS are null in metadata. - Schemas = `PRAGMA database_list` (main + attached); objects via `sqlite_master`; columns via `PRAGMA table_info`. - **Driver quirk:** Bun's SQLite adapter does not support `reserve()` — the per-file client is the dedicated connection; sessions wrap it directly. - `readOnly` opens the file read-only at the driver level. ### Redis - A distinct capability, not SQL: PING test, SCAN-bounded keyspace tree (10k keys/level, truncation marker node), `redis.get` for values (string/hash/list/set/zset, TTL, 100-entry cap). - **Driver quirk:** `HGETALL` returns an object in Bun's client, not a flat array. - No execution; `readOnly` is informational (browser is read-only anyway). ## Contract notes - `connectionMetadataSchema` host/port/username/tlsMode are nullable (SQLite). `databaseName` means database name, file path, or DB index per adapter (`databaseLabel` in capabilities). - `schemaNodeKindSchema` gains `db`, `prefix`, `key` — generic tree kinds, not dialect branches. - `workspace.open` includes `adapters: AdapterInfo[]` mirroring `ADAPTER_CAPABILITIES` so clients could consume server-driven capabilities later. ## Open questions - Whether Redis earns command execution later (would be a separate capability flag, not SQL). - Attached-database management UI for SQLite. - Whether MySQL should stream via unbuffered driver internals if Bun ever exposes them. --- # Spec — Authentication, authorization, and hardening **Status:** current **Phase:** 4 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/auth-and-hardening.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** the pre-auth stub notes in [`docs/spec/connections.md`](/specs/connections.md) (implements [`docs/initial_idea.md`](https://github.com/datagripe/datagripe/blob/main/docs/initial_idea.md) §10, §12; ADR 0002) ## Goal Every request is authenticated and workspace-authorized; credentials and targets are protected by session, CSRF, SSRF, and rate-limit controls that match the self-hosted deployment model. ## Non-goals - SAML, and OIDC providers other than Google (later, behind the same session contract — `oauth_identities` already carries a `provider`). - Security keys as a *second* factor on top of a password (they are an alternative to one; nothing forces both). - Attestation checks against the FIDO metadata service — registration asks for `attestationType: "none"`, so a deployment cannot today restrict which makes of key it accepts. - Multi-workspace switching UI (users belong to one default workspace). - Workspace invitation emails (members are added by existing owners). - Connect-time DNS rebinding defense (see SSRF limitations). ## Design ### Accounts and sessions - Local email+password accounts (ADR 0002), unless the deployment turned them off (`PASSWORD_AUTH_DISABLED`). `users.password_hash` uses `Bun.password` (bcrypt). Minimum password length 12. A security-key account has no password at all: `password_hash` is null and the credentials in `webauthn_credentials` are the whole story. - Signup: allowed while zero users exist (bootstrap), or when `ALLOW_SIGNUP=true` (development default; production operators enable it deliberately). The first real user also becomes `owner` of the migrated stub workspace (`Local`), inheriting its connections and history. - Sessions: opaque 32-byte tokens, SHA-256 hashed at rest (`sessions.token_hash`), 14-day expiry, hourly sweep of expired rows. Cookie `dg_session`: `HttpOnly`, `SameSite=Lax`, `Secure` when `NODE_ENV=production`, path `/`. - Logout revokes the session and closes every socket bound to it. ### Security keys (FIDO2 / WebAuthn) A key is an alternative to the password rather than a second factor on top of it, for both signup and sign-in, and one account may register as many keys as it likes — a spare in a drawer is the point. - **Usernameless sign-in.** `POST /api/auth/passkey/login/options` issues a challenge with an empty `allowCredentials`, so the browser offers whatever discoverable credentials it holds; the credential id that comes back names the account on its own, and there is no email to type and none to leak. Registration therefore asks for `residentKey: "required"`. - **User verification is required**, at registration and at every sign-in. On a hardware key that means a PIN. For a key-only account the key alone must not be enough, because there is no password behind it. - **Registration** is one of two things depending on who is asking. Signed in (session + CSRF): the key joins that account, with the account's existing credentials in `excludeCredentials` so the same key cannot be registered twice. Signed out: the request carries the email for a new account, gated by the same bootstrap / `ALLOW_SIGNUP` rule as password signup — checked when the challenge is issued and again when the credential comes back. - **The user handle is the account id.** It is decided before the account exists (a signup ceremony has to hand the authenticator *something*) and carried on the challenge row as `pending_user_id`; the account is created with that id when the credential verifies, or never, if the ceremony is abandoned. At sign-in the handle the authenticator reports must match the credential's owner. - **Challenges** live in `webauthn_challenges`, expire after 5 minutes, and are deleted by the statement that reads them: one challenge, one verification, whether or not that verification succeeds. The verify call finds its challenge by decoding the client data the authenticator signed — there is no session to keep it in. - **Signature counters** are stored and must advance. A counter that fails to move means a cloned authenticator, and the assertion is refused (`@simplewebauthn/server` enforces this). - **Removing the last credential** of an account with no password is refused: it is the only way in. Configuration: `WEBAUTHN_RP_ID` (defaults to `WEB_ORIGIN`'s hostname), `WEBAUTHN_RP_NAME`, `WEBAUTHN_EXTRA_ORIGINS`. Nothing switches the feature on — it is available whenever accounts are (`AUTH_DISABLED=false`) and the browser is in a secure context. | Route | Payload | Result | | --- | --- | --- | | `POST /api/auth/passkey/register/options` | `{ email? }` | creation options | | `POST /api/auth/passkey/register/verify` | `{ response, name? }` | the key, or a session cookie for a signup | | `POST /api/auth/passkey/login/options` | `{}` | request options | | `POST /api/auth/passkey/login/verify` | `{ response }` | session cookie | | `GET /api/auth/passkeys` | – | `{ passkeys, hasPassword }` | | `POST /api/auth/passkeys/rename` | `{ id, name }` | `{ ok }` | | `POST /api/auth/passkeys/delete` | `{ id }` | `{ ok }` | Rename, delete and registration onto a signed-in account require the `x-csrf-token` header; listing is a read and does not. All of them are absent (404) when `AUTH_DISABLED`. ### Google sign-in Optional, and off until a deployment configures an OAuth client: `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` together switch it on, and setting exactly one of them is refused at startup rather than quietly ignored. Like a security key, it is an alternative to the password, not a factor on top of one. - **Authorization code with PKCE.** `GET /api/auth/google/start` issues a state, a PKCE verifier and a nonce, seals all three into the short-lived `dg_oauth` cookie (HMAC over `SESSION_SECRET`, `SameSite=Lax` so it survives the trip back), and redirects to Google with `scope=openid email` — the address is all DataGripe stores. - **The callback** (`GET /api/auth/google/callback`) needs the state it issued, exchanges the code on a backchannel authenticated with the client secret, and checks the id token's `iss`, `aud`, `exp` and `nonce`. The token's signature is deliberately not verified: it came back over TLS from Google's token endpoint rather than through the browser, which is the case OIDC Core §3.1.3.7 exempts. - **Identities are matched on `sub`**, never on the email — people rename their address and the subject does not move. A *first* sign-in links to the existing account owning that address when Google reports `email_verified`, and otherwise creates one under the same bootstrap / `ALLOW_SIGNUP` rule as every other signup. An unverified address is refused outright. - **`GOOGLE_ALLOWED_DOMAINS`** (comma separated) restricts sign-in to Google Workspace domains, matched against the `hd` claim and falling back to the email's domain. Empty means any Google account, which with `ALLOW_SIGNUP=true` means anyone on the internet — a deployment that opens signup to Google should set this. - Failures come back as a redirect to `WEB_ORIGIN` with an `auth_error` query parameter, because a redirect is all a navigation can return; the sign-in screen shows it and scrubs it from the address bar. Configuration: `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI` (defaults to `WEB_ORIGIN` + `/api/auth/google/callback`), `GOOGLE_ALLOWED_DOMAINS`. | Route | Payload | Result | | --- | --- | --- | | `GET /api/auth/google/start` | – | 302 to Google + `dg_oauth` cookie | | `GET /api/auth/google/callback` | `?code&state` | session cookie, or 302 with `auth_error` | ### Roles Superseded by [`docs/spec/permissions.md`](/specs/permissions.md): a role is a project's own name and set of capabilities, and the three ranks are seeded rows with the capabilities they always had. What is still true here is where the check happens — on every message, in the dispatcher, against what the socket resolved at upgrade. ### Which methods a deployment offers Each method is independently switchable, so a server can be locked to one of them: | Variable | Effect | | --- | --- | | `PASSWORD_AUTH_DISABLED=true` | no login/signup routes, no password form | | `PASSKEY_AUTH_DISABLED=true` | every `/api/auth/passkey*` route is absent | | `GOOGLE_CLIENT_ID` + `GOOGLE_CLIENT_SECRET` | Google sign-in appears | Off means **absent** — 404, not a form that refuses. `GET /api/session` reports `passwordAuthEnabled`, `passkeysEnabled` and `googleAuthEnabled` so the sign-in screen only renders what exists. Turning off the last way in fails at startup: a server with accounts and no method to reach them is only discovered at the login screen otherwise. `AUTH_DISABLED=true` is exempt, having no accounts to sign in to. Removing a security key still refuses to remove the *only* way into an account — a linked Google identity now counts as one of those ways, alongside a password and a second key. ### CSRF and origin checks - `GET /api/session` returns `{ user, workspace, role, csrfToken, wsUrl, bootstrap }`. State-changing HTTP routes (login exempt — it has no session yet; logout) require the `x-csrf-token` header to match the session's token. - HTTP and WebSocket requests validate `Origin` against `WEB_ORIGIN`. The WS upgrade additionally requires the session cookie — browser WebSocket APIs cannot attach authorization headers. ### WebSocket authorization Upgrade binds the socket to `{ userId, sessionId, workspaceId, role }` (the user's default workspace: their first `workspace_members` row). Every action handler receives this context and the dispatcher enforces the role matrix on **every** message — socket authentication is not object authorization (initial_idea.md §10). | Action | viewer | editor | owner | | --- | :-: | :-: | :-: | | `workspace.open`, `schema.children`, `history.list`, `execution.subscribe`, `workspace.members` | ✓ | ✓ | ✓ | | `execution.start`, `execution.cancel` | – | ✓ | ✓ | | `connection.create/update/delete/test` | – | ✓ | ✓ | | `workspace.member.add`, `workspace.member.remove` | – | – | ✓ | | `document.*`, `layout.save` (server sync, later) | ✓ | ✓ | ✓ | Violations return `FORBIDDEN`. New actions: | Action | Payload | Result | | --- | --- | --- | | `workspace.members` | `{}` | `[{ userId, email, role, since }]` | | `workspace.member.add` | `{ email, role }` | the added member | | `workspace.member.remove` | `{ userId }` | `{}` (owners cannot remove the last owner) | ### SSRF policy (`security/ssrf.ts`) Before opening any target connection (test, introspection, execution), the hostname is resolved (`node:dns`) and **every** returned address is classified. Blocked by default: loopback, RFC1918, link-local (including 169.254.169.254 cloud metadata), ULA, multicast, unspecified, and reserved ranges. `TARGET_HOST_ALLOWLIST` (comma separated, exact or `*.suffix`) overrides per hostname — development uses `localhost,127.0.0.1,::1`. `SSRF_DISABLED=true` bypasses the policy entirely; intended only for trusted-network deployments, since the server will then connect to any host, including loopback and cloud metadata endpoints. Limitation (accepted): Bun.SQL resolves hostnames itself at connect time, so a DNS answer can change between our check and the driver's connection (rebinding TOCTOU). Full defense requires a connect-time address hook Bun.SQL does not expose; deployments that need it must enforce egress at the network layer. ### Rate limits (`security/rateLimit.ts`) In-memory token buckets, keyed per scope: | Scope | Key | Limit | | --- | --- | --- | | `auth.login` | IP | 30/min | | `auth.login` | email | 5/min | | `auth.passkey` | IP | 60/min | | `connection.test` | user | 10/min | | `execution.start` | user | 30/min | | `schema.children` | user | 120/min | Exhaustion returns `RATE_LIMITED` (HTTP 429 / WS error). ### Audit logging `log.audit(event, fields)` emits structured records for: `auth.signup`, `auth.login.success`, `auth.login.failure`, `auth.logout`, `auth.passkey.register`, `auth.passkey.register.failure`, `auth.passkey.login.failure`, `auth.passkey.remove`, `connection.create/update/delete`, `execution.start/cancel`, `workspace.member.add/remove`, and `ssrf.blocked`. Never passwords, secrets, or result values. Dispatch logs every action's duration and outcome at debug level. ## Open questions - Sliding session renewal vs fixed 14-day expiry (currently fixed). - Per-workspace role selection when users join multiple workspaces. - Lockout vs rate-limit-only for repeated login failures (currently rate-limit-only; audit events enable alerting). - Whether an account should be able to add a password after being created with a key alone (today it cannot: the only route to a password is signup). --- # Spec — Connection sources **Status:** current **Phase:** 2 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/connection-sources.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** nothing (extends [`docs/initial_idea.md`](https://github.com/datagripe/datagripe/blob/main/docs/initial_idea.md) §6, §8) ## Goal Connections are managed like DataGrip data sources: first-class, named, testable objects the user adds and organizes — not a deployment detail. In addition to user-created connections stored in the application database, operators can predefine connections via configuration (environment variables or a config file) so a deployment can ship with ready-made data sources. ## Non-goals - Editing predefined connections in the UI (they are read-only in the UI; change the config and restart/reload). - Per-user overrides of predefined connections (v1: predefined connections are visible to every member of the workspaces listed in their definition). - Hot reload of the config file (restart required; hot reload is an open question below). ## Definitions | Term | Meaning | | --- | --- | | Managed connection | Created in the UI, stored in `connections` + `connection_secrets` | | Predefined connection | Declared in server configuration; materialized read-only | | Git datasource | Read from a repository's `.datagripe/` ([`docs/spec/git-datasources.md`](/specs/git-datasources.md)) | Both kinds implement the same `ConnectionMetadata` contract (`packages/contracts/src/connections.ts`) so the explorer, editor, and execution paths never branch on origin. ## Design ### Configuration format Predefined connections come from one JSON file, path given by `CONNECTIONS_FILE` (default: `connections.json` at the repo root, ignored by git). Environment variables may supply secrets by reference so the file itself stays commit-safe: ```json { "connections": [ { "id": "local-dev", "name": "Local Dev Postgres", "adapter": "postgres", "host": "localhost", "port": 5432, "database": "datagripe", "username": "datagripe", "passwordEnv": "DEV_PG_PASSWORD", "tlsMode": "disable", "params": { "search_path": "sales,public" }, "readOnly": true, "workspaces": ["*"] } ] } ``` - `id` is stable, kebab-case, unique within the file. - Secrets are declared as `passwordEnv: "VAR_NAME"` (indirection into the process environment). Inline `password` is accepted but documented as development-only. - `params` are PostgreSQL runtime parameters sent when the connection opens (`packages/contracts/src/connectionParams.ts`). An allowlist, not free-form: an unrecognised one is a FATAL at connect time rather than a warning, so a typo here would be a datasource that cannot connect at all. Optional, and never a secret. - `workspaces` lists workspace names or `"*"` for all workspaces. - The file is validated at startup with a Zod schema in `packages/contracts` (`predefinedConnectionsFileSchema`); invalid files fail boot with per-entry error messages. ### Server behavior 1. At boot, `apps/server/src/config.ts` loads and validates the file. 2. Predefined connections are exposed through the same `workspace.open` payload and `connection.test` action as managed ones. The response marks them `source: "predefined"` (managed connections report `source: "managed"`); `ConnectionMetadata` gains this field. 3. Secrets for predefined connections never touch the application database: the server resolves `passwordEnv` into memory at boot, decrypt-where-used semantics do not apply, and the value is held only in the connection manager. 4. `connection.create/update/delete` targeting a predefined id returns `FORBIDDEN` with code `CONNECTION_READ_ONLY`. 5. Authorization: a predefined connection is usable only in the workspaces its definition names. ### Schema impact `connections` rows remain managed-only. Predefined connections are **not** inserted into `connections`; `query_executions.connection_id` gains a companion `connection_ref text` (e.g. `predefined:local-dev`) so history and audit survive without a foreign key. The `documents.default_connection_id` column cannot reference predefined ids; documents opened against a predefined connection store that preference client-side (IndexedDB) until a nullable `default_connection_ref` column is added in a later migration — tracked as an open question. `ConnectionMetadata.id` is a plain string (UUID for managed, kebab-case slug for predefined) with a `source: "managed" | "predefined"` field, so clients never branch on id shape. ### Conflict rules - A predefined `id` colliding with another predefined `id` fails boot. - Predefined and managed connections live in disjoint id namespaces (predefined ids are slugs, managed ids are UUIDs), so no runtime collision is possible. ## Client behavior - The explorer renders one list, with a subtle badge on predefined entries. - The connection dialog's save button is disabled for predefined entries; the form renders read-only with a "Defined by server configuration" hint. - Test-connection works identically for both kinds. ## Open questions - Hot reload of `CONNECTIONS_FILE` (SIGHUP or file watch) vs restart-only. - Whether documents should be able to persist `default_connection_ref` server-side (schema change) instead of client-side. - Per-workspace visibility scoping UX once real auth exists (`workspaces` currently matches by name; switch to ids when seeded workspaces exist). --- # Spec — Connections and schema explorer **Status:** current **Phase:** 2 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/connections.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** nothing (implements [`docs/initial_idea.md`](https://github.com/datagripe/datagripe/blob/main/docs/initial_idea.md) §6, §8, §11, §12; extends with [`docs/spec/connection-sources.md`](/specs/connection-sources.md)) ## Goal A user can save a PostgreSQL connection with an encrypted password, test it, and lazily browse schemas, tables, views, and columns — without the password ever leaving server memory or appearing in any client payload. ## Non-goals - Query execution against targets (Phase 3). - Authentication, per-user authorization, SSRF controls (Phase 4). - Connection organization beyond a flat list (folders/groups later). - Documents' default connection binding (Phase 3 with execution). ## Definitions | Term | Meaning | | --- | --- | | Managed connection | Created in the dialog, stored in `connections` + `connection_secrets` | | Predefined connection | Declared in `CONNECTIONS_FILE`; see `connection-sources.md` | | Resolved connection | Connection metadata plus decrypted password, held in server memory only (`ResolvedConnection` in `packages/database-adapters`) | ## Design ### Secret storage - Passwords are encrypted with AES-256-GCM by `apps/server/src/crypto/keyring.ts`. - The key ring derives a 32-byte key per version from deployment material (`CONNECTION_ENCRYPTION_KEY` = version 1) via domain-separated SHA-256. - Ciphertext layout: `iv (12) ‖ ciphertext ‖ tag (16)`, stored as bytea in `connection_secrets` beside `key_version`. Rotation adds a new version; every ring version decrypts, only the highest encrypts. - `ConnectionMetadata` (contracts) never contains a password; the dialog's password field is write-only (empty while editing keeps the stored secret, and `connection.test` by id exercises the stored secret). ### Actions (WebSocket) | Action | Behavior | | --- | --- | | `workspace.open` | Returns the workspace descriptor and all visible connections (predefined + managed), sorted by name | | `connection.create` | Validates, inserts metadata + encrypted secret in one transaction; idempotent via `idempotency_keys` (migration 0002) | | `connection.update` | Partial merge; `password` present → re-encrypt; targeting a predefined id → `CONNECTION_READ_ONLY` | | `connection.delete` | Cascades to the secret row; predefined → `CONNECTION_READ_ONLY` | | `connection.test` | By `connectionId` (managed: decrypt; predefined: in-memory) or by inline `draft` (unsaved dialog fields, never persisted) | | `schema.children` | Lazy introspection; see below | Idempotent mutations store their response in `idempotency_keys` keyed by `(workspace_id, action, key)`; retries replay the stored response. ### Pre-authentication stub There is no auth yet (Phase 4). At boot the server ensures a stub user (`local@datagripe.local`) and a `Local` workspace (`bootstrap.ts`); all rows are workspace-scoped to it. Every query already carries `workspace_id` so the Phase 4 session swap changes resolution, not the data model. ### Introspection The adapter (`packages/database-adapters`, `PostgresAdapter`) walks one tree level per call: ```text [] → schemas (system schemas excluded) [schema] → categories: tables, views [schema, tables] → base tables [schema, views] → views [schema, tables, table] → columns (name, data_type, nullable) [schema, views, view] → columns ``` - Target clients pool per connection fingerprint (host, port, database, username, TLS mode, password fingerprint) so a password change never reuses a stale pooled client. - The connections service caches children per `(connectionId, path)` for 30 s. `schema.children` with `refresh: true` bypasses and rewrites the cache; the explorer's refresh button refetches exactly its expanded paths plus the connection root with `refresh: true` and drops cached children of collapsed nodes. - DDL invalidation is deferred: Phase 3 refreshes affected nodes after successful DDL instead of parsing SQL now (initial_idea.md §11). ### Client - One multiplexed `WsClient` (`apps/web/src/api/ws.ts`): request/response correlation, send queue until open, pending requests rejected on close, 1 s reconnect; every reopen reloads `workspace.open` and resets the explorer. - The explorer tree is a Zustand store, not TanStack Query: refresh must propagate `refresh: true` into the request payload for exactly the expanded paths, which query caches cannot express. Children cache per node key (`connectionId/kind:name/…`) until refresh or reconnect. - The connection dialog (DataGrip-style): name, host, port, database, username, password (write-only), TLS mode, read-only flag, Test, Save. Predefined connections render read-only with a "Defined by server configuration" hint; Test works identically for both kinds. ## Open questions - Retention/cleanup of `idempotency_keys` rows (currently unbounded). - Whether introspection cache TTL should be configurable per deployment. - TLS `verify-full` currently maps to `rejectUnauthorized: true` without custom CA support; deployment CA configuration arrives with Phase 4 hardening. --- # Spec — Datasource paths **Status:** current **Phase:** 12 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/datasource-paths.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** nothing (extends [`docs/spec/connections.md`](/specs/connections.md), [`docs/spec/editor-workspace.md`](/specs/editor-workspace.md), [`docs/spec/multiplayer.md`](/specs/multiplayer.md); shares the host-filesystem gate with [`docs/spec/domains.md`](/specs/domains.md)) ## Goal A database does not arrive out of nowhere. Somewhere there is a checkout: the migrations that built it, the queries the team keeps, the domain dump the sync tab writes. Today DataGripe can see the database and cannot see any of that, so the file you want is in an editor in the other window and the query you paste in here is a copy that drifts. A **datasource path** is a `(name, directory)` pair configured on a datasource. While that datasource is the active one, each pair gets its own section in the left bar, above the workspace files, titled with the name. Inside it is a file tree; clicking a file opens it in the editor. The point is that it is *the file* — not an import of it. Opening one caches it as a workspace document so it gets the live multiplayer state every shared file has, and every save writes the bytes back to the path they came from. Work persists with the project, in the right location. ## Non-goals - Not a file manager. No create, rename, move or delete: the tree reads a directory and opens files out of it. Anything structural belongs to the tool that owns the checkout. - Not a general text editor. It is a SQL editor, so files over 2 MB and files with NUL bytes are refused rather than loaded and mangled on the next save. - Not per user. Paths are workspace-local configuration about a datasource, so everybody in the project sees the same sections — everybody is looking at the same server's disk. The exception is a git datasource, whose repository defines them and whose rows here are a mirror of that file ([`docs/spec/git-datasources.md`](/specs/git-datasources.md)). - Not a watcher. Nothing polls the filesystem; a change on disk is noticed the next time the file is opened (see "When the file moves underneath"). - Not a git client. [`docs/spec/domains.md`](/specs/domains.md) "Committing" and [`docs/spec/git-datasources.md`](/specs/git-datasources.md) own the places DataGripe runs git. ## Where they live `datasource_paths`, keyed by `(workspace_id, connection_ref)` and ordered by `position` (migration 0013). A table rather than a column on `connections`, for the same reason `datasource_export_paths` is one (migration 0012): a predefined connection has no row there, and it must be able to carry paths too. The path is not part of the datasource — it is what this project does with it, which is also why a predefined datasource stays read-only everywhere else on its page and still saves these. They ride on `ConnectionMetadata.paths`, so the sidebar has them the moment the connection list loads and needs no second round trip. ## Configuring them On the datasource edit page, in a `paths` section below `domain export`. Rows of name + directory, add and remove, and a per-row **check** that asks the server whether it can actually reach the directory. Saved by `datasource.set-paths` rather than `connection.update` — same reason as the export path — but on the form's single save, not a button of its own. A form with two save buttons makes someone press both (`docs/brand/mocks/datasource-settings.html` "The two save buttons"). `datasource.set-paths` replaces the whole list: - A row the request keeps is **updated in place**, so a renamed section keeps its id, and with it every file already open from it. The origin of a file-backed document is keyed by path id; a rename that reissued ids would orphan every open tab. - A row the request drops **archives** the documents opened from it rather than deleting them. Removing a pair is a sidebar decision and must not be able to destroy a member's unsaved work. Nothing on disk is touched either way. - An id belonging to a different datasource is refused, so a path id is not a workspace-wide handle that the connection ref merely decorates. - Two rows with the same name are refused: the sidebar could not tell the sections apart. ## What it may read Every host-filesystem path goes through one gate, shared with the domain export (`apps/server/src/domains/paths.ts`): - `HOST_FS_DISABLED` turns export *and* datasource paths off outright. This is what a hosted, multi-tenant deployment sets — there, the person clicking the tree does not own the disk. - `HOST_FS_ROOTS` is an **optional** colon-separated allowlist. Empty — the default — means no allowlist: the directory is already named explicitly per datasource, and making people configure the same thing twice bought nothing. Set it when the host is shared. `DOMAIN_EXPORT_ROOTS` is the pre-rename name and is still honoured. - The configured directory must be **absolute**. With no allowlist a relative path would quietly resolve against the server's working directory. - It is resolved with `realpath` on every request, not once at configuration time, so a symlink swapped in afterwards is caught. Inside a resolved root, the rules are deliberately *not* the export's `safeJoin`. That one guards generated components against a hostile database and allows only `[A-Za-z0-9._-]`; these are names that already exist on the user's own disk, where `0001 init (v2).sql` is an ordinary filename. What is checked instead is the only thing that matters for a path the browser supplied: - the joined path is lexically inside the root — this rejects `../` without touching the disk; - and the `realpath` of the result is still inside it, so a symlink inside the root pointing at `/etc` is refused rather than followed. Listings are capped at 2,000 entries per directory (a `node_modules` the sidebar could not render anyway) and load one directory at a time, on expand. Walking a checkout on open would stall the sidebar for the one file you actually wanted. Reading a file is `viewer`; opening one is `editor`, because opening caches it as a workspace document and saving writes to disk. ## The file as a document `file.open` returns a normal `Document` row that additionally carries an `origin`: `(connectionRef, pathId, filePath)`. That is the whole trick — everything downstream already works. Presence, followed cursors, revision-guarded saves and the conflict banner are the shared-file machinery from [`docs/spec/multiplayer.md`](/specs/multiplayer.md), untouched. - **One live document per file.** A partial unique index on `(workspace_id, origin_path_id, origin_file_path)` where the document is not archived. Two people opening the same file at the same moment land on one row; two caches of one file would mean two people editing two copies with the last save winning silently. - **The row is the cache, the file is the artifact.** `document.save` writes the database row and then the file. In that order, so a refused write leaves the row and the disk both readable rather than the row silently ahead. - **The path is re-resolved on every write**, never remembered from when the file was opened. The pair can be repointed, removed, or have a symlink swapped underneath it while a tab sits open, and none of those may turn into a write somewhere else. - File-backed documents are excluded from the *Workspace files* root: they belong to their path's section, and one file in two lists with two different names for what it is helps nobody. The language *was* `sql` for every document, so a `.md` opened out of a path got SQL highlighting — a wart this spec named and Phase 14 fixed. `documents.language` now follows the file's extension, and a markdown file opens rendered with its `sql` fences runnable in place ([`docs/spec/markdown-documents.md`](/specs/markdown-documents.md)). ## When the file moves underneath `disk_content_hash` records the bytes DataGripe last read from or wrote to the file. On `file.open` there are three cases: 1. **Disk matches the recorded hash.** Nothing moved; return the cached document. 2. **Disk moved, and the cached content still equals the recorded hash.** Only the file changed, so the file wins: its content is adopted as a new revision and broadcast. This is the `git pull` case, and asking about it would be noise. 3. **Both moved.** Neither can be adopted silently, so both are held. The editor shows a banner with the two options, and either answer is a *save* — the one action that makes the row, the file and every other viewer agree again. Nothing watches the filesystem. A file that changes while a tab is open is noticed on the next open, not live; a watcher across every configured root is a different feature with a different cost. ## What the sidebar shows ``` ┌ Files ──────────────────────┐ ← one section, three kinds of root │ ▾ migrations │ ← a path pair, under its own name │ ▾ 2026 │ │ · 0007_domains.sql │ │ · 0008_paths.sql ● │ ← dirty, same dot as the tab │ · README.md │ │ ▸ queries │ ← a second pair │ ▾ Workspace files [new] │ │ · shared.sql │ │ ▸ Scratchpads [new] │ └─────────────────────────────┘ ``` Path roots come first because they are the project's own files; the two below them are DataGripe's. They were three sidebar sections until the number of them started changing with the datasource — a sidebar whose section list re-shapes under you is a sidebar you cannot learn. They are all "files the editor can open", which is one question, so they are one section with three kinds of root: switching datasource now changes what is inside Files rather than which sections exist (docs/spec/editor-workspace.md "Sections"). `new` sits on the root it creates in, and opens that root — a file appearing in a folder you cannot see is indistinguishable from nothing happening. Which roots are open is `dg.sidebar.files`. A directory that will not read shows why, plus the absolute path it tried — "not readable" on its own sends people to the wrong machine. --- # Spec — Domains **Status:** current **Phase:** 12 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/domains.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** nothing (extends [`docs/spec/connections.md`](/specs/connections.md), [`docs/spec/workspaces.md`](/specs/workspaces.md), [`docs/spec/object-view.md`](/specs/object-view.md); the export's access reports are specified in [`docs/spec/access-report.md`](/specs/access-report.md)) ## Goal A schema is a flat pile of objects; a database is not. `public` holds tables that belong to authentication, to aggregation, to reporting, and nothing in the catalog says so. A **domain** is a user-maintained label for that missing dimension: every table, view, routine and sequence can carry one, the sidebar can group and colour by it, and the whole tagging plus the DDL it implies can be exported to a directory tree that lives in the project's git repository. The exported tree is the artifact people already hand-maintain as a shell script — a `declare -A TABLE_DOMAIN` map beside a pile of `pg_dump --table=` invocations, which silently omits every object added since the map was last edited. DataGripe already knows the object list and already produces DDL for each object; the only thing it lacks is the label. Once it has the label, the script is a button. ## Non-goals - Domains are not a security boundary. They do not affect grants or which objects a query can touch. A **hidden** domain affects what the sidebar draws and nothing else: the objects are still there, still queryable, still returned by `schema.children`. Hiding is tidying, not a permission. - Not a replacement for schemas. An object's namespace still comes from the engine; a domain cuts across namespaces. - Not nested. One flat list per datasource (see "Flat, deliberately"). - Not multi-tag. An object has at most one domain, because the sidebar expresses the tag as a single colour rail. - Not migration generation. The export is a snapshot of current structure, not an ordered, runnable rebuild script. See "What the export is not". - No automatic tagging in the first phase. Suggestions are phase 3. - Not the access report. The dump contains one and the sync tab links to it, but role × object grants are [`docs/spec/access-report.md`](/specs/access-report.md). ## Definitions | Term | Meaning | | --- | --- | | Domain | A named, coloured label scoped to one datasource in one workspace | | Tag | The assignment of one object to one domain | | Untagged | An object the datasource reports that carries no tag | | Shelf | A hidden domain: tagged, but deliberately out of the way and out of the export | | Domain root | The allowlisted directory the export writes into | | Dump | The exported tree plus its `domains.yaml`, as committed to git | | Drift | A tag whose object no longer exists, or an object no domain claims | ## Design ### Scope: workspace × datasource A domain belongs to a `(workspace_id, connection_ref)` pair, not to a workspace and not to a connection alone. Two datasources in one workspace — a production database and its analytics replica — deserve their own domain lists even when the schemas rhyme, and the same connection reached from two workspaces is two different projects' opinions about the same database. `connection_ref` is `ConnectionMetadata.id` — the same string the explorer, the object view and the table view already pass as `connectionId`. For a managed connection that is its UUID; for a predefined one it is the bare slug from `CONNECTIONS_FILE` (`local-demo`), **not** the `predefined:` form. Those two shapes both exist in the codebase and it is worth being explicit about which one this is: `workspaces.default_connection_ref` uses the prefixed form, while every per-object action uses the bare id. Domains follow the actions, because a domain is looked up in the same breath as the objects it tags. Deleting a managed connection leaves rows behind — `connection_ref` is text, not a foreign key, since a predefined slug has no row to reference — and a predefined connection disappearing from `CONNECTIONS_FILE` does the same. `domain.list` only ever returns rows for a ref the caller asked about, so orphans are inert rather than wrong. ### Flat, deliberately Hand-built trees drift toward paths — `source/sync/rival`, `infrastructure/util`. Domains stay one level, and a name may contain `-` to say the same thing: `sync-rival`, `infra-util`. The reasons are concrete rather than aesthetic: - The colour rail carries exactly one colour. A child inheriting its parent's colour makes the rail ambiguous; a child overriding it makes the parent's colour a lie. - Grouping the tree by domain adds a level of indentation to a sidebar the brand spec has already cut to three ([`docs/brand/brand-system.md`](https://github.com/datagripe/datagripe/blob/main/docs/brand/brand-system.md) "Sidebar / Structure"). - The export path is `domains//`, one segment, so a rename is a single directory rename in the diff. Nesting is recorded as an open question, not a rejected idea; the tag model does not need to change to add a `parent_id` later. ### Storage Migration `0010_domains.sql`: ```sql CREATE TABLE domains ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), workspace_id uuid NOT NULL REFERENCES workspaces (id) ON DELETE CASCADE, -- Managed UUID or `predefined:`; text because both shapes fit. connection_ref text NOT NULL, -- Lowercase, [a-z0-9-], because it becomes a directory name. name text NOT NULL CHECK (name ~ '^[a-z0-9]([a-z0-9-]{0,38}[a-z0-9])?$'), -- Palette slot, not a hex value: the theme owns the hue. colour smallint NOT NULL CHECK (colour BETWEEN 1 AND 8), description text NOT NULL DEFAULT '', -- Manual ordering in the manager and in the grouped tree. sort_order integer NOT NULL DEFAULT 0, -- Reference/config domains export INSERTs alongside their DDL. include_data boolean NOT NULL DEFAULT false, created_at timestamptz NOT NULL DEFAULT now() ); CREATE UNIQUE INDEX domains_unique_name ON domains (workspace_id, connection_ref, name); ``` Migration `0018_domain_hidden.sql` adds the shelf flag (see "Hidden domains"): ```sql ALTER TABLE domains ADD COLUMN hidden boolean NOT NULL DEFAULT false; ``` ```sql CREATE TABLE domain_tags ( domain_id uuid NOT NULL REFERENCES domains (id) ON DELETE CASCADE, -- Namespace and name exactly as the explorer tree shows them, so a -- PostgreSQL routine name carries its identity arguments and -- overloads tag independently. schema text NOT NULL, name text NOT NULL, kind text NOT NULL CHECK (kind IN ('table', 'view', 'function', 'procedure', 'sequence')), tagged_by uuid NOT NULL REFERENCES users (id) ON DELETE CASCADE, tagged_at timestamptz NOT NULL DEFAULT now(), PRIMARY KEY (domain_id, schema, name, kind) ); -- Finding an object's current tag means looking across every domain of -- the datasource, so the lookup leads with the object, not the domain. CREATE INDEX domain_tags_object ON domain_tags (schema, name, kind); -- Export run history (see "Run history"). Not keyed to `domains`: a run -- covers the datasource, and it must survive every domain in it being -- deleted. CREATE TABLE domain_exports ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), workspace_id uuid NOT NULL REFERENCES workspaces (id) ON DELETE CASCADE, connection_ref text NOT NULL, started_at timestamptz NOT NULL DEFAULT now(), finished_at timestamptz, actor uuid NOT NULL REFERENCES users (id) ON DELETE CASCADE, domain_count integer NOT NULL DEFAULT 0, object_count integer NOT NULL DEFAULT 0, written integer NOT NULL DEFAULT 0, deleted integer NOT NULL DEFAULT 0, refused integer NOT NULL DEFAULT 0, outcome text NOT NULL CHECK (outcome IN ('ok', 'failed', 'refused')), error text, -- Set when the run was committed from the sync tab. commit_sha text ); CREATE INDEX domain_exports_recent ON domain_exports (workspace_id, connection_ref, started_at DESC); ``` The primary key stops an object being tagged twice into the *same* domain. The stronger "at most one domain per object" rule cannot be a constraint on `domain_tags` at all, because the datasource identity lives on `domains`. It is enforced in the service: `domain.tag` deletes any existing tag for the target across every domain of that `(workspace_id, connection_ref)` before inserting, in one transaction. A test asserts the invariant directly against the database rather than through the API. Tags are **team-wide**, like gripe dismissals ([`docs/spec/gripes.md`](/specs/gripes.md) "Dismissal"): a domain is a fact about the schema, not a personal bookmark. `tagged_by` is stored so the decision can be revisited without losing who made it. ### Actions (WebSocket) | Action | Min role | Behaviour | | --- | --- | --- | | `domain.list` | viewer | Domains plus every tag for one `connectionRef` | | `domain.upsert` | editor | Create or update name, colour, description, order, `includeData`, `hidden` | | `domain.delete` | editor | Deletes the domain; its objects become untagged. Response reports the count. | | `domain.tag` | editor | Batch assign: `{ targets: [...], domainId \| null }`. `null` untags. | | `domain.export` | owner | Writes the dump. `dryRun` returns the plan without touching disk. | | `domain.import` | owner | Reads a manifest back and replaces the visible tag set; shelves survive | | `domain.runs` | viewer | Export run history for one `connectionRef` | | `domain.git` | owner | `status` / `commit` / `push` against the domain root | | `domain.set-export-path` | editor | The datasource's export directory. Editor, like the settings it sits beside; *using* it still needs `owner` | `domain.upsert`, `domain.tag` and `domain.export` carry an `idempotencyKey` and replay through `idempotency_keys` like every other mutation. Export runs long — one `object.describe` per tagged object — so it broadcasts `domain.export.progress` over the hub (`{ done, total, current }`) and the client shows a determinate bar. The final response is the plan that was executed. Contracts live in `packages/contracts/src/domains.ts`. ### Sidebar A **group by domain** toggle (`⌗`) sits in the tree's filter row, beside the object filter, with `⋯` for the domain manager next to it. It is a view mode, not a filter: every object stays reachable in both modes, and each mode keeps its own expansion state so toggling does not collapse the tree you were reading. ``` ┌─────────────────────────────┐ │ wallet-prod / public ▾ │ ├─────────────────────────────┤ │ filter objects… ⌗ ◎ ⋯ │ group-by-domain · show-hidden · manage ├─────────────────────────────┤ │▍▾ auth 6 │ domain row, colour rail │▍ ▤ basic_auth.users │ schema-qualified: domains cut across │▍ ▤ basic_auth.sessions │ │▍ ƒ login │ │▍▸ aggregation 23 │ │▍▸ reference 6 │ │▍▸ inbuilt ◎ 184 │ a shelf: dimmed, starts collapsed │ ▸ untagged 4 │ Ink dim, no rail, always last └─────────────────────────────┘ ``` - Object rows inside a domain group are **schema-qualified**, because the point of a domain is that it crosses namespaces. - **`untagged` is always present and always last**, even at zero, and its count is the number that matters: it is the drift the shell script could never show. At zero it renders in Ink faint like any synthetic empty row. - In the normal (schema-first) mode, tagged rows keep their colour rail. This is the mode most people stay in, and the rail is what makes a mis-tagged table visible without switching views. - Grouping needs the full object list for the schema, so switching to grouped mode forces the lazy category loads the tree would otherwise defer. The domain rows render immediately with `loading…` children, per the brand spec's rule that fetching must not look like empty. - Group order is **the project's domains, then the shelves, then `untagged`**. `untagged` is the row people act on, and a fixed final row is easier to find than one that moves as shelves come and go. - The **context menu is the same in both modes.** It is the keyboard path to every edit drag-and-drop offers, and the grouped tree shipped without one — "right click does nothing here" reads as the feature being half-built. `ContextMenu` therefore lives in its own module rather than inside either tree, and the selection both trees share lives in `stores/treeUi.ts`. ### Refresh and database changes The database is the source of truth for object membership. Datasource refresh reloads grouped domains and removes tags for objects absent from fresh database catalogs, including old routine signatures. Domain definitions remain, even when empty; a replacement signature is a new object and starts untagged. Sync performs the same reconciliation without requiring a refresh first, and deletes the old objects' exported files. Preview shows those deletions without changing tags or files. A failed catalog read aborts reconciliation instead of treating the failure as an empty database. ### Sorting The grouped tree is the sorting surface. Rows drag between groups, which is the gesture for the job people actually have: emptying `untagged`. - **The whole group accepts the drop**, header and member rows alike. Dropping onto a row inside `auth` obviously means "put it in auth", and demanding the header would make moving a table a precision test. - Dropping onto `untagged` untags, because "move to no domain" and "move to another domain" are the same edit (see `domain.tag`). - A drag carries the **multi-selection if the dragged row is in it**, and that row alone otherwise. The pointer is the more specific statement; silently moving two hundred still-selected objects is the failure this rule exists to prevent. It is the same rule the context menu follows. - **A drop that changes nothing writes nothing.** Re-tagging an object into the domain it already carries would still be a round trip and a fresh `tagged_by`. - The payload is **validated on drop**, not trusted: a drop can arrive from another tab or another application, and it ends in a write. - Drag is mouse-only by nature. The keyboard path to the same edit is the context menu's `domain` submenu on the focused row, which is why that menu is not optional. ### Hidden domains A domain can be **hidden**. It is still a domain — named, coloured, tagged, listed in the manager — but it is a shelf rather than a part of the project: - Its objects **leave the schema tree**, and the category counts above them count what is shown rather than what was fetched. A category reading 214 above eleven rows is a bug report waiting to be filed. - Its group in the grouped tree **starts collapsed** and is dimmed, with a mark beside the name. It is still a first-class row, because it has to be a drop target — that is the point of it. - The **export skips it entirely**: no `domains//` directory, no manifest entry, and — the reason the filter runs first — no `object.describe` call, which would be a round trip to the database for DDL nobody asked to commit. - It follows that an **import cannot see shelves**, so `domain.import` leaves them in place instead of replacing them. The one shelf that cannot survive is one the incoming file also names: the unique index on `(workspace_id, connection_ref, name)` leaves no third option, and a failed import is worse than an unhidden shelf. - Its objects are **not counted as untagged**, anywhere. That is the whole reason the flag exists. The last point is the argument for the feature. `untagged` is the drift number — the objects nobody has decided about yet — and it only means something if it is small enough to read. A database with two hundred extension functions, engine built-ins and plugin debris permanently sitting in it has no drift number at all. Tagging them into `inbuilt` and hiding it is the decision "these are not ours", recorded once. **A peek, not a mode.** A `show hidden` toggle sits in the filter row, **off by default**, and appears only once something is actually shelved: a toggle for a state that cannot exist is a question with one answer. It is deliberately not persisted — a peek that outlives the session stops being one. While it is on, shelved rows render dimmed and italic rather than as ordinary rows, so the answer is never a lie about where the object lives. **A filter overrides the shelf.** Typing a name and not being shown the object you named is a bug, not a feature, so a filter match surfaces a shelved object (dimmed, as above) whatever the toggle says. **A flag, not a reserved domain.** `inbuilt` and `plugin-generated` can be separate shelves, and either can be unhidden later without re-tagging a thing. Hiding and unhiding never touch `domain_tags`. ### The colour rail A 3px rail at the row's left edge, full row height, using `--dg-border-accent-row`. Rules: - The rail says **domain**, never severity and never project class. The sidebar tree carries no severity marks today (those live on the object view's tab strip, [`docs/spec/gripes.md`](/specs/gripes.md)), and this spec keeps it that way — introducing both to the same surface would make one of them unreadable. - Domain hues therefore may **not** be the four project accents. The eight palette slots need their own tokens (`--dg-domain-1` … `--dg-domain-8`), muted against the brand accents so a magenta-ish domain can never be mistaken for a production connection. - The hex values are **reserved for the brand pass**, the same way the gripe catalogue's wording was. This spec fixes the count (eight), the token names, and the constraint; [`docs/brand/brand-system.md`](https://github.com/datagripe/datagripe/blob/main/docs/brand/brand-system.md) fixes the values. - Colour is never the only carrier: the domain name is on the group row, and the field popover and object-view header both name the domain in text. ### Context menu The existing menu ([`docs/brand/brand-system.md`](https://github.com/datagripe/datagripe/blob/main/docs/brand/brand-system.md) "Context menu") grows one entry above `copy name`: ``` view rows dbl click ────────────────────────── columns … ddl ────────────────────────── domain ▸ ┌──────────────────┐ copy name │▍auth │ ────────────────────────── │▍aggregation ✓ │ danger zone… │▍reference │ │ ─────────────────│ │ untag │ │ new domain… │ └──────────────────┘ ``` - Each submenu item carries its colour rail and a `✓` on the current domain. `untag` appears only when the object is tagged. - Ctrl/Cmd click adds an object to a multi-selection, and with one active the submenu applies to all of it — the header reads `domain (7 objects)` and mixed current domains show no check, because a tick describing one of seven would be a lie. The right-clicked row joins the batch even if it was not selected, because it is what the pointer is on. A plain click clears the selection: leaving it live after you have obviously moved on is how a bulk action hits the wrong two hundred objects. - `new domain…` opens the manager with the name field focused and the pending assignment queued, so creating and tagging is one gesture. ### Domain manager A **tab**, opened from the breadcrumb overflow, the tree header's `⋯`, or `new domain…`. Not a modal: the rest of the app already decided that forms are tabs, so this one survives navigation and can sit beside the tree it describes (`docs/brand/mocks/datasource-selector.html`). One row per domain: colour swatch, name, description, object count, `include data` checkbox, `hidden` checkbox, delete. Deleting shows the object count that is about to become untagged and requires confirmation — a domain with 30 tags is 30 decisions. `hidden` is the shelf switch (see "Hidden domains"), and it sits beside `data` rather than near delete because it destroys nothing: the same objects stay in the same domain. `include data` is disabled on a hidden domain, since a domain that is not exported cannot export data. The untagged count lives in the grouped tree and in the sync tab's scope line rather than here. **Not built yet:** manual reordering (`sort_order` exists and is honoured; nothing sets it but creation order), and the stale-tag report — tags whose object the datasource no longer reports, with a "remove all" action. Stale tags are what a hand-maintained map accumulates silently, so this is the more valuable of the two. ### Export `domain.export` writes a deterministic tree under the workspace's domain root: ``` / domains.yaml domains/ auth/ tables/basic_auth.users.sql tables/basic_auth.sessions.sql routines/public.login__integer-text.sql views/public.v_sessions.sql aggregation/ tables/public.agg.sql sequences/public.agg_id_seq.sql reference/ tables/public.casinos.sql data/public.casinos.sql access/ matrix.md # roles × objects — docs/spec/access-report.md rls-policies.md default-acl.md findings.md pull.log ``` - One file per object. The filename is `..sql`, schema included because `basic_auth.users` and `public.users` are two files. - A PostgreSQL routine's name carries its identity arguments, which are not filesystem-safe. The file is `.__.sql`, where `argslug` lowercases the identity arguments and replaces every run of non-alphanumerics with `-`; over 48 characters it truncates and appends `-` plus the first 8 hex of the argument list's SHA-256. Overloads stay distinct files and stay stable across exports. - Subdirectories are `tables/`, `views/`, `routines/`, `sequences/`, `data/`. Functions and procedures share `routines/`, because the distinction is already in the DDL and a `procedures/` directory that is empty on every engine but MySQL is noise. - File bodies carry the object's DDL, including its attached triggers, then its grants. Triggers stay in their table or view's file and need no separate domain tag. PostgreSQL uses full catalog definitions and preserves disabled, replica and always modes; internal constraint triggers are excluded. MySQL/MariaDB uses SHOW CREATE TRIGGER in firing order; SQLite uses stored trigger SQL. PostgreSQL trigger functions remain independently tagged routines; this does not expand dependencies. The export remains a snapshot, not a dependency-ordered restore script. - The grant block is not decoration: under PostgREST a `CREATE FUNCTION` without one tells a reviewer nothing about whether the function is an endpoint. Grants to `PUBLIC` are written out explicitly, including PostgreSQL's implicit default `EXECUTE` on routines, so the default is visible in the diff. Full rules in [`docs/spec/access-report.md`](/specs/access-report.md), "Grants in the per-object DDL". - The DDL itself is exactly what the object view shows ([`docs/spec/object-view.md`](/specs/object-view.md) "DDL"), one adapter path for every engine — verbatim on MySQL/SQLite, reconstructed on PostgreSQL, and `pg_get_functiondef` verbatim for routines. **No external binary**: no `pg_dump`, no `psql`, no version-skew between the client tools on the server and the database being read. #### Hidden domains are not in it A hidden domain contributes nothing to the dump — no directory, no manifest entry, no `describe` call (see "Hidden domains"). Exporting a datasource whose every domain is hidden fails the same way exporting one with no domains does, with a message that says which of the two it is. #### Determinism is the requirement An export that churns is an export nobody commits. Every rule below exists so that re-running against an unchanged database produces a byte-identical tree and an empty `git diff`: - Domains in name order; objects in `(schema, name, kind)` order. - No timestamps, hostnames, server versions, row counts, sizes or oids in any generated file body. `statistics` is live data and is never exported. - `domains.yaml` is serialised with a fixed key order, two-space indent, block style, **no line folding**, and a trailing newline ([`docs/spec/git-datasources.md`](/specs/git-datasources.md) "domains.yaml"). Folding is the trap: a description that grows by one character would reflow a paragraph. - Every `.sql` file ends with exactly one newline. - Data dumps are ordered by primary key. A table with `includeData` and **no primary key** is refused with a named error rather than exported in whatever order the engine felt like — unstable ordering would put a thousand-line diff in front of you on every pull. - Data dumps are single-row `INSERT` statements with explicit column lists, capped at `DOMAIN_EXPORT_MAX_DATA_ROWS` (default 10,000) per table; exceeding the cap fails that table and reports it, rather than writing a truncated file that looks complete. #### The manifest `domains.yaml` is the round-trip source of truth and the only file the import reads. It replaced `manifest.json` in Phase 14 — one serialisation format for everything DataGripe puts in a repository — and the JSON is not read as a fallback: an existing dump re-exports once, which writes the YAML and prunes the JSON in the same run. A git datasource keeps its copy in `.datagripe/domains.yaml` instead, beside the config that defines it ([`docs/spec/git-datasources.md`](/specs/git-datasources.md)). ```yaml version: 1 connection: ref: predefined:wallet-prod name: wallet-prod engine: postgres domains: - name: auth colour: 1 description: "" includeData: false objects: - schema: basic_auth kind: table name: users - schema: public kind: function name: login(text, text) ``` It records `ref` and `name` so an import can match by reference first and fall back to the human name when the dump came from a different checkout. It records the engine so an import into the wrong engine fails loudly. It contains **no host, no port, no user, no credential of any kind** — the manifest is committed to a repository, and a connection string in git is the accident this project exists to complain about. #### Prune, and dry run first An export owns everything under `domains/` and `access/` plus the two top-level files, and nothing else. Files under those two directories that this export did not generate are deleted, so renaming a domain or untagging a table removes its file instead of leaving a stale one to be read as truth a year later. Because that deletes files in a git working tree, export follows the `object.alter` precedent ([`docs/spec/object-view.md`](/specs/object-view.md) "Editing columns"): **`dryRun` first**. The plan lists writes, unchanged files, deletions and refusals, and the user runs it only after reading it. A plan whose deletion count is non-zero says so at the top. `pull.log` is append-only and is the one file allowed to be nondeterministic: one line per export with timestamp, actor, object count and a written/deleted tally. It is excluded from the prune. #### Where it may write - `HOST_FS_DISABLED` (default off) — turns export off outright, along with every other host-filesystem feature. The sync tab says so rather than offering a button that always fails. - `HOST_FS_ROOTS` — an *optional* colon-separated allowlist of absolute directories, empty by default. **Empty means no allowlist**: the export directory is already named explicitly per datasource, and requiring the same thing twice bought nothing. Set it when the host is shared. `DOMAIN_EXPORT_ROOTS` is the pre-rename name and is still honoured. The gate is shared with the datasource paths the sidebar browses ([`docs/spec/datasource-paths.md`](/specs/datasource-paths.md)). - `datasource_export_paths` holds the chosen directory, keyed by `(workspace_id, connection_ref)` — **per datasource, not per project** (migration 0012). An export never crosses a datasource boundary, so a single path per workspace would have two datasources overwriting each other's tree, silently, because both trees are structurally valid. - It is set on the **datasource edit page**, beside the other per-datasource settings, and saved by `domain.set-export-path` rather than `connection.update`. That matters for predefined connections: `connection.update` refuses them outright, and the export path has to be settable on one. It can be, because the path is not part of the datasource — it is what this project does with it. - A table rather than a column on `connections`, because a predefined connection has no row there. - On every export the server resolves the path with `realpath`. The path must be absolute — with no allowlist a relative one would quietly resolve against the server's working directory — and where an allowlist *is* configured, the result must be a path-segment prefix match against one of its roots, segment-wise, so `/srv/repos-evil` does not pass for `/srv/repos`. Validation happens at export rather than at save, so a directory created later still works. - The resolution happens per export, not once at configuration time, so a symlink swapped in afterwards is caught. - Every generated path is re-checked after joining, and any component that is not `[A-Za-z0-9._-]` fails the export. Object names come from a database that a user may control; a schema called `../../etc` gets a named error, not a write. - Export is `owner`-only. It is the one action in the app that writes to the host filesystem. - Hosted deployments set `HOST_FS_ROOTS` explicitly, or turn host access off with `HOST_FS_DISABLED`. The desktop app (`apps/desktop`) needs neither: the person choosing the directory owns the machine. - `GIT_ENABLED` (default off) and `GIT_TIMEOUT_MS` (default 60,000) gate and bound the commit path — see "Committing". `DOMAIN_EXPORT_GIT` and `DOMAIN_GIT_TIMEOUT_MS` are the pre-rename names and are still honoured. - For a **git datasource** the domain root is not `datasource_export_paths` at all: it is `sync.dir` from the repository's own `.datagripe/sync.yaml`, resolved against the work tree root ([`docs/spec/git-datasources.md`](/specs/git-datasources.md)). One resolver answers for both cases, so an export and the commit that follows it can never be scoped to different directories. ### The sync tab Export is not a modal. It is a dock tab, `sync: `, because what it produces is a run you watch, read the output of, and act on — and because a modal cannot stay open beside the diff it just made. ``` ┌──────────────────────────────────────────────────────────────┐ │ sync: wallet-prod │ ├──────────────────────────────────────────────────────────────┤ │ target ~/repos/falsedynasty/datasource/schema │ │ (set on the datasource's edit page) │ │ git · main · 4 files changed │ │ scope 9 domains · 87 objects · 4 untagged │ ├──────────────────────────────────────────────────────────────┤ │ [ dry run ] [ export ] │ │ │ │ ████████████████░░░░░░░░ 61/87 public.transactions │ ├──────────────────────────────────────────────────────────────┤ │ plan │ │ 82 unchanged │ │ 4 written domains/source/tables/public.players.sql … │ │ 1 deleted domains/media/tables/public.tags.sql │ │ 1 refused public.audit_log — includeData, no primary │ │ key, so row order would churn every export │ ├──────────────────────────────────────────────────────────────┤ │ [ commit… ] [ commit and push… ] │ ├──────────────────────────────────────────────────────────────┤ │ history │ │ 2026-09-07 14:02 rick 87 objects 4 written 1 deleted ✓│ │ 2026-09-01 09:40 rick 85 objects 85 written │ │ 2026-08-24 16:11 sam 85 objects 0 written (no change) │ └──────────────────────────────────────────────────────────────┘ ``` The three sections answer the three questions in order: what am I pointed at, what is about to happen, what happened before. - **Refusals are first-class.** A table skipped because `includeData` is on and it has no primary key gets a line and a reason, never a silent omission. Silent omission is the failure mode of the scripts this replaces. - **Untagged is in the scope line**, not buried in the manager. The number of objects the export is *not* going to write is as important as the number it will. - The `access/` reports ([`docs/spec/access-report.md`](/specs/access-report.md)) are generated as part of the run, and the tab links to the access tab rather than inlining a matrix. ### Run history `domain_exports` (migration 0010), one row per completed run: `workspace_id`, `connection_ref`, `started_at`, `finished_at`, `actor`, `domain_count`, `object_count`, `written`, `deleted`, `refused`, `outcome` (`ok` / `failed` / `refused`), `error`, and the git commit sha when one was made. Server-side rather than parsed back out of `pull.log`, because it is attributable, queryable and survives the working tree being cleaned. `pull.log` still exists and still gets a line: it travels in the repository for people reading the dump without DataGripe, and it is derived from the same row. A run whose export fails partway writes a row with `outcome = failed` and the error. A dry run writes nothing — it did nothing. ### Committing The dump exists to be committed, so the last mile is in the tab. `domain.git` runs the local `git` binary in the resolved domain root. There is no credential management here and none is planned: DataGripe runs git the way you would, with your ambient configuration, and if the push has no credentials the push fails and you get git's own stderr, verbatim, in the tab. That is the correct amount of opinion for this feature to have. What it does have opinions about: - **Off by default.** `GIT_ENABLED` gates it (`DOMAIN_EXPORT_GIT` is the pre-rename name, still honoured); the buttons are absent, not disabled-with-a-tooltip, when it is off. - **`owner` role**, like export. It writes to the host and talks to a remote. - **argv, never a shell.** `Bun.spawn` with an argument array and no shell interpretation. The commit message is one argv element, so a message of `--amend` is a message. - **A fixed verb list**: `rev-parse --show-toplevel`, `status --porcelain`, `add --`, `commit -m`, `push`, and `log` for the sha. No user-supplied flags reach any of them. - **`git add` is scoped to the domain root pathspec.** The repository almost certainly has other work in progress; an export must never stage it. This is the constraint most likely to be dropped during implementation and the one most likely to be noticed by ruining somebody's afternoon. - The domain root must be inside the work tree `rev-parse` reports, or the operation refuses. - `GIT_TERMINAL_PROMPT=0` and an askpass that fails immediately, so a credential prompt is an error instead of a process that hangs until the timeout. `HOME`, `PATH` and `SSH_AUTH_SOCK` pass through, because that is where working credentials live. - Every invocation has a timeout (`GIT_TIMEOUT_MS`, default 60,000) and is killed at it. - stdout and stderr are streamed to the tab verbatim and the exit code is the verdict. No interpretation, no "something went wrong". - Every invocation writes an audit line, and a commit records its sha against the run. **Push is always a separate, explicit press.** It is never bundled into export and never automatic, because it is the only action in this spec that leaves the machine. `commit…` and `commit and push…` both show the `git status --porcelain` of the domain root and the message field before they run anything. ### Import `domain.import` reads `domains.yaml` from the domain root — or from `.datagripe/` for a git datasource — and replaces the datasource's domains and tags with what it finds. It is how a teammate who pulls the repo gets your tagging, and how tagging survives a workspace being recreated. It is a **replace** of the *visible* domains, previewed as a diff — domains added, removed, recoloured, and tags moved — and applied in one transaction. Tags for objects the datasource does not currently report are imported anyway and show up as stale in the manager, because the alternative is silently losing a tag while a migration is mid-flight. Hidden domains are not in the file and are not in the diff either; reporting one as `removed` would be a lie about what the import is about to do. They survive it (see "Hidden domains"). Import does not read the `.sql` files. They are output. ### What the export is not It is a **structure snapshot**, in the same sense the object view's DDL tab is: per-object DDL, no dependency ordering, no `CREATE SCHEMA`, no extension setup. Running the whole tree top to bottom against an empty database will fail on the first forward reference. That is fine — the artifact exists to be read and diffed in code review, which is what the scripts it replaces were used for too. Migration generation stays in the parking lot. ## Testing - Determinism: export twice against a fixture database, assert the two trees are byte-identical, including `domains.yaml` — and that a description long enough to tempt a line-folder still occupies one line. - Path safety: a schema named `../../etc`, a name with a null byte, a domain root that is a symlink into `/tmp`, a root outside the allowlist, and the `/srv/repos-evil` prefix case each get a named error and write nothing. - One-domain invariant: tagging an already-tagged object moves it, and a direct database query finds exactly one row. - Prune: untag an object, re-export, assert its file is gone and nothing outside `domains/` and `access/` was touched. - Round trip: export, delete every domain, import, assert the tag set is identical. - Refusals: `includeData` on a table with no primary key, and a table over the row cap, both fail that object and leave the rest of the export intact. - Drift: a tag whose object was dropped appears as stale and never as a silent omission. - Grouped tree: a domain containing objects from two schemas renders both schema-qualified under one group. - Group order: with a shelf present, the order is domains, shelves, `untagged` — and `untagged` is still last and still present at zero. - Shelving: an object in a hidden domain is absent from the schema tree, is not counted by the category above it, and is not counted as untagged. With `show hidden` on, or with a filter naming it, it comes back marked. - Drag payload: a dragged row inside the multi-selection moves the whole selection; one outside it moves alone. A payload that is not a list of targets is rejected rather than written. - No-op drop: dropping an object onto the domain it already carries issues no `domain.tag` at all. - Export exclusion: a datasource with one visible and one hidden domain exports exactly one directory, and the hidden domain's objects are never described. - Import survival: an import replaces the visible domains and leaves a shelf in place, unless the incoming file names it. - Git scoping: with an unrelated modified file elsewhere in the repository, commit stages and commits only the domain root. - Git argv: a commit message of `--amend` produces a commit with that message and does not amend. - Git refusal: a domain root outside any work tree, and a work tree whose root does not contain the domain root, both refuse and run nothing. - Git failure: a push with no credentials surfaces git's stderr and a non-zero exit, and does not hang waiting for a prompt. - Run history: a failed export records `outcome = failed` with the error; a dry run records nothing. ## What is not built - **Suggestions.** A domain carrying ordered glob patterns (`agg*`, `sync_rival_*`) that propose a domain for untagged objects. When it lands, proposals stay *proposals*: rendered in the untagged bucket in the candidate domain's colour at reduced opacity, with accept and dismiss. Nothing is ever tagged without a person saying so, because a wrong tag that appears by itself is worse than no tag. - **The stale-tag report** in the manager (see above). - **Manual domain reordering.** `sort_order` exists and is honoured; nothing sets it but creation order. Note that dragging *objects* between groups is built (see "Sorting") — it is dragging the *groups* that is not. ## Open questions - Should the gripes panel group and filter by domain? The mapping is free once tags exist — a finding's target is an object — and "aggregation has eleven blockers" is a more useful sentence than a flat list. Deferred so it does not widen Phase 11. - Nesting, if the flat list turns out to be genuinely insufficient in practice: `parent_id` plus a path-shaped export, with the colour rail rules re-decided rather than inherited. - Whether tags should follow a rename. The catalog has no rename event, so `ALTER TABLE … RENAME` currently reads as one stale tag plus one untagged object. Detecting it would mean tagging by oid, which is not portable across engines. - Whether the untagged count belongs in the status bar. It is the number most likely to be acted on, and the status bar already carries the gripe count. Hidden domains make this more attractive, not less: the count is now small enough to be worth a permanent home. - Multi-tag, if a table honestly belongs to two domains. It costs the colour rail its meaning, so it needs a different visual answer first. --- # Spec — Editor workspace **Status:** current **Phase:** 1 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/editor-workspace.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** nothing (implements [`docs/initial_idea.md`](https://github.com/datagripe/datagripe/blob/main/docs/initial_idea.md) §5) ## Goal An IDE-grade editor shell in the browser: many SQL documents open at once, tabs that move between groups, horizontal/vertical splits, and no lost changes across tab switches, splits, reloads, or browser crashes. ## Non-goals - Server-side document persistence. Phase 1 is deliberately client-only: there is no authenticated user or seeded workspace yet (auth lands in Phase 4), so IndexedDB is the source of truth. The `document.*` and `layout.save` WebSocket actions already exist in `packages/contracts/src/ws.ts`; wiring the stores to them is a later phase and must not change the store semantics defined here. - Query execution, result grids, connections, explorer (Phases 2–3). - Multiplayer/shared documents (Phase 6, [`docs/spec/multiplayer.md`](/specs/multiplayer.md)). - Rich tab context menus, drag-and-drop onto the sidebar, popout windows. ## Definitions | Term | Meaning | | --- | --- | | Document | A named SQL file. Domain state lives in the Zustand document store; durable copy in the Dexie `documents` table | | View | One Dockview panel hosting one Monaco editor on a document's model. Many views may share one document | | Model | The Monaco text model for a document (`datagripe://document/{id}.sql`); owns undo history | | Draft | Unsaved content checkpointed to the Dexie `drafts` table after each edit burst | | Layout | Dockview's serialized workspace (panels, groups, splits) in the Dexie `layouts` table | | View state | Cursor, selection, and scroll position per view, in the Dexie `viewStates` table | ## Design ### State ownership The layering from [`docs/initial_idea.md`](https://github.com/datagripe/datagripe/blob/main/docs/initial_idea.md) §5 applies unchanged: - **Zustand document store** is authoritative for domain state: document list, titles, saved vs current content, dirty flags, revisions. - **Monaco** is authoritative for active in-memory editing and undo history. Store updates flow from `onDidChangeModelContent`, never from re-setting editor values. - **IndexedDB (Dexie)** is the crash-recovery and reload layer. The UI never reads it after boot; all writes are fire-and-forget from store subscriptions and editor events. ### Document model ```ts type EditorDocument = { id: string; // uuid title: string; // display name, unique per list, e.g. "query-1.sql" language: "sql" | "markdown" | "plaintext"; // from the name savedContent: string; // content at last explicit save currentContent: string; // live content, may differ from savedContent revision: number; // increments on every save dirty: boolean; // currentContent !== savedContent createdAt: string; // ISO 8601 updatedAt: string; // last edit or save }; ``` `revision` exists so the later server sync can adopt `WHERE id = ? AND revision = ?` guarded saves without a model change. ### Naming a document Creating and renaming both ask for a name **in the row the file occupies**, prefilled with `query-N.sql` for a new one and with the current name for a rename, with the extension left out of the initial selection so typing replaces the stem and keeps the suffix that decides the language. Enter takes it, Escape abandons it, clicking away takes it. This replaced `window.prompt`, which is the browser's dialog in the middle of an application that has its own (roadmap: `ui · browser-chrome-in-an-app`) and could not show what the name was about to become. **A collision suggests rather than refuses.** A name already used in the same list gets `-1`, `-2`, … before its extension (`uniqueName` in contracts, case-insensitively — two files whose names differ only in case are two files nobody can tell apart in a sidebar), and the person can rename afterwards. Typing a name that is taken is far more often "I want another one of these" than a mistake, and a modal saying *that name is taken* makes somebody invent `notes2.md` by hand. Shared files and scratchpads are two lists and collide separately. A rename of a shared file saves: the title is the project's, not this browser's, and saving is how a title (and the language it decides) reaches everybody else. ### Reverting `revertDocument` puts `currentContent` back to `savedContent`, clears `dirty`, cancels the pending draft checkpoint and deletes the draft row. It is in the sidebar's context menu, on dirty documents only, and it asks first — it is the one action here that throws typed content away. It does not write: the saved row and the server revision are untouched, because going back to the last save is not a new save. ### Persistence schema (Dexie, database `datagripe`, version 1) | Table | Key | Row | | --- | --- | --- | | `documents` | `id` | `{ id, title, content, revision, createdAt, updatedAt }` — last saved state | | `drafts` | `id` (= document id) | `{ id, content, baseRevision, updatedAt }` — latest unsaved checkpoint | | `layouts` | `id` (= `"local"`) | `{ id, json, updatedAt }` — `SerializedDockview` | | `viewStates` | `id` (= panel id) | `{ id, documentId, state, updatedAt }` — `monaco.editor.ICodeEditorViewState` | There is exactly one implicit local workspace, so the layout row id is the constant `"local"`. Multi-workspace keying arrives with server sync. ### Write paths - **Edit** → Monaco change event → `updateContent` in the store → debounced (750 ms, per document) upsert into `drafts` with the document's saved `revision` as `baseRevision`. Pending checkpoints flush when the last view of a document unmounts and on `beforeunload` (best effort). - **Save** (Ctrl/Cmd+S or Save button) → write `currentContent` to `documents`, `revision += 1`, then delete the draft row. Order matters: a crash between the two writes leaves a recoverable draft, never a lost save. Ctrl/Cmd+S is handled by one window-level listener (capture phase, so Monaco's own bindings cannot consume shortcuts) routed to the most recent editor view (`lastEditorViewId` in the view store): Monaco text focus (`onDidFocusEditorText`) and Dockview panel activation both update it, so commands keep working while a non-editor panel (e.g. Results) is active. Per-editor `editor.addCommand` bindings are deliberately not used: Monaco command registrations are global, so identical keybindings from multiple editor instances collide and the latest editor shadows the rest. - **Layout change** → Dockview `onDidLayoutChange` → debounced (500 ms) `toJSON()` into `layouts`. - **View state change** (cursor/selection/scroll) → debounced (500 ms) `editor.saveViewState()` into `viewStates`; also written when a view unmounts. ### Boot / recovery 1. Load all `documents` and `drafts` rows. 2. Merge: a draft row wins over its document's saved content (`currentContent = draft.content`, `dirty = true`) whenever the draft is newer than the document row. A draft whose document row was deleted is dropped. This is the "never silently overwrite a newer draft" rule; comparing against a *server* revision joins this merge when sync lands. 3. Hydrate the document store. 4. Load the `layouts` row; validate the shape with Zod and drop panels whose `params.documentId` no longer exists (`sanitizeLayout`). Apply with `fromJSON`. Any failure → start with an empty workspace showing the watermark. A corrupt layout must never block editing. 5. Each restored panel mounts an `EditorView`, which acquires the document model and restores its view state from `viewStates`. ### Monaco model registry One model per document, owned by `createModelRegistry`: - `acquire(document)` returns the shared model, creating it from `currentContent` on first use; a reference count tracks live views. - `release(documentId)` decrements; at zero the registry disposes the model after a microtask delay so StrictMode remounts and Dockview panel moves (which unmount and remount the React component) do not churn model creation or lose undo history. - Disposal first invokes an `onRelease` flush so the final content is checkpointed to `drafts` before the model goes away. - Editors switch documents exclusively with `editor.setModel(model)`; `setValue` on tab switches is forbidden (it destroys undo history and view state). The registry takes its model factory as a parameter, so the reference counting and flush-on-dispose logic is unit-tested without Monaco. ### SQL completion A `CompletionItemProvider` for language `sql` (registered once in `monacoSetup.ts`) provides DataGrip-style schema-aware suggestions: - Catalog cache (`editor/completion/catalog.ts`): per-connection schema → table/view tree fetched via `schema.children`. Schemas and table lists load eagerly per connection (bounded concurrency); columns load lazily per table on first reference (the `schema.children` rate limit forbids bulk column fetch). Completion returns what is cached; Monaco re-queries on the next keystroke as data arrives. - Context (`editor/completion/context.ts`, pure and unit-tested): the statement under the cursor (sql-tools `statementAt`) is scanned for FROM/JOIN/UPDATE/INTO table refs and aliases. After `qualifier.` the qualifier resolves as alias/table (columns) then schema (tables) — falling through when a name matches no known table, so typing `FROM schema.` still completes. FROM/JOIN position suggests schemas, tables, and `schema.table` variants; elsewhere aliases, in-scope columns, tables, then keywords/functions. - Document → connection resolution mirrors the executions store (document pref, then workspace default); model URIs parse as `datagripe://document/.sql` with the id in `uri.path` (`authority === "document"`), via `documentIdFromModelUri`. ### Formatting **Ctrl+Alt+L reformats** — IntelliJ's binding, and DataGrip's. With a selection it formats the selection, without one the whole document. - The layout comes from `formatSql` in `@datagripe/sql-tools`: a token-stream formatter, not a parser, so it lays out a statement that is still half-typed. One clause per line, joins hanging under their FROM, comma lists and AND/OR chains inline while they fit inside 100 columns and one item per line when they do not, parens breaking as `(` … `)` around their contents, and CASE branches one per line. Only whitespace and keyword case change; identifiers, literals and comments are copied verbatim, and comments keep the line they were written on (a trailing `--` comment takes the comma with it, ahead of itself). - Every result is re-tokenized and compared with the input before it is returned. Any difference — a comma dropped from malformed input, an unbalanced paren — returns the original text untouched: a formatter that loses SQL is worse than no formatter. - Wired as Monaco formatting providers (`editor/formatting.ts`, registered from `monacoSetup.ts`), not as a command of our own, so one registration covers the SQL editor, the editor context menu, and Monaco's own Shift+Alt+F, and the table view's JSON value editor gets Ctrl+Alt+L for free from the JSON language service. - Indentation comes from Monaco's own tab settings per model, the dialect from the document's connection — MySQL backticks and Postgres dollar-quotes tokenize differently, and a formatter that guessed would rewrite a literal. - The keybinding is registered globally with `monaco.editor.addKeybindingRules`, two rules discriminated on `editorHasSelection`. Per-editor `editor.addCommand` is not used, for the same reason Ctrl/Cmd+S is not: command registrations are global, so identical per-editor bindings collide across split views. ### Selection - **Middle-click drag makes a columnar, multi-line selection**, as in IntelliJ: one cursor per line, all of them editable at once. Monaco implements this itself, but hands the middle button to the Linux primary clipboard unless `selectionClipboard` is off — so the option is off on every editor we create. A browser tab has no primary clipboard to paste from, so nothing is given up. - Alt+Shift+drag (Monaco's own column-select gesture) and Alt+click multi-cursor keep working; this only adds the mouse button IntelliJ users reach for. ### Views and Dockview - Each panel id is a stable `view-{uuid}`; the document binding travels in panel `params.documentId`, which Dockview serializes into the layout, so view→document bindings survive reload. - Editor panels use `renderer: "always"` so hidden tabs keep their editor instance; tab switches are pure visibility changes. - A custom tab component shows a kind glyph, the title and a dirty dot. Tab close removes the view only; the document and any dirty draft are retained until the user explicitly deletes the document in the sidebar. - **Middle click closes a tab**, same as its `×`. The listener is bound natively on Dockview's own `.dv-tab` element rather than as a React handler on the tab content: Dockview owns the interactive control and the drag listeners, so the whole tab responds including its padding, and no static element has to pretend to be a button. It listens on `mousedown` in the capture phase — `mousedown` so the browser never starts autoscroll, capture so Dockview does not read the press as the start of a tab drag. - Dragging tabs between groups and dropping on group edges creates horizontal/vertical splits — stock Dockview behavior, no custom DnD. - A Zustand view store mirrors Dockview's panel inventory (`viewId → { documentId }`, plus `activeViewId`) for React consumers (sidebar, save-command routing). Dockview remains the source of truth; the store is a read model fed by `onDidAddPanel` / `onDidRemovePanel` / `onDidActivePanelChange`. ### UI structure ```text ┌──────────────────────────────────────────────┐ │ >Datagripe:project_ … (av) │ ├───────────┬──────────────────────────────────┤ │ documents │ Dockview workspace │ │ sidebar │ (editor panels, splits, tabs) │ └───────────┴──────────────────────────────────┘ ``` The header is the prompt on the left (`ProjectPrompt`) and one avatar on the right (`AccountMenu`, docs/spec/updates.md) — role, settings, address and log-out all moved behind it, because the header is also the drag region of an installed window and every permanent control there is width the operating system may take. The status bar holds the version and the gripe count and nothing else: what it used to carry — the datasource, its namespace, the project and its class — the sidebar and the prompt already say, a foot up the same screen. The sidebar lists every document (dirty dot, click to focus or open, double-click to rename, delete to discard). It is plain React, not a Dockview panel — it has no close semantics and must always be reachable. The empty workspace shows a Dockview watermark with a New query button. ### Sections Below the explorer the sidebar is a stack of collapsible sections (`SidebarSections`): **Files**, **Repository** when the datasource has one, and **Online**. **MCP Server** is an owner-only navigation header: its title opens a dock tab, while its switch and status remain in the sidebar. Two rules, both about not moving: - **Every section starts collapsed.** A person opening one is stating a preference; `dg.sidebar.expanded` holds the ids they have opened and nothing else is remembered. (The older pair of keys — a collapse list *and* an expand list, because the default differed per section — is gone, and `dg.sidebar.collapsed` is ignored where it survives.) - **A section stays where it is in the list**, open or shut. Collapsed headers used to dock at the bottom, which meant opening one re-ordered the sidebar around it: the shape you learned was never the shape you were looking at. A section may carry `actions` — controls in its header, beside the toggle rather than inside it, live whether the section is open or shut — and `on`, which frames the whole section in green. Both exist for the MCP switch (docs/spec/mcp.md "The panel"): whether something outside the app can read this project is not a fact that should need a panel opened to see. ### Scale One number, `--dg-scale`, multiplying every type token in `styles/tokens.css`. The slider is in Account settings → Appearance (`SCALE_MIN`–`SCALE_MAX`, in steps of `SCALE_STEP`), and `stores/appearance.ts` owns it. It is **local to the browser** (`dg.appearance.scale`), not an account setting: the reason to turn it up is usually the screen in front of you, and the same account is also open on a phone. `main.tsx` applies it to `` before the first render, so the app never paints at one size and jumps to another. Two pieces of chrome are sized in px rather than type and have to be multiplied explicitly: the tab strip (`--dv-tabs-and-actions-container-height`, which Dockview sizes the row from) and the header, which is content-sized everywhere and `min-height` rather than `height` in an installed window — the reserved titlebar strip is a floor, not a ceiling, or the scale would crop the bar (docs/spec/updates.md). Nothing else in the application reads the scale. A component that sizes itself from it is a component that stops scaling the day the slider's range changes. Monaco is the one exception, because its font size is a JavaScript option rather than CSS: each editor is created at the scaled size and subscribes for changes, rather than being re-created — a font size is not worth an undo history. ## Client–server sync boundary (later phases) The contracts for `workspace.open`, `document.get/create/save/archive`, and `layout.save` are already defined. Sync plugs in at three seams without store-shape changes: 1. Boot merge gains a server-revision comparison in step 2. 2. `saveDocument` additionally sends `document.save` with the idempotency key; `409 Conflict` surfaces in the UI instead of overwriting. 3. Layout save additionally sends `layout.save` with its revision guard. ## Open questions - Draft retention policy for deleted-but-dirty documents (currently: delete removes document, draft, and view states immediately). - Whether the sidebar becomes a Dockview panel once tool windows (explorer, history) arrive in Phase 2–3. - Autosave cadence to the server once sync lands (explicit save only, or conservative background saves per [`docs/initial_idea.md`](https://github.com/datagripe/datagripe/blob/main/docs/initial_idea.md) §5). --- # Spec — Git datasources **Status:** current **Phase:** 14 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/git-datasources.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** the `manifest.json` half of [`docs/spec/domains.md`](/specs/domains.md) "The manifest" (extends [`docs/spec/connection-sources.md`](/specs/connection-sources.md), [`docs/spec/datasource-paths.md`](/specs/datasource-paths.md), [`docs/spec/domains.md`](/specs/domains.md); shares the host-filesystem gate with both) ## Goal [`docs/spec/datasource-paths.md`](/specs/datasource-paths.md) got DataGripe as far as *seeing* the checkout: you name a directory, it becomes a sidebar section, files open and save back. What it cannot do is say where that checkout came from, and it cannot carry its own configuration — the pairs live in a table in somebody's workspace, so the second person on the team sets them up again by hand, from memory, slightly differently. A **git datasource** inverts that. The repository is the definition. It carries a `.datagripe/` directory describing the connection, the directories worth showing, and the domain tagging; DataGripe reads it, and a teammate who clones the repo gets the same datasource, the same sidebar sections and the same domains without configuring anything. The corollary is that DataGripe now has to be honest about git. It runs the system `git` binary with your ambient configuration ([`docs/spec/domains.md`](/specs/domains.md) "Committing" already establishes this and the rules do not change), and it runs it **only when you press something**. Editing a file does not commit it. Saving does not push. The left bar gets a section that tells you what has changed and gives you the three buttons; nothing behind it happens on a timer. ## Non-goals - **Not a git client.** Status, commit, push, pull, fetch. No branch management, no merge conflict resolution, no rebase, no stash, no history browser, no blame. When git needs a decision DataGripe cannot represent, it stops and shows you git's own stderr, and you go to a terminal. The one exception is for callers with no terminal: MCP's `git_pull` and `git_push` rebase *unpushed* commits onto the upstream and abort on any conflict ([`docs/spec/mcp.md`](/specs/mcp.md) "Tools"). - **Nothing automatic.** No autocommit, no autopush, no pull on open, no background fetch. Every invocation is a press. This is the single most important sentence in this spec. - **No credential management.** Unchanged from [`docs/spec/domains.md`](/specs/domains.md): `GIT_TERMINAL_PROMPT=0`, an askpass that fails immediately, and `HOME`/`PATH`/`SSH_AUTH_SOCK` passed through because that is where working credentials live. A push with no credentials fails with git's message, verbatim. - **Not a secret store for the repo.** `.datagripe/config.yaml` is committed. It names an environment variable; it never contains a password. - **Not multi-repo per datasource.** One datasource, one work tree. Two repositories are two datasources. - **Not a submodule-aware anything.** A submodule is a directory to the file tree and invisible to the status list. ## Definitions | Term | Meaning | | --- | --- | | Git datasource | A datasource whose definition is read from a repository's `.datagripe/` directory | | Work tree | The repository root, as `git rev-parse --show-toplevel` reports it | | Clone | DataGripe created the checkout, from a URL, under the repos home | | Adopted | The user pointed at an existing checkout DataGripe did not create | | Sync dir | The directory inside the repo the domain export writes into | A git datasource implements the same `ConnectionMetadata` contract as a managed or predefined one, so the explorer, editor and execution paths never branch on origin — the third value of `source` is the only difference they can see. ## Design ### The `.datagripe` directory At the work tree root, committed, hand-editable: ``` .datagripe/ config.yaml connection identity, fields, branding, paths sync.yaml where the domain export writes, and how (optional) run.yaml commands DataGripe can run (optional) domains.yaml domains, colours and tagged objects (generated) ``` `run.yaml` is the one file here DataGripe *executes* rather than interprets, and it has its own gate, its own approval and its own document: [`docs/spec/repo-commands.md`](/specs/repo-commands.md). Three files rather than one, and the split is by feature rather than by size. A config that is "really just connection info and branding" stays readable in a diff and stays stable; a domain dump churns every time somebody tags a table, and burying the connection under three hundred lines of object list means every tagging change shows up as a change to the file that defines the database. New features get new files here rather than new top-level keys in `config.yaml`. Unknown top-level keys are **preserved on write and ignored on read**, so an older DataGripe does not silently delete a newer one's settings when it rewrites the file. Unknown keys inside a block DataGripe owns are an error, because that is almost always a typo. Every file starts with `version: 1`. A file with a version DataGripe does not know refuses the whole datasource rather than reading the parts it recognises — a half-understood connection definition points at the wrong database. #### `config.yaml` ```yaml version: 1 datasource: name: wallet-prod adapter: postgres host: db.internal port: 5432 database: wallet username: reader passwordEnv: WALLET_PG_PASSWORD tlsMode: verify-full # Optional. PostgreSQL runtime parameters, sent when the connection # opens, so a teammate who clones this resolves unqualified names the # same way you do. An allowlist — `search_path` and `application_name` — # because an unrecognised one is a FATAL at connect time rather than a # warning. Never a secret; this file is committed. params: search_path: wallet,public readOnly: true showAllSchemas: false branding: # Optional. Colours the datasource breadcrumb and the tab accents so # prod does not look like staging. One of the eight palette slots # domains already use (docs/spec/domains.md "The colour rail"). colour: 3 # Free text under the name in the datasource selector. description: Production wallet database. Read-only credentials. paths: - name: Some Folder path: foo - name: bar path: bar ``` - `datasource` is `predefinedConnectionSchema` minus `id` and `workspaces`, which are meaningless here: the repository *is* the identity, and visibility is whoever can see the checkout. - `paths` is the same `(name, directory)` pair as [`docs/spec/datasource-paths.md`](/specs/datasource-paths.md), with one difference that matters: **the path is relative to the work tree root**. `foo` and `./foo` are the same directory. An absolute path is refused, and so is any path that climbs out of the root — checked lexically and again with `realpath`, by the same `resolveInside` the file tree already uses. A repository that could name `/etc` in a committed file would be a remote code path with extra steps. - Two paths with the same `name` are refused for the reason they are refused in the form: the sidebar could not tell the sections apart. #### `sync.yaml` Optional. Absent means this datasource has no export target configured yet, which is a normal state — the button on the sync tab writes this file when you pick one. ```yaml version: 1 sync: # Relative to the work tree root, like every path in this directory. dir: domains includeAccessReports: true maxDataRows: 10000 ``` `sync.dir` replaces `datasource_export_paths` for a git datasource. It is deliberately *not* in `config.yaml`, even though the first sketch of this feature put it there: it is the one setting here that a second person might reasonably want set differently, and it belongs with the feature that reads it. #### `domains.yaml` Generated by the export and read by the import. It **replaces `manifest.json`** ([`docs/spec/domains.md`](/specs/domains.md) "The manifest") for every datasource, not only git ones — one serialisation format for the `.datagripe` directory is worth more than backwards compatibility with a file that has existed for one phase. It lives in **two places, by necessity**. A git datasource keeps it in `.datagripe/domains.yaml`, beside the config that defines the datasource, because that is where a reader looks for what this repository is. A managed or predefined datasource has no `.datagripe/` to put it in, so it goes at the export root, exactly where `manifest.json` was. The export root is what the import reads either way; the writer knows which case it is in and there is one function that decides. ```yaml version: 1 connection: ref: git:9b1f… name: wallet-prod engine: postgres domains: - name: auth colour: 1 description: "" includeData: false objects: - schema: basic_auth kind: table name: users - schema: public kind: function name: login(text, text) ``` The determinism rules from [`docs/spec/domains.md`](/specs/domains.md) carry over unchanged and gain one: **YAML is emitted with a fixed serialiser configuration** — block style, two-space indent, no line folding, no anchors or aliases, keys in the order above rather than alphabetically, and one trailing newline. Line folding is the trap. A description that grows by one character must not reflow a paragraph and put a twelve-line diff in front of somebody. `manifest.json` is not read as a fallback. An existing dump is re-exported once, which produces `domains.yaml` and deletes `manifest.json` in the same run — a prune, which the export already does for files it no longer generates. Both names are in `OWNED_ROOT_FILES`, so a stale copy at the export root goes the same way when the datasource keeps its own in `.datagripe/`. ### Importing one **Import is its own tab**, reached from the datasource menu beside *new datasource*, and not a block inside the create form. They are different acts: creating asks you for a host and a password, importing asks you for a URL and reads everything else out of the repository. One form holding both made people read the half that did not apply to them. The import tab does one thing and then gets out of the way: on success it is replaced by the datasource's own edit page, because the next thing anybody wants is to test it and — if it needs one — give it a password. Two ways in, one destination: - **Clone.** Paste an `https://` or `ssh://`/`scp`-style git URL. DataGripe runs `git clone` into the repos home (below) and reads `.datagripe/config.yaml` out of the result. - **Adopt.** Give an absolute path to a checkout that already exists. `git rev-parse --show-toplevel` must succeed and must report a root that contains the path given, and the path given must be the root itself — adopting `~/repo/src` and then writing `~/repo/.datagripe` would be a surprise. Either way the repository must contain `.datagripe/config.yaml`. A directory that does not is refused with the path it looked in, rather than being adopted as an empty datasource nobody can use. A clone URL is validated by the same SSRF policy as a connection host (`apps/server/src/security/ssrf.ts`), because it is the same thing: a user-supplied name the server is about to connect to. `file://` and local-path remotes are refused outright — a clone from `/etc` is not a feature. #### Where clones live `GIT_REPOS_DIR`, defaulting to `repos/` beside the embedded data directory — `./data/repos` in the repo checkout, and `/repos` under the desktop shell, which is what `apps/desktop/src/main.ts` already does for `pg/`. One directory per datasource, named `-`, so two repositories called `datasource` do not collide and the directory name is still readable. DataGripe may delete a directory it created, and only one it created — that is what `managed_clone` records. Removing an adopted datasource removes the row and touches nothing on disk. #### Secrets The repository names an environment variable; the server resolves it. This is `passwordEnv` from [`docs/spec/connection-sources.md`](/specs/connection-sources.md), and it is the documented path. **`noPassword: true`** is the other one: this database takes no credential at all. It exists for the self-contained case — a trust-auth cluster on loopback that the repository starts for itself ([`docs/spec/repo-commands.md`](/specs/repo-commands.md)), which is what makes `datagripe-example` importable with nothing to configure. Explicit rather than inferred from a missing `passwordEnv`, because inferring it would turn a typo in a variable name into a silent attempt to connect with no credential. The two are alternatives: setting both, or neither, refuses the file. When the variable is not set, the datasource is **listed and not connectable**, with the variable's name in the message. It is not hidden and it is not an error at load: the person who clones the repo on a laptop where `WALLET_PG_PASSWORD` is not exported should see the datasource and be told exactly which variable to set, not wonder why the sidebar is empty. As an escape hatch, the password may be supplied in DataGripe once and stored encrypted at rest with the existing keyring (`apps/server/src/crypto/keyring.ts`), keyed to `(workspace, datasource id)`. It never goes near the repository. An inline `password:` in `config.yaml` is **refused**, not deprecated: `connections.json` can get away with "development only" because it is gitignored, and this file is the opposite of gitignored. ### What the repo owns and what the workspace still owns | | Source | | --- | --- | | name, adapter, host, port, database, username, tls, readOnly, showAllSchemas | `config.yaml` | | branding colour and description | `config.yaml` | | path pairs | `config.yaml` | | sync dir and export options | `sync.yaml` | | domains and tags | the app database; `domains.yaml` on export/import | | password | a password stored here, else `passwordEnv`, else `noPassword` | | `read only`, `show all schemas` | the repository's value is the default; **this project may override either** | | default connection for a document, layout, drafts | the workspace, as before | On the datasource edit page the *connection fields* render like a predefined one — read-only, with "Defined by `.datagripe/config.yaml`" and a button that opens that file in the editor. Editing the file and saving it is how you change what the datasource **is**, which is the whole point; the connection list reloads when a `.datagripe/` file is saved or a pull changes one. The page is not otherwise dead, and this is the difference from predefined. Three settings are **this project's**, saved by `git.datasource.set-options` (migration 0017) and never written to the repository: - **A password.** Encrypted with the same keyring as every other connection secret. It wins over `passwordEnv`, because storing one is a deliberate act somebody took on this page — usually precisely because the variable was missing or wrong, and falling back to it afterwards would make the field look broken. Clearing it falls back. - **`read only`** and **`show all schemas`**. `NULL` means "whatever the repository says", which is distinguishable from an explicit override of the same value, and the form shows what the repository asks for beside the toggle. Somebody who imported a repository to look at production should be able to keep read-only on without opening a pull request against a repository they may not own. The override reaches the *connection*, not just the form — a `read only` that did not refuse writes would be a label. Everything describing what the datasource *is* — engine, host, port, database, user — stays the repository's. The split is "what this is" versus "how I use it", and it is the same split that already puts the export path and the domain tagging where they are. Note the reversal against [`docs/spec/datasource-paths.md`](/specs/datasource-paths.md): there, paths were workspace-local configuration *about* a datasource, precisely so a read-only predefined connection could carry them. Here the datasource brings its own, and the repo's list is **mirrored into `datasource_paths` rather than merged with it** — the table becomes a cache of what the file says, the form cannot edit it, and every reload rewrites it from the file. Merging would mean the sidebar showed sections a teammate did not have, from a file that claims to define the sidebar. Mirroring rather than special-casing is the whole reason this feature is small: `file.list`, `file.open`, the document origin and the archive-on-removal behaviour are the Phase 12 machinery, untouched. Rows are matched **by name**, so renaming a directory in `config.yaml` keeps the section's id and every file already open from it; a row whose name disappears archives its documents rather than deleting them, exactly as `datasource.set-paths` does. ### Exporting a config from an existing datasource On the edit page of any managed or predefined datasource, **export config**. It generates the `.datagripe/` file set from what the datasource already has — connection fields, its `domainExportPath` as `sync.dir` when that path is inside the target repo, its `datasource_paths` relativised against the target repo, and its domains as `domains.yaml` — and writes it into a directory you name. - **The password is never written.** The generated `config.yaml` carries `passwordEnv: ` with a name derived from the datasource (`WALLET_PROD_PASSWORD`), which you are told about in the panel above the button, because a placeholder that looks like a setting is worse than a missing one. - A path that does not sit inside the target repository is written as a comment with the absolute path and a note, not as a broken relative path. Silently dropping it is how a teammate ends up with three of your four sections. - The generated YAML is shown before it is written, with copy-to- clipboard, because half the time the destination is a repository on a different machine. - Writing over an existing `.datagripe/config.yaml` asks first and shows what differs. Exporting a config does not convert the datasource. It stays managed or predefined; you get a directory you can commit, and adding it back as a git datasource is a separate, deliberate act. ### The repository section A new left-bar section, present only while a git datasource is the active one, above its path sections ([`docs/spec/datasource-paths.md`](/specs/datasource-paths.md) "What the sidebar shows"): ``` ┌ repository (main) (3) ⟳┐ ← branch, changed count, refresh │ 3 changed · 2 staged │ │ ☑ M foo/blah.sql │ ← staged for the next commit │ ☑ M .datagripe/config.yaml │ │ ☐ ? bar/baz/query22.sql │ │ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ │ │ [ commit… ] [ push 2 ][ pull 0 ]│ ├ Some Folder ──────────────────┤ │ ▾ foo │ ``` **The branch, the change count and refresh live in the section header**, where they are legible whether or not the section is open. "Which branch am I on" is a question people ask without wanting the file list, and it used to cost a click and two rows of panel. What is left inside is one summary line and the buttons. **The ahead/behind counts ride on `push` and `pull` themselves** — `push 2`, `pull 0` — rather than sitting in a pair of arrows above them. The count belongs to the press that changes it, and `↑2 ↓0` needs its words said out loud to mean anything anyway. - **The list is `git status --porcelain -uall` at the work tree root**, not scoped to a sub-path. This is a repository view, and hiding a changed file because it is outside a configured path is how you commit half of something. - **Every file is a checkbox and nothing is checked by default.** The commit stages exactly what you ticked — `git add -- ` with the selected pathspecs and no `-A` anywhere — then `commit -m`. The rule from [`docs/spec/domains.md`](/specs/domains.md) survives in the form that matters: a commit from DataGripe never stages a file the person did not name. Files already staged when the section loaded show as checked and stay checked; unticking one runs `git restore --staged`. - **`commit…` opens a message field.** It is a press, then a second press. `push` is separate and always separate, because it is the only button here that leaves the machine. - **`pull` is `git pull --ff-only`.** Anything that would need a merge commit, a rebase, or a conflict resolution refuses with git's own message. DataGripe cannot represent a conflicted work tree and must not pretend to. - A row's status letters are git's, unexplained and unabbreviated: ` M`, `M `, `??`, `A `, `D `, `R `, `UU`. People who use git read these already, and people who do not are not served by a paraphrase. - Clicking a row opens the file, **when a configured path covers it**. The two path vocabularies meet here and nowhere else: git names files relative to the work tree root, the editor names them relative to a path pair. A row outside every pair says so and opens nothing, rather than opening a different file. `git.status` carries the work tree root for exactly this — without it the client has two vocabularies and no way to relate them. - **`refresh` is the only way the list updates**, plus after any operation this section ran. Nothing polls. A status call per second across every open workspace is a `git` process per second. - **Refresh fetches first, where there is an upstream** (`git.fetch` → `git fetch --quiet`, then a status). `rev-list` counts against the remote-tracking ref, which is a memory of the last fetch, so a refresh that only ran `status` would answer "0 behind" from whenever somebody last pulled — a button that says there is nothing to pull when there is. A fetch moves no file and touches no work tree, and it is still a press: nothing fetches on a timer. Git's verdict comes back verbatim like every other command here, so a fetch that failed on credentials says so instead of showing stale counts. With no upstream it is a plain status. The section is capped at 500 rows; past that it shows the count and a line saying to use a terminal, because a 12,000-file status is not a list anybody is going to tick through. ### After a pull A successful pull can change files that are open in tabs. The machinery for this already exists — `disk_content_hash` and the three cases in [`docs/spec/datasource-paths.md`](/specs/datasource-paths.md) "When the file moves underneath" — and this is exactly the `git pull` case that spec names. What changes: the server broadcasts `repo.changed` with the datasource ref after a pull that moved `HEAD`, and each client re-runs the open check for every file-backed document it holds from that datasource. Documents whose cached content still matches the recorded hash adopt the new bytes silently. Documents with edits of their own raise the existing banner. Directory listings for the datasource's paths are invalidated, and a `.datagripe/` file that changed reloads the connection list. A pull is refused while any file-backed document from that datasource is dirty *and* would be touched by the pull. DataGripe knows the changed file list from `git status`, and offering to fast-forward over somebody's unsaved work is not a service. ### Storage Migration 0014: ```sql CREATE TABLE git_datasources ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), workspace_id uuid NOT NULL REFERENCES workspaces (id) ON DELETE CASCADE, -- Absolute path of the work tree root on the host the server runs on. repo_path text NOT NULL, -- The remote it was cloned from; NULL when an existing checkout was -- adopted. Not read back for anything except display: the remote of -- record is whatever `git remote` says now. remote_url text, -- True only when DataGripe created this directory, and therefore the -- only case in which it may delete it. managed_clone boolean NOT NULL DEFAULT false, created_by uuid REFERENCES users (id) ON DELETE SET NULL, created_at timestamptz NOT NULL DEFAULT now(), UNIQUE (workspace_id, repo_path) ); ``` Plus `git_datasource_secrets`, the escape hatch above: one encrypted password per datasource, same layout and same keyring as `connection_secrets`, cascading off the datasource row so removing the datasource takes the secret with it. A pointer, not a copy. Nothing from `config.yaml` is denormalised into this table, because then there would be two answers to what the datasource is and one of them would be stale. The file is read on `workspace.open` and cached in memory with the file's mtime and size as the cache key. `ConnectionMetadata.id` for a git datasource is `git:`, so nothing branches on id shape: managed ids are bare UUIDs, predefined ids are slugs, git ids carry their prefix. `connectionSourceSchema` gains `"git"`. Domains, tags, gripe dismissals, `datasource_roles` and query history all key on `connection_ref` as text already and need no change. ### Actions (WebSocket) | Action | Role | Notes | | --- | --- | --- | | `git.datasource.add` | `owner` | Clone or adopt; talks to a remote, writes the host | | `git.datasource.remove` | `owner` | Deletes the checkout only when `managed_clone` | | `git.datasource.reload` | `editor` | Re-read `.datagripe/`, bypassing the mtime cache | | `git.datasource.set-options` | `editor` | The password and the two overrides — this project's settings, like the export path beside them | | `git.status` | `editor` | Branch, upstream ahead/behind, porcelain rows | | `git.stage` | `editor` | `add --` / `restore --staged` for named paths | | `git.commit` | `editor` | Stages the named paths, then commits | | `git.pull` | `editor` | `pull --ff-only` | | `git.push` | `owner` | The only one that leaves the machine | | `datasource.export-config` | `editor` | Generate, preview, write `.datagripe/` | `commit` and `pull` being `editor` is a deliberate departure from [`docs/spec/domains.md`](/specs/domains.md), where every git operation is `owner`. The reason that rule exists is that `domain.git` writes a generated tree and pushes it; an editor who may already open a file, edit it and write it back to disk is not meaningfully more dangerous for being able to record that in the local history. `push` and `add` stay `owner` because they reach the network. Every invocation keeps the guarantees from [`docs/spec/domains.md`](/specs/domains.md) "Committing" without restatement: `Bun.spawn` with an argv array and no shell, a fixed verb list with no user-supplied flags, `--` before every user-supplied value, `GIT_TERMINAL_PROMPT=0` with a failing askpass, a `GIT_TIMEOUT_MS` kill, verbatim stdout/stderr with the exit code as the verdict, and an audit line per call. `apps/server/src/domains/git.ts` moves to `apps/server/src/git/` and grows the new verbs; the domain export keeps its own root-scoped `add`. ### Configuration | Variable | Default | Meaning | | --- | --- | --- | | `GIT_ENABLED` | `false` | Gates every git feature, including git datasources. `DOMAIN_EXPORT_GIT` is the pre-rename name and is still honoured | | `GIT_REPOS_DIR` | `/repos` | Where clones go | | `GIT_TIMEOUT_MS` | `60000` | Per invocation. `DOMAIN_GIT_TIMEOUT_MS` is the pre-rename name | | `GIT_CLONE_TIMEOUT_MS` | `600000` | Clone separately, because a big repository is not a hung one | `HOST_FS_DISABLED` turns git datasources off regardless, the same way it turns the export and the path tree off: all three are the server reading and writing a disk the person pressing the button may not own. ## Testing - **Round trip.** Export a config from a managed datasource into a temporary repo, add that repo as a git datasource, assert the connection fields, the path pairs and the domains match the original, and that no password was written. - **Determinism.** Export domains twice against a fixture database and assert the two `domains.yaml` files are byte-identical, including after a description is edited to a length that would tempt a folder. - **Relative paths.** `foo`, `./foo` and `foo/` resolve to one directory; `/etc`, `../outside` and a symlink in the repo pointing at `/etc` are each refused with a named error and list nothing. - **Config refusals.** Missing `.datagripe/config.yaml`, an unknown `version`, two paths with the same name, and an inline `password:` each refuse the datasource and say which file and which key. - **Unknown keys.** A `config.yaml` with a top-level block DataGripe does not know survives a write from DataGripe unchanged. - **Staging.** With five modified files, ticking two and committing produces a commit containing exactly those two and leaves the other three modified in the work tree. - **Argv.** A commit message of `--amend`, and a filename beginning with `-`, both round-trip as values. - **No credentials.** A push against a remote with no credentials surfaces git's stderr and a non-zero exit and does not hang until the timeout. - **Pull refusal.** A dirty file-backed document whose file the pull would touch refuses the pull and names the file; a dirty document the pull would not touch does not. - **Pull adoption.** A file changed only on disk is adopted silently as a new revision; a file changed on both sides raises the banner. Both are the existing `file.open` cases, asserted through a real pull. - **Missing secret.** With `passwordEnv` naming an unset variable, the datasource is listed, the connection test fails, and the message names the variable. - **Clone safety.** `file:///etc`, a bare local path, and a URL resolving to a private range with SSRF enabled are each refused before any process is spawned. - **Adopted removal.** Removing an adopted datasource leaves the directory on disk; removing a cloned one deletes only its own directory under `GIT_REPOS_DIR`. ## What is not built - **Branch switching.** The section shows the branch and will not change it. Checking out a branch with open dirty tabs is a whole design. - **Conflict resolution.** `--ff-only` is the position, not a first step toward a merge UI. - **Watching the work tree.** Consistent with [`docs/spec/datasource-paths.md`](/specs/datasource-paths.md): nothing polls, and `refresh` is a button. A watcher across every repository in every workspace is a different feature with a different cost. - **Per-user overrides of a repo-defined datasource.** If the committed host is wrong for you, the answer today is a second datasource. - **`.datagripe` in a subdirectory.** It is at the work tree root or it does not exist. ## Open questions - Whether `branding.colour` here and the domain colour palette should be the same eight slots. Reusing them is cheap and might make a prod datasource look like a domain rail. - Whether a git datasource should be able to carry *additional* workspace-local paths on top of the repo's, for the one person who keeps a scratch directory beside the checkout. Currently no, because merged lists make "what does this file define" unanswerable. - Whether `sync.yaml` should be able to name more than one sync dir, once a repository holds two datasources. Two datasources in one repository is not supported and this is the first thing that would need to change if it were. - Whether the commit list should offer hunk-level staging. Almost certainly not — that is the point where a terminal is better — but it is the most likely request. - Whether a managed datasource's `domains.yaml` should also move into a `.datagripe/` directory at its export root, so there is one answer instead of two. It would mean inventing that directory for a datasource that has no repository, which reads like a promise the feature does not make. --- # Spec — Gripes engine **Status:** draft — machinery and the editor/panel surfaces are built **Phase:** 11 (planned) **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/gripes.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** nothing (implements [`docs/brand/brand-system.md`](https://github.com/datagripe/datagripe/blob/main/docs/brand/brand-system.md) "Voice", "Attitude levels", "Writing gripes", and the tier-3 mascot rule) > Built: `packages/gripes` (rule shape, runner, renderer, catalogue and > its assertions), the wire types in contracts, `scanTokens` in > sql-tools, the rule catalogue, a client `SchemaInput`, dismissal at all > three scopes, and four > surfaces — editor gutter and squiggle, annotation rail, gripes panel, > object-view annotations, status-bar count. > > Not built: the server-side runner on the execution path, which waits on > a rule with `execution` inputs. ## Goal DataGripe notices things about your query and your schema, and says so — accurately, specifically, and rudely. A gripe is a finding with a stable rule id, a severity, a location, and a set of facts, rendered as one sentence whose register depends on the project's attitude level. The product thesis is a single sentence from the brand spec, and every decision below follows from it: > A gripe that is merely rude is noise and users disable it within a > week. A gripe that is correct and rude is a feature they screenshot > into the team channel. So correctness outranks coverage, and silence outranks a guess. ## Non-goals - **A general SQL style linter.** sqlfluff exists. A gripe earns its place by knowing something about *this* database — the row count, the missing index, the column that is nullable in practice. A finding derivable from the query text alone, with no reference to the schema or the cost, is usually a lint rule and belongs to a formatter. - **Generated wording.** No LLM, no template shuffling, no synonym rotation. The brand spec is explicit: "The same rule firing twice uses the same wording. Never generate variants to seem clever." Four fixed strings per rule, written by a human, reviewed like copy. - **Blocking anything.** A gripe never prevents a run, never gates a save, never fails a build. It is an opinion held loudly. - **Auto-fix.** Suggesting `create index` text to copy is in scope later; editing the user's SQL is not. - **Scoring.** No grade, no "query health 62%". A number invites gaming and averages away the one blocker that mattered. ## Design ### The shape of a rule A rule is a pure function. It declares what it needs; the runner gives it exactly that and nothing else. ```ts interface Rule { /** Stable, namespaced, greppable: `.`. */ id: string; // "index.missing" severity: "blocker" | "warning" | "style"; /** What this rule needs to decide. The runner only calls rules whose * inputs it can supply. */ inputs: RuleInput[]; // ["statement", "schema"] /** Returns nothing when it cannot tell. Never a hedge. */ evaluate(context: GripeContext): Finding[]; } interface Finding { ruleId: string; severity: Severity; /** Where to point: a document range, an object, or a result set. */ at: GripeLocation; /** Named values the wording interpolates. Never pre-formatted prose. */ facts: Record; } ``` Two properties matter more than the types: **A rule that cannot tell returns nothing.** The third calibration example in the brand spec — *"Hmm, this query might be a little slow, maybe?"* — is rejected for being "hedged into uselessness. If the tool is not sure, it says nothing." That is a hard property, not a style note: a rule needing the schema must stay silent when the schema is not cached, rather than firing on a guess. **A finding carries facts, not sentences.** `{ rows: 41203882, column: "status" }`, not `"41M rows and no index on status"`. This is what makes attitude a presentation layer rather than four parallel analyses. ### Findings are analysis; wording is presentation The brand spec fixes this: "The technical content never changes between levels — only the register. Attitude is a presentation layer over a fixed set of correct observations, never a different analysis. Implementation follows: one finding, four strings, selected at render." So the split is: | Stage | Produces | Depends on attitude | | --- | --- | --- | | Evaluate | `Finding` — rule id, severity, location, facts | no | | Render | one sentence | yes | Consequences worth naming: - Changing the attitude level **re-renders**; it never re-analyses. The dropdown is instant and cannot change what was found. - The server never sends prose. It would have to know the reader's attitude, and attitude is a per-session setting that changes without any new analysis. - A gripe is auditable: the footer carries `severity · rule id · line`, and the rule id is the same string at every level and in every locale. ### Where the code lives A new `packages/gripes`, pure, no I/O — the same shape as `packages/sql-tools`: ``` packages/gripes/ src/ rules/ one file per rule, each a pure function + its tests types.ts Rule, GripeContext, the five input shapes catalogue.ts the rule registry messages.ts ruleId → { notice, warning, fatal, panic } render.ts Finding + attitude → string, and the footer runner.ts given available inputs, run the applicable rules statement.ts tokenize once per statement, not once per rule assertions.ts the mechanical checks over the catalogue ``` The wire types — severity, attitude, location, finding — live in `packages/contracts/src/gripes.ts` instead, because a finding crosses the socket when the server evaluates an execution rule. Static analysis rests on `scanTokens` in `packages/sql-tools`: a token walk that shares the statement splitter's awareness of comments, string literals, quoted identifiers and dollar-quoted bodies. That sharing is the whole point — a rule scanning raw text with a regex fires on the word `join` inside a comment, and a wrong gripe destroys trust in every other gripe. Quoted words keep their case, because `"From"` and `"from"` are different columns. Both the server and the web app import it: the server to evaluate rules needing a connection, the client to evaluate rules over things it already holds and to render every finding it displays. ### Where rules run, and why it is split A rule's `inputs` decide this. Nothing else does. | Input | Available to | Notes | | --- | --- | --- | | `statement` | client, server | the statement under the cursor, via sql-tools | | `schema` | client, server | the client's completion catalog; the server's introspection | | `object` | client | an `object.describe` result the object view already has | | `execution` | server | row count, elapsed, truncation, affected rows | | `plan` | server | `EXPLAIN` output — opt-in, see Open questions | Static and structural rules run **in the client**: their inputs are already in memory, so findings appear as you type with no round trip. Execution and plan rules run **in the server**, which is where the connection is, and arrive as part of the execution's event stream. This is not two engines. It is one pure catalogue and two runners, and a rule does not know or care which one called it. ### Severity, and why there are three Three, because `tokens.css` defines exactly three and the brand spec's gripe-row treatment is built on them: | Severity | Token | Means | | --- | --- | --- | | `blocker` | `--dg-sev-blocker` (magenta) | this will hurt: a cross product, a full scan of a large table, a missing index behind a hot filter | | `warning` | `--dg-sev-warning` (violet) | this is probably wrong: a nullable column compared with `=`, an unbounded `select *` | | `style` | `--dg-sev-style` (green) | this is untidy and cheap to fix | Severity is **how bad the finding is**. Attitude is **how rudely it is phrased**. They are independent: a `style` finding at `fatal` swears about something trivial, which is exactly the failure mode the brand spec warns about — "A gripe that swears at a missing semicolon has spent the currency for nothing." So the message catalogue is where that gets controlled, per rule, by the person writing the strings. The engine does not couple them. ### Presentation Per the interface rules: a 3px severity border on the left, the background tinted to roughly 4% of that accent over Void, square corners because the border is single-sided, and **a severity cue that is not colour** — a hard requirement, since all four accents sit at similar lightness and someone with a colour vision deficiency must still be able to read the panel. Five surfaces, in descending order of how often you see them: 1. **Editor glyph margin and squiggle.** Built. A parallel decoration collection to the one `statementMarkers` already uses for execution outcomes, with a severity glyph in the gutter and a wavy underline on the range. Underline rather than a filled background: a filled range would fight the syntax colours it sits on. The hover carries the rendered sentence and the footer. 2. **Gripes panel.** Built. Rows grouped by document, worst first, each with its severity glyph and footer. Clicking a row reveals the offset in whichever view holds that document and focuses the editor — `revealPositionInCenterIfOutsideViewport`, so a finding already on screen does not throw away the reader's sense of place. The attitude selector drives the wording live. 3. **Object view.** Built. An annotation block above the tab's content, in the shape the tree-interactions mock shows. A finding carries the tab it belongs to, so an index complaint stays on `indexes` and does not follow you to `grants`; a finding with no tab shows on all of them. Above the content rather than replacing it — a complaint about an index reads best next to the indexes. A finding is scoped to a tab, which means a tab can hide one. The strip therefore marks any tab carrying findings with the severity glyph the gripe rows use, worst-first, and a routine opens on `ddl` rather than on the first tab in its list — that is where its gripes live and what you opened it to read. Without both, a routine's security-definer finding was in the store, counted in the status bar, and invisible on screen. Object rules run in the object view itself, because the describe result is their whole input. The findings are published into the gripes store so the panel and the status-bar count agree with what the tab is showing: a count reading "no gripes" beside two visible gripes would undermine every other number the tool prints. 4. **Annotation rail.** Built, on Monaco's overview ruler, which is exactly this: marks beside the vertical scrollbar showing where findings are in the *whole* document, not just the visible window. Capped at forty, because "a rail with two hundred marks is a gradient, not a map" — and since findings are sorted worst first, the cap drops style notes before blockers. 5. **Status bar.** Built. A count, and magenta when any finding is a blocker — so the one that matters is not averaged away by a pile of style notes. **The mascot appears on none of them.** Tier 3 is "empty state only (`no gripes` gets faint approval). **Never per-gripe.**" A mascot on every finding stops being funny within a day. ### Attitude levels Four, from the brand spec, and they are Postgres severities on purpose: `notice`, `warning` (default), `fatal`, `panic`. Resolution order, first hit wins: 1. Session override — `set datagripe.attitude = 'notice'`, honoured as a real statement in the editor so it can be dropped before a screen share without opening settings. Lives for the session. 2. Project setting. 3. `warning`. Three constraints on `panic`, all from the brand spec and all implementable rather than aspirational: - It can never be the default. - It cannot be set for an organisation — only by the person reading it, for their own session or their own project. - **It resets on upgrade.** Store the app version alongside the setting; if the version has changed and the level is `panic`, drop to `warning`. "A joke that cannot be escaped is not a joke." **Blocked on the same thing the danger zone is.** Attitude and project class both live in `apps/web/src/stores/branding.ts` in `localStorage`, marked MOCK. A per-project attitude that the server does not know is a per-browser attitude. Promoting both to the workspace model is one migration and unblocks both features — see [`docs/spec/object-view.md`](/specs/object-view.md) "Danger zone". ### Dismissal "Always dismissible. Per rule and per file, at every attitude level." Three scopes, coarsest last: | Scope | Key | Means | | --- | --- | --- | | occurrence | rule id + location | not here | | target | rule id + document or object | not in this file | | project | rule id | never, in this project | Stored server-side per workspace (`gripe_dismissals`) so a dismissal survives a browser and is visible to the team. It is **team-wide**, because a schema finding is a team fact — still an open question, so `dismissed_by` is recorded on every row, which is enough to make it personal later without losing history. A scratchpad's dismissal references a document id the server does not otherwise know; harmless, and worth knowing. **An occurrence dismissal cannot key on a document offset.** Offsets move the moment anything above them is typed, so the dismissal would evaporate on the next keystroke. It keys on a hash of the statement's normalized text instead, which gets the behaviour right in both directions: editing an unrelated statement — or running the formatter — keeps the dismissal, and editing *this* statement drops it, because the finding may no longer hold once the text changed. Dismissing the same thing twice is a no-op rather than an error, via a unique index over `(workspace, rule, scope, key)`. `project` scope has no key and stores `''`, so one index covers all three scopes. Dismissal is never silent. The panel keeps a count of what is hidden and a "show all again" beside it, or the feature becomes a way to make the tool lie quietly. A dismissed finding also leaves the gutter and the rail — silencing the panel while the squiggle argues with it would be worse than not dismissing at all. Findings stay in the store either way, so restoring costs no re-analysis. The scope labels say what they mean rather than what they are called: "not here", "not in this file" (or "not on this object"), "never in this project". ### Rule ids are a public contract A rule id appears in the gripe footer, in dismissal rows, and in whatever anyone greps their logs for. Renaming one silently un-dismisses it for every user. So: ids are stable, and a rule that changes meaning gets a new id rather than new behaviour under the old one. Format is `.`, lower-kebab: `join.no-condition`, `index.missing`, `table.no-primary-key`, `select.unqualified-star`, `routine.volatile-but-readonly`. ### The catalogue Every rule earns its place by knowing something the query text alone does not, or by being about damage rather than tidiness. What is deliberately absent is style filler — a formatter's job, and the fastest way to get the whole thing switched off. | Rule | Severity | Inputs | Fires on | | --- | --- | --- | --- | | `routine.definer-no-search-path` | blocker | object | `security definer` with no `search_path` pinned | | `join.no-condition` | blocker | statement | a join with no `on` or `using` | | `delete.no-where` | blocker | statement | a delete that removes every row | | `update.no-where` | blocker | statement | an update that rewrites every row | | `column.nullable-inequality` | warning | statement, schema | `col <> 'x'` where the schema says `col` is nullable | | `subquery.not-in` | warning | statement | `NOT IN (SELECT ...)`, which returns nothing at all if the subquery yields a null | | `table.no-primary-key` | warning | object | a base table with no addressable row | | `view.select-star` | warning | statement | a view whose column list the star froze at creation | | `index.not-concurrent` | warning | statement | `CREATE INDEX` with no `CONCURRENTLY`, which blocks writes for the build | | `index.duplicate` | style | object | an index whose keys prefix another's | | `routine.volatile-but-readonly` | style | object | a read-only `sql` routine left volatile | Two of these are about something the statement text actively misrepresents. A star in a view reads as a standing instruction and is not one: the database expands it once, at creation, and records the result, so a column added later never appears. And `NOT IN` against a subquery reads as the negation of `IN` and is not one: a single null in that subquery makes the whole predicate unknown, so the query answers zero rows with no error and no clue. The shape is the finding in both cases — nothing at the call site says whether the subquery's column is nullable, and it can become nullable later without this query being touched. `column.nullable-inequality` is the first rule that needs the schema, and it is deliberately about the *inequality* and not the equality. Most columns are nullable, and `col = 'x'` excluding nulls is what everyone expects, so griping there would fire on half the queries in the tool and get the whole feature switched off. The negation is where the reading and the behaviour come apart: "status <> 'void'" reads as everything that is not void, and nulls are obviously not void, but the comparison is unknown for those rows so they are dropped in silence. `routine.definer-no-search-path` is the one that is about a vulnerability rather than a cost: a definer routine runs with the owner's privileges but resolves unqualified names using the *caller's* `search_path`, so anyone who can create a schema can shadow something the body calls and have it run as the owner. **Where the correctness discipline actually bit.** Each of these has a "looks like the finding and is not" fixture class, because that is where a wrong gripe would come from: - `delete.no-where` reads the statement's *main verb*, looking through a leading `WITH`. Without that, `create trigger t after delete on x` reads as an unqualified delete — and it would fire on every trigger in the database. - `index.duplicate` treats a prefix as covered but not an equal column list, and never a unique index: a unique index enforces something the wider index does not, so dropping it changes behaviour rather than saving writes. - `view.select-star` distinguishes a projection star from multiplication by what surrounds it: a star follows `select`, a comma or a qualifier's `.`, and precedes a comma or `from`. Depth alone is not enough — the `*` in `select qty * price` sits at the same depth as a real one, so without this the rule fires on arithmetic. - `index.not-concurrent` is gated on the *dialect*, not the adapter id. `CONCURRENTLY` is not merely unhelpful advice on MySQL and SQLite; it is a syntax error, so a gripe suggesting it would be actively wrong. - `subquery.not-in` stays on the adjacent `NOT IN` form and never fires on `NOT EXISTS`, which is the fix, nor on a written-out list, where a null is visible to whoever reads it. - `routine.volatile-but-readonly` reads the routine *body*, not the definition. Checking the whole definition finds `CREATE` in every routine, so the rule never fires at all — which is how it was first written, and what its test caught. It also declines to judge a plpgsql body, which can write through dynamic SQL that no amount of reading will reveal. ### The client's SchemaInput Built, backed by the completion catalog, and honest about what that cache does and does not hold. `isNullable` is real; `rowsFor` and `indexLeadsWith` always answer `null`, because the catalog carries neither. They are declared rather than omitted so a rule needing them compiles and stays quiet, instead of the runner having to know which parts of the schema each caller can supply. The catalog fetches a table's columns on demand, so a schema rule's first look usually knows nothing and correctly says nothing. Asking for the columns is the side effect; when they arrive the catalog notifies, the store re-analyses, and the finding appears — a beat late, but never wrong. That is the whole reason `null` may not be read as `false`: a rule treating "not fetched" as "NOT NULL" would go silent on a real problem, and one treating it as "nullable" would invent one. Two things had to be fixed before a schema or dialect rule could be trusted at all, and both were silent: - `scanTokens` dropped any character it did not recognise, and `!` was not in its punctuation set. `a != 1` therefore tokenized identically to `a = 1`, so a rule about equality would have fired on its exact opposite. Multi-character operators are now single tokens. - `SqlDialect` was `keyof Record`, which is `string`, so every caller type-checked and the editor was passing a hardcoded `"postgres"` for every connection. A dialect-gated rule would have fired on MySQL. The dialect is now a real union, resolved from the adapter's `sqlDialect` capability — a capability and not the adapter id, since Redis is an adapter with no dialect at all. ### Still on the list - **Statement rules over an object's DDL.** The `ddl` tab holds the reconstructed definition as text, and nothing analyses it. A view created with `select *` is a finding this catalogue already has — `view.select-star` — and it fires in a query document but not on the view itself, which is the one place someone would look. Running the statement rules over the DDL would close that, and is the cheapest large win left. - `index.missing` needs to know a relation's indexes, which the completion catalog does not carry. The object describe result does, so this waits on either widening the catalog or a server-side runner. - `execution.truncated` needs the server-side runner. - `select.unqualified-star` is **not** going to ship as specified. In a SQL client, `select * from t limit 100` is the single most common legitimate query there is, and griping at it is precisely the style filler this catalogue exists to avoid. The freezing case has real teeth and shipped as `view.select-star`; the ad-hoc case does not. ### Analysis is debounced, and per document The client runner re-analyses on document content change, debounced 400ms per document through the same keyed debouncer the draft checkpoints use — long enough that typing mid-word does not flicker findings. Each statement is analysed separately, with its document offset passed in, so a finding in the third statement still points at where it is in the whole document rather than at an offset into that statement. That is the one thing easy to get wrong here and it has a test. ### Testing The rules are pure, so the tests are the cheap part and there is no excuse for thin coverage. Per rule: a fixture that fires, a fixture that does not, and a fixture that *cannot tell* and must stay silent. Beyond that, the brand spec's own acceptance checks become assertions over the catalogue (`assertions.ts`), run once for every rule: - Every rule has all four attitude strings, none blank. A missing `fatal` cannot fall back to `warning`; that is how a level silently stops existing. - No string exceeds 90 characters at `notice` or `warning`. - No string at **any** level, including `panic`, contains a barred term. "Swearing at a query is funny; punching downward is not, and it is the one thing that would follow the product around." - `notice` contains no sanctioned profanity. - Rule ids are `.` in lower-kebab, and unique. - No rule declares zero inputs, and no wording exists for a rule that does not. - Rendering a rule's own fixtures leaves no unresolved `{placeholder}`. Matching is on word boundaries throughout: a substring check that flags "hello" for containing "hell" is worse than no check, because it trains people to work around it. **Two things are deliberately not asserted.** *"Every string interpolates a fact"* was in an earlier draft of this spec and is wrong. The brand spec's own approved calibration example — "This join has no condition. I'll allow it. I won't forget it." — names nothing at all. Specificity for a statement rule comes from the *location*, which points at the construct; a fact is how a rule is specific when it has a number worth quoting. Asserting the fact would have failed the brand spec's own copy. *Whether a gripe is funny.* That is a review gate, not a test. ### The barred-term list, and the disclaimer instead of a gate `BARRED_TERMS` starts empty and grows alongside the wording: a rule arrives with four strings, anything those strings prove they need fencing off is added, and `assertNoBarredTerms` keeps it fenced for every rule that follows. An enumerate-it-up-front gate was considered and dropped — this is free software with no public release to gate, and a list nobody has a rule to test against is guesswork wearing a checklist. What actually holds the line is the brand spec's rule, which is a review standard rather than a string match: "nothing touching race, gender, sexuality, disability or religion at any level", and "it criticises the query, never the person who wrote it". A string match cannot enforce either; it can only catch a regression in something already decided, which is what it is for. The reader gets told as much. `DISCLAIMER`, shown beside the attitude control in the panel: > Gripes criticise the query, never the person who wrote it. fatal and > panic swear; choose notice if that is unwelcome. It sits next to the attitude control because that is where someone picks the register, and it is written in the plain product voice, not the gripe voice — "if the whole interface is sarcastic then nothing is". ## Open questions - **Which rules ship.** The reserved decision. Everything above is machinery for a catalogue that does not exist. - **Is a dismissal personal or team-wide?** Proposed team-wide, on the grounds that a schema finding is a team fact. The counter-argument is real: one member silencing a blocker for everyone is a way to lose the finding that mattered. A middle option is team-wide for `object` targets and personal for documents. - **`EXPLAIN`, and whether it is automatic.** Plan-based rules are the ones that would justify the product, and they cost a target-database round trip per statement. Automatic on every edit is clearly wrong; automatic on every execution is arguable; a button is safe and will be under-used. Undecided. - **Firing volume.** No rule for how many gripes is too many. A panel with 200 rows is the annotation rail problem again — it stops being a map. Options: cap per document, collapse repeats of one rule into a count, or rank and show the top n. - **Localisation.** From the brand spec's open items: "Profanity does not translate at consistent strength. Each locale needs its own calibration, not a translated string table." So a locale is a re-authoring job per rule, which changes the shape of `messages.ts` from a map to a per-locale catalogue. Not designed here. - **Severity vs the project class.** A `blocker` in a `local` sandbox and a `blocker` in `production` are not equally urgent. Whether the class modulates severity, or only the wording, or neither, is open. - **Does a session `set datagripe.attitude` statement really execute?** Treating it as a real statement the editor intercepts is a lovely touch and a parsing special case. The alternative is a plain control in the panel, which already exists. --- # Spec — Markdown documents **Status:** current **Phase:** 14 **Source:** https://github.com/datagripe/datagripe/blob/main/docs/spec/markdown-documents.md > An engineering document, not user documentation. It records what > a subsystem does and why the alternatives were rejected, and some > of it describes things that are not built. For using DataGripe, > read [the documentation](/docs.md) instead. **Supersedes:** the "the language stays `sql`" paragraph in [`docs/spec/datasource-paths.md`](/specs/datasource-paths.md) "The file as a document" (extends [`docs/spec/editor-workspace.md`](/specs/editor-workspace.md), [`docs/spec/query-execution.md`](/specs/query-execution.md), [`docs/spec/gripes.md`](/specs/gripes.md)) ## Goal [`docs/spec/datasource-paths.md`](/specs/datasource-paths.md) shipped with a wart it named out loud: a `.md` opened out of a datasource path gets SQL highlighting, "and the fix is a language field on the document rather than anything here". This is that fix, and the reason it is worth a spec rather than a one-line enum change is what a `.md` file in a database repository actually contains. It contains a runbook. `maintenance.md` explains what the nightly job does, and then — in a fenced `sql` block — the query you run when it fails. Today that query is read in one window and retyped in another, which is how it drifts from the prose that explains it. DataGripe can run it where it is written. So: markdown is the second document language, decided by extension, in every place DataGripe shows files. It **opens rendered**, because a runbook is something you read. Its SQL blocks are executable in place, against the document's connection, through exactly the execution path a selection goes through. And a button in the bottom right flips to the editor, where the fences get the SQL editor's completion, formatting and gripes, and flips back. ## Non-goals - **Not a markdown authoring tool.** No preview-beside-editor, no toolbar, no WYSIWYG, no paste-image. It is two modes and one button. - **Not a general document store.** Markdown arrives because a file is called `.md`. There is no "new markdown document" verb beyond naming one — see "Where a language comes from". - **Not HTML.** Raw HTML in markdown is escaped and rendered as text, never passed through. A file from a checkout is content somebody else may have written. - **Not a syntax-highlighting project.** JSON, YAML, TOML and the rest are `plaintext`: a name for "no formatting", not a feature. Markdown earns a language because it has runnable content; a highlighter per extension has no end. (This was "they stay `sql`", which meant a `.csv` opened out of a datasource path was a query with a thousand findings in it.) - **Not a notebook.** Blocks do not share state, do not run in order, have no persisted outputs, and produce no artifact. Running one is running a query. ## Design ### Where a language comes from One rule, everywhere: **the extension of the name decides.** | Name | Language | | --- | --- | | `maintenance.md`, `README.markdown` | `markdown` | | `blah.sql`, `query22.sql` | `sql` | | `notes`, `Makefile`, `.env` | `sql` | | `analysis.json`, `rows.csv`, `notes.txt` | `plaintext` | A name with no extension is `sql` because a document nobody has named yet is a query — `query-1.sql` without the suffix is still the thing it was. A name *with* an extension that is neither is plain text, which is the honest answer: DataGripe opens the file and does not pretend to understand it. The rule applies to file-backed documents (`origin.filePath`), to shared workspace files and to local scratchpads, because all three are named by a person and all three show up in a files area. Renaming a document from `notes.sql` to `notes.md` changes its language, which is the behaviour somebody renaming it is asking for. `documentSchema.language` is `z.enum(["sql", "markdown", "plaintext"])`; migration 0015 added `documents.language text NOT NULL DEFAULT 'sql'` with a check constraint and 0026 widened it. It is stored rather than derived on read because the client needs it before it has the content — the language is recomputed on save, with the title, and broadcast like any other document change. **A rename retunes the open model rather than rebuilding it.** Monaco cannot change a model's URI, and rebuilding the model to get a new one would throw away the undo history the person just typed, so `modelRegistry.setLanguage` calls `setModelLanguage` on the live model and the URI keeps the extension it was born with. Everything a reader sees — the highlighting, the markdown pane, the dialect the gripes engine uses — reads the store's `language`, so the change is immediate and a reload is not part of it. The default for `document.create` stays `sql`: a new scratchpad is a scratchpad. Name it `notes.md` and it is markdown on the next save. ### Two modes A markdown document's editor panel has a mode, `view` or `edit`, and opens in `view`. ``` ┌ maintenance.md ─────────────────────────────────────────┐ │ │ │ Nightly reconciliation │ │ ───────────────────── │ │ When the job reports a gap, find the orphans: │ │ │ │ ┌───────────────────────────────────── sql ──── ▶ ──┐ │ │ │ select * from ledger.entries e │ │ │ │ where not exists (…) │ │ │ └───────────────────────────────────────────────────┘ │ │ │ │ Then re-run the job with `--from`. │ │ │ │ [ edit ] │ └─────────────────────────────────────────────────────────┘ ``` The button sits in the bottom right of the pane and says `edit`; in edit mode it stays in the same place and says `view`. One control, one position, two labels — a control that moves when you press it costs people the second press. Mode is per **view**, not per document. Splitting a markdown document to read it beside its own source is a thing people do, and it is free if mode is a view property. It rides on the Dockview panel's parameters — which are already serialised with the layout — rather than a store of its own, so a tab you left in edit mode reopens in edit mode with no new persistence to keep in sync. A `sql` document has no button and no mode. Nothing in the pane moves for the language that does not need it. ### Rendering `marked`, configured with GitHub-flavoured markdown, no raw HTML, and no external resource loading: - **HTML is escaped, not sanitized.** A ``, `` and an `