🔍 DiskTracker

Created by Pratham Parikh

DiskTracker is a high-performance, real-time Windows file-system observation daemon and an AI-driven CLI interface.

Decoupled through a log-structured state machine model, it reads NTFS USN (Update Sequence Number) change logs to track volume modifications instantly without locking your drive or overloading system resources. Equipped with an LLM-driven LangGraph orchestration agent, you can query your disk status, find large files, track file mutation history, and manage space interactively using natural language.

💡 Why DiskTracker?

Traditional disk space analyzers crawl the file system from scratch every time they run. This is slow, resource-heavy, and doesn't tell you when or how your space is growing.

DiskTracker resolves this by:

• Instant Tracking: Hooks into the NTFS USN Journal to catch every file creation, deletion, renaming, or modification in microseconds.
• Append-Only Performance: Real-time events are appended to a fast SQLite WAL-mode log (mutation_log) and processed by a background drain engine, preventing database locks and I/O bottlenecks.
• Conversational Control: Integrates an AI agent (disktracker ask) utilizing LangGraph to execute reads, diagnostics, and pruning actions safely under human approval.

🏗️ System Architecture

DiskTracker uses a log-structured state machine model to separate fast write paths from relational analysis:

graph TB
    subgraph Windows Kernel
        USN["NTFS USN Journal"]
    end

    subgraph "Background Daemon (disktracker daemon)"
        Watcher["USN Watcher Thread"] -->|Append-Only| MutLog[("SQLite mutation_log")]
        Scanner["Baseline Scanner"] -->|Bulk Inserts Fact-Tree| Facts[("SQLite facts DB")]
        
        Drain["Drain Engine"] -->|1. Polling Log| MutLog
        Drain -->|2. Resolve file size/meta| USN
        Drain -->|3. Merge UPSERT/DELETE| Facts

        IPC["JSON-RPC Named Pipe Server"]
    end

    subgraph "CLI Tool (disktracker CLI)"
        User["User Interface"] --> CLI["CLI Core Command Dispatch"]
        CLI --> Client["Named Pipe Client Client"]
        Client -->|Named Pipe: \\.\pipe\disktracker| IPC
        
        subgraph "AI Subsystem (disktracker ask)"
            Agent["LangGraph Agent Graph"] -->|Tool Execution| CLI
            Agent -->|ChatModel Engine| LLM["OpenAI / OpenRouter API"]
        end
    end

1. Baseline Scanner: Walks the filesystem on startup (using fast Win32 directory APIs) to index initial facts.
2. USN Watcher Thread: Streams NTFS change events in real-time, instantly logging them into SQLite WAL.
3. Drain Engine: Replays mutation logs, queries file metadata, and merges state into the facts table (transitions from BaselineScanning -> Reconciling -> Live).
4. AI Subsystem: Combines a state graph with 12 built-in CLI and DB tools to allow conversational system diagnostics and file pruning.

📦 Installation

DiskTracker is built for Windows (x64 and ARM64). You can install it through several package managers or a standalone script.

Method 1: Standalone Powershell (Recommended)

Open Command Prompt or PowerShell as Administrator and run:

# Via PowerShell (Admin)
irm https://raw.githubusercontent.com/pratham15541/disktracker/main/install.ps1 | iex

Method 2: NPM Package

npm install -g disktracker

Method 3: Chocolatey

choco install disktracker

Method 4: WinGet

winget install pratham15541.disktracker

🚀 CLI Usage Overview

Below is the quick reference card for DiskTracker commands.

For detailed parameters and flags for each subcommand, read the CLI Command Reference Card.

🤖 The AI Assistant (disktracker ask)

DiskTracker embeds an AI agent powered by LangGraph which operates as a ReAct (Reasoning and Acting) state machine. It is equipped with 12 tools to query and modify system state:

               [User Input]
                    │
                    ▼
          ┌───────────────────┐
          │    Agent Node     │◄───────────────────┐
          └───────────────────┘                    │
                    │                              │
           [Tools Condition Check]                 │
                    │                              │
           Is tool needed?                         │
             /         \                           │
          (Yes)        (No)                        │
           /             \                         │
          ▼               ▼                        │
   ┌─────────────┐   ┌───────────────┐             │
   │ Tools Node  │   │ Final Answer  │             │
   └─────────────┘   └───────────────┘             │
          │                                        │
     [Runs Tool] ──────────────────────────────────┘

The agent is protected by Human-In-The-Loop (HITL) gates. If the AI tries to run a mutating command (such as deleting files or altering snapshots), the CLI pauses, displays the exact script to run, and asks for your explicit permission (y/n).

To read more about the agent tools, prompt structures, and configuration, see AI Agent Guide.

🎯 Next Goals

We plan to expand DiskTracker along these key vectors:

1. Testing Suite Expansion:
   • Write comprehensive E2E tests for the named pipe JSON-RPC interface.
   • Mock NTFS USN Journal logs to test edge cases in our state reducer / drain engine.
   • Run clippy, formatting, and unit tests under GitHub Actions (ci.yml) on every pull request.
2. Cross-Platform Support:
   • Extend the core platform traits with platform-macos utilizing Apple's FSEvents API.
   • Add platform-linux utilizing the Linux inotify system.
3. GUI Desktop Client:
   • Develop a modern, lightweight desktop interface built with Tauri and React.
   • Provide interactive sunburst charts, real-time file modification feeds, and visual timeline analysis.

DiskTracker — System Architecture & Design

DiskTracker is a high-performance, real-time Windows file-system observation utility. It decouples fast filesystem events from slower relational database index updates using a Log-Structured State Machine model.

1. Core Decoupling Model (Log-Structured State Machine)

To track every file write on a system without causing database lockouts or high I/O latency, DiskTracker separates the Write Path (streaming filesystem events) from the State Reduction Path (computing the factual current layout of the disk).

graph TD
    A["NTFS USN Journal"] -->|Read events| B("Watcher Thread")
    B -->|Append-only| C[("mutation_log table")]
    
    D["Baseline Scanner"] -->|Bulk inserts| E[("facts table")]
    
    C -->|Poll mutations| F("Drain Engine")
    F -->|get_file_size_by_id| A
    F -->|UPSERT / DELETE| E

• Why decouple? Direct updates (e.g. executing UPDATE facts SET size = N WHERE file_id = X) for every file operation in real-time would create massive lock contention on SQLite, blocking reads and slowing down the OS.
• The Solution:
  • The Watcher streams USN records and executes simple, unindexed, append-only INSERT INTO mutation_log statements. This is extremely fast (sub-millisecond) and operates lock-free in SQLite WAL (Write-Ahead Log) mode.
  • The Drain Engine runs as a background worker task, pulling batches from mutation_log periodically, resolving file metadata (sizes/paths), and applying consolidated updates (UPSERT / DELETE) to the facts projection table.

2. Watched/Observation Layer & NTFS USN Algorithm

The Windows observation layer binds to the native NTFS Change Journal. The NTFS file system maintains an internal database called the USN (Update Sequence Number) Journal. Every change to a file or directory is recorded here with a unique sequence number.

A. Journal Reading Algorithm

DiskTracker opens the volume handle (e.g., \\.\C:) with GENERIC_READ | GENERIC_WRITE permissions and queries/reads journal records using Win32 DeviceIoControl APIs:

1. Query Journal: Uses control code FSCTL_QUERY_USN_JOURNAL to get the current journal parameters (e.g., NextUsn, FirstUsn, MaxUsn).
2. Read Records: Uses control code FSCTL_READ_USN_JOURNAL in a polling/waiting loop. It passes a READ_USN_JOURNAL_DATA_V0 configuration struct specifying the starting USN cursor (StartUsn), reason masks, and the journal ID.

B. USN Record Parsing

The memory block returned by FSCTL_READ_USN_JOURNAL is aligned and parsed into native USN_RECORD_V2 (or V3) structures:

C. Mapping USN Reason Flags

The watcher maps NTFS reason bitflags to DiskTracker mutations:

• USN_REASON_FILE_CREATE ──► Created
• USN_REASON_DATA_OVERWRITE | USN_REASON_DATA_EXTEND | USN_REASON_BASIC_INFO_CHANGE ──► Modified
• USN_REASON_RENAME_OLD_NAME | USN_REASON_RENAME_NEW_NAME ──► Renamed
• USN_REASON_FILE_DELETE | USN_REASON_CLOSE ──► Deleted

3. The Drain Engine & State Reduction Algorithm

Decoupling introduces a classic concurrency challenge: how to initialize the database with a baseline scan while concurrently capturing real-time updates? DiskTracker solves this through a State Gating Synchronization Protocol.

A. Lifecycle State Transitions

Each volume progress moves through three phases:

stateDiagram-v2
    [*] --> BaselineScanning : Crawler starts baseline indexing
    BaselineScanning --> Reconciling : Scanner completes
    Reconciling --> Live : Replayed all overlap mutations
    Live --> [*]

1. BaselineScanning (State 1):
   • The Watcher starts immediately. It queries the current NTFS USN cursor (startup_usn) and starts streaming real-time writes into mutation_log.
   • Concurrently, the Scanner begins walking the directory tree, writing the initial file state into the facts table.
   • The Drain Engine is completely blocked (gated) during this crawl to prevent it from updating file sizes/paths in the database that the Scanner has not yet indexed.
2. Reconciling (State 2):
   • The Scanner completes and records its end timestamp.
   • The Drain Engine wakes up. It reads the drain_state (which records the sequence cursor) and begins processing all entries in the mutation_log where sequence > last_sequence.
   • This phase replays all modifications that occurred during the scanner crawl (the overlap window), ensuring that any new modifications override old baseline values.
3. Live (State 3):
   • Once the Drain Engine has processed all logs up to the latest sequence, it transitions to Live.
   • It polls the mutation_log every 500ms, immediately applying incoming file edits to the facts database.

B. State Reduction Math

For each processed mutation log item, the Drain Engine performs a state reduction on the facts projection:

• Created / Modified:
  • Open file handle by 64-bit NTFS identifier (OpenFileById) using platform_windows::open_file_by_id. This is incredibly fast because it accesses the file directly through NTFS index lookups, bypassing slow string path resolutions.
  • Query file size and modification timestamps from the handle.
  • Execute an UPSERT on the facts table:

    INSERT INTO facts (volume, file_id, parent_file_id, name, is_directory, size, created_at, modified_at)
    VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)
    ON CONFLICT(volume, file_id) DO UPDATE SET
      parent_file_id = excluded.parent_file_id,
      name = excluded.name,
      size = excluded.size,
      modified_at = excluded.modified_at;
    

• Renamed:
  • Update name and parent_file_id for the matching file_id.
• Deleted:
  • Run a recursive delete to remove the file and any child directories:

    DELETE FROM facts WHERE volume = ?1 AND file_id = ?2;
    

4. Named Pipe IPC Architecture

The CLI frontend interacts with the background daemon over Windows Named Pipes:

• Endpoint: \\.\pipe\disktracker
• Protocol: JSON-RPC 2.0.
• Server Implementation: Binds using Win32 named pipe APIs (CreateNamedPipeW, ConnectNamedPipe). Spawns a dedicated connection handler thread per connected CLI client.
• Client Implementation: CLI commands send requests (e.g. {"jsonrpc": "2.0", "method": "status", "id": 1}) and read structured responses.

5. AI Reasoning & Graph Tools

The AI assistant utilizes a LangGraph orchestrator to solve questions using the ReAct (Reasoning and Acting) loop pattern.

A. Graph Loop Optimization

Instead of passing raw directories list to the LLM (which would exceed token limits), the agent uses specialized local system tools:

• disktracker_top (Size & Growth rollups): Solves the space analysis problem using a recursive SQL rollup query.

  -- High-speed recursive CTE to calculate directory sizing
  WITH RECURSIVE FolderHierarchy(file_id, size) AS (
      SELECT parent_file_id, size FROM facts WHERE is_directory = 0
      UNION ALL
      SELECT f.parent_file_id, fh.size
      FROM facts f
      JOIN FolderHierarchy fh ON f.file_id = fh.file_id
      WHERE f.is_directory = 1 AND f.parent_file_id != 0
  )
  SELECT file_id, SUM(size) as total_size FROM FolderHierarchy GROUP BY file_id;
  

• disktracker_search: Utilizes SQLite indexes or Tantivy search indices to solve substring filename lookups instantly.
• sqlite_read_query: Lets the LLM execute arbitrary SELECT queries to find specific files, installations, or logs using standard SQL patterns.

6. SQLite Performance Tuning (WAL Engine)

SQLite is tuned for maximum concurrent performance:

• WAL Mode (journal_mode = WAL): Enables readers (CLI search/status) to query the database concurrently without blocking or being blocked by the writer (Drain Engine / Watcher).
• Normal Synchronization (synchronous = NORMAL): The database engine syncs logs to disk at critical checkpoints, reducing disk write wear and maximizing write speed.
• High Cache Size (cache_size = -64000): Keeps 64MB of database pages in-memory to cache queries and avoid filesystem read calls.

7. Algorithms & Core Logic

DiskTracker resolves core filesystem challenges using specialized, high-performance algorithms:

A. Variable-Length USN Memory Parsing

Because NTFS USN records vary in length due to custom filename lengths, the Watcher streams the change journal into a raw memory buffer and walks it using record offset math: $$\text{offset}{next} = \text{offset}{current} + \text{record}.\text{RecordLength}$$ The watcher uses bitwise reason masks to ignore intermediate updates, processing the file changes only when the file handles are closed:

#![allow(unused)]
fn main() {
if (reason & USN_REASON_CLOSE) != 0 {
    // Process the final state change
}
}

B. Directory Crawler (Scanner) & Transaction Writing

To index large drives under a minute without blocking the system, the Scanner crawls directories using Win32 batch walking APIs (FindFirstFileW/FindNextFileW) and aggregates write queries:

• Batching Transactions: Instead of executing an SQL insert for every single file found, the crawler accumulates files and writes them in batches of 1,000 files inside a single transaction (BEGIN TRANSACTION ... COMMIT). This reduces disk sync write commits from 1,000 to 1, speeding up baseline crawlers by up to 25x.

C. Overlap Change Reconciliation

If a file is scanned at $T_1$, and a user writes to that same file at $T_2$ (while the scanner is still crawling the rest of the disk), the baseline index values could overwrite the new change. DiskTracker resolves this overlap:

1. The Watcher records the write event at $T_2$ in the mutation_log table.
2. Once the crawler is complete, the Drain Engine reconciles the state.
3. For every log item, it opens the file directly via its 64-bit NTFS index identifier (OpenFileById). This is an $O(1)$ Win32 pointer call that bypasses slow string parent-path resolution.
4. It queries the current size and modifies the facts table using UPSERT, ensuring that the final database state reflects the most up-to-date physical size on the disk.

D. Recursive CTE Size Rollups

Aggregating the sizing sums of deeply nested directories in relational databases is computationally expensive. DiskTracker implements a leaf-to-root recursive Common Table Expression (CTE) to sum up directories sizes:

1. It selects all files where is_directory = 0 as the anchor.
2. It recursively queries up the directory tree by matching child parent IDs to their parent folders.
3. It groups the results by parent folder ID, executing high-speed memory sums that compute complete drive hierarchies in milliseconds.

🚀 DiskTracker CLI Subcommand Reference Card

This reference card details every subcommand, parameter, flag, and usage example in DiskTracker.

1. Core Daemon Lifecycle Management

disktracker init

Starts or registers the DiskTracker background daemon as a Windows Service and initiates baseline scanning.

• Usage: disktracker init
• Admin privileges required: Yes.

disktracker status

Queries the running background daemon's state and monitored volumes.

• Usage: disktracker status
• Parameters:
  • --json: Returns raw JSON (useful for tooling/scripts) instead of human-readable text.

disktracker doctor

Runs health diagnostics on the local installation, including checks for Admin permissions, NTFS Journal access, SQLite integrity, and database WAL-mode config.

• Usage: disktracker doctor
• Admin privileges required: Yes.

disktracker uninstall

Stops the running daemon, terminates the service registry, removes the named pipe, and cleans up database files.

• Usage: disktracker uninstall [flags]
• Admin privileges required: Yes.
• Flags:
  • --delete-snapshot: Deletes all historical snapshot index files as well (default keeping snapshots).
  • -y, --yes: Auto-approves the confirmation prompt.

2. Windows Service Control

disktracker service <subcommand>

Direct Win32 service wrapper operations.

• Admin privileges required: Yes.
• Subcommands:
  • register: Registers DiskTracker as a Windows Service named DiskTracker.
  • unregister: Stops and unregisters the DiskTracker service.
  • start: Starts the registered Windows Service.
  • stop: Stops the registered Windows Service.

3. Database Search and Statistics

disktracker search [query]

Performs ultra-fast substring searches on indexed filenames. By default, it runs an exact case-insensitive substring search.

• Usage: disktracker search "log" [options]
• Options:
  • --advanced (alias --advance): Enable multi-tier scoring (phrase, prefix, ngram, fuzzy matching).
  • --fuzzy: Enable fuzzy matching scoring tier.
  • --path <path>: Filter files by folder path prefix (e.g. C:\Users).
  • --ext <extension>: Filter by exact file extension (e.g. pdf).
  • --volume <volume>: Filter by volume designation (e.g. C:).
  • --min-size <bytes>: Filter files larger than N bytes.
  • --max-size <bytes>: Filter files smaller than N bytes.
  • --modified-after <duration/datetime>: Filter files modified after a relative duration (e.g., 2d, 12h) or absolute UTC datetime (RFC3339).
  • --modified-before <duration/datetime>: Filter files modified before a relative duration or UTC datetime.
  • --hidden: Filter hidden files.
  • --system: Filter system files.
  • --limit <limit>: Maximum results to return (default: 100).
  • --json: Outputs raw JSON results.

disktracker history [path]

Displays the mutation timeline of a folder or file.

• Display rules: Created/Deleted/Renamed always shown (including 0B). Modified only when |size_delta| >= 2. Consecutive identical Created/Deleted/Renamed rows for the same file are aggregated; Modified rows stay unique.
• Usage: disktracker history [path] [options]
• Options:
  • --since <duration/datetime>: Filter events after a relative time (e.g., 2d, 12h) or UTC datetime.
  • --until <duration/datetime>: Filter events before a relative time or UTC datetime.
  • --kind <Created/Modified/Deleted/Renamed>: Filter by change action.
  • --collapse: Also merges consecutive same-file same-kind rows (including Modified size deltas). Identical Created/Deleted/Renamed rows are already aggregated without this flag.
  • --limit <limit>: Max entries to return (default: 100).
  • --json: Outputs raw JSON results.

disktracker top

Ranks files and directories by size, growth, or modifications.

• Display rules (growth mode): 0B create/delete events can appear; tiny non-zero growth under 2B is hidden.
• Usage: disktracker top [options]
• Options:
  • --path <path>: Restricts ranking to a specific folder path.
  • --volume <volume>: Restricts ranking to a specific volume (e.g. C:).
  • --folders: Group results by folder rollup sizes (conflicts with --files).
  • --files: Flat listing of largest files (conflicts with --folders).
  • --since <duration/datetime>: Filter growth or churn since a relative duration.
  • --between <SNAP_A> <SNAP_B>: Compare growth/churn delta between two snapshots.
  • --growth: Rank results by size delta (default for interval mode).
  • --churn: Rank results by modification frequency.
  • --limit <limit>: Max items to display (default: 20).

4. Snapshot Management

disktracker snapshot <subcommand>

Manage and inspect volume snapshots.

• Subcommands:
  • create [--label <name>] [path]: Takes a snapshot index of the specified volume or folder with a custom label (auto-generated if omitted).
  • list [--volume <vol>]: Lists all saved snapshots in the system.
  • diff <snapshot_a> <snapshot_b> [--path <prefix>]: Computes the net mutations and size differences between two snapshots. Created/Deleted/Renamed always shown (including 0B); Modified only when |size_delta| >= 2.

5. Conversational AI Agent

disktracker ai <subcommand>

Configure the AI model bindings.

• Subcommands:
  • config [options]:
    • --base-url <url>: Set the LLM endpoint base URL.
    • --api-key <token>: Commit the LLM API token to the Windows Credential Manager.
    • --model <name>: Define the chat model name (e.g., gpt-4o, meta-llama/llama-3-70b).
    • --chat-session-store <true/false>: Choose whether to save AI conversation histories.
    • --websearch-provider <name>: Configure web search provider (e.g. duckduckgo, google).
    • --websearch-key <key>: Commit the Web Search API key to Credential Manager.
  • test: Runs a structural ping handshake test with the configured LLM API.
  • session <list/show>: Manages saved conversation session history.

disktracker ask "<question>"

Invokes the LangGraph agent to inspect the disk state using natural language.

• Usage: disktracker ask "Which folders in my AppData grew the most yesterday?"
• Options:
  • -i, --interactive: Enable interactive Action mode. Allows the AI agent to propose file system mutations (like deleting caches or temporary log files) under human-in-the-loop (HITL) approval gates.
  • --session <id>: Resume a previous conversation session.

6. Self-Update Utility

disktracker update

Checks the GitHub repository for the latest release, downloads the appropriate target archive, stops any running background daemons/services, performs a zero-downtime binary swap, restarts the daemon, and cleans up old temporary binaries.

• Usage: disktracker update
• Admin privileges required: Yes (on Windows if installed in program files or registered as a Windows Service).

🤖 DiskTracker AI Orchestrator & LangGraph Integration

DiskTracker integrates an advanced, local LLM-driven AI agent that can converse about your file system state. Built using LangGraph, it enables structured reasoning (via LLMs) combined with deterministic tool execution.

1. Graph State Machine Architecture

The AI subsystem operates as a ReAct (Reasoning and Acting) state machine. It coordinates conversational turns and tool invocations:

stateDiagram-v2
    [*] --> AgentNode : User Prompt
    AgentNode --> RouteCondition : Check response
    RouteCondition --> ToolsNode : If LLM requests tool call
    ToolsNode --> AgentNode : Returns tool execution output
    RouteCondition --> FinalAnswer : If LLM has completed response
    FinalAnswer --> [*]

1. Agent Node: Formulates a plan based on the conversation history and prompt. It decides whether it has enough information to answer or if it needs to query the system using a tool.
2. Conditional Routing: Inspects the assistant's message. If tool calls are requested, it routes execution to the Tools Node. Otherwise, it outputs the final response.
3. Tools Node: Resolves tool parameters, executes the requested system or database tool locally, appends the tool results as a ToolMessage to the graph state, and loops back to the Agent Node.

2. Human-in-the-Loop (HITL) Safety Gates

For safety, tools are classified into Passive (Read-Only) and Active (Mutating):

• Passive Tools (e.g. searching the database, checking disk space, running diagnostic commands) execute immediately and transparently.
• Active Tools (e.g. deleting files, compressing snapshots, altering configurations) require elevated permissions.
  • If the agent attempts a mutating action, the execution pauses.
  • The CLI displays the exact command to be executed.
  • The user is prompted for approval: Do you want to run this command? (y/n).
  • If rejected, a warning is returned to the LLM, and the operation is aborted.

3. Built-In Local System Tools

The agent has access to 12 specialized tools to query or interact with your system:

4. Configuration and Credentials

Credentials and API keys are stored securely using Windows Credential Manager via the keyring crate. They are never saved in plain text files.

Set up OpenAI/OpenRouter connection:

# Configure endpoint base URL
disktracker ai config --base-url "https://openrouter.ai/api/v1"

# Set model name
disktracker ai config --model "meta-llama/llama-3-70b-instruct"

# Commit API token securely
disktracker ai config --api-key "your-api-key-here"

Test API Handshake

Verify that DiskTracker can communicate with your LLM provider:

disktracker ai test

Conversational Session History

You can manage and view previous conversational sessions:

# List all saved sessions
disktracker ai session list

# Show the full chat transcript of a session
disktracker ai session show <session-id>

Release & Publishing Guide