me0nline/LOGAR
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity
Files
c5d364b289b1041a7e334573df3f4508df396492
LOGAR/README.md
T

11 KiB
Raw Blame History

LOGAR: Edge-Thin Log Analysis & Temporal Verification System

LOGAR is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies OpenPGP encryption, authenticated TCP streaming, temporal persistence tracking across 12-hour evaluation windows, and an automated 4-run rule to filter out transient infrastructure blips before reporting verified anomalies to Hermes.

Table of Contents

  1. Core Philosophy
  2. Architecture & Data Flow
  3. Security & Cryptographic Model
  4. Cloud-Side Temporal Persistence & 4-Run Rule
  5. Agentic Hermes Integration
  6. Dynamic Machine & Domain Identification
  7. Repository & Shippables Structure
  8. Getting Started & Installation
  9. Running Tests

Core Philosophy

1. Edge Thinness & Zero State

Site agents running on Windows and Linux act strictly as lightweight forwarders:

  • No Local Database: Clients maintain zero state and no local SQLite or cache files.
  • Source-Level Noise Stripping: Conversational, informational, and debugging log noise (INFO, DEBUG, audit entries) is dropped directly at the source.
  • End-to-End Encryption: Logs are encrypted using the server's OpenPGP public key before leaving the edge node.
  • Secure TCP Sockets: Ingestion occurs over low-overhead authenticated TCP sockets rather than bulky HTTP/HTTPS endpoints.

2. Cloud-Side Temporal Persistence

The central Python/TCP hub handles the heavy lifting:

  • State tracking is managed centrally in SQLite (logar_state.db).
  • Candidate issues are evaluated over a 12-hour temporal evaluation window.
  • An issue must persist across at least 4 consecutive runs / cycles to be confirmed as a genuine system anomaly. Transient blips and sporadic spikes are filtered out automatically.

3. Agentic Integration with Hermes

Instead of human engineers manually diving through noisy logs, Hermes ingests pre-filtered, 4-run validated anomalies directly from the cloud hub (GET /api/hermes/report), treating them as verified system artifacts to trigger precise team notifications.

Architecture & Data Flow

graph TB
    subgraph Edge Nodes [Zero-State Edge Forwarders]
        W[Win_Client.py / Win_Client.exe<br/>Windows Event Log Application]
        L[Linux_Client.py / Linux_Client.bin<br/>systemd journalctl -p warning]
    end

    subgraph Security Layer [Security & Framing]
        E[OpenPGP Payload Encryption<br/>Server Public Key & Fingerprint]
        S[Length-Prefixed Framing<br/>4-byte Big-Endian + Auth Envelope]
    end

    subgraph Cloud Hub [LOGAR Central Server Hub]
        TCP[Authenticated TCP Listener<br/>Port 9443]
        DEC[OpenPGP Decryption<br/>Server Private Key]
        DB[(SQLite Persistence<br/>active_issues & ingest_runs)]
        RULE{12h Window &<br/>4-Run Rule}
    end

    subgraph Agentic Reporting [Downstream Integration]
        API[FastAPI / Uvicorn Reporting<br/>Port 8443]
        HERMES[Hermes Agent<br/>GET /api/hermes/report]
    end

    W --> E
    L --> E
    E --> S
    S -->|TCP Stream| TCP
    TCP --> DEC
    DEC --> RULE
    RULE --> DB
    DB --> API
    API --> HERMES

Security & Cryptographic Model

Pure-Python OpenPGP (RFC 4880)

  • Zero OS Binary Dependency: Utilizes pgpy and cryptography in pure Python. No native GnuPG or gpg binary installation is required on the server, Windows nodes, or Linux nodes.
  • First-Run Automatic Key Generation: On the first launch, if server_config.json is missing, Server.py automatically generates:
    • An OpenPGP RSA 2048 keypair with encryption-only usage flags.
    • An armored private key (private_key) and public key (public_key).
    • A SHA-256 public encryption fingerprint (server_fingerprint).
    • A cryptographically random authentication secret token (auth_token).
  • Client Configuration Exporter: 
    python Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
    Produces an anonymous client config containing only the server socket coordinates, authentication token, and the encryption-only public key & fingerprint.
  • Socket Protocol Framing:
    • [4 bytes big-endian unsigned int] : Total envelope length.
    • [JSON Envelope] : 
      {
  "auth_token": "<SECRET_TOKEN>",
  "timestamp": "2026-09-03T...",
  "encrypted_payload": "-----BEGIN PGP MESSAGE-----\n..."
}
    • Unauthorized clients or invalid authentication tokens are rejected immediately.

Cloud-Side Temporal Persistence & 4-Run Rule

Incoming candidate logs are tracked in SQLite table active_issues:

  • Issue Fingerprint: Formatted as {site_name}:{server}:{signature}.
  • 12-Hour Evaluation Window:
    • When an issue is observed, the hub compares (now - last_seen).
    • If more than 12 hours have passed since the issue was last recorded, the previous window is expired and the cycle resets to run_count = 1 with status TRANSIENT.
  • 4-Run Rule:
    • For each distinct run batch, run_count increments.
    • Issues with run_count < 4 are marked as TRANSIENT and ignored by downstream reporting.
    • When run_count >= 4 within the active 12-hour window, the status transitions to VERIFIED.

Agentic Hermes Integration

The server hub serves a REST reporting API (default port 8443):

GET /api/hermes/report

Returns exclusively verified anomalies that have satisfied the 4-run rule within the active 12-hour evaluation window:

[
  {
    "fingerprint": "corp.internal:web-app-01.corp.internal:NginxWorkerCrash",
    "site": "corp.internal",
    "server": "web-app-01.corp.internal",
    "signature": "NginxWorkerCrash",
    "severity": "ERROR",
    "message": "Worker process 4120 terminated with signal 11",
    "os_type": "linux",
    "first_seen": "2026-09-03T09:00:00+00:00",
    "last_seen": "2026-09-03T21:00:00+00:00",
    "consecutive_runs": 4,
    "evaluation_window": "12h",
    "verified": true,
    "status": "VERIFIED"
  }
]

GET /api/hermes/all

Diagnostic endpoint listing all active issues (both TRANSIENT candidate blips and VERIFIED anomalies).

GET /health

Returns hub health, encryption fingerprint, and listener ports.

Dynamic Machine & Domain Identification

Client configurations intentionally contain no machine name or site name. Both forwarders dynamically identify their host and domain at runtime via get_machine_identifier():

  1. Fully Qualified Domain Name (FQDN): Checked via socket.getfqdn().
  2. OS-Specific Domain Discovery:
    • Windows: Checks Active Directory environment variable USERDNSDOMAIN / USERDOMAIN.
    • Linux: Parses /etc/resolv.conf domain and search directives.
  3. Reverse DNS Lookup: Resolves canonical hostname via socket.gethostbyaddr.
  4. Fallback: Local hostname socket.gethostname().

The server automatically infers site attribution from domain qualifiers (e.g. node01.corp.internal \rightarrow site corp.internal).

Repository & Shippables Structure

LOGAR/
├── .gitignore                      # Ignore venv, caches, DBs, and private keys
├── requirements.txt                # Unified dependencies
├── README.md                       # Comprehensive documentation
├── Server.py                       # Central TCP server and Hermes API
├── Win_Client.py                   # Windows edge forwarder
├── Linux_Client.py                 # Linux edge forwarder
├── test_pipeline.py                # End-to-end integration test
└── out/                            # Standalone shippable distributions
    ├── server/
    │   ├── Server.exe              # Standalone Windows executable
    │   ├── Server.py               # Python source
    │   ├── server_config.sample.json
    │   ├── requirements.txt
    │   ├── README.md
    │   └── test/
    │       └── test_server.py      # Server unit tests
    ├── win_client/
    │   ├── Win_Client.exe          # Standalone Windows executable
    │   ├── Win_Client.py           # Python source
    │   ├── client_config.sample.json
    │   ├── requirements.txt
    │   ├── README.md
    │   └── test/
    │       └── test_win_client.py  # Windows client unit tests
    └── linux_client/
        ├── Linux_Client.bin        # Standalone executable binary (zipapp)
        ├── build_bin.sh            # PyInstaller ELF compiler script
        ├── Linux_Client.py         # Python source
        ├── client_config.sample.json
        ├── requirements.txt
        ├── README.md
        └── test/
            └── test_linux_client.py# Linux client unit tests

Getting Started & Installation

1. Central Server Hub

  1. Install dependencies: 
    pip install -r requirements.txt
  2. Start the server (generates server_config.json and keypair on first run): 
    python Server.py
# Or run the standalone executable:
./out/server/Server.exe
  3. Export a client configuration: 
    python Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json

2. Windows Client Deployment

  1. Copy Win_Client.exe (or Win_Client.py) and client_config.json to the target machine.
  2. Run manually or schedule via Task Scheduler (every 3 hours): 
    Win_Client.exe --hours 6

3. Linux Client Deployment

  1. Copy Linux_Client.bin (or Linux_Client.py) and client_config.json to /opt/logar/.
  2. Ensure executable permissions: 
    chmod +x /opt/logar/Linux_Client.bin
  3. Run via cron or systemd timer: 
    0 */3 * * * /opt/logar/Linux_Client.bin --hours 6

Running Tests

1. Component-Specific Unit Tests

Each component in out/ includes its own isolated test suite:

# Server tests (config generation, SQLite persistence, 4-run rule)
python out/server/test/test_server.py

# Windows client tests (config anonymity, machine ID, OpenPGP encryption)
python out/win_client/test/test_win_client.py

# Linux client tests (config anonymity, journalctl priority filter, OpenPGP)
python out/linux_client/test/test_linux_client.py

2. End-to-End Pipeline Integration Test

Start the server in one shell and run the pipeline test:

python test_pipeline.py

This tests invalid token rejection, encrypted socket streaming, database persistence, status promotion upon the 4th run, and the Hermes API output.

Reference in New Issue View Git Blame Copy Permalink