94 Commits
Author SHA1 Message Date
me0nline f5ff8ab6cc ci(windows): use native PowerShell git checkout to eliminate Node.js dependency on Windows runner
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m45s
Release Binaries & Installers / Build & Release Windows Binaries & Installers (push) Failing after 16s
Release Binaries & Installers / Build & Release Linux Binaries (push) Successful in 2m8s
2026-09-04 23:54:13 +02:00
me0nline fd6da560ed docs: update release tag instructions for v2.0.1 in root README.md
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 2m15s
2026-09-04 23:31:44 +02:00
me0nline 956dfc5d93 docs(release): publish release notes for LOGAR v2.0.1 2026-09-04 23:31:29 +02:00
me0nline 35e6a8df3e ci: bump default release tag to v2.0.1 in release.yml 2026-09-04 23:31:15 +02:00
me0nline c121291dda build(installer): bump server AppVersion to 2.0.1 in installer_server.iss 2026-09-04 23:31:03 +02:00
me0nline d19bbb2ec1 build(installer): bump client AppVersion to 2.0.1 in installer_client.iss 2026-09-04 23:30:50 +02:00
me0nline e6dc82ff55 chore(server): bump server version to 2.0.1 2026-09-04 23:30:36 +02:00
me0nline 56d8516427 docs: update README.md to reference consolidated release.yml workflow
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 2m24s
2026-09-04 23:26:05 +02:00
me0nline 94ad3f9461 ci: remove obsolete release-windows.yml in favor of consolidated release.yml 2026-09-04 23:25:38 +02:00
me0nline 26a4509429 ci: remove obsolete release-linux.yml in favor of consolidated release.yml 2026-09-04 23:25:35 +02:00
me0nline 80ae42088f ci: add consolidated release.yml combining Linux and Windows release jobs 2026-09-04 23:25:32 +02:00
me0nline 42f01addca docs: document 25 unit tests and tripartite release options in root README.md
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 2m21s
2026-09-04 23:23:08 +02:00
me0nline a9f096ef1b ci(windows): optimize tag triggers and add default v2.0.0 tag to release-windows.yml 2026-09-04 23:22:52 +02:00
me0nline 26d3d59812 ci(linux): optimize tag triggers and add default v2.0.0 tag to release-linux.yml 2026-09-04 23:22:38 +02:00
me0nline 5fec32327f ci: prevent ci.yml from triggering on tag pushes
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 2m22s
2026-09-04 23:08:19 +02:00
me0nline 478807d873 ci(linux): add v* tag pattern to release-linux.yml 2026-09-04 23:08:06 +02:00
me0nline 149ba6dfec ci(windows): add release event and multi-pattern tag triggers to release-windows.yml 2026-09-04 23:07:54 +02:00
me0nline a11b05f0a9 docs: update release tag instructions for v2.0.0 in root README.md
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 2m21s
2026-09-04 22:57:27 +02:00
me0nline 722d2a1fec docs(release): publish release notes for LOGAR v2.0.0 2026-09-04 22:57:17 +02:00
me0nline 5883b78822 build(installer): bump server AppVersion to 2.0.0 in installer_server.iss 2026-09-04 22:57:02 +02:00
me0nline d79de301bf build(installer): bump client AppVersion to 2.0.0 in installer_client.iss 2026-09-04 22:56:53 +02:00
me0nline 184fdc6bc6 docs: add certificate auto-renewal details to out/linux_client/README.md
CI Test Suite / Run Component Tests & Pipeline Verification (push) Has been cancelled
2026-09-04 22:55:09 +02:00
me0nline 4625b650e5 docs: add certificate auto-renewal details to out/win_client/README.md 2026-09-04 22:54:59 +02:00
me0nline f38bbfb540 docs: add certificate watchdog details to out/linux_server/README.md 2026-09-04 22:54:47 +02:00
me0nline 18f0286692 docs: add certificate watchdog details to out/win_server/README.md 2026-09-04 22:54:36 +02:00
me0nline c9f769ef2b docs: document certificate validity watchdog and auto-renewal in root README.md 2026-09-04 22:54:25 +02:00
me0nline c01c09ecbd test(linux_client): add test_client_certificate_validity_and_proactive_check in test_linux_client.py 2026-09-04 22:52:49 +02:00
me0nline 2afe94fb36 test(win_client): add test_client_certificate_validity_and_proactive_check in test_win_client.py 2026-09-04 22:52:34 +02:00
me0nline 4650cbcafc test(server): add test_cert_validity_and_hub_pki_renewal in test_server.py 2026-09-04 22:52:18 +02:00
me0nline 0d613b3d22 feat(linux_client): add proactive certificate expiry check and reactive self-healing in Linux_Client.py 2026-09-04 22:52:00 +02:00
me0nline 916d3764a2 feat(win_client): add proactive certificate expiry check and reactive self-healing in Win_Client.py 2026-09-04 22:51:28 +02:00
me0nline 0e7d299594 feat(server): add in-flight certificate validity watchdog and dynamic SSLContext reloading in Server.py 2026-09-04 22:50:50 +02:00
me0nline 2cff629e23 feat(pki): add certificate expiration check and auto-renewal in server_enrollment.py 2026-09-04 22:49:54 +02:00
me0nline cfc633c398 docs(linux_server): update README.md with mTLS 1.3, dynamic PKI licensing, and systemd installer script instructions
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 2m3s
2026-09-04 22:39:48 +02:00
me0nline d61e343fbe docs(win_server): update README.md with mTLS 1.3, dynamic PKI licensing, and Inno Setup installer instructions 2026-09-04 22:39:25 +02:00
me0nline 4a446a2e73 docs(linux_client): update README.md with mTLS 1.3, dynamic PKI enrollment, and systemd installer script instructions 2026-09-04 22:39:09 +02:00
me0nline b24108a788 docs(win_client): update README.md with mTLS 1.3, dynamic PKI enrollment, and Inno Setup installer instructions 2026-09-04 22:37:20 +02:00
me0nline 1c985c85c8 docs: update README.md with mTLS architecture, dynamic PKI licensing, service installers, and updated test suite
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 5m9s
2026-09-04 22:29:30 +02:00
me0nline 249b754423 test(pipeline): update end-to-end integration test for mTLS and dynamic PKI enrollment in test_pipeline.py
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 2m17s
2026-09-04 20:55:12 +02:00
me0nline 16faad063d test(linux_client): add mTLS client certificate handling test in test_linux_client.py 2026-09-04 20:55:07 +02:00
me0nline c705be57eb test(win_client): add mTLS client certificate handling test in test_win_client.py 2026-09-04 20:55:01 +02:00
me0nline 3d6a2b86d6 test(server): add PKI, enrollment, and license quota tests in test_server.py 2026-09-04 20:54:54 +02:00
me0nline 1d0845b548 ci: include src/server_enrollment.py in syntax verification step 2026-09-04 20:54:49 +02:00
me0nline 11f16ed8e1 ci(windows): update release-windows.yml to trigger on v* tags and compile Inno Setup installers 2026-09-04 20:54:40 +02:00
me0nline 8cb3089e8e feat(installer): add Inno Setup script for LOGAR Server 2026-09-04 20:54:35 +02:00
me0nline 5bbb56b4c3 feat(installer): add Inno Setup script for LOGAR Client 2026-09-04 20:54:30 +02:00
me0nline a04d9bac9f chore(deps): bundle NSSM 64-bit binary for Windows service management 2026-09-04 20:54:24 +02:00
me0nline 8813d2d865 chore(git): unignore compilation/nssm.exe in .gitignore 2026-09-04 20:54:18 +02:00
me0nline 7e9f7a56f8 feat(install): add Linux server systemd installation script 2026-09-04 20:54:12 +02:00
me0nline 99920ef395 feat(install): add Linux client systemd installation script 2026-09-04 20:54:06 +02:00
me0nline ccb23d65e5 feat(linux_client): add mTLS connection and automatic enrollment in src/Linux_Client.py 2026-09-04 20:54:00 +02:00
me0nline fb9ef6e6cb feat(win_client): add mTLS connection and automatic enrollment in src/Win_Client.py 2026-09-04 20:53:54 +02:00
me0nline 491d2b1194 feat(server): add mTLS listener, client licensing schema, and enrollment endpoint in src/Server.py 2026-09-04 20:53:48 +02:00
me0nline e83a5b3e0f feat(pki): add dynamic PKI and client certificate generation in src/server_enrollment.py 2026-09-04 20:53:42 +02:00
me0nline 0901ccb3eb Merge branch 'origin/main' into main: adopt restructured repository layout 2026-09-04 20:29:29 +02:00
me0nline cb4c763e0e Add hierarchical wildcard to tags trigger in release-windows.yml
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m41s
2026-09-04 17:54:57 +02:00
me0nline 84579d8719 Add hierarchical wildcard to tags trigger in release-linux.yml 2026-09-04 17:54:50 +02:00
me0nline 08aa4edfb1 Update ci.yml tags-ignore to ignore all tags
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m38s
2026-09-04 17:46:09 +02:00
me0nline e4f6a9295b De-hardcode workflow initiator in release-windows.yml to support release events and all tags 2026-09-04 17:46:00 +02:00
me0nline 7f6bf4442f De-hardcode workflow initiator in release-linux.yml to support release events and all tags 2026-09-04 17:45:48 +02:00
me0nline db80e7f5a2 Update README.md release examples to reference v1.0.2
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 2m16s
2026-09-04 17:34:14 +02:00
me0nline 1562285034 Update release-windows.yml default tag to v1.0.2 2026-09-04 17:34:03 +02:00
me0nline 74942d4c00 Update release-linux.yml default tag to v1.0.2 2026-09-04 17:33:57 +02:00
me0nline 874d693dac Update RELEASE_NOTES.md for v1.0.2 release 2026-09-04 17:33:51 +02:00
me0nline 15bfb4940b Update Windows server deployment guide for warning persistence and immediate error pass
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m34s
2026-09-04 16:12:04 +02:00
me0nline 364fefea47 Update Linux server deployment guide for warning persistence and immediate error pass 2026-09-04 16:11:51 +02:00
me0nline 9624935a81 Document warning persistence and immediate error pass in release notes 2026-09-04 16:11:37 +02:00
me0nline 867271e8b7 Update README to document 4-run rule for warnings and immediate pass for errors 2026-09-04 16:11:30 +02:00
me0nline 205d0cfbad Update integration pipeline to test warning persistence and error immediate pass 2026-09-04 16:10:59 +02:00
me0nline d37191e302 Update unit tests to verify 4-run rule on warnings and immediate pass on errors 2026-09-04 16:10:36 +02:00
me0nline 35a736dacb Restrict 4-run rule to warnings and pass errors immediately as verified 2026-09-04 16:10:01 +02:00
me0nline f871344da4 Update README.md to document out/linux_server and out/win_server deployment packages
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m33s
2026-09-04 16:05:04 +02:00
me0nline e1dbe32063 Add Windows server deployment guide and sample configuration to out/win_server 2026-09-04 16:04:51 +02:00
me0nline 98a227a234 Add Linux server deployment guide and sample configuration to out/linux_server 2026-09-04 16:04:20 +02:00
me0nline 355c6e1bc0 Update README.md documentation and commands to reflect compilation/ and tests/ paths
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m34s
2026-09-04 15:58:03 +02:00
me0nline 907616511b Update release-windows.yml workflow to use compilation/ directory 2026-09-04 15:57:14 +02:00
me0nline 939fa4270a Update release-linux.yml workflow to use compilation/ directory 2026-09-04 15:57:02 +02:00
me0nline 1f4bf3219b Update CI workflow with compilation/ and tests/ paths 2026-09-04 15:56:50 +02:00
me0nline df03c52a05 Update tests/test_pipeline.py to resolve paths relative to repository root and src/ directory 2026-09-04 15:56:37 +02:00
me0nline d63a623763 Update compilation/upload_release.py to resolve paths relative to repo root 2026-09-04 15:56:24 +02:00
me0nline 7ed264db5a Update compilation/package_dist.py to resolve paths relative to repo root 2026-09-04 15:56:01 +02:00
me0nline 6fec838344 Move test_pipeline.py from repository root to tests/ directory 2026-09-04 15:55:51 +02:00
me0nline 4c160924f7 Move build and release tools (package_dist.py, upload_release.py, requirements.txt) into compilation/ directory 2026-09-04 15:55:44 +02:00
me0nline 99be50ebc6 Update Server.py reference to src/Server.py in forwarder deployment READMEs
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m32s
2026-09-04 15:51:37 +02:00
me0nline 85f0d94805 Update README.md documentation and repo structure to reflect src/ directory 2026-09-04 15:51:23 +02:00
me0nline 1a18c4c079 Update CI workflow to execute server and client scripts from src/ directory 2026-09-04 15:50:34 +02:00
me0nline a770e24f26 Update test_pipeline.py to import client forwarders from src/ directory 2026-09-04 15:50:26 +02:00
me0nline f7ebc6c0a1 Update unit test suites to import components from src/ directory 2026-09-04 15:50:14 +02:00
me0nline 86649f796d Update package_dist.py to resolve source files from src/ directory 2026-09-04 15:49:52 +02:00
me0nline 78fc2ac8c5 Move source files Server.py, Win_Client.py, and Linux_Client.py into src/ directory 2026-09-04 15:49:32 +02:00
me0nline 082b839965 Configure separate Windows and Linux Gitea release workflows with dedicated SHA-256 checksums
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 3m48s
2026-09-04 15:46:35 +02:00
me0nline e7bf277fb9 Add v1.0.1 release notes documenting removal of client filter logic
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m36s
Release Binaries / release (push) Successful in 2m51s
2026-09-04 15:38:10 +02:00
me0nline 7052e68589 Implement edge filtering, state tracking, clean out/ directory, and add Gitea CI workflow
CI Test Suite / Run Component Tests & Pipeline Verification (push) Successful in 1m40s
2026-09-04 15:32:39 +02:00
me0nline 4e2242e0ae Name checksum files distinctively as SHA256SUMS-windows.txt and SHA256SUMS-linux.txt 2026-09-03 22:13:36 +02:00
48 changed files with 4759 additions and 2708 deletions
+76
View File
@@ -0,0 +1,76 @@
name: CI Test Suite
on:
push:
branches:
- '**'
tags-ignore:
- '*'
- '**'
- 'v*'
pull_request:
workflow_dispatch:
jobs:
test:
name: Run Component Tests & Pipeline Verification
if: "!startsWith(github.ref, 'refs/tags/')"
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Install System Dependencies & Python
run: |
if command -v apt-get >/dev/null 2>&1; then
apt-get update -y
apt-get install -y python3 python3-pip python3-venv curl
fi
python3 -m pip install --upgrade pip --break-system-packages || python3 -m pip install --upgrade pip || true
pip3 install -r compilation/requirements.txt --break-system-packages || pip3 install -r compilation/requirements.txt
- name: Verify Python Syntax
run: |
python3 -m py_compile src/Server.py src/server_enrollment.py src/Win_Client.py src/Linux_Client.py compilation/package_dist.py compilation/upload_release.py tests/test_pipeline.py tests/*.py
- name: Run Component Unit Tests
run: |
python3 -m unittest discover -s tests -v
- name: Run End-to-End Pipeline Integration Test
run: |
# Clean up any leftover test configs or database
rm -f server_config.json client_config.json logar_state.db client_state.json
# 1. Initialize server config and export client configuration
python3 src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
# 2. Launch LOGAR server in the background
python3 src/Server.py &
SERVER_PID=$!
echo "[*] Server launched in background with PID $SERVER_PID"
# 3. Poll Hermes health / report endpoint until server is listening
READY=0
for i in $(seq 1 20); do
if curl -s http://127.0.0.1:8443/api/hermes/report >/dev/null 2>&1; then
echo "[+] LOGAR Server is ready after ${i}s."
READY=1
break
fi
sleep 1
done
if [ $READY -ne 1 ]; then
echo "[!] Server failed to start within 20 seconds."
kill $SERVER_PID || true
exit 1
fi
# 4. Execute end-to-end integration test
python3 tests/test_pipeline.py
# 5. Cleanly terminate background server
kill $SERVER_PID || true
wait $SERVER_PID 2>/dev/null || true
echo "[+] Server stopped successfully."
+97 -9
View File
@@ -1,36 +1,124 @@
name: Release Binaries name: Release Binaries & Installers
on: on:
push: push:
tags: tags:
- 'v*' - 'v*'
workflow_dispatch: workflow_dispatch:
inputs:
tag:
description: 'Release tag to publish assets to (default: v2.0.1)'
required: false
default: 'v2.0.1'
jobs: jobs:
release: release-linux:
name: Build & Release Linux Binaries
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Checkout Code - name: Checkout Code
uses: actions/checkout@v4 uses: actions/checkout@v4
- name: Install Python and Dependencies - name: Install Python and Build Dependencies
run: | run: |
if command -v apt-get >/dev/null 2>&1; then if command -v apt-get >/dev/null 2>&1; then
apt-get update -y apt-get update -y
apt-get install -y python3 python3-pip python3-venv binutils zip apt-get install -y python3 python3-pip python3-venv binutils zip
fi fi
python3 -m pip install --upgrade pip --break-system-packages || python3 -m pip install --upgrade pip || true python3 -m pip install --upgrade pip --break-system-packages || python3 -m pip install --upgrade pip || true
pip3 install pyinstaller -r requirements.txt --break-system-packages || pip3 install pyinstaller -r requirements.txt pip3 install pyinstaller -r compilation/requirements.txt --break-system-packages || pip3 install pyinstaller -r compilation/requirements.txt
- name: Compile Standalone Binaries - name: Compile Standalone Linux Binaries
run: | run: |
python3 package_dist.py python3 compilation/package_dist.py --target linux
- name: Publish Release - name: Publish Linux Release Assets
env: env:
GITEA_TOKEN: ${{ secrets.TAG_TOKEN || github.token }} GITEA_TOKEN: ${{ secrets.TAG_TOKEN || github.token }}
GITEA_SERVER_URL: ${{ github.server_url }} GITEA_SERVER_URL: ${{ github.server_url }}
GITEA_REPOSITORY: ${{ github.repository }} GITEA_REPOSITORY: ${{ github.repository }}
GITEA_REF_NAME: ${{ github.ref_name }} GITEA_REF_NAME: ${{ inputs.tag || github.event.release.tag_name || github.ref_name }}
run: | run: |
python3 upload_release.py --skip-build python3 compilation/upload_release.py --skip-build
release-windows:
name: Build & Release Windows Binaries & Installers
# Note: Requires a registered Gitea Act Runner with label 'windows-latest'
runs-on: windows-latest
steps:
- name: Checkout Repository
shell: powershell
run: |
$server = "${{ github.server_url }}"
$token = "${{ secrets.TAG_TOKEN || github.token }}"
$repo = "${{ github.repository }}"
$cleanUrl = $server -replace "^https?://", ""
$authUrl = "https://${token}@${cleanUrl}/${repo}.git"
if (-not (Test-Path ".git")) {
git init
git remote add origin $authUrl
} else {
git remote set-url origin $authUrl
}
git fetch --depth 1 origin "${{ github.sha }}"
git checkout -f FETCH_HEAD
- name: Install Dependencies
shell: powershell
run: |
$py = "python"
if (-not (Get-Command "python" -ErrorAction SilentlyContinue)) {
if (Get-Command "py" -ErrorAction SilentlyContinue) {
$py = "py -3.12"
}
}
& $py -m pip install --upgrade pip
& $py -m pip install pyinstaller cryptography -r compilation/requirements.txt
- name: Compile Standalone Windows Binaries
shell: powershell
run: |
$py = "python"
if (-not (Get-Command "python" -ErrorAction SilentlyContinue)) {
if (Get-Command "py" -ErrorAction SilentlyContinue) {
$py = "py -3.12"
}
}
& $py compilation/package_dist.py --target windows
- name: Compile Inno Setup Installers
shell: powershell
run: |
$iscc = $null
if (Test-Path "C:\Program Files (x86)\Inno Setup 6\ISCC.exe") {
$iscc = "C:\Program Files (x86)\Inno Setup 6\ISCC.exe"
} elseif (Test-Path "C:\Program Files\Inno Setup 6\ISCC.exe") {
$iscc = "C:\Program Files\Inno Setup 6\ISCC.exe"
} elseif (Get-Command "ISCC.exe" -ErrorAction SilentlyContinue) {
$iscc = "ISCC.exe"
}
if ($iscc) {
Write-Host "[*] Compiling Windows Inno Setup installers using $iscc..."
& $iscc compilation/installer_client.iss
& $iscc compilation/installer_server.iss
} else {
Write-Host "[!] Inno Setup compiler (ISCC.exe) not found on runner host. Skipping installer compilation."
}
- name: Publish Windows Release Assets
shell: powershell
env:
GITEA_TOKEN: ${{ secrets.TAG_TOKEN || github.token }}
GITEA_SERVER_URL: ${{ github.server_url }}
GITEA_REPOSITORY: ${{ github.repository }}
GITEA_REF_NAME: ${{ inputs.tag || github.event.release.tag_name || github.ref_name }}
run: |
$py = "python"
if (-not (Get-Command "python" -ErrorAction SilentlyContinue)) {
if (Get-Command "py" -ErrorAction SilentlyContinue) {
$py = "py -3.12"
}
}
& $py compilation/upload_release.py --skip-build
+3
View File
@@ -11,6 +11,7 @@ build/
dist/ dist/
*.spec *.spec
*.exe *.exe
!compilation/nssm.exe
*.bin *.bin
*.dll *.dll
*.so *.so
@@ -20,6 +21,8 @@ dist/
*.db *.db
*.sqlite *.sqlite
*.sqlite3 *.sqlite3
client_state.json
*.tmp
# Live configuration with generated private keys & tokens (samples are tracked) # Live configuration with generated private keys & tokens (samples are tracked)
server_config.json server_config.json
-206
View File
@@ -1,206 +0,0 @@
import os
import sys
import json
import socket
import struct
import argparse
import subprocess
import warnings
from datetime import datetime, timezone
# Suppress cryptography / pgpy deprecation notices
warnings.filterwarnings("ignore")
import pgpy
CONFIG_FILE_NAME = "client_config.json"
def load_config(config_path: str = CONFIG_FILE_NAME):
if not os.path.exists(config_path):
raise FileNotFoundError(
f"Client configuration file not found at: {config_path}\n"
f"Generate one from the server using: python Server.py --create-client-config --client-out {config_path}"
)
with open(config_path, "r", encoding="utf-8") as f:
return json.load(f)
def get_machine_identifier() -> str:
"""
Returns the hostname of the machine sending the logs,
and appends the network/DNS domain if available.
"""
# 1. Try fully-qualified domain name (FQDN)
fqdn = socket.getfqdn()
if fqdn and "." in fqdn and not fqdn.startswith("localhost"):
return fqdn
hostname = socket.gethostname()
# 2. Check /etc/resolv.conf domain or search directive
try:
if os.path.exists("/etc/resolv.conf"):
with open("/etc/resolv.conf", "r", encoding="utf-8") as f:
for line in f:
parts = line.strip().split()
if parts and parts[0] in ["domain", "search"] and len(parts) > 1:
domain = parts[1]
if domain and not domain.startswith("."):
return f"{hostname}.{domain}"
except Exception:
pass
# 3. Try reverse DNS lookup
try:
host_ip = socket.gethostbyname(hostname)
canonical_name = socket.gethostbyaddr(host_ip)[0]
if canonical_name and "." in canonical_name and not canonical_name.startswith("localhost"):
return canonical_name
except Exception:
pass
return hostname
def get_recent_linux_logs(hours: int = 6):
"""
Collects warnings and errors from systemd journalctl over the lookback window.
Edge Thinness: Drops INFO and DEBUG entries at the source.
"""
cmd = ["journalctl", "--since", f"{hours} hours ago", "-p", "warning", "--output=json"]
try:
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
except FileNotFoundError:
print("[!] journalctl command not found. Ensure this script runs on a systemd-enabled Linux system.")
return []
except Exception as e:
print(f"[!] Error running journalctl: {e}")
return []
logs = []
machine_id = get_machine_identifier()
for line in result.stdout.splitlines():
line_str = line.strip()
if not line_str:
continue
try:
entry = json.loads(line_str)
priority = str(entry.get("PRIORITY", "4"))
# Priority 0: Emerg, 1: Alert, 2: Crit, 3: Err, 4: Warning.
# Strip anything above 4 (5: Notice, 6: Info, 7: Debug)
if int(priority) > 4:
continue
sev = "WARNING" if priority == "4" else "ERROR"
logs.append({
"server": machine_id,
"os_type": "linux",
"signature": entry.get("SYSLOG_IDENTIFIER", "unknown"),
"severity": sev,
"message": entry.get("MESSAGE", "")[:2048]
})
except (json.JSONDecodeError, ValueError):
continue
return logs
def send_encrypted_logs_over_socket(config: dict, logs: list):
"""
Encrypts the payload using the server's OpenPGP public key and streams
over an authenticated TCP socket. Zero local state is maintained on the client.
"""
server_host = config["server_host"]
server_port = int(config["server_port"])
auth_token = config["auth_token"]
pub_key_armored = config["server_public_key"]
expected_fp = config.get("server_fingerprint", "").replace(" ", "").upper()
# Load and verify server public key
pub_key, _ = pgpy.PGPKey.from_blob(pub_key_armored)
actual_fp = str(pub_key.fingerprint).replace(" ", "").upper()
if expected_fp and actual_fp != expected_fp:
raise ValueError(
f"Server fingerprint mismatch! Expected {expected_fp}, but key has {actual_fp}."
)
machine_id = get_machine_identifier()
# Prepare zero-state candidate batch
payload = {
"server": machine_id,
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
payload_json = json.dumps(payload)
# Encrypt payload with server's encryption-only key
pgp_msg = pgpy.PGPMessage.new(payload_json)
encrypted_msg = pub_key.encrypt(pgp_msg)
encrypted_armored = str(encrypted_msg)
# Envelope with socket authentication header
envelope = {
"auth_token": auth_token,
"timestamp": datetime.now(timezone.utc).isoformat(),
"encrypted_payload": encrypted_armored
}
envelope_bytes = json.dumps(envelope).encode("utf-8")
# Connect over TCP socket and transmit with 4-byte length prefix framing
print(f"[*] Connecting to LOGAR server at {server_host}:{server_port} over secure TCP socket...")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.settimeout(15.0)
sock.connect((server_host, server_port))
# Send frame: length (4 bytes big-endian) + envelope
frame = struct.pack(">I", len(envelope_bytes)) + envelope_bytes
sock.sendall(frame)
# Receive response length
resp_len_bytes = sock.recv(4)
if not resp_len_bytes:
raise ConnectionError("Server closed connection without response.")
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = bytearray()
while len(resp_bytes) < resp_len:
chunk = sock.recv(min(4096, resp_len - len(resp_bytes)))
if not chunk:
break
resp_bytes.extend(chunk)
response = json.loads(resp_bytes.decode("utf-8"))
print(f"[+] Server response: {response}")
return response
def main():
parser = argparse.ArgumentParser(description="LOGAR Linux Edge Log Forwarder (Zero State)")
parser.add_argument("--config", default=CONFIG_FILE_NAME, help="Path to client_config.json")
parser.add_argument("--hours", type=int, default=6, help="Lookback window in hours for journalctl logs")
args = parser.parse_args()
try:
config = load_config(args.config)
except Exception as e:
print(f"[!] Configuration error: {e}")
sys.exit(1)
machine_id = get_machine_identifier()
print(f"[*] Edge Forwarder Node: {machine_id}")
print(f"[*] Scanning Linux journalctl for candidate anomalies (last {args.hours} hours)...")
candidate_logs = get_recent_linux_logs(hours=args.hours)
print(f"[*] Found {len(candidate_logs)} candidate anomalies (noise stripped at source).")
try:
send_encrypted_logs_over_socket(config, candidate_logs)
except Exception as e:
print(f"[!] Failed to stream logs to server: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
+255 -149
View File
@@ -1,19 +1,21 @@
# LOGAR: Edge-Thin Log Analysis & Temporal Verification System # LOGAR: Edge-Thin Log Analysis & Temporal Verification System
**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies OpenPGP encryption, authenticated TCP streaming, temporal persistence tracking across 12-hour evaluation windows, and an automated 4-run rule to filter out transient infrastructure blips before reporting verified anomalies to **Hermes**. **LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, an automated 4-run rule to filter transient warnings, and immediate pass-through for critical errors before reporting verified anomalies to **Hermes**.
--- ---
## Table of Contents ## Table of Contents
1. [Core Philosophy](#core-philosophy) 1. [Core Philosophy](#core-philosophy)
2. [Architecture & Data Flow](#architecture--data-flow) 2. [Architecture & Data Flow](#architecture--data-flow)
3. [Security & Cryptographic Model](#security--cryptographic-model) 3. [mTLS Security, Dynamic PKI & Licensing](#mtls-security-dynamic-pki--licensing)
4. [Cloud-Side Temporal Persistence & 4-Run Rule](#cloud-side-temporal-persistence--4-run-rule) 4. [Cloud-Side Temporal Persistence & 4-Run Rule](#cloud-side-temporal-persistence--4-run-rule)
5. [Agentic Hermes Integration](#agentic-hermes-integration) 5. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api)
6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification) 6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
7. [Repository & Shippables Structure](#repository--shippables-structure) 7. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows)
8. [Getting Started & Installation](#getting-started--installation) 8. [Repository & Shippables Structure](#repository--shippables-structure)
9. [Running Tests](#running-tests) 9. [Getting Started & Deployment](#getting-started--deployment)
10. [Running Tests](#running-tests)
11. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions)
--- ---
@@ -21,19 +23,20 @@
### 1. Edge Thinness & Zero State ### 1. Edge Thinness & Zero State
Site agents running on Windows and Linux act strictly as lightweight forwarders: Site agents running on Windows and Linux act strictly as lightweight forwarders:
- **No Local Database**: Clients maintain zero state and no local SQLite or cache files. - **No Local Database**: Clients maintain zero local SQLite or heavy cache files.
- **Source-Level Noise Stripping**: Conversational, informational, and debugging log noise (`INFO`, `DEBUG`, audit entries) is dropped directly at the source. - **Source-Level Filtering**: Agents stream candidate entries from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), stripping verbose debugging noise (`DEBUG`, trace entries) and skipping events older than 24 hours.
- **End-to-End Encryption**: Logs are encrypted using the server's OpenPGP public key before leaving the edge node. - **Transport Security (mTLS 1.3)**: Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates.
- **Secure TCP Sockets**: Ingestion occurs over low-overhead authenticated TCP sockets rather than bulky HTTP/HTTPS endpoints. - **Zero Configuration Overhead**: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret.
### 2. Cloud-Side Temporal Persistence ### 2. Cloud-Side Temporal Persistence & Severity Routing
The central Python/TCP hub handles the heavy lifting: The central Python hub handles state and verification:
- State tracking is managed centrally in SQLite (`logar_state.db`). - State tracking is managed centrally in SQLite (`logar_state.db`).
- Candidate issues are evaluated over a **12-hour temporal evaluation window**. - Candidate issues are evaluated over a **12-hour temporal evaluation window**.
- An issue must persist across **at least 4 consecutive runs / cycles** to be confirmed as a genuine system anomaly. Transient blips and sporadic spikes are filtered out automatically. - **Warning Persistence (4-Run Rule)**: `WARNING` issues must persist across **at least 4 consecutive runs / cycles** within the 12-hour window to be confirmed as genuine anomalies, automatically filtering transient infrastructure blips.
- **Immediate Error Pass**: Critical errors (`ERROR`, `CRITICAL`, `FATAL`) bypass the 4-run threshold and are promoted immediately to `VERIFIED` on their first occurrence.
### 3. Agentic Integration with Hermes ### 3. Agentic Integration with Hermes
Instead of human engineers manually diving through noisy logs, **Hermes** ingests pre-filtered, 4-run validated anomalies directly from the cloud hub (`GET /api/hermes/report`), treating them as verified system artifacts to trigger precise team notifications. Instead of engineers manually sifting through logs, **Hermes** ingests pre-filtered anomalies directly from the cloud hub (`GET /api/hermes/report`), treating verified errors and 4-run validated warnings as actionable system artifacts to trigger precise notifications and remediations.
--- ---
@@ -41,34 +44,43 @@ Instead of human engineers manually diving through noisy logs, **Hermes** ingest
```mermaid ```mermaid
graph TB graph TB
subgraph Edge Nodes [Zero-State Edge Forwarders] subgraph Edge Nodes [Lightweight Edge Forwarders]
W[Win_Client.py / Win_Client.exe<br/>Windows Event Log Application] W[Win_Client.exe / Win_Client.py<br/>Windows Event Log Ingestion]
L[Linux_Client.py / Linux_Client.bin<br/>systemd journalctl -p warning] L[Linux_Client.bin / Linux_Client.py<br/>systemd journalctl -p warning]
end end
subgraph Security Layer [Security & Framing] subgraph Enrollment [Dynamic PKI & Licensing]
E[OpenPGP Payload Encryption<br/>Server Public Key & Fingerprint] ENR[POST /api/client/enroll<br/>License Quota & Secret Validation]
S[Length-Prefixed Framing<br/>4-byte Big-Endian + Auth Envelope] CA[Internal Root CA<br/>Signs RSA-2048 Client Cert]
end end
subgraph Cloud Hub [LOGAR Central Server Hub] subgraph Transport [mTLS 1.3 Security Layer]
TCP[Authenticated TCP Listener<br/>Port 9443] MTLS[Mutual TLS 1.3 Handshake<br/>Port 9443 - Client Cert Required]
DEC[OpenPGP Decryption<br/>Server Private Key] AUTH[Extract Client CN & Fingerprint<br/>Validate Active License in SQLite]
DB[(SQLite Persistence<br/>active_issues & ingest_runs)] end
subgraph Hub [LOGAR Server Hub]
INGEST[Length-Prefixed Frame Ingestion]
DB[(SQLite Persistence<br/>active_issues, clients, license_config)]
RULE{12h Window &<br/>4-Run Rule} RULE{12h Window &<br/>4-Run Rule}
end end
subgraph Agentic Reporting [Downstream Integration] subgraph Downstream [Hermes Agent & Monitoring]
API[FastAPI / Uvicorn Reporting<br/>Port 8443] API[FastAPI Reporting & Management<br/>Port 8443]
HERMES[Hermes Agent<br/>GET /api/hermes/report] HERMES[Hermes Agent<br/>GET /api/hermes/report]
end end
W --> E W -->|Auto-Enrollment| ENR
L --> E L -->|Auto-Enrollment| ENR
E --> S ENR --> CA
S -->|TCP Stream| TCP CA -->|ca.crt, client.crt, client.key| W
TCP --> DEC CA -->|ca.crt, client.crt, client.key| L
DEC --> RULE
W -->|mTLS Stream| MTLS
L -->|mTLS Stream| MTLS
MTLS --> AUTH
AUTH --> INGEST
INGEST --> RULE
RULE --> DB RULE --> DB
DB --> API DB --> API
API --> HERMES API --> HERMES
@@ -76,31 +88,40 @@ graph TB
--- ---
## Security & Cryptographic Model ## mTLS Security, Dynamic PKI & Licensing
### Pure-Python OpenPGP (RFC 4880) ### 1. TLS 1.3 Mutual Authentication (mTLS)
- **Zero OS Binary Dependency**: Utilizes `pgpy` and `cryptography` in pure Python. **No native GnuPG or `gpg` binary installation is required** on the server, Windows nodes, or Linux nodes. - **Port 9443**: Ingestion occurs exclusively over TLS 1.3 sockets with `ssl.CERT_REQUIRED`.
- **First-Run Automatic Key Generation**: On the first launch, if `server_config.json` is missing, `Server.py` automatically generates: - Both the hub and edge clients verify each other's certificates:
- An OpenPGP RSA 2048 keypair with encryption-only usage flags. - Client verifies server certificate against `ca.crt`.
- An armored private key (`private_key`) and public key (`public_key`). - Server verifies client certificate against the Root CA.
- A SHA-256 public encryption fingerprint (`server_fingerprint`). - **Client CN Identification**: In the TLS handshake, the server extracts the `commonName` attribute (`client_id`), validates that the client is marked `active` in the `clients` table, updates the `last_seen` timestamp, and drops unregistered or revoked certificates immediately.
- A cryptographically random authentication secret token (`auth_token`).
- **Client Configuration Exporter**: ### 2. Dynamic PKI Hub Engine (`src/server_enrollment.py`)
```bash - **Root CA**: On first run, `Server.py` creates a self-signed Root CA (`ca.crt` / `ca.key`) valid for 10 years.
python Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json - **Server TLS Certificate**: Generated automatically with Subject Alternative Names (SANs) for `localhost`, `127.0.0.1`, server IP, and hostnames.
``` - **Authority Key Identifiers**: Full compliance with OpenSSL 3.x and Python 3.123.14 via `SubjectKeyIdentifier` and `AuthorityKeyIdentifier` extensions.
Produces an anonymous client config containing only the server socket coordinates, authentication token, and the encryption-only public key & fingerprint. - **Dynamic Client Certificates**: RSA-2048 keys and X.509 client certificates are issued on the fly via the enrollment API.
- **Socket Protocol Framing**:
- `[4 bytes big-endian unsigned int]` : Total envelope length. ### 3. Seat Accounting & Licensing
- `[JSON Envelope]` : - Stored in SQLite table `license_config`:
```json - `max_seats`: Maximum concurrent active client licenses (default: 10).
{ - `enrollment_secret`: Cryptographic secret required for initial client enrollment.
"auth_token": "<SECRET_TOKEN>", - Stored in SQLite table `clients`:
"timestamp": "2026-09-03T...", - `client_id`: Unique client identifier (machine GUID or hardware hash).
"encrypted_payload": "-----BEGIN PGP MESSAGE-----\n..." - `hostname`, `os_type`, `cert_fingerprint`, `status` (`active` / `revoked`), `first_seen`, `last_seen`.
} - When a new client enrolls:
``` - If `active_seats >= max_seats`, the hub rejects registration with `HTTP 403 (License seat limit reached)`.
- Unauthorized clients or invalid authentication tokens are rejected immediately. - Existing registered clients can re-enroll / renew seamlessly without consuming additional seats.
### 4. In-Flight Certificate Watchdog & Automated Self-Healing Renewal
- **Continuous Hub PKI Watchdog**:
- The server hub runs a continuous background watchdog coroutine (`cert_validity_watchdog`, running every 12 hours) alongside startup checks.
- The hub automatically inspects expiration dates of both the Root CA (`ca.crt`) and the Server TLS certificate (`server.crt`).
- If either certificate is within 30 days of expiration, the server regenerates certificates (backing up previous keys as `ca.crt.<timestamp>.bak`) and dynamically reloads its active `ssl.SSLContext` in memory without dropping socket connections or restarting the service.
- **Client Proactive Check & Reactive Self-Healing**:
- **Proactive Renewal**: Edge clients inspect `client.crt` before every run cycle. If the certificate expires in less than 30 days, it automatically contacts `/api/client/enroll` to renew its certificate.
- **Reactive Self-Healing**: If the server hub Root CA rotates or a handshake fails with `ssl.SSLError` / `SSLCertVerificationError`, edge clients catch the verification exception, re-bootstrap certificate enrollment against the hub, and re-establish the connection cleanly without human intervention.
--- ---
@@ -109,34 +130,77 @@ graph TB
Incoming candidate logs are tracked in SQLite table `active_issues`: Incoming candidate logs are tracked in SQLite table `active_issues`:
- **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`. - **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`.
- **12-Hour Evaluation Window**: - **12-Hour Evaluation Window**:
- When an issue is observed, the hub compares `(now - last_seen)`. - The hub compares `(now - last_seen)`.
- If more than 12 hours have passed since the issue was last recorded, the previous window is expired and the cycle resets to `run_count = 1` with status `TRANSIENT`. - If more than 12 hours have elapsed since the issue was last recorded, the previous window expires and the cycle resets to `run_count = 1`.
- **4-Run Rule**: - **4-Run Rule for Warnings**:
- For each distinct run batch, `run_count` increments. - The 4-run persistence threshold applies to `WARNING` (and `INFO`) events to eliminate transient operational noise.
- Issues with `run_count < 4` are marked as `TRANSIENT` and ignored by downstream reporting. - Each distinct run batch increments `run_count`.
- When `run_count >= 4` within the active 12-hour window, the status transitions to `VERIFIED`. - Warnings with `run_count < 4` are marked as `TRANSIENT` and excluded from Hermes reports.
- When `run_count >= 4` within the active 12-hour window, the warning transitions to `VERIFIED`.
- **Immediate Verification for Errors**:
- High-severity events (`ERROR`, `CRITICAL`, `FATAL`) **always pass immediately**.
- On their very first ingestion (`run_count = 1`), errors are promoted directly to `VERIFIED` and surfaced to Hermes without waiting for 4 runs.
--- ---
## Agentic Hermes Integration ## Agentic Hermes & Client Management API
The server hub serves a REST reporting API (default port `8443`): The server hub exposes a management and reporting REST API (default port `8443`):
### `POST /api/client/enroll`
Client enrollment endpoint:
- **Request**:
```json
{
"client_id": "web-worker-01.corp.internal",
"hostname": "web-worker-01",
"os": "linux",
"enrollment_secret": "<SECRET>"
}
```
- **Response**:
```json
{
"ca_cert": "-----BEGIN CERTIFICATE-----\n...",
"client_cert": "-----BEGIN CERTIFICATE-----\n...",
"client_key": "-----BEGIN RSA PRIVATE KEY-----\n..."
}
```
### `GET /api/clients`
Returns seat quota status and registered client telemetry:
```json
{
"active_seats": 2,
"max_seats": 10,
"clients": [
{
"client_id": "web-worker-01.corp.internal",
"hostname": "web-worker-01",
"os_type": "linux",
"cert_fingerprint": "7D5B660B...",
"status": "active",
"first_seen": "2026-09-04 18:00:00",
"last_seen": "2026-09-04 19:15:00"
}
]
}
```
### `GET /api/hermes/report` ### `GET /api/hermes/report`
Returns exclusively **verified anomalies** that have satisfied the 4-run rule within the active 12-hour evaluation window: Returns all **verified anomalies** (immediate critical errors and warnings verified after 4 consecutive runs within the 12-hour window):
```json ```json
[ [
{ {
"fingerprint": "corp.internal:web-app-01.corp.internal:NginxWorkerCrash", "fingerprint": "corp.internal:web-worker-01.corp.internal:PostgresPoolExhausted",
"site": "corp.internal", "site": "corp.internal",
"server": "web-app-01.corp.internal", "server": "web-worker-01.corp.internal",
"signature": "NginxWorkerCrash", "signature": "PostgresPoolExhausted",
"severity": "ERROR", "severity": "WARNING",
"message": "Worker process 4120 terminated with signal 11", "message": "Connection pool saturated (>95%) across 4 runs",
"os_type": "linux", "os_type": "linux",
"first_seen": "2026-09-03T09:00:00+00:00", "first_seen": "2026-09-04T07:00:00+00:00",
"last_seen": "2026-09-03T21:00:00+00:00", "last_seen": "2026-09-04T19:00:00+00:00",
"consecutive_runs": 4, "consecutive_runs": 4,
"evaluation_window": "12h", "evaluation_window": "12h",
"verified": true, "verified": true,
@@ -146,19 +210,19 @@ Returns exclusively **verified anomalies** that have satisfied the 4-run rule wi
``` ```
### `GET /api/hermes/all` ### `GET /api/hermes/all`
Diagnostic endpoint listing all active issues (both `TRANSIENT` candidate blips and `VERIFIED` anomalies). Diagnostic endpoint listing all candidate issues (`TRANSIENT` and `VERIFIED`).
### `GET /health` ### `GET /health`
Returns hub health, encryption fingerprint, and listener ports. Returns hub health, encryption fingerprint, listener ports, and mTLS status.
--- ---
## Dynamic Machine & Domain Identification ## Dynamic Machine & Domain Identification
Client configurations intentionally contain **no machine name or site name**. Both forwarders dynamically identify their host and domain at runtime via `get_machine_identifier()`: Client configurations intentionally contain **no hardcoded machine name or site name**. Both forwarders dynamically identify their host and domain at runtime via `get_machine_identifier()`:
1. **Fully Qualified Domain Name (FQDN)**: Checked via `socket.getfqdn()`. 1. **Fully Qualified Domain Name (FQDN)**: Checked via `socket.getfqdn()`.
2. **OS-Specific Domain Discovery**: 2. **OS-Specific Domain Discovery**:
- **Windows**: Checks Active Directory environment variable `USERDNSDOMAIN` / `USERDOMAIN`. - **Windows**: Checks Active Directory environment variables (`USERDNSDOMAIN`, `USERDOMAIN`).
- **Linux**: Parses `/etc/resolv.conf` `domain` and `search` directives. - **Linux**: Parses `/etc/resolv.conf` `domain` and `search` directives.
3. **Reverse DNS Lookup**: Resolves canonical hostname via `socket.gethostbyaddr`. 3. **Reverse DNS Lookup**: Resolves canonical hostname via `socket.gethostbyaddr`.
4. **Fallback**: Local hostname `socket.gethostname()`. 4. **Fallback**: Local hostname `socket.gethostname()`.
@@ -167,123 +231,165 @@ The server automatically infers site attribution from domain qualifiers (e.g. `n
--- ---
## Automated Service Installers (Linux & Windows)
LOGAR provides production-grade installation scripts and installer builders for automated service deployment:
### 1. Linux Service Installers
- **Client Installer (`compilation/install_linux_client.sh`)**:
- Non-interactive script deploying to `/opt/logar-client`.
- Automatically queries `/etc/machine-id` and enrolls with the hub via `curl`.
- Installs and enables `logar-client.service` systemd unit.
```bash
sudo ./compilation/install_linux_client.sh "http://hub.example.com:8443" "<ENROLLMENT_SECRET>"
```
- **Server Installer (`compilation/install_linux_server.sh`)**:
- Deploys server to `/opt/logar-server`.
- Configures logging and installs `logar-server.service` with `LimitNOFILE=65536`.
```bash
sudo ./compilation/install_linux_server.sh
```
### 2. Windows Inno Setup Installers
- Built using **Inno Setup 6** and bundled with **NSSM** (`compilation/nssm.exe`):
- **Client Setup (`compilation/installer_client.iss`)**: Compiles `LOGAR-Client-Setup.exe`. Installs `Win_Client.exe` into `{autopf}\LOGAR`, sets up `LOGAR_Client` service via NSSM with stdout/stderr redirection to `{commonappdata}\LOGAR\client.log`, and starts the service. Clean uninstallation stops and removes the service.
- **Server Setup (`compilation/installer_server.iss`)**: Compiles `LOGAR-Server-Setup.exe`. Installs `Server.exe` and sets up `LOGAR_Server` Windows service via NSSM.
---
## Repository & Shippables Structure ## Repository & Shippables Structure
``` ```
LOGAR/ LOGAR/
├── .gitignore # Ignore venv, caches, DBs, and private keys ├── .gitea/
├── requirements.txt # Unified dependencies │ └── workflows/
├── README.md # Comprehensive documentation │ ├── ci.yml # CI pipeline: syntax, 25 unit tests & mTLS pipeline test
├── Server.py # Central TCP server and Hermes API │ └── release.yml # Consolidated release workflow (Linux binaries & Windows installers)
├── Win_Client.py # Windows edge forwarder ├── compilation/ # Packaging, installers, and release automation
├── Linux_Client.py # Linux edge forwarder │ ├── install_linux_client.sh # Automated Linux client systemd installation script
├── test_pipeline.py # End-to-end integration test │ ├── install_linux_server.sh # Automated Linux server systemd installation script
└── out/ # Standalone shippable distributions │ ├── installer_client.iss # Inno Setup Windows Client installer script
├── server/ ├── installer_server.iss # Inno Setup Windows Server installer script
│ ├── Server.py # Python source │ ├── nssm.exe # Official 64-bit NSSM service manager binary
│ ├── server_config.sample.json │ ├── package_dist.py # Standalone binary compiler & packager
│ ├── requirements.txt │ ├── requirements.txt # Unified project dependencies
│ └── upload_release.py # Gitea REST API release asset publisher
├── src/ # Core application source modules
│ ├── __init__.py
│ ├── Server.py # Central mTLS server, temporal engine, and Hermes REST API
│ ├── server_enrollment.py # Dynamic PKI, Root CA, and client certificate generator
│ ├── Win_Client.py # Windows edge forwarder with auto-enrollment
│ └── Linux_Client.py # Linux edge forwarder with auto-enrollment
├── tests/ # Automated test suites
│ ├── test_linux_client.py # Linux client unit tests & mTLS certificate validation
│ ├── test_pipeline.py # End-to-end mTLS integration & 4-run verification test
│ ├── test_server.py # Server unit tests, PKI generation, and seat quota tests
│ └── test_win_client.py # Windows client unit tests & mTLS certificate validation
├── .gitignore # Ignores venv, caches, DBs, and private keys
├── README.md # Architecture and usage documentation
├── RELEASE_NOTES.md # Release history and changelog
├── server_config.sample.json # Reference server configuration
└── out/ # Component guides and sample configs
├── linux_server/
│ ├── README.md │ ├── README.md
│ └── test/ │ └── server_config.sample.json
│ └── test_server.py # Server unit tests ├── win_server/
├── win_client/
│ ├── Win_Client.py # Python source
│ ├── client_config.sample.json
│ ├── requirements.txt
│ ├── README.md │ ├── README.md
│ └── test/ │ └── server_config.sample.json
│ └── test_win_client.py # Windows client unit tests ├── linux_client/
└── linux_client/ │ ├── README.md
── build_bin.sh # PyInstaller native ELF compiler script ── client_config.sample.json
── Linux_Client.py # Python source ── win_client/
├── client_config.sample.json
├── requirements.txt
├── README.md ├── README.md
└── test/ └── client_config.sample.json
└── test_linux_client.py# Linux client unit tests
``` ```
--- ---
## Getting Started & Installation ## Getting Started & Deployment
### 1. Central Server Hub ### 1. Central Server Hub
1. **Install dependencies**: 1. **Install dependencies**:
```bash ```bash
pip install -r requirements.txt pip install -r compilation/requirements.txt
``` ```
2. **Start the server** (generates `server_config.json` and keypair on first run): 2. **Start the server** (generates `server_config.json`, Root CA, and server certs on first run):
```bash ```bash
python Server.py python src/Server.py
``` ```
3. **Export a client configuration**: 3. **Export a client configuration**:
```bash ```bash
python Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json python src/Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json
``` ```
### 2. Windows Client Deployment ### 2. Windows Client Deployment
1. Download `LOGAR-Client-Setup.exe` from releases and run it, or place `Win_Client.exe` and `client_config.json` in `C:\Program Files\LOGAR`.
1. Copy `Win_Client.py` (and `requirements.txt`) plus `client_config.json` to the target machine. 2. On first run with `client_config.json`, `Win_Client.exe` automatically enrolls with the hub, receives its mTLS certificates, and establishes secure streaming.
2. Run manually or schedule via Task Scheduler (every 3 hours):
```powershell
python Win_Client.py --hours 6
```
### 3. Linux Client Deployment ### 3. Linux Client Deployment
1. Run the automated installer:
1. Copy `Linux_Client.py` (and `requirements.txt`) plus `client_config.json` to `/opt/logar/`.
2. (Optional) Run `build_bin.sh` to compile a standalone ELF binary if desired.
3. Run via cron or systemd timer:
```bash ```bash
0 */3 * * * python3 /opt/logar/Linux_Client.py --hours 6 sudo ./compilation/install_linux_client.sh "http://<HUB_HOST>:8443" "<ENROLLMENT_SECRET>"
``` ```
2. The installer enrolls the client, configures `/etc/logar/certs`, and activates `logar-client.service`.
--- ---
## Running Tests ## Running Tests
### 1. Component-Specific Unit Tests ### 1. Component Unit Tests
Each component in `out/` includes its own isolated test suite:
```bash ```bash
# Server tests (config generation, SQLite persistence, 4-run rule) python -m unittest discover -s tests -v
python out/server/test/test_server.py
# Windows client tests (config anonymity, machine ID, OpenPGP encryption)
python out/win_client/test/test_win_client.py
# Linux client tests (config anonymity, journalctl priority filter, OpenPGP)
python out/linux_client/test/test_linux_client.py
``` ```
Runs all **25 unit tests**, covering:
- Dynamic Root CA generation and server TLS certificate issuance.
- Dynamic client certificate issuance with CN and authority key extensions.
- Enrollment secret authentication, seat limits, and certificate revocation.
- Proactive certificate validity checks, Root CA auto-renewal, and in-flight server SSLContext reload.
- Windows & Linux event log collection, deduplication, and mTLS certificate verification.
- 12-hour evaluation window and 4-run rule progression.
### 2. End-to-End Pipeline Integration Test ### 2. End-to-End Pipeline Integration Test
Start the server in one shell and run the pipeline test:
```bash ```bash
python test_pipeline.py # 1. Initialize test configuration
python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
# 2. Launch server in background
python src/Server.py &
# 3. Run integration test
python tests/test_pipeline.py
``` ```
This tests invalid token rejection, encrypted socket streaming, database persistence, status promotion upon the 4th run, and the Hermes API output. Tests client enrollment, secret rejection, mTLS TLS 1.3 socket handshake, warning 4-run rule promotion, immediate error promotion, and Hermes report output.
--- ---
## Automated Releases via Gitea Actions ## Releases & Binary Distribution
Releases are automated via [`.gitea/workflows/release.yml`](.gitea/workflows/release.yml) using your Gitea action runner: Releases can be built and published through three complementary channels:
### Publishing a Release ### 1. Tag Push Automation (Gitea Actions)
Whenever you want to release a new version with compiled standalone binaries: Pushing a release tag automatically triggers the build workflows:
```bash ```bash
git tag v1.0.0 git tag v2.0.1
git push origin v1.0.0 git push origin v2.0.1
``` ```
The consolidated workflow **`release.yml`** defines two parallel jobs:
- **`release-linux`** (`ubuntu-latest`): Compiles standalone native ELF binaries (`Linux_Client.bin`, `Server.bin`) and checksums.
- **`release-windows`** (`windows-latest`): Compiles Windows executables (`Win_Client.exe`, `Server.exe`), builds Inno Setup installers, and publishes checksums (requires self-hosted Windows Act Runner).
### What Gitea Actions Does Automatically: ### 2. Manual Workflow Dispatch (Gitea UI)
1. Gitea runner executes the workflow on tag push. Workflows can be manually triggered on demand from the Gitea web interface:
2. Runs `package_dist.py` to compile native standalone binaries: 1. Navigate to **Actions** $\rightarrow$ **Release Binaries & Installers** (`release.yml`).
- `Linux_Client.bin` (standalone binary) 2. Click **Run workflow**, set the release tag (defaults to `v2.0.1`), and run.
- `Server.bin` (standalone server binary)
- `Win_Client.pyz` (standalone executable zipapp)
- `SHA256SUMS.txt` (checksums)
3. Publishes the Gitea release using `gitea-release-action` and attaches the compiled binary assets.
*(Note: You can also use `upload_release.py` from your Windows machine to upload Windows `.exe` binaries directly if desired).* ### 3. Native Local Windows Build & Direct Release Upload
For environments without a registered Windows CI runner, Windows executables can be built and published directly to Gitea releases:
```powershell
# 1. Build Windows binaries locally
python compilation/package_dist.py --target windows
# 2. Upload assets and release notes directly to the Gitea release
python compilation/upload_release.py --tag v2.0.1 --token <GITEA_TOKEN> --skip-build
```
+60
View File
@@ -0,0 +1,60 @@
# LOGAR Release v2.0.1
Maintenance and deployment release consolidating Gitea Actions release automation into a unified single workflow file, standardizing release dispatching across platforms, and bumping version definitions across server hub and Windows installers.
### Key Highlights & Changes in v2.0.1:
- **Consolidated Single Release Automation Workflow (`.gitea/workflows/release.yml`)**:
- Unified separate platform release files into a single, cohesive workflow (`release.yml`) running parallel jobs (`release-linux` on `ubuntu-latest` and `release-windows` on `windows-latest`).
- Standardized tag matching for Gitea Actions on `push: tags: ['v*']`.
- Added streamlined manual `workflow_dispatch` triggers with automated release tag defaulting (`v2.0.1`).
- Retired deprecated `release-linux.yml` and `release-windows.yml` files.
- **Installer & Engine Version Bump**:
- Updated FastAPI Hub engine version to `2.0.1` in `src/Server.py`.
- Bumped Inno Setup Windows Client Installer (`compilation/installer_client.iss`) `AppVersion` to `2.0.1`.
- Bumped Inno Setup Windows Server Installer (`compilation/installer_server.iss`) `AppVersion` to `2.0.1`.
- **Distribution & Release Documentation**:
- Updated root and distribution documentation across all 5 deployment targets to reflect the 25 passing unit tests and tripartite release options.
---
# LOGAR Release v2.0.0
Major architectural release introducing Mutual TLS 1.3 (mTLS) transport security, built-in dynamic PKI & license accounting, in-flight certificate validity monitoring and auto-renewal, and automated Windows and Linux service installers.
### Key Highlights & Changes in v2.0.0:
- **mTLS 1.3 Transport Security & Runtime Licensing**:
- Replaced legacy plain TCP sockets with mutual TLS 1.3 authentication (`ssl.CERT_REQUIRED`, TLS 1.3 minimum version).
- Hub dynamically validates incoming client Common Name (`client_id`) against active license seats in SQLite during the TLS handshake.
- Drops unauthorized, un-enrolled, or revoked clients at the transport layer before payload reading.
- **Dynamic Hub PKI Engine (`src/server_enrollment.py`)**:
- Automatically initializes an internal RSA-4096 Root CA (`ca.crt` / `ca.key`).
- Generates RSA-2048 Server TLS certificates with SANs for localhost, loopback, and server hostnames.
- Full OpenSSL 3.x and Python 3.123.14 compatibility via `SubjectKeyIdentifier` and `AuthorityKeyIdentifier` certificate extensions.
- Generates and signs client certificates on demand via `POST /api/client/enroll`.
- **In-Flight Certificate Validity Watchdog & Dynamic SSLContext Reloading**:
- Server hub runs a continuous background watchdog coroutine (`cert_validity_watchdog`, evaluated every 12 hours) alongside startup checks.
- Automatically checks Root CA and server TLS certificate expiration against a 30-day threshold.
- Generates renewed certificates on disk with timestamped backups (`.bak`), and reloads active `ssl.SSLContext` in memory dynamically without dropping socket listeners or restarting the background service.
- **Client Proactive Expiry Check & Reactive Self-Healing Auto-Renewal**:
- **Proactive**: Forwarders (`Win_Client.py` and `Linux_Client.py`) evaluate `client.crt` validity before each run, auto-renewing via `/api/client/enroll` if expiring within 30 days.
- **Reactive**: If the hub rotates its Root CA or a TLS verification error (`ssl.SSLError` / `SSLCertVerificationError`) occurs, clients automatically catch the error, re-enroll with the hub using their enrollment secret, and reconnect cleanly.
- **Database Schema & License Quota Accounting**:
- SQLite tables `license_config` (`max_seats`, `enrollment_secret`) and `clients` (`client_id`, `hostname`, `os_type`, `cert_fingerprint`, `status`, timestamps).
- Enforces seat limits on enrollment (`HTTP 403 License seat limit reached`) while allowing active registered nodes to re-enroll/renew indefinitely.
- Added `GET /api/clients` endpoint for license auditing and telemetry tracking.
- **Automated Service Installers**:
- **Windows**: Self-contained Inno Setup installers (`LOGAR-Client-Setup.exe` and `LOGAR-Server-Setup.exe`) bundling `nssm.exe` to register, configure, and start Windows services automatically.
- **Linux**: Automated installer scripts (`compilation/install_linux_client.sh` and `install_linux_server.sh`) deploying systemd service units with auto-restart policies.
- **CI/CD Release Workflows**:
- Windows workflow (`.gitea/workflows/release-windows.yml`) and Linux workflow (`.gitea/workflows/release-linux.yml`) automated to build native executables, installers, and upload release assets on tag push.
-465
View File
@@ -1,465 +0,0 @@
import os
import sys
import json
import uuid
import struct
import socket
import sqlite3
import argparse
import asyncio
import secrets
import warnings
from datetime import datetime, timezone, timedelta
from typing import Dict, Any, List, Optional
# Suppress cryptography / pgpy deprecation notices for a clean terminal output
warnings.filterwarnings("ignore")
import pgpy
from pgpy.constants import (
PubKeyAlgorithm,
KeyFlags,
HashAlgorithm,
SymmetricKeyAlgorithm,
CompressionAlgorithm
)
from fastapi import FastAPI, HTTPException
import uvicorn
CONFIG_FILE_NAME = "server_config.json"
DEFAULT_DB_FILE = "logar_state.db"
EVALUATION_WINDOW_HOURS = 12
RUN_THRESHOLD = 4
app = FastAPI(title="LOGAR Cloud Ingestion & Hermes Hub", version="2.0.0")
# Global context holding server state
SERVER_STATE: Dict[str, Any] = {}
def generate_server_keypair(server_name: str):
"""Generates an OpenPGP RSA 2048 key with encryption capability."""
key = pgpy.PGPKey.new(PubKeyAlgorithm.RSAEncryptOrSign, 2048)
uid = pgpy.PGPUID.new(server_name)
key.add_uid(
uid,
usage={KeyFlags.EncryptCommunications, KeyFlags.EncryptStorage},
hashes=[HashAlgorithm.SHA256],
ciphers=[SymmetricKeyAlgorithm.AES256],
compression=[CompressionAlgorithm.Uncompressed]
)
private_key_armored = str(key)
public_key_armored = str(key.pubkey)
fingerprint = str(key.pubkey.fingerprint)
return private_key_armored, public_key_armored, fingerprint
def load_or_init_config(config_path: str = CONFIG_FILE_NAME) -> Dict[str, Any]:
"""Loads existing server_config.json or creates a new one on first run."""
if os.path.exists(config_path):
print(f"[*] Loading server configuration from: {os.path.abspath(config_path)}")
with open(config_path, "r", encoding="utf-8") as f:
config = json.load(f)
return config
print(f"[!] Config '{config_path}' not found. Initializing first-run configuration...")
server_name = "LOGAR-Cloud-Hub"
private_key, public_key, fingerprint = generate_server_keypair(server_name)
auth_token = secrets.token_hex(24)
config = {
"server_name": server_name,
"tcp_host": "0.0.0.0",
"tcp_port": 9443,
"hermes_host": "0.0.0.0",
"hermes_port": 8443,
"auth_token": auth_token,
"db_path": DEFAULT_DB_FILE,
"evaluation_window_hours": EVALUATION_WINDOW_HOURS,
"min_persistence_runs": RUN_THRESHOLD,
"server_fingerprint": fingerprint,
"public_key": public_key,
"private_key": private_key
}
with open(config_path, "w", encoding="utf-8") as f:
json.dump(config, f, indent=2)
print(f"[+] Successfully generated new server config and OpenPGP keypair.")
print(f"[+] Server Encryption Fingerprint: {fingerprint}")
print(f"[+] Saved to: {os.path.abspath(config_path)}")
return config
def create_client_config(
server_host: str,
server_port: int,
output_path: str,
config_path: str = CONFIG_FILE_NAME
) -> Dict[str, Any]:
"""Creates a client configuration file containing the server address, auth token, and encryption-only key/fingerprint."""
server_conf = load_or_init_config(config_path)
client_conf = {
"server_host": server_host,
"server_port": server_port,
"server_fingerprint": server_conf["server_fingerprint"],
"server_public_key": server_conf["public_key"],
"auth_token": server_conf["auth_token"]
}
out_dir = os.path.dirname(os.path.abspath(output_path))
if out_dir and not os.path.exists(out_dir):
os.makedirs(out_dir, exist_ok=True)
with open(output_path, "w", encoding="utf-8") as f:
json.dump(client_conf, f, indent=2)
print(f"[+] Client configuration successfully written to: {os.path.abspath(output_path)}")
print(f" - Server Target: {server_host}:{server_port}")
print(f" - Encryption Fingerprint: {server_conf['server_fingerprint']}")
return client_conf
def init_db(db_path: str):
"""Initializes the SQLite schema for multi-run temporal tracking."""
conn = sqlite3.connect(db_path)
conn.execute("""
CREATE TABLE IF NOT EXISTS active_issues (
fingerprint TEXT PRIMARY KEY,
site_name TEXT,
server TEXT,
signature TEXT,
severity TEXT,
message TEXT,
os_type TEXT,
first_seen TEXT,
last_seen TEXT,
run_count INTEGER,
status TEXT,
last_run_id TEXT
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS ingest_runs (
run_id TEXT PRIMARY KEY,
site_name TEXT,
server TEXT,
timestamp TEXT,
log_count INTEGER
)
""")
conn.commit()
conn.close()
def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: int, min_runs: int) -> Dict[str, Any]:
"""
Evaluates candidate issues against the 12-hour evaluation window and 4-run rule.
Zero-state clients send raw candidate entries; this engine handles temporal state.
"""
client_server = payload.get("server", "unknown-host")
site_name = payload.get("site_name") or (client_server.split(".", 1)[1] if "." in client_server else "default")
logs = payload.get("logs", [])
run_id = str(uuid.uuid4())
now = datetime.now(timezone.utc)
now_iso = now.isoformat()
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
# Record the batch run
cursor.execute(
"INSERT INTO ingest_runs (run_id, site_name, server, timestamp, log_count) VALUES (?, ?, ?, ?, ?)",
(run_id, site_name, client_server, now_iso, len(logs))
)
processed_count = 0
promoted_to_verified = 0
for log in logs:
severity = str(log.get("severity", "WARNING")).upper()
# Edge forwarder filter safeguard (strip informational noise)
if severity in ["INFO", "DEBUG"]:
continue
signature = log.get("signature", "unknown")
server = log.get("server", client_server)
message = log.get("message", "")
os_type = log.get("os_type", "unknown")
fp = f"{site_name}:{server}:{signature}"
cursor.execute(
"SELECT run_count, first_seen, last_seen, status, last_run_id FROM active_issues WHERE fingerprint = ?",
(fp,)
)
row = cursor.fetchone()
if row:
run_count, first_seen_str, last_seen_str, current_status, last_run_id = row
try:
last_seen_dt = datetime.fromisoformat(last_seen_str)
except Exception:
last_seen_dt = now
# 12-hour evaluation window expiry check
if (now - last_seen_dt) > timedelta(hours=window_hours):
# Window elapsed: reset to new cycle
new_runs = 1
new_first_seen = now_iso
new_status = "TRANSIENT"
else:
# Same run guard: only increment count once per distinct run batch
if last_run_id != run_id:
new_runs = run_count + 1
else:
new_runs = run_count
new_first_seen = first_seen_str
# 4-run rule enforcement
new_status = "VERIFIED" if new_runs >= min_runs else "TRANSIENT"
if new_status == "VERIFIED" and current_status != "VERIFIED":
promoted_to_verified += 1
cursor.execute("""
UPDATE active_issues
SET run_count = ?, last_seen = ?, first_seen = ?, status = ?, last_run_id = ?, message = ?, severity = ?
WHERE fingerprint = ?
""", (new_runs, now_iso, new_first_seen, new_status, run_id, message, severity, fp))
else:
initial_status = "VERIFIED" if 1 >= min_runs else "TRANSIENT"
cursor.execute("""
INSERT INTO active_issues
(fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status, last_run_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (fp, site_name, server, signature, severity, message, os_type, now_iso, now_iso, 1, initial_status, run_id))
processed_count += 1
conn.commit()
conn.close()
return {
"status": "success",
"run_id": run_id,
"processed": processed_count,
"promoted_verified": promoted_to_verified
}
async def handle_socket_client(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
"""
Authenticated TCP socket handler.
Protocol:
- 4-byte big-endian prefix: payload length
- Payload: JSON with auth_token and encrypted_payload (OpenPGP ASCII armored)
- Response: 4-byte length + JSON confirmation
"""
addr = writer.get_extra_info("peername")
try:
# Read 4-byte length prefix
length_bytes = await reader.readexactly(4)
length = struct.unpack(">I", length_bytes)[0]
if length <= 0 or length > 10 * 1024 * 1024: # 10MB limit
raise ValueError(f"Invalid frame size: {length}")
payload_bytes = await reader.readexactly(length)
envelope = json.loads(payload_bytes.decode("utf-8"))
# Authenticate socket client
expected_token = SERVER_STATE["config"]["auth_token"]
provided_token = envelope.get("auth_token")
if not secrets.compare_digest(str(provided_token), str(expected_token)):
err_msg = json.dumps({"status": "error", "message": "Authentication failed"}).encode("utf-8")
writer.write(struct.pack(">I", len(err_msg)) + err_msg)
await writer.drain()
writer.close()
await writer.wait_closed()
return
# Decrypt payload using server's OpenPGP private key
encrypted_armored = envelope.get("encrypted_payload", "")
pgp_msg = pgpy.PGPMessage.from_blob(encrypted_armored)
priv_key = SERVER_STATE["private_key_obj"]
decrypted_obj = priv_key.decrypt(pgp_msg)
decrypted_json_str = decrypted_obj.message
log_payload = json.loads(decrypted_json_str)
# Ingest and apply 12h window / 4-run rule
res = process_ingested_logs(
log_payload,
db_path=SERVER_STATE["config"]["db_path"],
window_hours=SERVER_STATE["config"]["evaluation_window_hours"],
min_runs=SERVER_STATE["config"]["min_persistence_runs"]
)
resp_bytes = json.dumps(res).encode("utf-8")
writer.write(struct.pack(">I", len(resp_bytes)) + resp_bytes)
await writer.drain()
except Exception as e:
err = json.dumps({"status": "error", "message": str(e)}).encode("utf-8")
try:
writer.write(struct.pack(">I", len(err)) + err)
await writer.drain()
except Exception:
pass
finally:
writer.close()
try:
await writer.wait_closed()
except Exception:
pass
@app.get("/api/hermes/report")
def get_hermes_report():
"""
Agentic Integration endpoint: Consumed by Hermes to fetch anomalies that have persisted
across the 12-hour evaluation window and satisfied the 4-run rule.
"""
db_path = SERVER_STATE["config"]["db_path"]
window_hours = SERVER_STATE["config"]["evaluation_window_hours"]
min_runs = SERVER_STATE["config"]["min_persistence_runs"]
now = datetime.now(timezone.utc)
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("""
SELECT fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status
FROM active_issues
WHERE status = 'VERIFIED' AND run_count >= ?
""", (min_runs,))
rows = cursor.fetchall()
conn.close()
report = []
for r in rows:
last_seen_dt = datetime.fromisoformat(r[8])
# Only return anomalies active within the evaluation window
if (now - last_seen_dt) <= timedelta(hours=window_hours):
report.append({
"fingerprint": r[0],
"site": r[1],
"server": r[2],
"signature": r[3],
"severity": r[4],
"message": r[5],
"os_type": r[6],
"first_seen": r[7],
"last_seen": r[8],
"consecutive_runs": r[9],
"evaluation_window": f"{window_hours}h",
"verified": True,
"status": r[10]
})
return report
@app.get("/api/hermes/all")
def get_all_issues():
"""Diagnostic endpoint to inspect both transient candidate blips and verified anomalies."""
db_path = SERVER_STATE["config"]["db_path"]
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("""
SELECT fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status
FROM active_issues
""")
rows = cursor.fetchall()
conn.close()
return [
{
"fingerprint": r[0],
"site": r[1],
"server": r[2],
"signature": r[3],
"severity": r[4],
"message": r[5],
"os_type": r[6],
"first_seen": r[7],
"last_seen": r[8],
"run_count": r[9],
"status": r[10]
}
for r in rows
]
@app.get("/health")
def health_check():
return {
"status": "healthy",
"server_name": SERVER_STATE["config"]["server_name"],
"fingerprint": SERVER_STATE["config"]["server_fingerprint"],
"tcp_port": SERVER_STATE["config"]["tcp_port"],
"hermes_port": SERVER_STATE["config"]["hermes_port"]
}
async def run_server():
"""Runs the TCP socket listener and the Hermes REST API concurrently."""
config = SERVER_STATE["config"]
tcp_host = config["tcp_host"]
tcp_port = int(config["tcp_port"])
hermes_host = config["hermes_host"]
hermes_port = int(config["hermes_port"])
# Start TCP Socket Server
tcp_server = await asyncio.start_server(handle_socket_client, tcp_host, tcp_port)
print(f"[*] LOGAR TCP Socket Server listening on {tcp_host}:{tcp_port}")
# Start FastAPI / Uvicorn server for Hermes
uv_config = uvicorn.Config(app, host=hermes_host, port=hermes_port, log_level="warning")
uv_server = uvicorn.Server(uv_config)
print(f"[*] Hermes Reporting API available at http://{hermes_host}:{hermes_port}/api/hermes/report")
await asyncio.gather(
tcp_server.serve_forever(),
uv_server.serve()
)
def main():
parser = argparse.ArgumentParser(description="LOGAR Cloud Hub & TCP Socket Ingestion Server")
parser.add_argument("--config", default=CONFIG_FILE_NAME, help="Path to server_config.json")
parser.add_argument("--create-client-config", action="store_true", help="Generate a client config with encryption-only fingerprint and server address")
parser.add_argument("--client-out", default="client_config.json", help="Output file path for generated client config")
parser.add_argument("--server-host", default="127.0.0.1", help="Server address to embed in client config")
parser.add_argument("--server-port", type=int, default=None, help="TCP port to embed in client config")
args = parser.parse_args()
config = load_or_init_config(args.config)
init_db(config["db_path"])
# Load OpenPGP private key into memory
priv_key_obj, _ = pgpy.PGPKey.from_blob(config["private_key"])
SERVER_STATE["config"] = config
SERVER_STATE["private_key_obj"] = priv_key_obj
if args.create_client_config:
port = args.server_port or config["tcp_port"]
create_client_config(
server_host=args.server_host,
server_port=port,
output_path=args.client_out,
config_path=args.config
)
sys.exit(0)
print("=" * 60)
print(f" LOGAR Server Hub: {config['server_name']}")
print(f" Encryption Fingerprint: {config['server_fingerprint']}")
print(f" Evaluation Window: {config['evaluation_window_hours']} hours | Rule: {config['min_persistence_runs']}+ consecutive runs")
print("=" * 60)
try:
asyncio.run(run_server())
except KeyboardInterrupt:
print("\n[!] Server shutting down.")
if __name__ == "__main__":
main()
-209
View File
@@ -1,209 +0,0 @@
import os
import sys
import json
import socket
import struct
import argparse
import warnings
from datetime import datetime, timezone, timedelta
# Suppress cryptography / pgpy deprecation notices
warnings.filterwarnings("ignore")
import pgpy
try:
import win32evtlog
except ImportError:
win32evtlog = None
CONFIG_FILE_NAME = "client_config.json"
def load_config(config_path: str = CONFIG_FILE_NAME):
if not os.path.exists(config_path):
raise FileNotFoundError(
f"Client configuration file not found at: {config_path}\n"
f"Generate one from the server using: python Server.py --create-client-config --client-out {config_path}"
)
with open(config_path, "r", encoding="utf-8") as f:
return json.load(f)
def get_machine_identifier() -> str:
"""
Returns the hostname of the machine sending the logs,
and appends the network/DNS domain if available.
"""
fqdn = socket.getfqdn()
if fqdn and "." in fqdn and not fqdn.startswith("localhost"):
return fqdn
hostname = socket.gethostname()
user_dns_domain = os.environ.get("USERDNSDOMAIN")
if user_dns_domain and user_dns_domain.lower() != hostname.lower():
return f"{hostname}.{user_dns_domain.lower()}"
try:
host_ip = socket.gethostbyname(hostname)
canonical_name = socket.gethostbyaddr(host_ip)[0]
if canonical_name and "." in canonical_name and not canonical_name.startswith("localhost"):
return canonical_name
except Exception:
pass
return hostname
def get_recent_windows_logs(hours: int = 6):
"""
Scans the Windows Application Event Log backwards for events within the window.
Edge Thinness & Noise Stripping: INFO and DEBUG events are dropped at the source.
"""
if win32evtlog is None:
print("[!] pywin32 is not installed or not running on Windows. Returning mock/empty candidate list.")
return []
server = "localhost"
log_type = "Application"
flags = win32evtlog.EVENTLOG_BACKWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
try:
hand = win32evtlog.OpenEventLog(server, log_type)
except Exception as e:
print(f"[!] Error opening Windows event log: {e}")
return []
logs = []
cutoff_time = datetime.now() - timedelta(hours=hours)
machine_id = get_machine_identifier()
sev_map = {
1: "CRITICAL",
2: "ERROR",
3: "WARNING"
}
while True:
events = win32evtlog.ReadEventLog(hand, flags, 0)
if not events:
break
for event in events:
if event.TimeGenerated < cutoff_time:
break
# Drop conversational or informational noise (INFO=4, etc.) at source
# Only retain Critical (1), Error (2), and Warning (3)
if event.EventType in sev_map:
msg = " ".join(event.StringInserts) if event.StringInserts else "Event Log Entry"
logs.append({
"server": machine_id,
"os_type": "windows",
"signature": event.SourceName or "Windows-Event",
"severity": sev_map[event.EventType],
"message": msg[:2048] # Limit message length
})
if events[-1].TimeGenerated < cutoff_time:
break
win32evtlog.CloseEventLog(hand)
return logs
def send_encrypted_logs_over_socket(config: dict, logs: list):
"""
Encrypts the payload using the server's OpenPGP public key and streams
over an authenticated TCP socket. Zero local state is maintained on the client.
"""
server_host = config["server_host"]
server_port = int(config["server_port"])
auth_token = config["auth_token"]
pub_key_armored = config["server_public_key"]
expected_fp = config.get("server_fingerprint", "").replace(" ", "").upper()
# Load and verify server public key
pub_key, _ = pgpy.PGPKey.from_blob(pub_key_armored)
actual_fp = str(pub_key.fingerprint).replace(" ", "").upper()
if expected_fp and actual_fp != expected_fp:
raise ValueError(
f"Server fingerprint mismatch! Expected {expected_fp}, but key has {actual_fp}."
)
machine_id = get_machine_identifier()
# Prepare zero-state candidate batch
payload = {
"server": machine_id,
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
payload_json = json.dumps(payload)
# Encrypt payload with server's encryption-only key
pgp_msg = pgpy.PGPMessage.new(payload_json)
encrypted_msg = pub_key.encrypt(pgp_msg)
encrypted_armored = str(encrypted_msg)
# Envelope with socket authentication header
envelope = {
"auth_token": auth_token,
"timestamp": datetime.now(timezone.utc).isoformat(),
"encrypted_payload": encrypted_armored
}
envelope_bytes = json.dumps(envelope).encode("utf-8")
# Connect over TCP socket and transmit with 4-byte length prefix framing
print(f"[*] Connecting to LOGAR server at {server_host}:{server_port} over secure TCP socket...")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.settimeout(15.0)
sock.connect((server_host, server_port))
# Send frame: length (4 bytes big-endian) + envelope
frame = struct.pack(">I", len(envelope_bytes)) + envelope_bytes
sock.sendall(frame)
# Receive response length
resp_len_bytes = sock.recv(4)
if not resp_len_bytes:
raise ConnectionError("Server closed connection without response.")
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = bytearray()
while len(resp_bytes) < resp_len:
chunk = sock.recv(min(4096, resp_len - len(resp_bytes)))
if not chunk:
break
resp_bytes.extend(chunk)
response = json.loads(resp_bytes.decode("utf-8"))
print(f"[+] Server response: {response}")
return response
def main():
parser = argparse.ArgumentParser(description="LOGAR Windows Edge Log Forwarder (Zero State)")
parser.add_argument("--config", default=CONFIG_FILE_NAME, help="Path to client_config.json")
parser.add_argument("--hours", type=int, default=6, help="Lookback window in hours for event logs")
args = parser.parse_args()
try:
config = load_config(args.config)
except Exception as e:
print(f"[!] Configuration error: {e}")
sys.exit(1)
print(f"[*] Scanning Windows Application event log for candidate anomalies (last {args.hours} hours)...")
candidate_logs = get_recent_windows_logs(hours=args.hours)
print(f"[*] Found {len(candidate_logs)} candidate anomalies (noise stripped at source).")
try:
send_encrypted_logs_over_socket(config, candidate_logs)
except Exception as e:
print(f"[!] Failed to stream logs to server: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env bash
set -euo pipefail
HUB_URL="${1:-http://hub.example.com:8443}"
ENROLL_SECRET="${2:-}"
INSTALL_DIR="/opt/logar-client"
CONFIG_DIR="/etc/logar"
echo "[+] Installing LOGAR Client..."
mkdir -p "${INSTALL_DIR}" "${CONFIG_DIR}/certs"
if [ -f "dist/Linux_Client.bin" ]; then
cp dist/Linux_Client.bin "${INSTALL_DIR}/Linux_Client"
elif [ -f "dist/Linux_Client" ]; then
cp dist/Linux_Client "${INSTALL_DIR}/Linux_Client"
else
echo "[!] Warning: dist/Linux_Client binary not found in current directory. Continuing with existing binary if present."
fi
if [ -f "${INSTALL_DIR}/Linux_Client" ]; then
chmod +x "${INSTALL_DIR}/Linux_Client"
fi
# Bootstrap certificate if missing and enrollment secret is provided
if [ ! -f "${CONFIG_DIR}/certs/client.crt" ] && [ -n "${ENROLL_SECRET}" ]; then
echo "[+] Enrolling client with LOGAR Hub..."
MACHINE_ID=$(cat /etc/machine-id 2>/dev/null || hostname)
RESPONSE=$(curl -s -X POST "${HUB_URL}/api/client/enroll" \
-H "Content-Type: application/json" \
-d "{\"client_id\": \"${MACHINE_ID}\", \"hostname\": \"$(hostname)\", \"os\": \"linux\", \"enrollment_secret\": \"${ENROLL_SECRET}\"}")
echo "${RESPONSE}" | grep -q "client_cert" || {
echo "[!] Enrollment failed: ${RESPONSE}"
exit 1
}
if command -v jq >/dev/null 2>&1; then
echo "${RESPONSE}" | jq -r .ca_cert > "${CONFIG_DIR}/certs/ca.crt"
echo "${RESPONSE}" | jq -r .client_cert > "${CONFIG_DIR}/certs/client.crt"
echo "${RESPONSE}" | jq -r .client_key > "${CONFIG_DIR}/certs/client.key"
else
python3 -c "import sys, json; data=json.loads(sys.stdin.read()); open('${CONFIG_DIR}/certs/ca.crt','w').write(data['ca_cert']); open('${CONFIG_DIR}/certs/client.crt','w').write(data['client_cert']); open('${CONFIG_DIR}/certs/client.key','w').write(data['client_key'])" <<< "${RESPONSE}"
fi
chmod 600 "${CONFIG_DIR}/certs/client.key"
echo "[+] Certificates written to ${CONFIG_DIR}/certs"
fi
cat <<EOF > /etc/systemd/system/logar-client.service
[Unit]
Description=LOGAR Edge Log Aggregator Client
After=network.target
[Service]
Type=simple
ExecStart=${INSTALL_DIR}/Linux_Client --config ${CONFIG_DIR}/config.json
Restart=always
RestartSec=5s
User=root
[Install]
WantedBy=multi-user.target
EOF
if command -v systemctl >/dev/null 2>&1; then
systemctl daemon-reload
systemctl enable --now logar-client.service || true
echo "[+] LOGAR Client service configured and activated."
else
echo "[+] Systemd service installed at /etc/systemd/system/logar-client.service"
fi
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
set -euo pipefail
INSTALL_DIR="/opt/logar-server"
CONFIG_DIR="/etc/logar"
echo "[+] Installing LOGAR Server..."
mkdir -p "${INSTALL_DIR}" "${CONFIG_DIR}" "/var/log/logar"
if [ -f "dist/Server.bin" ]; then
cp dist/Server.bin "${INSTALL_DIR}/Server"
elif [ -f "dist/Server" ]; then
cp dist/Server "${INSTALL_DIR}/Server"
elif [ -f "dist/LOGAR_Server" ]; then
cp dist/LOGAR_Server "${INSTALL_DIR}/Server"
else
echo "[!] Warning: dist/Server.bin binary not found in current directory. Continuing with existing binary if present."
fi
if [ -f "${INSTALL_DIR}/Server" ]; then
chmod +x "${INSTALL_DIR}/Server"
fi
cat <<EOF > /etc/systemd/system/logar-server.service
[Unit]
Description=LOGAR Hub and Aggregator Engine
After=network.target
[Service]
Type=simple
WorkingDirectory=${INSTALL_DIR}
ExecStart=${INSTALL_DIR}/Server --config ${CONFIG_DIR}/server_config.json
Restart=always
RestartSec=5s
User=root
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
if command -v systemctl >/dev/null 2>&1; then
systemctl daemon-reload
systemctl enable --now logar-server.service || true
echo "[+] LOGAR Server service installed and activated."
else
echo "[+] Systemd service installed at /etc/systemd/system/logar-server.service"
fi
+27
View File
@@ -0,0 +1,27 @@
[Setup]
AppName=LOGAR Client
AppVersion=2.0.1
DefaultDirName={autopf}\LOGAR
OutputDir=..\dist
OutputBaseFilename=LOGAR-Client-Setup
PrivilegesRequired=admin
Compression=lzma
SolidCompression=yes
[Files]
Source: "..\dist\Win_Client.exe"; DestDir: "{app}"; Flags: ignoreversion
Source: "..\compilation\nssm.exe"; DestDir: "{app}"; Flags: ignoreversion
[Dirs]
Name: "{commonappdata}\LOGAR"; Permissions: users-modify
[Run]
Filename: "{app}\nssm.exe"; Parameters: "install LOGAR_Client ""{app}\Win_Client.exe"""; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "set LOGAR_Client AppDirectory ""{app}"""; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "set LOGAR_Client AppStdout ""{commonappdata}\LOGAR\client.log"""; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "set LOGAR_Client AppStderr ""{commonappdata}\LOGAR\client_err.log"""; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "start LOGAR_Client"; Flags: runhidden
[UninstallRun]
Filename: "{app}\nssm.exe"; Parameters: "stop LOGAR_Client"; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "remove LOGAR_Client confirm"; Flags: runhidden
+27
View File
@@ -0,0 +1,27 @@
[Setup]
AppName=LOGAR Server
AppVersion=2.0.1
DefaultDirName={autopf}\LOGAR-Server
OutputDir=..\dist
OutputBaseFilename=LOGAR-Server-Setup
PrivilegesRequired=admin
Compression=lzma
SolidCompression=yes
[Files]
Source: "..\dist\Server.exe"; DestDir: "{app}"; Flags: ignoreversion
Source: "..\compilation\nssm.exe"; DestDir: "{app}"; Flags: ignoreversion
[Dirs]
Name: "{commonappdata}\LOGAR-Server"; Permissions: users-modify
[Run]
Filename: "{app}\nssm.exe"; Parameters: "install LOGAR_Server ""{app}\Server.exe"""; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "set LOGAR_Server AppDirectory ""{app}"""; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "set LOGAR_Server AppStdout ""{commonappdata}\LOGAR-Server\server.log"""; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "set LOGAR_Server AppStderr ""{commonappdata}\LOGAR-Server\server_err.log"""; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "start LOGAR_Server"; Flags: runhidden
[UninstallRun]
Filename: "{app}\nssm.exe"; Parameters: "stop LOGAR_Server"; Flags: runhidden
Filename: "{app}\nssm.exe"; Parameters: "remove LOGAR_Server confirm"; Flags: runhidden
Binary file not shown.
+206
View File
@@ -0,0 +1,206 @@
import os
import sys
import shutil
import zipapp
import hashlib
import platform
import subprocess
import argparse
ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
SRC_DIR = os.path.join(ROOT_DIR, "src")
DIST_DIR = os.path.join(ROOT_DIR, "dist")
BUILD_TEMP = os.path.join(ROOT_DIR, "build_temp")
def clean_and_prep():
if os.path.exists(DIST_DIR):
shutil.rmtree(DIST_DIR)
os.makedirs(DIST_DIR, exist_ok=True)
if os.path.exists(BUILD_TEMP):
shutil.rmtree(BUILD_TEMP)
os.makedirs(BUILD_TEMP, exist_ok=True)
def build_pyinstaller_binary(script_path, binary_name):
print(f"[*] Compiling {binary_name} with PyInstaller...")
cmd = [
sys.executable, "-m", "PyInstaller",
"--onefile",
"--clean",
"--distpath", DIST_DIR,
"--workpath", os.path.join(BUILD_TEMP, f"work_{binary_name}"),
"--specpath", os.path.join(BUILD_TEMP, f"spec_{binary_name}"),
"--name", binary_name,
script_path
]
res = subprocess.run(cmd, capture_output=True, text=True)
if res.returncode != 0:
print(f"[!] Compilation error for {binary_name}:\n{res.stderr}")
raise RuntimeError(f"Failed to build {binary_name}")
print(f"[+] Successfully compiled {binary_name}")
def calculate_sha256(filepath):
h = hashlib.sha256()
with open(filepath, "rb") as f:
while chunk := f.read(65536):
h.update(chunk)
return h.hexdigest()
def write_checksum_file(filename, digest, binary_filename):
out_path = os.path.join(DIST_DIR, filename)
with open(out_path, "w", encoding="utf-8") as f:
f.write(f"{digest} {binary_filename}\n")
print(f"[+] Generated checksum file: {filename} ({digest[:16]}...)")
def build_linux_zipapp_fallback():
print("[*] Packaging Linux standalone zipapp fallback binaries...")
# Linux Client zipapp
app_dir = os.path.join(BUILD_TEMP, "linux_app")
os.makedirs(app_dir, exist_ok=True)
shutil.copy(os.path.join(SRC_DIR, "Linux_Client.py"), os.path.join(app_dir, "Linux_Client.py"))
client_out = os.path.join(DIST_DIR, "Linux_Client.bin")
zipapp.create_archive(
source=app_dir,
target=client_out,
interpreter="/usr/bin/env python3",
main="Linux_Client:main"
)
# Server zipapp
srv_dir = os.path.join(BUILD_TEMP, "linux_srv")
os.makedirs(srv_dir, exist_ok=True)
shutil.copy(os.path.join(SRC_DIR, "Server.py"), os.path.join(srv_dir, "Server.py"))
server_out = os.path.join(DIST_DIR, "Server.bin")
zipapp.create_archive(
source=srv_dir,
target=server_out,
interpreter="/usr/bin/env python3",
main="Server:main"
)
def build_windows():
print("[*] Compiling Windows standalone executables...")
win_client_script = os.path.join(SRC_DIR, "Win_Client.py")
build_pyinstaller_binary(win_client_script, "Win_Client")
server_script = os.path.join(SRC_DIR, "Server.py")
build_pyinstaller_binary(server_script, "Server")
client_bin = os.path.join(DIST_DIR, "Win_Client.exe")
server_bin = os.path.join(DIST_DIR, "Server.exe")
sums = []
if os.path.exists(client_bin):
client_hash = calculate_sha256(client_bin)
write_checksum_file("win_client_sha256sum", client_hash, "Win_Client.exe")
write_checksum_file("win_agent_sha256sum", client_hash, "Win_Client.exe")
sums.append(f"{client_hash} Win_Client.exe")
else:
print(f"[!] Warning: Expected {client_bin} was not found.")
if os.path.exists(server_bin):
server_hash = calculate_sha256(server_bin)
write_checksum_file("win_server_sha256sum", server_hash, "Server.exe")
sums.append(f"{server_hash} Server.exe")
else:
print(f"[!] Warning: Expected {server_bin} was not found.")
sums_path = os.path.join(DIST_DIR, "SHA256SUMS_windows.txt")
with open(sums_path, "w", encoding="utf-8") as f:
f.write("\n".join(sums) + "\n")
def build_linux():
print("[*] Compiling Linux standalone binaries...")
is_linux_host = platform.system() == "Linux"
if is_linux_host:
linux_client_script = os.path.join(SRC_DIR, "Linux_Client.py")
build_pyinstaller_binary(linux_client_script, "Linux_Client.bin")
server_script = os.path.join(SRC_DIR, "Server.py")
build_pyinstaller_binary(server_script, "Server.bin")
# Normalize extensions in case PyInstaller dropped .bin
for name in ["Linux_Client", "Server"]:
plain_path = os.path.join(DIST_DIR, name)
bin_path = os.path.join(DIST_DIR, f"{name}.bin")
if os.path.exists(plain_path) and not os.path.exists(bin_path):
os.rename(plain_path, bin_path)
# Ensure executable permissions on Linux
for b in ["Linux_Client.bin", "Server.bin"]:
p = os.path.join(DIST_DIR, b)
if os.path.exists(p):
try:
os.chmod(p, 0o755)
except Exception:
pass
else:
print("[!] Note: Host platform is not Linux. Generating executable zipapps for Linux target.")
build_linux_zipapp_fallback()
client_bin = os.path.join(DIST_DIR, "Linux_Client.bin")
server_bin = os.path.join(DIST_DIR, "Server.bin")
sums = []
if os.path.exists(client_bin):
client_hash = calculate_sha256(client_bin)
write_checksum_file("linux_client_sha256sum", client_hash, "Linux_Client.bin")
write_checksum_file("linux_agent_sha256sum", client_hash, "Linux_Client.bin")
sums.append(f"{client_hash} Linux_Client.bin")
else:
print(f"[!] Warning: Expected {client_bin} was not found.")
if os.path.exists(server_bin):
server_hash = calculate_sha256(server_bin)
write_checksum_file("linux_server_sha256sum", server_hash, "Server.bin")
sums.append(f"{server_hash} Server.bin")
else:
print(f"[!] Warning: Expected {server_bin} was not found.")
sums_path = os.path.join(DIST_DIR, "SHA256SUMS_linux.txt")
with open(sums_path, "w", encoding="utf-8") as f:
f.write("\n".join(sums) + "\n")
def main(target=None):
if target is None:
parser = argparse.ArgumentParser(description="LOGAR Standalone Binary Compiler & Packager")
parser.add_argument(
"--target", "-t",
choices=["windows", "win", "linux", "auto"],
default="auto",
help="Target platform to compile binaries for (default: auto-detect)"
)
args, _ = parser.parse_known_args()
target = args.target
if target == "auto":
target = "windows" if platform.system() == "Windows" else "linux"
elif target == "win":
target = "windows"
print("=" * 60)
print(" LOGAR Binary Packaging")
print(f" Host Platform: {platform.system()} ({platform.machine()})")
print(f" Target Platform: {target.upper()}")
print("=" * 60)
clean_and_prep()
if target == "windows":
build_windows()
elif target == "linux":
build_linux()
else:
raise ValueError(f"Unsupported target: {target}")
# Clean temporary build directory
if os.path.exists(BUILD_TEMP):
shutil.rmtree(BUILD_TEMP, ignore_errors=True)
print("\n[+] Binary shipping artifacts assembled in 'dist/':")
for f in sorted(os.listdir(DIST_DIR)):
sz = os.path.getsize(os.path.join(DIST_DIR, f))
print(f" - {f} ({sz / (1024*1024):.2f} MB)" if sz > 1024*1024 else f" - {f} ({sz} bytes)")
print("=" * 60)
if __name__ == "__main__":
main()
+205
View File
@@ -0,0 +1,205 @@
import os
import sys
import json
import argparse
import urllib.request
import urllib.parse
ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
sys.path.insert(0, os.path.dirname(__file__))
import package_dist
import time
DEFAULT_GITEA_URL = os.environ.get("GITEA_SERVER_URL", "https://gitea.eibl.tech")
DEFAULT_REPO = os.environ.get("GITEA_REPOSITORY", "me0nline/LOGAR")
def get_existing_assets(base_url, repo, release_id, token):
url = f"{base_url}/api/v1/repos/{repo}/releases/{release_id}/assets"
req = urllib.request.Request(url, headers={"Authorization": f"token {token}", "Accept": "application/json"})
try:
with urllib.request.urlopen(req) as resp:
return json.loads(resp.read().decode("utf-8"))
except Exception:
return []
def delete_asset(base_url, repo, release_id, asset_id, token):
url = f"{base_url}/api/v1/repos/{repo}/releases/{release_id}/assets/{asset_id}"
req = urllib.request.Request(url, method="DELETE", headers={"Authorization": f"token {token}"})
try:
with urllib.request.urlopen(req) as resp:
pass
except Exception:
pass
def upload_file_to_release(base_url, repo, release_id, token, file_path, max_retries=3):
filename = os.path.basename(file_path)
# Clean up existing asset with same name if already present
existing_assets = get_existing_assets(base_url, repo, release_id, token)
for asset in existing_assets:
if asset.get("name") == filename:
print(f"[*] Removing existing asset '{filename}' (ID: {asset['id']})...")
delete_asset(base_url, repo, release_id, asset["id"], token)
url = f"{base_url}/api/v1/repos/{repo}/releases/{release_id}/assets?name={urllib.parse.quote(filename)}"
with open(file_path, "rb") as f:
file_bytes = f.read()
for attempt in range(1, max_retries + 1):
req = urllib.request.Request(url, data=file_bytes, method="POST")
req.add_header("Authorization", f"token {token}")
req.add_header("Content-Type", "application/octet-stream")
req.add_header("Accept", "application/json")
try:
with urllib.request.urlopen(req) as resp:
data = json.loads(resp.read().decode("utf-8"))
print(f"[+] Attached {filename} ({len(file_bytes)} bytes) to release.")
return data
except urllib.error.HTTPError as e:
err = e.read().decode("utf-8", errors="ignore")
print(f"[!] Attempt {attempt}/{max_retries} - Error uploading {filename}: HTTP {e.code} - {err}")
if attempt < max_retries:
time.sleep(2 * attempt)
else:
return None
except Exception as ex:
print(f"[!] Attempt {attempt}/{max_retries} - Exception uploading {filename}: {ex}")
if attempt < max_retries:
time.sleep(2 * attempt)
else:
return None
def create_or_get_release(base_url, repo, tag, token, title=None, notes=None):
url = f"{base_url}/api/v1/repos/{repo}/releases"
headers = {
"Authorization": f"token {token}",
"Content-Type": "application/json",
"Accept": "application/json"
}
# Check if release exists
check_url = f"{base_url}/api/v1/repos/{repo}/releases/tags/{urllib.parse.quote(tag)}"
check_req = urllib.request.Request(check_url, headers={"Authorization": f"token {token}"})
try:
with urllib.request.urlopen(check_req) as resp:
existing = json.loads(resp.read().decode("utf-8"))
print(f"[*] Found existing release for tag {tag} (ID: {existing['id']})")
if notes or title:
patch_url = f"{base_url}/api/v1/repos/{repo}/releases/{existing['id']}"
patch_payload = {}
if title:
patch_payload["name"] = title
if notes:
patch_payload["body"] = notes
patch_req = urllib.request.Request(
patch_url,
data=json.dumps(patch_payload).encode("utf-8"),
headers=headers,
method="PATCH"
)
try:
with urllib.request.urlopen(patch_req) as p_resp:
print(f"[+] Updated release description for tag {tag}")
except Exception as e:
print(f"[!] Warning: Could not update existing release description: {e}")
return existing["id"]
except urllib.error.HTTPError:
pass
# Create new release
payload = {
"tag_name": tag,
"name": title or f"LOGAR Release {tag}",
"body": notes or f"Automated binary release for {tag}.",
"draft": False,
"prerelease": False
}
req = urllib.request.Request(url, data=json.dumps(payload).encode("utf-8"), headers=headers, method="POST")
try:
with urllib.request.urlopen(req) as resp:
created = json.loads(resp.read().decode("utf-8"))
print(f"[+] Created release {tag} (ID: {created['id']})")
return created["id"]
except urllib.error.HTTPError as e:
print(f"[*] Release creation returned HTTP {e.code}. Checking if peer runner created it concurrently...")
for attempt in range(1, 6):
time.sleep(2)
try:
with urllib.request.urlopen(check_req) as resp:
existing = json.loads(resp.read().decode("utf-8"))
print(f"[+] Retrieved peer-created release for tag {tag} (ID: {existing['id']})")
return existing["id"]
except Exception:
pass
raise
def main():
parser = argparse.ArgumentParser(description="Upload LOGAR compiled binaries directly to Gitea Release")
parser.add_argument("--tag", default=os.environ.get("GITEA_REF_NAME"), help="Release tag name (e.g. v1.0.1)")
parser.add_argument("--token", default=os.environ.get("GITEA_TOKEN"), help="Gitea Personal Access Token (or set GITEA_TOKEN env var)")
parser.add_argument("--url", default=DEFAULT_GITEA_URL, help="Base Gitea instance URL")
parser.add_argument("--repo", default=DEFAULT_REPO, help="Repository owner/name")
parser.add_argument("--title", default=os.environ.get("RELEASE_TITLE"), help="Release title")
parser.add_argument("--notes", default=os.environ.get("RELEASE_NOTES"), help="Release description / notes")
parser.add_argument("--notes-file", default=None, help="Path to markdown file with release notes")
parser.add_argument("--skip-build", action="store_true", help="Skip running package_dist.py before upload")
args = parser.parse_args()
tag = args.tag
if not tag:
tag = input("Enter tag name (e.g. v1.0.1): ").strip()
token = args.token
if not token:
token = input("Enter Gitea Token: ").strip()
if not tag or not token:
print("[!] Tag and Token are required.")
sys.exit(1)
# Resolve release notes
notes = args.notes
if not notes and args.notes_file and os.path.exists(args.notes_file):
with open(args.notes_file, "r", encoding="utf-8") as nf:
notes = nf.read()
elif not notes and os.path.exists(os.path.join(ROOT_DIR, "RELEASE_NOTES.md")):
with open(os.path.join(ROOT_DIR, "RELEASE_NOTES.md"), "r", encoding="utf-8") as nf:
notes = nf.read()
elif not notes:
notes = (
f"## LOGAR Release {tag}\n\n"
"### Changes in this Release:\n"
"- **Dual Platform Gitea Release Automation**: Added dedicated Windows (`release-windows.yml`) and Linux (`release-linux.yml`) Gitea Actions to compile native executables and publish assets concurrently.\n"
"- **Dedicated SHA-256 Checksums**: Release assets now include dedicated checksum files matching `[win/linux]_[client/agent]_sha256sum` (e.g., `win_client_sha256sum`, `win_agent_sha256sum`, `win_server_sha256sum`, `linux_client_sha256sum`, `linux_agent_sha256sum`, `linux_server_sha256sum`).\n"
"- **Removed Client Filter Logic**: Removed restrictive source-level noise filtering on edge forwarders. Clients now stream all candidate events from `INFO` up to `ERROR` across the lookback window instead of discarding them at the source.\n"
"- **State Tracking & Deduplication**: Added persistent state tracking (`client_state.json`) with cursor and record number deduplication so previously transmitted events are never resent.\n"
"- **24-Hour Lookback Window**: Forwarders now scan and upload events from the last 24 hours, skipping older entries.\n"
"- **Lightweight Distribution Structure**: Cleaned `out/` to strictly contain deployment documentation and sample configurations.\n"
"- **Automated Gitea CI/CD**: Integrated push testing workflow (`ci.yml`) and automated release asset packaging.\n"
)
title = args.title or f"LOGAR Release {tag}"
if not args.skip_build:
print("[*] Assembling compiled binaries...")
package_dist.main()
dist_dir = os.path.join(ROOT_DIR, "dist")
if not os.path.exists(dist_dir) or not os.listdir(dist_dir):
print(f"[!] No binaries found in {dist_dir}. Run package_dist.py first.")
sys.exit(1)
print(f"[*] Connecting to Gitea: {args.url} (repo: {args.repo})...")
release_id = create_or_get_release(args.url, args.repo, tag, token, title=title, notes=notes)
print(f"[*] Uploading binary assets from '{dist_dir}'...")
for f in sorted(os.listdir(dist_dir)):
fpath = os.path.join(dist_dir, f)
if os.path.isfile(fpath):
upload_file_to_release(args.url, args.repo, release_id, token, fpath)
print(f"\n[+] Release successfully published with binary assets: {args.url}/{args.repo}/releases/tag/{tag}")
if __name__ == "__main__":
main()
-194
View File
@@ -1,194 +0,0 @@
import os
import sys
import json
import socket
import struct
import argparse
import subprocess
import warnings
from datetime import datetime, timezone
# Suppress cryptography / pgpy deprecation notices
warnings.filterwarnings("ignore")
import pgpy
CONFIG_FILE_NAME = "client_config.json"
def load_config(config_path: str = CONFIG_FILE_NAME):
if not os.path.exists(config_path):
raise FileNotFoundError(
f"Client configuration file not found at: {config_path}\n"
f"Generate one from the server using: python Server.py --create-client-config --client-out {config_path}"
)
with open(config_path, "r", encoding="utf-8") as f:
return json.load(f)
def get_machine_identifier() -> str:
"""
Returns the hostname of the machine sending the logs,
and appends the network/DNS domain if available.
"""
fqdn = socket.getfqdn()
if fqdn and "." in fqdn and not fqdn.startswith("localhost"):
return fqdn
hostname = socket.gethostname()
try:
if os.path.exists("/etc/resolv.conf"):
with open("/etc/resolv.conf", "r", encoding="utf-8") as f:
for line in f:
parts = line.strip().split()
if parts and parts[0] in ["domain", "search"] and len(parts) > 1:
domain = parts[1]
if domain and not domain.startswith("."):
return f"{hostname}.{domain}"
except Exception:
pass
try:
host_ip = socket.gethostbyname(hostname)
canonical_name = socket.gethostbyaddr(host_ip)[0]
if canonical_name and "." in canonical_name and not canonical_name.startswith("localhost"):
return canonical_name
except Exception:
pass
return hostname
def get_recent_linux_logs(hours: int = 6):
"""
Collects warnings and errors from systemd journalctl over the lookback window.
Edge Thinness: Drops INFO and DEBUG entries at the source.
"""
cmd = ["journalctl", "--since", f"{hours} hours ago", "-p", "warning", "--output=json"]
try:
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
except FileNotFoundError:
print("[!] journalctl command not found. Ensure this script runs on a systemd-enabled Linux system.")
return []
except Exception as e:
print(f"[!] Error running journalctl: {e}")
return []
logs = []
machine_id = get_machine_identifier()
for line in result.stdout.splitlines():
line_str = line.strip()
if not line_str:
continue
try:
entry = json.loads(line_str)
priority = str(entry.get("PRIORITY", "4"))
if int(priority) > 4:
continue
sev = "WARNING" if priority == "4" else "ERROR"
logs.append({
"server": machine_id,
"os_type": "linux",
"signature": entry.get("SYSLOG_IDENTIFIER", "unknown"),
"severity": sev,
"message": entry.get("MESSAGE", "")[:2048]
})
except (json.JSONDecodeError, ValueError):
continue
return logs
def send_encrypted_logs_over_socket(config: dict, logs: list):
"""
Encrypts the payload using the server's OpenPGP public key and streams
over an authenticated TCP socket. Zero local state is maintained on the client.
"""
server_host = config["server_host"]
server_port = int(config["server_port"])
auth_token = config["auth_token"]
pub_key_armored = config["server_public_key"]
expected_fp = config.get("server_fingerprint", "").replace(" ", "").upper()
pub_key, _ = pgpy.PGPKey.from_blob(pub_key_armored)
actual_fp = str(pub_key.fingerprint).replace(" ", "").upper()
if expected_fp and actual_fp != expected_fp:
raise ValueError(
f"Server fingerprint mismatch! Expected {expected_fp}, but key has {actual_fp}."
)
machine_id = get_machine_identifier()
payload = {
"server": machine_id,
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
payload_json = json.dumps(payload)
pgp_msg = pgpy.PGPMessage.new(payload_json)
encrypted_msg = pub_key.encrypt(pgp_msg)
encrypted_armored = str(encrypted_msg)
envelope = {
"auth_token": auth_token,
"timestamp": datetime.now(timezone.utc).isoformat(),
"encrypted_payload": encrypted_armored
}
envelope_bytes = json.dumps(envelope).encode("utf-8")
print(f"[*] Connecting to LOGAR server at {server_host}:{server_port} over secure TCP socket...")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.settimeout(15.0)
sock.connect((server_host, server_port))
frame = struct.pack(">I", len(envelope_bytes)) + envelope_bytes
sock.sendall(frame)
resp_len_bytes = sock.recv(4)
if not resp_len_bytes:
raise ConnectionError("Server closed connection without response.")
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = bytearray()
while len(resp_bytes) < resp_len:
chunk = sock.recv(min(4096, resp_len - len(resp_bytes)))
if not chunk:
break
resp_bytes.extend(chunk)
response = json.loads(resp_bytes.decode("utf-8"))
print(f"[+] Server response: {response}")
return response
def main():
parser = argparse.ArgumentParser(description="LOGAR Linux Edge Log Forwarder (Zero State)")
parser.add_argument("--config", default=CONFIG_FILE_NAME, help="Path to client_config.json")
parser.add_argument("--hours", type=int, default=6, help="Lookback window in hours for journalctl logs")
args = parser.parse_args()
try:
config = load_config(args.config)
except Exception as e:
print(f"[!] Configuration error: {e}")
sys.exit(1)
machine_id = get_machine_identifier()
print(f"[*] Edge Forwarder Node: {machine_id}")
print(f"[*] Scanning Linux journalctl for candidate anomalies (last {args.hours} hours)...")
candidate_logs = get_recent_linux_logs(hours=args.hours)
print(f"[*] Found {len(candidate_logs)} candidate anomalies (noise stripped at source).")
try:
send_encrypted_logs_over_socket(config, candidate_logs)
except Exception as e:
print(f"[!] Failed to stream logs to server: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
+113 -38
View File
@@ -1,62 +1,137 @@
# LOGAR Linux Edge Forwarder # LOGAR Linux Edge Forwarder
Lightweight edge log forwarder for Linux servers running systemd. Standalone compiled binary and automated systemd service distribution for Linux edge servers.
## Features ---
- **Zero Local State**: No local SQLite database or state tracking on the edge server.
- **Edge Noise Stripping**: Strips conversational/informational noise (`INFO`, `DEBUG`) directly at the source via `journalctl -p warning`.
- **End-to-End OpenPGP Encryption**: Encrypts logs using the server's public key; decrypted exclusively on the cloud hub.
- **Authenticated TCP Socket**: Direct, low-overhead TCP streaming with token authentication.
- **No GPG Binary Required**: Pure-Python implementation (`pgpy` + `cryptography`).
## Installation ## Overview
`Linux_Client.bin` is a self-contained, pre-compiled executable that queries `systemd-journald` via `journalctl`, filters logs directly at the source, auto-enrolls with the central LOGAR hub, and streams candidate events over mutual TLS 1.3 (**mTLS**) to the central hub.
### Key Capabilities
- **Pre-compiled & Dependency-Free**: Ships as a standalone native binary (`Linux_Client.bin`). No Python environment, pip packages, or GnuPG binaries are required on the host.
- **Mutual TLS 1.3 (mTLS) Ingestion**: Streams directly over hardware-authenticated TLS 1.3 sockets with machine-bound client certificates.
- **Automated Client Enrollment**: On first run with an `enrollment_secret`, the client automatically calls `POST /api/client/enroll` on the hub, saves its certificates into `/etc/logar/certs/`, and establishes secure mTLS streaming.
- **Proactive Expiry Check & Reactive Self-Healing**: Before each run, the client evaluates `client.crt` validity. If within 30 days of expiry, it automatically contacts the hub to renew certificates. If the server Root CA rotates or a TLS handshake error occurs, the client catch-heals by re-enrolling immediately and re-establishing connection without human intervention.
- **Source-Level Filtering**: Retains events spanning `INFO`, `WARNING`, and `ERROR` (`journalctl -p warning`). Drops debug noise and skips events older than 24 hours.
- **State Tracking & Deduplication**: Maintains persistent client state in `client_state.json` (tracking systemd journalctl cursors and microsecond timestamps) so every log record is forwarded exactly once without duplicates.
- **Fail-Safe State Commit**: State is committed only when the server returns a verified `success` response. In the event of a network outage, state remains unchanged and unsent events are retried automatically on the next run.
---
## 1. Automated Installation via Script (Recommended)
Run the automated installer script:
```bash ```bash
python3 -m pip install -r requirements.txt sudo ./compilation/install_linux_client.sh "http://<HUB_HOST>:8443" "<ENROLLMENT_SECRET>"
```
This script:
1. Installs the binary to `/opt/logar-client/Linux_Client`.
2. Creates `/etc/logar/certs` with strict permissions.
3. Automatically queries `/etc/machine-id` and enrolls with the hub via `curl`.
4. Deploys, enables, and starts the systemd service unit `/etc/systemd/system/logar-client.service`.
---
## 2. Generating & Deploying the Configuration File
### Step 1: Generate `client_config.json` on the Server
Run the following command on your central LOGAR server:
```bash
python src/Server.py --create-client-config --server-host <SERVER_IP_OR_DNS> --server-port 9443 --client-out client_config.json
```
- Replace `<SERVER_IP_OR_DNS>` with the reachable IP address or FQDN of your central LOGAR server hub.
- Default mTLS socket port is `9443`; Hermes REST API port is `8443`.
### Step 2: Configuration Structure
The generated `client_config.json` contains:
```json
{
"server_host": "192.168.1.100",
"server_port": 9443,
"hermes_host": "192.168.1.100",
"hermes_port": 8443,
"enrollment_secret": "a1b2c3d4e5f6...",
"cert_dir": "certs",
"server_fingerprint": "375388960531264EA0648EC0D2C4E4ABC6F22AC2",
"server_public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
"auth_token": "a1b2c3d4e5f6..."
}
``` ```
## Configuration > [!NOTE]
Place `client_config.json` generated by the server (`Server.py --create-client-config`) in the same directory as `Linux_Client.py`. > The configuration contains **no host-specific names or site names** to ensure client anonymity and easy redistribution.
## Running the Forwarder ### Step 3: Copy to Edge Node
Place `Linux_Client.bin` and `client_config.json` into the target directory (e.g. `/opt/logar/`):
```bash ```bash
python3 Linux_Client.py --hours 6 sudo mkdir -p /opt/logar
sudo cp Linux_Client.bin client_config.json /opt/logar/
sudo chmod +x /opt/logar/Linux_Client.bin
``` ```
## Cron / Systemd Timer Deployment ---
### Option A: Cron Job (Every 3 hours)
```bash
0 */3 * * * cd /opt/logar && /usr/bin/python3 Linux_Client.py --hours 6 >> /var/log/logar_client.log 2>&1
```
### Option B: Systemd Service & Timer ## 3. Running Manually
1. Create `/etc/systemd/system/logar-forwarder.service`:
Test the forwarder interactively:
```bash
cd /opt/logar
./Linux_Client.bin --hours 24
```
On first run, the client contacts `http://<hermes_host>:<hermes_port>/api/client/enroll`, downloads `ca.crt`, `client.crt`, and `client.key` into `certs/`, and streams logs over mTLS.
### Command-Line Arguments
| Argument | Default | Description |
| :--- | :--- | :--- |
| `--config` | `client_config.json` | Path to client configuration file |
| `--hours` | `24` | Lookback window in hours for journal logs |
| `--state-file` | `client_state.json` | Path to persistent state file |
| `--no-state` | `False` | Disable state tracking and send all events matching lookback window |
---
## 4. Manual Systemd Service & Timer Setup
### Step 1: Create the Systemd Service Unit
Create `/etc/systemd/system/logar-client.service`:
```ini ```ini
[Unit] [Unit]
Description=LOGAR Edge Forwarder Description=LOGAR Edge Log Forwarder
After=network.target After=network-online.target
Wants=network-online.target
[Service] [Service]
Type=oneshot Type=simple
WorkingDirectory=/opt/logar WorkingDirectory=/opt/logar
ExecStart=/usr/bin/python3 /opt/logar/Linux_Client.py --hours 6 ExecStart=/opt/logar/Linux_Client.bin --hours 24
``` Restart=always
RestartSec=5s
2. Create `/etc/systemd/system/logar-forwarder.timer`: User=root
```ini StandardOutput=journal
[Unit] StandardError=journal
Description=Run LOGAR Edge Forwarder every 3 hours
[Timer]
OnBootSec=5min
OnUnitActiveSec=3h
Persistent=true
[Install] [Install]
WantedBy=timers.target WantedBy=multi-user.target
``` ```
3. Enable and start: ### Step 2: Enable and Start the Service
```bash ```bash
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo systemctl enable --now logar-forwarder.timer sudo systemctl enable --now logar-client.service
```
### Step 3: Check Logs
```bash
sudo journalctl -u logar-client.service -n 50 -f
```
---
## 5. Uninstallation & Removal
```bash
sudo systemctl disable --now logar-client.service
sudo rm -f /etc/systemd/system/logar-client.service
sudo systemctl daemon-reload
sudo rm -rf /opt/logar-client /opt/logar /etc/logar
``` ```
-11
View File
@@ -1,11 +0,0 @@
#!/usr/bin/env bash
# Build script to compile Linux_Client into a standalone native ELF binary on Linux
set -e
echo "[*] Installing build requirements..."
pip3 install pyinstaller pgpy cryptography standard-imghdr
echo "[*] Compiling Linux_Client native binary..."
pyinstaller --onefile --clean --name Linux_Client.bin Linux_Client.py
echo "[+] Compilation successful: dist/Linux_Client.bin"
+1 -2
View File
@@ -3,6 +3,5 @@
"server_port": 9443, "server_port": 9443,
"server_fingerprint": "PASTE_SERVER_FINGERPRINT_HERE", "server_fingerprint": "PASTE_SERVER_FINGERPRINT_HERE",
"server_public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...\n-----END PGP PUBLIC KEY BLOCK-----\n", "server_public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...\n-----END PGP PUBLIC KEY BLOCK-----\n",
"auth_token": "PASTE_AUTH_TOKEN_HERE", "auth_token": "PASTE_AUTH_TOKEN_HERE"
"site_name": "Frankfurt-DC"
} }
-3
View File
@@ -1,3 +0,0 @@
pgpy>=0.6.0
standard-imghdr>=3.13.0; python_version >= "3.13"
cryptography>=42.0.0
-107
View File
@@ -1,107 +0,0 @@
import os
import sys
import json
import struct
import unittest
import warnings
warnings.filterwarnings("ignore")
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
import Linux_Client
import pgpy
from pgpy.constants import PubKeyAlgorithm, KeyFlags, HashAlgorithm, SymmetricKeyAlgorithm, CompressionAlgorithm
class TestLinuxClientComponent(unittest.TestCase):
def setUp(self):
self.dummy_config = "test_linux_client_config.json"
# Generate dummy PGP key for testing
key = pgpy.PGPKey.new(PubKeyAlgorithm.RSAEncryptOrSign, 2048)
uid = pgpy.PGPUID.new("TestHub")
key.add_uid(
uid,
usage={KeyFlags.EncryptCommunications, KeyFlags.EncryptStorage},
hashes=[HashAlgorithm.SHA256],
ciphers=[SymmetricKeyAlgorithm.AES256],
compression=[CompressionAlgorithm.Uncompressed]
)
self.server_priv = key
self.server_pub = key.pubkey
self.fingerprint = str(key.pubkey.fingerprint)
with open(self.dummy_config, "w", encoding="utf-8") as f:
json.dump({
"server_host": "127.0.0.1",
"server_port": 9443,
"server_fingerprint": self.fingerprint,
"server_public_key": str(self.server_pub),
"auth_token": "secret-test-token"
}, f)
def tearDown(self):
if os.path.exists(self.dummy_config):
try:
os.remove(self.dummy_config)
except Exception:
pass
def test_client_config_anonymity(self):
config = Linux_Client.load_config(self.dummy_config)
self.assertNotIn("server_name", config)
self.assertNotIn("name", config)
self.assertNotIn("site_name", config)
self.assertEqual(config["server_fingerprint"], self.fingerprint)
def test_get_machine_identifier(self):
machine_id = Linux_Client.get_machine_identifier()
self.assertIsInstance(machine_id, str)
self.assertGreater(len(machine_id), 0)
self.assertNotEqual(machine_id, "localhost")
def test_journalctl_parsing_and_priority_filter(self):
sample_journal_lines = [
json.dumps({"PRIORITY": "3", "SYSLOG_IDENTIFIER": "sshd", "MESSAGE": "Failed password for root"}),
json.dumps({"PRIORITY": "4", "SYSLOG_IDENTIFIER": "systemd", "MESSAGE": "Unit entered failed state"}),
json.dumps({"PRIORITY": "6", "SYSLOG_IDENTIFIER": "cron", "MESSAGE": "Informational session opened"}),
]
logs = []
machine_id = Linux_Client.get_machine_identifier()
for line_str in sample_journal_lines:
entry = json.loads(line_str)
priority = str(entry.get("PRIORITY", "4"))
if int(priority) > 4:
continue
sev = "WARNING" if priority == "4" else "ERROR"
logs.append({
"server": machine_id,
"os_type": "linux",
"signature": entry.get("SYSLOG_IDENTIFIER", "unknown"),
"severity": sev,
"message": entry.get("MESSAGE", "")
})
# Priority 6 must be stripped (INFO noise)
self.assertEqual(len(logs), 2)
self.assertEqual(logs[0]["severity"], "ERROR")
self.assertEqual(logs[1]["severity"], "WARNING")
def test_encryption_and_decryption(self):
config = Linux_Client.load_config(self.dummy_config)
pub_key, _ = pgpy.PGPKey.from_blob(config["server_public_key"])
payload = {
"server": Linux_Client.get_machine_identifier(),
"logs": [{"signature": "kernel", "severity": "ERROR", "message": "Kernel panic - not syncing"}]
}
msg = pgpy.PGPMessage.new(json.dumps(payload))
enc = pub_key.encrypt(msg)
self.assertTrue(str(enc).startswith("-----BEGIN PGP MESSAGE-----"))
dec = self.server_priv.decrypt(enc)
restored = json.loads(dec.message)
self.assertEqual(restored["logs"][0]["signature"], "kernel")
if __name__ == "__main__":
unittest.main()
+123
View File
@@ -0,0 +1,123 @@
# LOGAR Linux Server Hub
Standalone compiled binary and automated systemd service distribution for Linux server environments (`Server.bin`).
---
## Overview
`Server.bin` is a self-contained, pre-compiled Linux ELF executable that operates as the central coordination, log analysis, dynamic PKI, and reporting hub of the LOGAR telemetry architecture.
### Key Architecture & Capabilities
- **Pre-compiled & Dependency-Free**: Ships as a standalone native Linux ELF binary (`Server.bin`). No Python runtime, pip dependencies, or GnuPG binaries are required on the host system.
- **Mutual TLS 1.3 (mTLS) Ingestion (Port 9443)**: Enforces mutual TLS 1.3 authentication for all incoming edge connections. Validates client certificates against an internal Root CA and verifies active licensing in SQLite.
- **Dynamic PKI & License Accounting**: Built-in Root CA generates server TLS certificates with SANs and dynamically signs client certificates via `POST /api/client/enroll` while enforcing seat limits (`max_seats`).
- **In-Flight Certificate Watchdog & Dynamic Reloading**: Continuously monitors Root CA (`ca.crt`) and Server TLS certificate (`server.crt`) validity in the background (every 12 hours). When nearing expiration (< 30 days), certificates are automatically regenerated with timestamped backups, and active `ssl.SSLContext` structures are reloaded dynamically without dropping socket connections or restarting the systemd service.
- **Warning Persistence & Immediate Error Routing**: High-severity `ERROR`, `CRITICAL`, and `FATAL` events are promoted to `VERIFIED` immediately on their first occurrence. Operational `WARNING` and `INFO` events require persistence across at least 4 distinct client transmission cycles within a rolling 12-hour evaluation window.
- **Embedded Hermes Reporting & Management API (Port 8443)**: Integrated REST API exposing `/api/hermes/report`, `/api/clients`, and `/api/client/enroll`.
- **State Database**: Stores issue lifecycle records, client telemetry, and licensing quotas in a local SQLite database (`logar_state.db`).
---
## 1. Automated Installation via Script (Recommended)
Deploy using the automated installer:
```bash
sudo ./compilation/install_linux_server.sh
```
This script:
1. Installs the server binary to `/opt/logar-server/Server`.
2. Creates `/etc/logar` and `/var/log/logar`.
3. Deploys, enables, and starts the systemd service unit `/etc/systemd/system/logar-server.service` with `LimitNOFILE=65536`.
---
## 2. Initializing & Generating Server Configuration
### Step 1: Automatic First-Run Generation
When launched without an existing `server_config.json`, `Server.bin` automatically generates:
1. An internal Root CA (`certs/ca.crt` and `certs/ca.key`).
2. A server TLS certificate (`certs/server.crt` and `certs/server.key`) with SANs.
3. An OpenPGP RSA-2048 keypair (`private_key` and `public_key`).
4. Cryptographically random authentication tokens and enrollment secrets.
5. Default network socket coordinates (mTLS 9443, Hermes API 8443).
Run `Server.bin` once to initialize:
```bash
./Server.bin
```
Output:
```
[!] Config 'server_config.json' not found. Initializing first-run configuration...
[+] Successfully generated new server config and OpenPGP keypair.
[+] Server Encryption Fingerprint: 375388960531264EA0648EC0D2C4E4ABC6F22AC2
[+] Saved to: server_config.json
============================================================
LOGAR Server Hub: LOGAR-Cloud-Hub
Transport Security: mTLS (TLS 1.3)
License Quota: 10 Active Seats
Server Encryption Fingerprint: 375388960531264EA0648EC0D2C4E4ABC6F22AC2
Evaluation Window: 12 hours | 4-Run Rule: Warnings | Immediate Pass: Errors
============================================================
[*] LOGAR mTLS TLSv1.3 Socket Server listening on 0.0.0.0:9443
[*] Hermes Reporting API available at http://0.0.0.0:8443/api/hermes/report
[*] Client Enrollment API available at http://0.0.0.0:8443/api/client/enroll
```
### Step 2: Configuration Fields Reference
The generated `server_config.json` contains:
```json
{
"server_name": "LOGAR-Linux-Hub",
"tcp_host": "0.0.0.0",
"tcp_port": 9443,
"hermes_host": "0.0.0.0",
"hermes_port": 8443,
"auth_token": "a1b2c3d4e5f67890abcdef1234567890...",
"enrollment_secret": "e1f2a3b4c5d6...",
"max_seats": 10,
"cert_dir": "certs",
"tls_enabled": true,
"db_path": "logar_state.db",
"evaluation_window_hours": 12,
"min_persistence_runs": 4,
"server_fingerprint": "375388960531264EA0648EC0D2C4E4ABC6F22AC2",
"public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
"private_key": "-----BEGIN PGP PRIVATE KEY BLOCK-----\n..."
}
```
---
## 3. Exporting Client Configurations
Generate a client configuration bundle to deploy onto Windows or Linux forwarders:
```bash
./Server.bin --create-client-config --server-host 192.168.1.100 --server-port 9443 --client-out client_config.json
```
The output file contains the server coordinates, enrollment secret, and fingerprint, ready for client deployment.
---
## 4. Manual Systemd Service Management
Check service status:
```bash
sudo systemctl status logar-server.service
```
Inspect live service logs:
```bash
sudo journalctl -u logar-server.service -f -n 50
```
---
## 5. Uninstallation & Removal
```bash
sudo systemctl disable --now logar-server.service
sudo rm -f /etc/systemd/system/logar-server.service
sudo systemctl daemon-reload
sudo rm -rf /opt/logar-server /etc/logar /var/log/logar
```
@@ -0,0 +1,14 @@
{
"server_name": "LOGAR-Linux-Hub",
"tcp_host": "0.0.0.0",
"tcp_port": 9443,
"hermes_host": "0.0.0.0",
"hermes_port": 8443,
"auth_token": "replace_with_secure_random_hex_token",
"db_path": "logar_state.db",
"evaluation_window_hours": 12,
"min_persistence_runs": 4,
"server_fingerprint": "AUTO_GENERATED_ON_FIRST_RUN",
"public_key": "AUTO_GENERATED_ON_FIRST_RUN",
"private_key": "AUTO_GENERATED_ON_FIRST_RUN"
}
-36
View File
@@ -1,36 +0,0 @@
# LOGAR Server Hub
Central Python/TCP ingestion server for the LOGAR Log Analysis System.
## Features
- **Zero External GPG Requirement**: Uses pure-Python OpenPGP (`pgpy` + `cryptography`), no native GnuPG binary needed.
- **First-Run Key & Config Auto-generation**: Generates OpenPGP keypairs, auth tokens, and `server_config.json` automatically on first launch.
- **Client Config Exporter**: Generates `client_config.json` bundles containing the server's encryption-only fingerprint and address.
- **Cloud-Side Temporal Persistence**: SQLite database tracking candidate anomalies over 12-hour evaluation windows.
- **4-Run Persistence Rule**: Filters out transient infrastructure blips, promoting issues to `VERIFIED` anomalies only after persisting across $\ge 4$ runs.
- **Agentic Hermes Endpoint**: REST API (`GET /api/hermes/report`) providing verified system artifacts for Hermes agent alerts.
## Installation
```bash
pip install -r requirements.txt
```
## Running the Server
```bash
# Starts both the TCP socket listener (port 9443) and the Hermes API (port 8443)
python Server.py
```
## Generating Client Configurations
To deploy edge forwarders, generate a client config file:
```bash
python Server.py --create-client-config --server-host <SERVER_IP_OR_DNS> --server-port 9443 --site-name "Frankfurt-DC" --client-out client_config.json
```
Copy the generated `client_config.json` into the deployment directory of `Win_Client.py` or `Linux_Client.py`.
## Hermes Agent Integration
Hermes queries the verified anomalies via:
```
GET http://<SERVER_IP>:8443/api/hermes/report
```
Only issues meeting the 4-run persistence rule within the active 12-hour evaluation window are returned.
-437
View File
@@ -1,437 +0,0 @@
# Copy of Server.py without site_name in client_config
import os
import sys
import json
import uuid
import struct
import socket
import sqlite3
import argparse
import asyncio
import secrets
import warnings
from datetime import datetime, timezone, timedelta
from typing import Dict, Any, List, Optional
# Suppress cryptography / pgpy deprecation notices for a clean terminal output
warnings.filterwarnings("ignore")
import pgpy
from pgpy.constants import (
PubKeyAlgorithm,
KeyFlags,
HashAlgorithm,
SymmetricKeyAlgorithm,
CompressionAlgorithm
)
from fastapi import FastAPI, HTTPException
import uvicorn
CONFIG_FILE_NAME = "server_config.json"
DEFAULT_DB_FILE = "logar_state.db"
EVALUATION_WINDOW_HOURS = 12
RUN_THRESHOLD = 4
app = FastAPI(title="LOGAR Cloud Ingestion & Hermes Hub", version="2.0.0")
SERVER_STATE: Dict[str, Any] = {}
def generate_server_keypair(server_name: str):
"""Generates an OpenPGP RSA 2048 key with encryption capability."""
key = pgpy.PGPKey.new(PubKeyAlgorithm.RSAEncryptOrSign, 2048)
uid = pgpy.PGPUID.new(server_name)
key.add_uid(
uid,
usage={KeyFlags.EncryptCommunications, KeyFlags.EncryptStorage},
hashes=[HashAlgorithm.SHA256],
ciphers=[SymmetricKeyAlgorithm.AES256],
compression=[CompressionAlgorithm.Uncompressed]
)
private_key_armored = str(key)
public_key_armored = str(key.pubkey)
fingerprint = str(key.pubkey.fingerprint)
return private_key_armored, public_key_armored, fingerprint
def load_or_init_config(config_path: str = CONFIG_FILE_NAME) -> Dict[str, Any]:
"""Loads existing server_config.json or creates a new one on first run."""
if os.path.exists(config_path):
print(f"[*] Loading server configuration from: {os.path.abspath(config_path)}")
with open(config_path, "r", encoding="utf-8") as f:
config = json.load(f)
return config
print(f"[!] Config '{config_path}' not found. Initializing first-run configuration...")
server_name = "LOGAR-Cloud-Hub"
private_key, public_key, fingerprint = generate_server_keypair(server_name)
auth_token = secrets.token_hex(24)
config = {
"server_name": server_name,
"tcp_host": "0.0.0.0",
"tcp_port": 9443,
"hermes_host": "0.0.0.0",
"hermes_port": 8443,
"auth_token": auth_token,
"db_path": DEFAULT_DB_FILE,
"evaluation_window_hours": EVALUATION_WINDOW_HOURS,
"min_persistence_runs": RUN_THRESHOLD,
"server_fingerprint": fingerprint,
"public_key": public_key,
"private_key": private_key
}
with open(config_path, "w", encoding="utf-8") as f:
json.dump(config, f, indent=2)
print(f"[+] Successfully generated new server config and OpenPGP keypair.")
print(f"[+] Server Encryption Fingerprint: {fingerprint}")
print(f"[+] Saved to: {os.path.abspath(config_path)}")
return config
def create_client_config(
server_host: str,
server_port: int,
output_path: str,
config_path: str = CONFIG_FILE_NAME
) -> Dict[str, Any]:
"""Creates a client configuration file containing the server address, auth token, and encryption-only key/fingerprint."""
server_conf = load_or_init_config(config_path)
client_conf = {
"server_host": server_host,
"server_port": server_port,
"server_fingerprint": server_conf["server_fingerprint"],
"server_public_key": server_conf["public_key"],
"auth_token": server_conf["auth_token"]
}
out_dir = os.path.dirname(os.path.abspath(output_path))
if out_dir and not os.path.exists(out_dir):
os.makedirs(out_dir, exist_ok=True)
with open(output_path, "w", encoding="utf-8") as f:
json.dump(client_conf, f, indent=2)
print(f"[+] Client configuration successfully written to: {os.path.abspath(output_path)}")
print(f" - Server Target: {server_host}:{server_port}")
print(f" - Encryption Fingerprint: {server_conf['server_fingerprint']}")
return client_conf
def init_db(db_path: str):
"""Initializes the SQLite schema for multi-run temporal tracking."""
conn = sqlite3.connect(db_path)
conn.execute("""
CREATE TABLE IF NOT EXISTS active_issues (
fingerprint TEXT PRIMARY KEY,
site_name TEXT,
server TEXT,
signature TEXT,
severity TEXT,
message TEXT,
os_type TEXT,
first_seen TEXT,
last_seen TEXT,
run_count INTEGER,
status TEXT,
last_run_id TEXT
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS ingest_runs (
run_id TEXT PRIMARY KEY,
site_name TEXT,
server TEXT,
timestamp TEXT,
log_count INTEGER
)
""")
conn.commit()
conn.close()
def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: int, min_runs: int) -> Dict[str, Any]:
"""
Evaluates candidate issues against the 12-hour evaluation window and 4-run rule.
Zero-state clients send raw candidate entries; this engine handles temporal state.
"""
client_server = payload.get("server", "unknown-host")
site_name = payload.get("site_name") or (client_server.split(".", 1)[1] if "." in client_server else "default")
logs = payload.get("logs", [])
run_id = str(uuid.uuid4())
now = datetime.now(timezone.utc)
now_iso = now.isoformat()
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute(
"INSERT INTO ingest_runs (run_id, site_name, server, timestamp, log_count) VALUES (?, ?, ?, ?, ?)",
(run_id, site_name, client_server, now_iso, len(logs))
)
processed_count = 0
promoted_to_verified = 0
for log in logs:
severity = str(log.get("severity", "WARNING")).upper()
if severity in ["INFO", "DEBUG"]:
continue
signature = log.get("signature", "unknown")
server = log.get("server", client_server)
message = log.get("message", "")
os_type = log.get("os_type", "unknown")
fp = f"{site_name}:{server}:{signature}"
cursor.execute(
"SELECT run_count, first_seen, last_seen, status, last_run_id FROM active_issues WHERE fingerprint = ?",
(fp,)
)
row = cursor.fetchone()
if row:
run_count, first_seen_str, last_seen_str, current_status, last_run_id = row
try:
last_seen_dt = datetime.fromisoformat(last_seen_str)
except Exception:
last_seen_dt = now
if (now - last_seen_dt) > timedelta(hours=window_hours):
new_runs = 1
new_first_seen = now_iso
new_status = "TRANSIENT"
else:
if last_run_id != run_id:
new_runs = run_count + 1
else:
new_runs = run_count
new_first_seen = first_seen_str
new_status = "VERIFIED" if new_runs >= min_runs else "TRANSIENT"
if new_status == "VERIFIED" and current_status != "VERIFIED":
promoted_to_verified += 1
cursor.execute("""
UPDATE active_issues
SET run_count = ?, last_seen = ?, first_seen = ?, status = ?, last_run_id = ?, message = ?, severity = ?
WHERE fingerprint = ?
""", (new_runs, now_iso, new_first_seen, new_status, run_id, message, severity, fp))
else:
initial_status = "VERIFIED" if 1 >= min_runs else "TRANSIENT"
cursor.execute("""
INSERT INTO active_issues
(fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status, last_run_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (fp, site_name, server, signature, severity, message, os_type, now_iso, now_iso, 1, initial_status, run_id))
processed_count += 1
conn.commit()
conn.close()
return {
"status": "success",
"run_id": run_id,
"processed": processed_count,
"promoted_verified": promoted_to_verified
}
async def handle_socket_client(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
try:
length_bytes = await reader.readexactly(4)
length = struct.unpack(">I", length_bytes)[0]
if length <= 0 or length > 10 * 1024 * 1024:
raise ValueError(f"Invalid frame size: {length}")
payload_bytes = await reader.readexactly(length)
envelope = json.loads(payload_bytes.decode("utf-8"))
expected_token = SERVER_STATE["config"]["auth_token"]
provided_token = envelope.get("auth_token")
if not secrets.compare_digest(str(provided_token), str(expected_token)):
err_msg = json.dumps({"status": "error", "message": "Authentication failed"}).encode("utf-8")
writer.write(struct.pack(">I", len(err_msg)) + err_msg)
await writer.drain()
writer.close()
await writer.wait_closed()
return
encrypted_armored = envelope.get("encrypted_payload", "")
pgp_msg = pgpy.PGPMessage.from_blob(encrypted_armored)
priv_key = SERVER_STATE["private_key_obj"]
decrypted_obj = priv_key.decrypt(pgp_msg)
decrypted_json_str = decrypted_obj.message
log_payload = json.loads(decrypted_json_str)
res = process_ingested_logs(
log_payload,
db_path=SERVER_STATE["config"]["db_path"],
window_hours=SERVER_STATE["config"]["evaluation_window_hours"],
min_runs=SERVER_STATE["config"]["min_persistence_runs"]
)
resp_bytes = json.dumps(res).encode("utf-8")
writer.write(struct.pack(">I", len(resp_bytes)) + resp_bytes)
await writer.drain()
except Exception as e:
err = json.dumps({"status": "error", "message": str(e)}).encode("utf-8")
try:
writer.write(struct.pack(">I", len(err)) + err)
await writer.drain()
except Exception:
pass
finally:
writer.close()
try:
await writer.wait_closed()
except Exception:
pass
@app.get("/api/hermes/report")
def get_hermes_report():
db_path = SERVER_STATE["config"]["db_path"]
window_hours = SERVER_STATE["config"]["evaluation_window_hours"]
min_runs = SERVER_STATE["config"]["min_persistence_runs"]
now = datetime.now(timezone.utc)
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("""
SELECT fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status
FROM active_issues
WHERE status = 'VERIFIED' AND run_count >= ?
""", (min_runs,))
rows = cursor.fetchall()
conn.close()
report = []
for r in rows:
last_seen_dt = datetime.fromisoformat(r[8])
if (now - last_seen_dt) <= timedelta(hours=window_hours):
report.append({
"fingerprint": r[0],
"site": r[1],
"server": r[2],
"signature": r[3],
"severity": r[4],
"message": r[5],
"os_type": r[6],
"first_seen": r[7],
"last_seen": r[8],
"consecutive_runs": r[9],
"evaluation_window": f"{window_hours}h",
"verified": True,
"status": r[10]
})
return report
@app.get("/api/hermes/all")
def get_all_issues():
db_path = SERVER_STATE["config"]["db_path"]
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("""
SELECT fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status
FROM active_issues
""")
rows = cursor.fetchall()
conn.close()
return [
{
"fingerprint": r[0],
"site": r[1],
"server": r[2],
"signature": r[3],
"severity": r[4],
"message": r[5],
"os_type": r[6],
"first_seen": r[7],
"last_seen": r[8],
"run_count": r[9],
"status": r[10]
}
for r in rows
]
@app.get("/health")
def health_check():
return {
"status": "healthy",
"server_name": SERVER_STATE["config"]["server_name"],
"fingerprint": SERVER_STATE["config"]["server_fingerprint"],
"tcp_port": SERVER_STATE["config"]["tcp_port"],
"hermes_port": SERVER_STATE["config"]["hermes_port"]
}
async def run_server():
config = SERVER_STATE["config"]
tcp_host = config["tcp_host"]
tcp_port = int(config["tcp_port"])
hermes_host = config["hermes_host"]
hermes_port = int(config["hermes_port"])
tcp_server = await asyncio.start_server(handle_socket_client, tcp_host, tcp_port)
print(f"[*] LOGAR TCP Socket Server listening on {tcp_host}:{tcp_port}")
uv_config = uvicorn.Config(app, host=hermes_host, port=hermes_port, log_level="warning")
uv_server = uvicorn.Server(uv_config)
print(f"[*] Hermes Reporting API available at http://{hermes_host}:{hermes_port}/api/hermes/report")
await asyncio.gather(
tcp_server.serve_forever(),
uv_server.serve()
)
def main():
parser = argparse.ArgumentParser(description="LOGAR Cloud Hub & TCP Socket Ingestion Server")
parser.add_argument("--config", default=CONFIG_FILE_NAME, help="Path to server_config.json")
parser.add_argument("--create-client-config", action="store_true", help="Generate a client config with encryption-only fingerprint and server address")
parser.add_argument("--client-out", default="client_config.json", help="Output file path for generated client config")
parser.add_argument("--server-host", default="127.0.0.1", help="Server address to embed in client config")
parser.add_argument("--server-port", type=int, default=None, help="TCP port to embed in client config")
args = parser.parse_args()
config = load_or_init_config(args.config)
init_db(config["db_path"])
priv_key_obj, _ = pgpy.PGPKey.from_blob(config["private_key"])
SERVER_STATE["config"] = config
SERVER_STATE["private_key_obj"] = priv_key_obj
if args.create_client_config:
port = args.server_port or config["tcp_port"]
create_client_config(
server_host=args.server_host,
server_port=port,
output_path=args.client_out,
config_path=args.config
)
sys.exit(0)
print("=" * 60)
print(f" LOGAR Server Hub: {config['server_name']}")
print(f" Encryption Fingerprint: {config['server_fingerprint']}")
print(f" Evaluation Window: {config['evaluation_window_hours']} hours | Rule: {config['min_persistence_runs']}+ consecutive runs")
print("=" * 60)
try:
asyncio.run(run_server())
except KeyboardInterrupt:
print("\n[!] Server shutting down.")
if __name__ == "__main__":
main()
-6
View File
@@ -1,6 +0,0 @@
pgpy>=0.6.0
standard-imghdr>=3.13.0; python_version >= "3.13"
cryptography>=42.0.0
fastapi>=0.110.0
uvicorn>=0.28.0
pydantic>=2.6.0
-119
View File
@@ -1,119 +0,0 @@
import os
import sys
import json
import socket
import struct
import sqlite3
import unittest
import urllib.request
import warnings
from datetime import datetime, timezone, timedelta
warnings.filterwarnings("ignore")
# Ensure parent directory is in path to import Server
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
import Server
import pgpy
class TestServerComponent(unittest.TestCase):
def setUp(self):
self.test_db = "test_server_state.db"
self.test_config = "test_server_config.json"
if os.path.exists(self.test_db):
os.remove(self.test_db)
if os.path.exists(self.test_config):
os.remove(self.test_config)
def tearDown(self):
if os.path.exists(self.test_db):
try:
os.remove(self.test_db)
except Exception:
pass
if os.path.exists(self.test_config):
try:
os.remove(self.test_config)
except Exception:
pass
def test_first_run_config_and_keypair_generation(self):
config = Server.load_or_init_config(self.test_config)
self.assertTrue(os.path.exists(self.test_config))
self.assertIn("server_fingerprint", config)
self.assertIn("public_key", config)
self.assertIn("private_key", config)
self.assertIn("auth_token", config)
self.assertNotIn("site_name", config)
# Verify keypair
priv_key, _ = pgpy.PGPKey.from_blob(config["private_key"])
pub_key, _ = pgpy.PGPKey.from_blob(config["public_key"])
self.assertEqual(str(pub_key.fingerprint), config["server_fingerprint"])
def test_create_client_config(self):
Server.load_or_init_config(self.test_config)
client_out = "test_client_out.json"
try:
client_conf = Server.create_client_config(
server_host="10.0.0.1",
server_port=9443,
output_path=client_out,
config_path=self.test_config
)
self.assertTrue(os.path.exists(client_out))
self.assertEqual(client_conf["server_host"], "10.0.0.1")
self.assertEqual(client_conf["server_port"], 9443)
# Verify no machine name or site_name is included
self.assertNotIn("server_name", client_conf)
self.assertNotIn("name", client_conf)
self.assertNotIn("site_name", client_conf)
finally:
if os.path.exists(client_out):
os.remove(client_out)
def test_4_run_rule_and_12h_window(self):
Server.init_db(self.test_db)
log_entry = {
"server": "app-worker-01.corp.local",
"signature": "PostgresConnTimeout",
"severity": "ERROR",
"message": "Connection to database pool timed out after 30s",
"os_type": "linux"
}
payload = {
"server": "app-worker-01.corp.local",
"logs": [log_entry]
}
# Runs 1 to 3: should remain TRANSIENT
for run_idx in range(1, 4):
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
self.assertEqual(res["status"], "success")
self.assertEqual(res["promoted_verified"], 0)
conn = sqlite3.connect(self.test_db)
c = conn.cursor()
c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnTimeout",))
row = c.fetchone()
conn.close()
self.assertEqual(row[0], 3)
self.assertEqual(row[1], "TRANSIENT")
# Run 4: promotes to VERIFIED!
res4 = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
self.assertEqual(res4["promoted_verified"], 1)
conn = sqlite3.connect(self.test_db)
c = conn.cursor()
c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnTimeout",))
row = c.fetchone()
conn.close()
self.assertEqual(row[0], 4)
self.assertEqual(row[1], "VERIFIED")
if __name__ == "__main__":
unittest.main()
+118 -17
View File
@@ -1,31 +1,132 @@
# LOGAR Windows Edge Forwarder # LOGAR Windows Edge Forwarder
Lightweight edge log forwarder for Windows servers. Standalone compiled executable and installer distribution for Windows Server and workstation environments.
## Features ---
- **Zero Local State**: No local database or state tracking. Forwarder simply scans recent logs and streams candidates.
- **Edge Noise Stripping**: Strips conversational/informational noise (INFO, DEBUG, Audit) at the source.
- **End-to-End OpenPGP Encryption**: Encrypts logs using the server's public key so that only the server can decrypt them.
- **Authenticated TCP Socket**: Connects directly via raw TCP framing with token verification.
- **No GPG Binary Required**: Pure-Python cryptography (`pgpy` + `cryptography`).
## Installation ## Overview
`Win_Client.exe` is a self-contained executable that queries the Windows Application Event Log, filters candidate events at the source, auto-enrolls with the central LOGAR hub to receive signed mTLS certificates, and streams records over mutual TLS 1.3 (**mTLS**) socket connection.
### Key Capabilities
- **Pre-compiled & Dependency-Free**: Ships as a standalone native Windows executable (`Win_Client.exe`) or full installer (`LOGAR-Client-Setup.exe`). No Python installation, pip packages, or GnuPG binaries are required on the host.
- **Mutual TLS 1.3 (mTLS) Ingestion**: Streams directly over hardware-authenticated TLS 1.3 sockets with hardware/machine-bound client certificates.
- **Automated Client Enrollment**: On first run with an `enrollment_secret`, the client automatically calls `POST /api/client/enroll` on the hub, saves its certificates into `certs/`, and establishes secure mTLS streaming.
- **Proactive Expiry Check & Reactive Self-Healing**: Before each run, the client evaluates `client.crt` validity. If within 30 days of expiry, it automatically contacts the hub to renew certificates. If the server Root CA rotates or a TLS handshake error occurs, the client catch-heals by re-enrolling immediately and re-establishing connection without human intervention.
- **Source-Level Filtering**: Retains events spanning `INFO`, `WARNING`, and `ERROR`. Strips audit events and debug noise, skipping events older than 24 hours.
- **State Tracking & Deduplication**: Maintains persistent client state in `client_state.json` (tracking event record numbers and timestamp signatures) so every log record is forwarded exactly once without duplicates.
- **Fail-Safe State Commit**: State is committed only when the server returns a verified `success` response. In the event of a network outage, state remains unchanged and unsent events are retried automatically on the next run.
---
## 1. Automated Installation via Inno Setup (Recommended)
Run the self-contained installer built from `compilation/installer_client.iss`:
```powershell ```powershell
python -m pip install -r requirements.txt .\LOGAR-Client-Setup.exe
```
This installer:
1. Installs `Win_Client.exe` and bundled `nssm.exe` to `C:\Program Files\LOGAR\`.
2. Sets up directory permissions in `C:\ProgramData\LOGAR\`.
3. Registers and starts the `LOGAR_Client` Windows service automatically via NSSM.
4. Redirects stdout and stderr logs to `C:\ProgramData\LOGAR\client.log` and `client_err.log`.
---
## 2. Generating & Deploying the Configuration File
### Step 1: Generate `client_config.json` on the Server
Run the following command on your central LOGAR server:
```bash
python src/Server.py --create-client-config --server-host <SERVER_IP_OR_DNS> --server-port 9443 --client-out client_config.json
```
- Replace `<SERVER_IP_OR_DNS>` with the reachable IP address or FQDN of your central LOGAR server hub.
- Default mTLS socket port is `9443`; Hermes REST API port is `8443`.
### Step 2: Configuration Structure
The generated `client_config.json` contains:
```json
{
"server_host": "192.168.1.100",
"server_port": 9443,
"hermes_host": "192.168.1.100",
"hermes_port": 8443,
"enrollment_secret": "a1b2c3d4e5f6...",
"cert_dir": "certs",
"server_fingerprint": "375388960531264EA0648EC0D2C4E4ABC6F22AC2",
"server_public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
"auth_token": "a1b2c3d4e5f6..."
}
``` ```
## Configuration > [!NOTE]
Place the `client_config.json` generated by the server (`Server.py --create-client-config`) in the same directory as `Win_Client.py`. > The configuration contains **no host-specific names or site names** to ensure client anonymity and easy redistribution.
## Running the Forwarder ### Step 3: Copy to Edge Node
Place `client_config.json` next to `Win_Client.exe` (e.g. `C:\Program Files\LOGAR\` or `C:\LOGAR\`):
```powershell ```powershell
python Win_Client.py --hours 6 New-Item -ItemType Directory -Path "C:\LOGAR" -Force
Copy-Item "Win_Client.exe", "client_config.json" -Destination "C:\LOGAR\"
``` ```
## Scheduled Task Deployment ---
To run periodically via Windows Task Scheduler (e.g., every 3 hours):
## 3. Running Manually
Test the forwarder interactively from PowerShell or Command Prompt:
```powershell ```powershell
$Action = New-ScheduledTaskAction -Execute "python.exe" -Argument "C:\LOGAR\Win_Client.py --hours 6" -WorkingDirectory "C:\LOGAR" cd C:\LOGAR
.\Win_Client.exe --hours 24
```
On first run, the client contacts `http://<hermes_host>:<hermes_port>/api/client/enroll`, downloads `ca.crt`, `client.crt`, and `client.key` into `certs/`, and streams logs over mTLS.
### Command-Line Arguments
| Argument | Default | Description |
| :--- | :--- | :--- |
| `--config` | `client_config.json` | Path to client configuration file |
| `--hours` | `24` | Lookback window in hours for event logs |
| `--state-file` | `client_state.json` | Path to persistent state tracking file |
| `--no-state` | `False` | Disable state tracking and send all events matching lookback window |
---
## 4. Manual Service Installation (NSSM or Scheduled Task)
### Method A: Windows Service via Bundled NSSM
```powershell
# From the compilation directory or with bundled nssm.exe:
.\nssm.exe install LOGAR_Client "C:\LOGAR\Win_Client.exe" "--hours 24"
.\nssm.exe set LOGAR_Client AppDirectory "C:\LOGAR"
.\nssm.exe set LOGAR_Client AppStdout "C:\ProgramData\LOGAR\client.log"
.\nssm.exe set LOGAR_Client AppStderr "C:\ProgramData\LOGAR\client_err.log"
.\nssm.exe start LOGAR_Client
```
### Method B: Windows Scheduled Task via PowerShell
```powershell
$Action = New-ScheduledTaskAction -Execute "C:\LOGAR\Win_Client.exe" -Argument "--hours 24" -WorkingDirectory "C:\LOGAR"
$Trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) -RepetitionInterval (New-TimeSpan -Hours 3) $Trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) -RepetitionInterval (New-TimeSpan -Hours 3)
Register-ScheduledTask -TaskName "LOGAR_Windows_Forwarder" -Action $Action -Trigger $Trigger -Description "LOGAR Edge Forwarder" $Settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Minutes 15)
Register-ScheduledTask -TaskName "LOGAR_Forwarder" `
-Action $Action `
-Trigger $Trigger `
-Settings $Settings `
-User "NT AUTHORITY\SYSTEM" `
-RunLevel Highest `
-Description "LOGAR Windows Edge Log Forwarder Service"
Start-ScheduledTask -TaskName "LOGAR_Forwarder"
```
---
## 5. Uninstallation
If installed via the Inno Setup installer, use **Windows Add/Remove Programs** or run `unins000.exe` in `C:\Program Files\LOGAR\`.
If installed manually via NSSM:
```powershell
.\nssm.exe stop LOGAR_Client
.\nssm.exe remove LOGAR_Client confirm
Remove-Item -Recurse -Force "C:\LOGAR"
``` ```
-211
View File
@@ -1,211 +0,0 @@
import os
import sys
import json
import socket
import struct
import argparse
import warnings
from datetime import datetime, timezone, timedelta
# Suppress cryptography / pgpy deprecation notices
warnings.filterwarnings("ignore")
import pgpy
try:
import win32evtlog
except ImportError:
win32evtlog = None
CONFIG_FILE_NAME = "client_config.json"
def load_config(config_path: str = CONFIG_FILE_NAME):
if not os.path.exists(config_path):
raise FileNotFoundError(
f"Client configuration file not found at: {config_path}\n"
f"Generate one from the server using: python Server.py --create-client-config --client-out {config_path}"
)
with open(config_path, "r", encoding="utf-8") as f:
return json.load(f)
def get_machine_identifier() -> str:
"""
Returns the hostname of the machine sending the logs,
and appends the network/DNS domain if available.
"""
fqdn = socket.getfqdn()
if fqdn and "." in fqdn and not fqdn.startswith("localhost"):
return fqdn
hostname = socket.gethostname()
user_dns_domain = os.environ.get("USERDNSDOMAIN")
if user_dns_domain and user_dns_domain.lower() != hostname.lower():
return f"{hostname}.{user_dns_domain.lower()}"
try:
host_ip = socket.gethostbyname(hostname)
canonical_name = socket.gethostbyaddr(host_ip)[0]
if canonical_name and "." in canonical_name and not canonical_name.startswith("localhost"):
return canonical_name
except Exception:
pass
return hostname
def get_recent_windows_logs(hours: int = 6):
"""
Scans the Windows Application Event Log backwards for events within the window.
Edge Thinness & Noise Stripping: INFO and DEBUG events are dropped at the source.
"""
if win32evtlog is None:
print("[!] pywin32 is not installed or not running on Windows. Returning mock/empty candidate list.")
return []
server = "localhost"
log_type = "Application"
flags = win32evtlog.EVENTLOG_BACKWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
try:
hand = win32evtlog.OpenEventLog(server, log_type)
except Exception as e:
print(f"[!] Error opening Windows event log: {e}")
return []
logs = []
cutoff_time = datetime.now() - timedelta(hours=hours)
machine_id = get_machine_identifier()
sev_map = {
1: "CRITICAL",
2: "ERROR",
3: "WARNING"
}
while True:
events = win32evtlog.ReadEventLog(hand, flags, 0)
if not events:
break
for event in events:
if event.TimeGenerated < cutoff_time:
break
# Drop conversational or informational noise (INFO=4, etc.) at source
# Only retain Critical (1), Error (2), and Warning (3)
if event.EventType in sev_map:
msg = " ".join(event.StringInserts) if event.StringInserts else "Event Log Entry"
logs.append({
"server": machine_id,
"os_type": "windows",
"signature": event.SourceName or "Windows-Event",
"severity": sev_map[event.EventType],
"message": msg[:2048] # Limit message length
})
if events[-1].TimeGenerated < cutoff_time:
break
win32evtlog.CloseEventLog(hand)
return logs
def send_encrypted_logs_over_socket(config: dict, logs: list):
"""
Encrypts the payload using the server's OpenPGP public key and streams
over an authenticated TCP socket. Zero local state is maintained on the client.
"""
server_host = config["server_host"]
server_port = int(config["server_port"])
auth_token = config["auth_token"]
pub_key_armored = config["server_public_key"]
expected_fp = config.get("server_fingerprint", "").replace(" ", "").upper()
# Load and verify server public key
pub_key, _ = pgpy.PGPKey.from_blob(pub_key_armored)
actual_fp = str(pub_key.fingerprint).replace(" ", "").upper()
if expected_fp and actual_fp != expected_fp:
raise ValueError(
f"Server fingerprint mismatch! Expected {expected_fp}, but key has {actual_fp}."
)
machine_id = get_machine_identifier()
# Prepare zero-state candidate batch
payload = {
"server": machine_id,
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
payload_json = json.dumps(payload)
# Encrypt payload with server's encryption-only key
pgp_msg = pgpy.PGPMessage.new(payload_json)
encrypted_msg = pub_key.encrypt(pgp_msg)
encrypted_armored = str(encrypted_msg)
# Envelope with socket authentication header
envelope = {
"auth_token": auth_token,
"timestamp": datetime.now(timezone.utc).isoformat(),
"encrypted_payload": encrypted_armored
}
envelope_bytes = json.dumps(envelope).encode("utf-8")
# Connect over TCP socket and transmit with 4-byte length prefix framing
print(f"[*] Connecting to LOGAR server at {server_host}:{server_port} over secure TCP socket...")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.settimeout(15.0)
sock.connect((server_host, server_port))
# Send frame: length (4 bytes big-endian) + envelope
frame = struct.pack(">I", len(envelope_bytes)) + envelope_bytes
sock.sendall(frame)
# Receive response length
resp_len_bytes = sock.recv(4)
if not resp_len_bytes:
raise ConnectionError("Server closed connection without response.")
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = bytearray()
while len(resp_bytes) < resp_len:
chunk = sock.recv(min(4096, resp_len - len(resp_bytes)))
if not chunk:
break
resp_bytes.extend(chunk)
response = json.loads(resp_bytes.decode("utf-8"))
print(f"[+] Server response: {response}")
return response
def main():
parser = argparse.ArgumentParser(description="LOGAR Windows Edge Log Forwarder (Zero State)")
parser.add_argument("--config", default=CONFIG_FILE_NAME, help="Path to client_config.json")
parser.add_argument("--hours", type=int, default=6, help="Lookback window in hours for event logs")
args = parser.parse_args()
try:
config = load_config(args.config)
except Exception as e:
print(f"[!] Configuration error: {e}")
sys.exit(1)
machine_id = get_machine_identifier()
print(f"[*] Edge Forwarder Node: {machine_id}")
print(f"[*] Scanning Windows Application event log for candidate anomalies (last {args.hours} hours)...")
candidate_logs = get_recent_windows_logs(hours=args.hours)
print(f"[*] Found {len(candidate_logs)} candidate anomalies (noise stripped at source).")
try:
send_encrypted_logs_over_socket(config, candidate_logs)
except Exception as e:
print(f"[!] Failed to stream logs to server: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
+1 -2
View File
@@ -3,6 +3,5 @@
"server_port": 9443, "server_port": 9443,
"server_fingerprint": "PASTE_SERVER_FINGERPRINT_HERE", "server_fingerprint": "PASTE_SERVER_FINGERPRINT_HERE",
"server_public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...\n-----END PGP PUBLIC KEY BLOCK-----\n", "server_public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...\n-----END PGP PUBLIC KEY BLOCK-----\n",
"auth_token": "PASTE_AUTH_TOKEN_HERE", "auth_token": "PASTE_AUTH_TOKEN_HERE"
"site_name": "Frankfurt-DC"
} }
-4
View File
@@ -1,4 +0,0 @@
pgpy>=0.6.0
standard-imghdr>=3.13.0; python_version >= "3.13"
cryptography>=42.0.0
pywin32>=306
-95
View File
@@ -1,95 +0,0 @@
import os
import sys
import json
import struct
import unittest
import warnings
warnings.filterwarnings("ignore")
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
import Win_Client
import pgpy
from pgpy.constants import PubKeyAlgorithm, KeyFlags, HashAlgorithm, SymmetricKeyAlgorithm, CompressionAlgorithm
class TestWinClientComponent(unittest.TestCase):
def setUp(self):
self.dummy_config = "test_win_client_config.json"
# Generate dummy PGP key for testing
key = pgpy.PGPKey.new(PubKeyAlgorithm.RSAEncryptOrSign, 2048)
uid = pgpy.PGPUID.new("TestHub")
key.add_uid(
uid,
usage={KeyFlags.EncryptCommunications, KeyFlags.EncryptStorage},
hashes=[HashAlgorithm.SHA256],
ciphers=[SymmetricKeyAlgorithm.AES256],
compression=[CompressionAlgorithm.Uncompressed]
)
self.server_priv = key
self.server_pub = key.pubkey
self.fingerprint = str(key.pubkey.fingerprint)
with open(self.dummy_config, "w", encoding="utf-8") as f:
json.dump({
"server_host": "127.0.0.1",
"server_port": 9443,
"server_fingerprint": self.fingerprint,
"server_public_key": str(self.server_pub),
"auth_token": "secret-test-token"
}, f)
def tearDown(self):
if os.path.exists(self.dummy_config):
try:
os.remove(self.dummy_config)
except Exception:
pass
def test_client_config_anonymity(self):
config = Win_Client.load_config(self.dummy_config)
self.assertNotIn("server_name", config)
self.assertNotIn("name", config)
self.assertNotIn("site_name", config)
self.assertEqual(config["server_fingerprint"], self.fingerprint)
def test_get_machine_identifier(self):
machine_id = Win_Client.get_machine_identifier()
self.assertIsInstance(machine_id, str)
self.assertGreater(len(machine_id), 0)
self.assertNotEqual(machine_id, "localhost")
def test_encryption_and_envelope_creation(self):
config = Win_Client.load_config(self.dummy_config)
logs = [{
"server": Win_Client.get_machine_identifier(),
"signature": "TestWinSignature",
"severity": "WARNING",
"message": "Disk space threshold warning"
}]
pub_key, _ = pgpy.PGPKey.from_blob(config["server_public_key"])
payload = {
"server": Win_Client.get_machine_identifier(),
"logs": logs
}
msg = pgpy.PGPMessage.new(json.dumps(payload))
enc = pub_key.encrypt(msg)
self.assertTrue(str(enc).startswith("-----BEGIN PGP MESSAGE-----"))
# Decrypt with private key to verify end-to-end payload integrity
dec = self.server_priv.decrypt(enc)
restored = json.loads(dec.message)
self.assertEqual(restored["logs"][0]["signature"], "TestWinSignature")
def test_framing_protocol(self):
envelope_data = json.dumps({"test": "data"}).encode("utf-8")
frame = struct.pack(">I", len(envelope_data)) + envelope_data
self.assertEqual(len(frame), 4 + len(envelope_data))
length = struct.unpack(">I", frame[:4])[0]
self.assertEqual(length, len(envelope_data))
if __name__ == "__main__":
unittest.main()
+122
View File
@@ -0,0 +1,122 @@
# LOGAR Windows Server Hub
Standalone compiled executable and installer distribution for Windows Server environments (`Server.exe`).
---
## Overview
`Server.exe` is a self-contained, pre-compiled native Windows PE executable that serves as the central log aggregation, temporal persistence analyzer, dynamic PKI certificate authority, and reporting hub of the LOGAR infrastructure.
### Key Architecture & Capabilities
- **Pre-compiled & Dependency-Free**: Ships as a standalone Windows executable (`Server.exe`) or full installer (`LOGAR-Server-Setup.exe`). No Python installation, pip packages, or GnuPG binaries are required on Windows Server.
- **Mutual TLS 1.3 (mTLS) Ingestion (Port 9443)**: Enforces mutual TLS 1.3 authentication for all incoming edge connections. Validates client certificates against an internal Root CA and verifies active licensing in SQLite.
- **Dynamic PKI & License Accounting**: Built-in Root CA generates server TLS certificates with SANs and dynamically signs client certificates via `POST /api/client/enroll` while enforcing seat limits (`max_seats`).
- **In-Flight Certificate Watchdog & Dynamic Reloading**: Continuously monitors Root CA (`ca.crt`) and Server TLS certificate (`server.crt`) validity in the background (every 12 hours). When nearing expiration (< 30 days), certificates are automatically regenerated with timestamped backups, and active `ssl.SSLContext` structures are reloaded dynamically without dropping socket connections or restarting the Windows service.
- **Warning Persistence & Immediate Error Routing**: High-severity `ERROR`, `CRITICAL`, and `FATAL` events are promoted to `VERIFIED` immediately on their first occurrence. Operational `WARNING` and `INFO` events require persistence across at least 4 distinct transmission cycles within a rolling 12-hour evaluation window.
- **Embedded Hermes Reporting & Management API (Port 8443)**: Integrated REST API exposing `/api/hermes/report`, `/api/clients`, and `/api/client/enroll`.
- **State Database**: Stores issue lifecycle records, client telemetry, and licensing quotas in a local SQLite database (`logar_state.db`).
---
## 1. Automated Installation via Inno Setup (Recommended)
Run the self-contained installer built from `compilation/installer_server.iss`:
```powershell
.\LOGAR-Server-Setup.exe
```
This installer:
1. Installs `Server.exe` and bundled `nssm.exe` to `C:\Program Files\LOGAR-Server\`.
2. Registers and starts the `LOGAR_Server` Windows service automatically via NSSM.
3. Redirects stdout and stderr logs to `C:\ProgramData\LOGAR-Server\server.log` and `server_err.log`.
---
## 2. Initializing & Generating Server Configuration
### Step 1: Automatic First-Run Generation
When launched without an existing `server_config.json`, `Server.exe` automatically initializes:
1. An internal Root CA (`certs/ca.crt` and `certs/ca.key`).
2. A server TLS certificate (`certs/server.crt` and `certs/server.key`) with SANs.
3. An OpenPGP RSA-2048 keypair (`private_key` and `public_key`).
4. Cryptographically random authentication tokens and enrollment secrets.
5. Default network socket coordinates (mTLS 9443, Hermes API 8443).
Open PowerShell and run:
```powershell
.\Server.exe
```
Output:
```
[!] Config 'server_config.json' not found. Initializing first-run configuration...
[+] Successfully generated new server config and OpenPGP keypair.
[+] Server Encryption Fingerprint: 375388960531264EA0648EC0D2C4E4ABC6F22AC2
[+] Saved to: server_config.json
============================================================
LOGAR Server Hub: LOGAR-Cloud-Hub
Transport Security: mTLS (TLS 1.3)
License Quota: 10 Active Seats
Server Encryption Fingerprint: 375388960531264EA0648EC0D2C4E4ABC6F22AC2
Evaluation Window: 12 hours | 4-Run Rule: Warnings | Immediate Pass: Errors
============================================================
[*] LOGAR mTLS TLSv1.3 Socket Server listening on 0.0.0.0:9443
[*] Hermes Reporting API available at http://0.0.0.0:8443/api/hermes/report
[*] Client Enrollment API available at http://0.0.0.0:8443/api/client/enroll
```
### Step 2: Configuration Fields Reference
The generated `server_config.json` contains:
```json
{
"server_name": "LOGAR-Windows-Hub",
"tcp_host": "0.0.0.0",
"tcp_port": 9443,
"hermes_host": "0.0.0.0",
"hermes_port": 8443,
"auth_token": "a1b2c3d4e5f67890abcdef1234567890...",
"enrollment_secret": "e1f2a3b4c5d6...",
"max_seats": 10,
"cert_dir": "certs",
"tls_enabled": true,
"db_path": "logar_state.db",
"evaluation_window_hours": 12,
"min_persistence_runs": 4,
"server_fingerprint": "375388960531264EA0648EC0D2C4E4ABC6F22AC2",
"public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
"private_key": "-----BEGIN PGP PRIVATE KEY BLOCK-----\n..."
}
```
---
## 3. Exporting Client Configurations
Generate a client configuration bundle to deploy onto Windows or Linux forwarders:
```powershell
.\Server.exe --create-client-config --server-host 192.168.1.100 --server-port 9443 --client-out client_config.json
```
The output file contains the server coordinates, enrollment secret, and fingerprint, ready for client deployment.
---
## 4. Manual Windows Service Setup (via NSSM)
```powershell
.\nssm.exe install LOGAR_Server "C:\LOGAR-Server\Server.exe"
.\nssm.exe set LOGAR_Server AppDirectory "C:\LOGAR-Server"
.\nssm.exe set LOGAR_Server AppStdout "C:\ProgramData\LOGAR-Server\server.log"
.\nssm.exe set LOGAR_Server AppStderr "C:\ProgramData\LOGAR-Server\server_err.log"
.\nssm.exe start LOGAR_Server
```
---
## 5. Uninstallation
If installed via the Inno Setup installer, use **Windows Add/Remove Programs**.
If installed manually via NSSM:
```powershell
.\nssm.exe stop LOGAR_Server
.\nssm.exe remove LOGAR_Server confirm
```
+14
View File
@@ -0,0 +1,14 @@
{
"server_name": "LOGAR-Windows-Hub",
"tcp_host": "0.0.0.0",
"tcp_port": 9443,
"hermes_host": "0.0.0.0",
"hermes_port": 8443,
"auth_token": "replace_with_secure_random_hex_token",
"db_path": "logar_state.db",
"evaluation_window_hours": 12,
"min_persistence_runs": 4,
"server_fingerprint": "AUTO_GENERATED_ON_FIRST_RUN",
"public_key": "AUTO_GENERATED_ON_FIRST_RUN",
"private_key": "AUTO_GENERATED_ON_FIRST_RUN"
}
-122
View File
@@ -1,122 +0,0 @@
import os
import sys
import shutil
import zipapp
import hashlib
import platform
import subprocess
DIST_DIR = os.path.abspath("dist")
OUT_DIR = os.path.abspath("out")
BUILD_TEMP = os.path.abspath("build_temp")
def clean_and_prep():
if os.path.exists(DIST_DIR):
shutil.rmtree(DIST_DIR)
os.makedirs(DIST_DIR, exist_ok=True)
if os.path.exists(BUILD_TEMP):
shutil.rmtree(BUILD_TEMP)
os.makedirs(BUILD_TEMP, exist_ok=True)
def build_pyinstaller_binary(script_path, binary_name):
print(f"[*] Compiling {binary_name} with PyInstaller...")
cmd = [
sys.executable, "-m", "PyInstaller",
"--onefile",
"--clean",
"--distpath", DIST_DIR,
"--workpath", os.path.join(BUILD_TEMP, f"work_{binary_name}"),
"--specpath", os.path.join(BUILD_TEMP, f"spec_{binary_name}"),
"--name", binary_name,
script_path
]
res = subprocess.run(cmd, capture_output=True, text=True)
if res.returncode != 0:
print(f"[!] Compilation error for {binary_name}:\n{res.stderr}")
raise RuntimeError(f"Failed to build {binary_name}")
print(f"[+] Successfully compiled {binary_name}")
def build_linux_zipapp_binary():
print("[*] Packaging Linux_Client.bin executable binary...")
app_dir = os.path.join(BUILD_TEMP, "linux_app")
os.makedirs(app_dir, exist_ok=True)
shutil.copy(os.path.join(OUT_DIR, "linux_client", "Linux_Client.py"), os.path.join(app_dir, "Linux_Client.py"))
bin_output = os.path.join(DIST_DIR, "Linux_Client.bin")
zipapp.create_archive(
source=app_dir,
target=bin_output,
interpreter="/usr/bin/env python3",
main="Linux_Client:main"
)
print(f"[+] Successfully generated {bin_output}")
def generate_checksums():
checksum_file = os.path.join(DIST_DIR, "SHA256SUMS.txt")
lines = []
for fname in sorted(os.listdir(DIST_DIR)):
if fname == "SHA256SUMS.txt":
continue
fpath = os.path.join(DIST_DIR, fname)
if os.path.isfile(fpath):
with open(fpath, "rb") as f:
digest = hashlib.sha256(f.read()).hexdigest()
lines.append(f"{digest} {fname}")
with open(checksum_file, "w", encoding="utf-8") as f:
f.write("\n".join(lines) + "\n")
def main():
print("=" * 60)
print(" LOGAR Binary Packaging (Binaries Only)")
print(f" Platform: {platform.system()} ({platform.machine()})")
print("=" * 60)
clean_and_prep()
is_windows = platform.system() == "Windows"
if is_windows:
# Build Windows client executable
win_client_script = os.path.join(OUT_DIR, "win_client", "Win_Client.py")
build_pyinstaller_binary(win_client_script, "Win_Client")
# Build Windows server executable
server_script = os.path.join(OUT_DIR, "server", "Server.py")
build_pyinstaller_binary(server_script, "Server")
# Build Linux client standalone binary
build_linux_zipapp_binary()
else:
# On Linux runner: Build native Linux binaries
linux_client_script = os.path.join(OUT_DIR, "linux_client", "Linux_Client.py")
build_pyinstaller_binary(linux_client_script, "Linux_Client.bin")
server_script = os.path.join(OUT_DIR, "server", "Server.py")
build_pyinstaller_binary(server_script, "Server.bin")
# Also package standalone Windows zipapp executable
win_app_dir = os.path.join(BUILD_TEMP, "win_app")
os.makedirs(win_app_dir, exist_ok=True)
shutil.copy(os.path.join(OUT_DIR, "win_client", "Win_Client.py"), os.path.join(win_app_dir, "Win_Client.py"))
win_bin_output = os.path.join(DIST_DIR, "Win_Client.pyz")
zipapp.create_archive(
source=win_app_dir,
target=win_bin_output,
interpreter="/usr/bin/env python3",
main="Win_Client:main"
)
generate_checksums()
# Clean temporary build directory
if os.path.exists(BUILD_TEMP):
shutil.rmtree(BUILD_TEMP, ignore_errors=True)
print("\n[+] Binary shipping artifacts assembled in 'dist/':")
for f in os.listdir(DIST_DIR):
sz = os.path.getsize(os.path.join(DIST_DIR, f))
print(f" - {f} ({sz / (1024*1024):.2f} MB)" if sz > 1024*1024 else f" - {f} ({sz} bytes)")
print("=" * 60)
if __name__ == "__main__":
main()
+490
View File
@@ -0,0 +1,490 @@
import os
import sys
import json
import socket
import ssl
import struct
import argparse
import subprocess
import urllib.request
import warnings
from datetime import datetime, timezone, timedelta
from typing import Optional, Dict, Any, List
# Suppress cryptography / pgpy deprecation notices
warnings.filterwarnings("ignore")
import pgpy
CONFIG_FILE_NAME = "client_config.json"
STATE_FILE_NAME = "client_state.json"
def is_cert_expiring_soon(cert_path: str, threshold_days: int = 30) -> bool:
"""Checks if client certificate at cert_path is expiring within threshold_days."""
if not os.path.exists(cert_path):
return True
try:
from cryptography import x509
with open(cert_path, "r", encoding="utf-8") as f:
cert = x509.load_pem_x509_certificate(f.read().encode("utf-8"))
expiry = getattr(cert, "not_valid_after_utc", None)
if expiry is None:
expiry = cert.not_valid_after.replace(tzinfo=timezone.utc)
now = datetime.now(timezone.utc)
return expiry <= (now + timedelta(days=threshold_days))
except Exception:
return True
def enroll_client_if_needed(
hub_url: str,
enrollment_secret: str,
cert_dir: str,
client_id: str,
hostname: str,
os_type: str = "linux",
force_renew: bool = False,
threshold_days: int = 30
):
"""Bootstraps client enrollment if certificates are missing or expiring soon."""
os.makedirs(cert_dir, exist_ok=True)
ca_path = os.path.join(cert_dir, "ca.crt")
cert_path = os.path.join(cert_dir, "client.crt")
key_path = os.path.join(cert_dir, "client.key")
if not force_renew and os.path.exists(ca_path) and os.path.exists(cert_path) and os.path.exists(key_path):
if not is_cert_expiring_soon(cert_path, threshold_days=threshold_days):
return True
print(f"[*] Client certificate at {cert_path} is expiring within {threshold_days} days. Auto-renewing...")
action_name = "re-enrolling" if os.path.exists(cert_path) else "enrolling"
print(f"[*] Bootstrapping client {action_name} with LOGAR Hub at {hub_url}...")
enroll_endpoint = f"{hub_url.rstrip('/')}/api/client/enroll"
payload = {
"client_id": client_id,
"hostname": hostname,
"os": os_type,
"enrollment_secret": enrollment_secret
}
req = urllib.request.Request(
enroll_endpoint,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req, timeout=10) as resp:
if resp.status != 200:
raise RuntimeError(f"Enrollment failed with status code {resp.status}")
data = json.loads(resp.read().decode("utf-8"))
with open(ca_path, "w", encoding="utf-8") as f:
f.write(data["ca_cert"])
with open(cert_path, "w", encoding="utf-8") as f:
f.write(data["client_cert"])
with open(key_path, "w", encoding="utf-8") as f:
f.write(data["client_key"])
try:
os.chmod(key_path, 0o600)
except Exception:
pass
print(f"[+] Client certificates updated successfully in {os.path.abspath(cert_dir)}")
return True
def get_tls_socket(hub_host: str, hub_port: int, cert_dir: str):
"""Establishes an mTLS connection with the LOGAR hub using client certificates."""
ca_path = os.path.join(cert_dir, "ca.crt")
cert_path = os.path.join(cert_dir, "client.crt")
key_path = os.path.join(cert_dir, "client.key")
if not (os.path.exists(ca_path) and os.path.exists(cert_path) and os.path.exists(key_path)):
raise FileNotFoundError(f"mTLS certificates not found in '{cert_dir}'. Enroll client first.")
ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=ca_path)
ctx.load_cert_chain(certfile=cert_path, keyfile=key_path)
ctx.minimum_version = ssl.TLSVersion.TLSv1_3
ctx.check_hostname = False
raw_sock = socket.create_connection((hub_host, hub_port), timeout=15)
return ctx.wrap_socket(raw_sock, server_hostname=hub_host)
def get_state_path(config_path: str, custom_state_path: Optional[str] = None) -> str:
if custom_state_path:
return custom_state_path
config_dir = os.path.dirname(os.path.abspath(config_path))
return os.path.join(config_dir, STATE_FILE_NAME)
def load_state(state_path: str) -> dict:
if os.path.exists(state_path):
try:
with open(state_path, "r", encoding="utf-8") as f:
return json.load(f)
except Exception as e:
print(f"[!] Warning: Failed to read state file '{state_path}': {e}")
return {}
return {}
def save_state(state_path: str, state: dict):
try:
temp_path = f"{state_path}.tmp"
with open(temp_path, "w", encoding="utf-8") as f:
json.dump(state, f, indent=2)
os.replace(temp_path, state_path)
except Exception as e:
print(f"[!] Warning: Could not save client state to '{state_path}': {e}")
def commit_state(state: dict, state_path: str):
if "new_last_cursor" in state:
val = state.pop("new_last_cursor")
if val:
state["last_cursor"] = val
if "new_last_timestamp_us" in state:
val = state.pop("new_last_timestamp_us")
if val:
state["last_timestamp_us"] = val
if "new_sent_cursors" in state:
state["sent_cursors"] = state.pop("new_sent_cursors")
save_state(state_path, state)
def load_config(config_path: str = CONFIG_FILE_NAME):
if not os.path.exists(config_path):
raise FileNotFoundError(
f"Client configuration file not found at: {config_path}\n"
f"Generate one from the server using: python Server.py --create-client-config --client-out {config_path}"
)
with open(config_path, "r", encoding="utf-8") as f:
return json.load(f)
def get_machine_identifier() -> str:
"""
Returns the hostname of the machine sending the logs,
and appends the network/DNS domain if available.
"""
# 1. Try fully-qualified domain name (FQDN)
fqdn = socket.getfqdn()
if fqdn and "." in fqdn and not fqdn.startswith("localhost"):
return fqdn
hostname = socket.gethostname()
# 2. Check /etc/resolv.conf domain or search directive
try:
if os.path.exists("/etc/resolv.conf"):
with open("/etc/resolv.conf", "r", encoding="utf-8") as f:
for line in f:
parts = line.strip().split()
if parts and parts[0] in ["domain", "search"] and len(parts) > 1:
domain = parts[1]
if domain and not domain.startswith("."):
return f"{hostname}.{domain}"
except Exception:
pass
# 3. Try reverse DNS lookup
try:
host_ip = socket.gethostbyname(hostname)
canonical_name = socket.gethostbyaddr(host_ip)[0]
if canonical_name and "." in canonical_name and not canonical_name.startswith("localhost"):
return canonical_name
except Exception:
pass
return hostname
def get_recent_linux_logs(hours: int = 24, state: Optional[dict] = None) -> list:
"""
Collects info, warnings, and errors from systemd journalctl over the lookback window.
Edge Filtering: Retains INFO, WARNING, and ERROR. Strips DEBUG (priority 7) and skips events older than lookback window (default: 24h).
State Tracking: Skips events older than lookback window (default 24h) and events
that have already been sent in previous runs.
"""
last_cursor = None
last_timestamp_us = 0.0
sent_cursors = set()
if state:
last_cursor = state.get("last_cursor")
try:
last_timestamp_us = float(state.get("last_timestamp_us", 0))
except (ValueError, TypeError):
last_timestamp_us = 0.0
sent_cursors = set(state.get("sent_cursors", []))
cmd = ["journalctl", "--since", f"{hours} hours ago", "-p", "info", "--output=json"]
result = None
if last_cursor:
cmd_with_cursor = ["journalctl", "--since", f"{hours} hours ago", "--after-cursor", str(last_cursor), "-p", "info", "--output=json"]
try:
res = subprocess.run(cmd_with_cursor, capture_output=True, text=True, check=False)
if res.returncode == 0:
result = res
except FileNotFoundError:
print("[!] journalctl command not found. Ensure this script runs on a systemd-enabled Linux system.")
return []
except Exception:
pass
if result is None:
try:
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
except FileNotFoundError:
print("[!] journalctl command not found. Ensure this script runs on a systemd-enabled Linux system.")
return []
except Exception as e:
print(f"[!] Error running journalctl: {e}")
return []
logs = []
machine_id = get_machine_identifier()
cutoff_epoch_us = (datetime.now(timezone.utc) - timedelta(hours=hours)).timestamp() * 1_000_000
newest_cursor = None
newest_timestamp_us = last_timestamp_us
collected_cursors = []
for line in result.stdout.splitlines():
line_str = line.strip()
if not line_str:
continue
try:
entry = json.loads(line_str)
entry_cursor = entry.get("__CURSOR")
entry_ts_us_raw = entry.get("__REALTIME_TIMESTAMP")
entry_ts_us = 0.0
if entry_ts_us_raw:
try:
entry_ts_us = float(entry_ts_us_raw)
except (ValueError, TypeError):
pass
# 1. Skip entries older than lookback window (default: 24h)
if entry_ts_us and entry_ts_us < cutoff_epoch_us:
continue
# 2. Skip already sent events
if entry_cursor and (entry_cursor in sent_cursors or entry_cursor == last_cursor):
continue
if last_timestamp_us > 0 and entry_ts_us > 0 and entry_ts_us < last_timestamp_us:
continue
# Advance newest tracking for new entries
if entry_cursor:
newest_cursor = entry_cursor
collected_cursors.append(entry_cursor)
if entry_ts_us > newest_timestamp_us:
newest_timestamp_us = entry_ts_us
priority = int(entry.get("PRIORITY", "6"))
# Priority 0: Emerg, 1: Alert, 2: Crit, 3: Err (-> ERROR)
# Priority 4: Warning, 5: Notice (-> WARNING)
# Priority 6: Info (-> INFO)
# Priority 7: Debug (skip)
if priority > 6:
continue
if priority <= 3:
sev = "ERROR"
elif priority in (4, 5):
sev = "WARNING"
else:
sev = "INFO"
logs.append({
"server": machine_id,
"os_type": "linux",
"signature": entry.get("SYSLOG_IDENTIFIER", "unknown"),
"severity": sev,
"message": entry.get("MESSAGE", "")[:2048]
})
except (json.JSONDecodeError, ValueError):
continue
if state is not None:
state["new_last_cursor"] = newest_cursor or last_cursor
state["new_last_timestamp_us"] = max(newest_timestamp_us, last_timestamp_us)
state["new_sent_cursors"] = (list(sent_cursors) + collected_cursors)[-1000:]
state["last_run_timestamp"] = datetime.now(timezone.utc).isoformat()
return logs
def send_encrypted_logs_over_socket(config: dict, logs: list):
"""
Streams logs to the LOGAR hub.
Uses mutual TLS 1.3 (mTLS) with client certificates if available,
or falls back to OpenPGP encrypted envelope over TCP.
"""
server_host = config["server_host"]
server_port = int(config["server_port"])
cert_dir = config.get("cert_dir", "certs")
enrollment_secret = config.get("enrollment_secret")
machine_id = get_machine_identifier()
# Attempt automatic enrollment bootstrap if certs are missing and secret is provided
hub_url = None
if enrollment_secret:
hermes_host = config.get("hermes_host", server_host)
hermes_port = config.get("hermes_port", 8443)
hub_url = f"http://{hermes_host}:{hermes_port}"
try:
enroll_client_if_needed(hub_url, enrollment_secret, cert_dir, machine_id, machine_id, os_type="linux")
except Exception as e:
print(f"[!] Warning: Enrollment bootstrap failed: {e}")
ca_path = os.path.join(cert_dir, "ca.crt")
cert_path = os.path.join(cert_dir, "client.crt")
key_path = os.path.join(cert_dir, "client.key")
has_mtls_certs = os.path.exists(ca_path) and os.path.exists(cert_path) and os.path.exists(key_path)
if has_mtls_certs:
print(f"[*] Connecting to LOGAR server at {server_host}:{server_port} over mTLS (TLS 1.3)...")
sock = None
try:
sock = get_tls_socket(server_host, server_port, cert_dir)
except (ssl.SSLError, ssl.CertificateError, ConnectionResetError) as tls_err:
if enrollment_secret and hub_url:
print(f"[!] TLS handshake error ({tls_err}). Re-enrolling with LOGAR Hub...")
try:
enroll_client_if_needed(hub_url, enrollment_secret, cert_dir, machine_id, machine_id, os_type="linux", force_renew=True)
sock = get_tls_socket(server_host, server_port, cert_dir)
except Exception as retry_err:
print(f"[!] Re-enrollment or reconnection retry failed: {retry_err}")
raise
else:
raise
with sock:
payload = {
"server": machine_id,
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
payload_bytes = json.dumps(payload).encode("utf-8")
frame = struct.pack(">I", len(payload_bytes)) + payload_bytes
sock.sendall(frame)
resp_len_bytes = sock.recv(4)
if not resp_len_bytes:
raise ConnectionError("Server closed mTLS connection without response.")
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = bytearray()
while len(resp_bytes) < resp_len:
chunk = sock.recv(min(4096, resp_len - len(resp_bytes)))
if not chunk:
break
resp_bytes.extend(chunk)
response = json.loads(resp_bytes.decode("utf-8"))
print(f"[+] Server response: {response}")
return response
# Fallback to OpenPGP envelope over plain TCP socket
auth_token = config.get("auth_token", "")
pub_key_armored = config.get("server_public_key")
if not pub_key_armored:
raise ValueError("No server public key or mTLS certificates available for connection.")
expected_fp = config.get("server_fingerprint", "").replace(" ", "").upper()
pub_key, _ = pgpy.PGPKey.from_blob(pub_key_armored)
actual_fp = str(pub_key.fingerprint).replace(" ", "").upper()
if expected_fp and actual_fp != expected_fp:
raise ValueError(f"Server fingerprint mismatch! Expected {expected_fp}, but key has {actual_fp}.")
payload = {
"server": machine_id,
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
payload_json = json.dumps(payload)
pgp_msg = pgpy.PGPMessage.new(payload_json)
encrypted_msg = pub_key.encrypt(pgp_msg)
encrypted_armored = str(encrypted_msg)
envelope = {
"auth_token": auth_token,
"timestamp": datetime.now(timezone.utc).isoformat(),
"encrypted_payload": encrypted_armored
}
envelope_bytes = json.dumps(envelope).encode("utf-8")
print(f"[*] Connecting to LOGAR server at {server_host}:{server_port} over secure TCP socket...")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.settimeout(15.0)
sock.connect((server_host, server_port))
frame = struct.pack(">I", len(envelope_bytes)) + envelope_bytes
sock.sendall(frame)
resp_len_bytes = sock.recv(4)
if not resp_len_bytes:
raise ConnectionError("Server closed connection without response.")
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = bytearray()
while len(resp_bytes) < resp_len:
chunk = sock.recv(min(4096, resp_len - len(resp_bytes)))
if not chunk:
break
resp_bytes.extend(chunk)
response = json.loads(resp_bytes.decode("utf-8"))
print(f"[+] Server response: {response}")
return response
def main():
parser = argparse.ArgumentParser(description="LOGAR Linux Edge Log Forwarder with State Tracking")
parser.add_argument("--config", default=CONFIG_FILE_NAME, help="Path to client_config.json")
parser.add_argument("--hours", type=int, default=24, help="Lookback window in hours for journalctl logs (default: 24)")
parser.add_argument("--state-file", default=None, help="Path to state tracking file (default: client_state.json next to config)")
parser.add_argument("--no-state", action="store_true", help="Disable state tracking and send all events matching lookback window")
args = parser.parse_args()
try:
config = load_config(args.config)
except Exception as e:
print(f"[!] Configuration error: {e}")
sys.exit(1)
state_path = get_state_path(args.config, args.state_file)
state = None if args.no_state else load_state(state_path)
machine_id = get_machine_identifier()
print(f"[*] Edge Forwarder Node: {machine_id}")
if state and ("last_cursor" in state or "last_timestamp_us" in state):
print(f"[*] State tracking active: resuming after previous cursor/timestamp (state file: {state_path})")
elif not args.no_state:
print(f"[*] State tracking initialized (state file: {state_path})")
print(f"[*] Scanning Linux journalctl for unsent entries (last {args.hours} hours)...")
candidate_logs = get_recent_linux_logs(hours=args.hours, state=state)
print(f"[*] Found {len(candidate_logs)} unsent candidate entries (INFO to ERROR, entries > {args.hours}h and already-sent skipped).")
if not candidate_logs:
print("[*] No new unsent events to transmit.")
if state is not None:
commit_state(state, state_path)
return
try:
resp = send_encrypted_logs_over_socket(config, candidate_logs)
if state is not None and resp and resp.get("status") == "success":
commit_state(state, state_path)
print(f"[+] State successfully committed to {state_path}")
except Exception as e:
print(f"[!] Failed to stream logs to server: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
+781
View File
@@ -0,0 +1,781 @@
import os
import sys
import json
import uuid
import struct
import socket
import sqlite3
import argparse
import asyncio
import secrets
import warnings
from datetime import datetime, timezone, timedelta
from typing import Dict, Any, List, Optional
# Suppress cryptography / pgpy deprecation notices for a clean terminal output
warnings.filterwarnings("ignore")
import pgpy
from pgpy.constants import (
PubKeyAlgorithm,
KeyFlags,
HashAlgorithm,
SymmetricKeyAlgorithm,
CompressionAlgorithm
)
import ssl
from pydantic import BaseModel
from fastapi import FastAPI, HTTPException
import uvicorn
try:
from src import server_enrollment as enrollment
except ImportError:
import server_enrollment as enrollment
CONFIG_FILE_NAME = "server_config.json"
DEFAULT_DB_FILE = "logar_state.db"
EVALUATION_WINDOW_HOURS = 12
RUN_THRESHOLD = 4
app = FastAPI(title="LOGAR Cloud Ingestion & Hermes Hub", version="2.0.1")
# Global context holding server state
SERVER_STATE: Dict[str, Any] = {}
class ClientEnrollRequest(BaseModel):
client_id: str
hostname: str
os: str
enrollment_secret: str
def generate_server_keypair(server_name: str):
"""Generates an OpenPGP RSA 2048 key with encryption capability."""
key = pgpy.PGPKey.new(PubKeyAlgorithm.RSAEncryptOrSign, 2048)
uid = pgpy.PGPUID.new(server_name)
key.add_uid(
uid,
usage={KeyFlags.EncryptCommunications, KeyFlags.EncryptStorage},
hashes=[HashAlgorithm.SHA256],
ciphers=[SymmetricKeyAlgorithm.AES256],
compression=[CompressionAlgorithm.Uncompressed]
)
private_key_armored = str(key)
public_key_armored = str(key.pubkey)
fingerprint = str(key.pubkey.fingerprint)
return private_key_armored, public_key_armored, fingerprint
def load_or_init_config(config_path: str = CONFIG_FILE_NAME) -> Dict[str, Any]:
"""Loads existing server_config.json or creates a new one on first run."""
if os.path.exists(config_path):
print(f"[*] Loading server configuration from: {os.path.abspath(config_path)}")
with open(config_path, "r", encoding="utf-8") as f:
config = json.load(f)
if "enrollment_secret" not in config:
config["enrollment_secret"] = secrets.token_hex(24)
if "max_seats" not in config:
config["max_seats"] = 10
if "cert_dir" not in config:
config["cert_dir"] = "certs"
if "tls_enabled" not in config:
config["tls_enabled"] = True
return config
print(f"[!] Config '{config_path}' not found. Initializing first-run configuration...")
server_name = "LOGAR-Cloud-Hub"
private_key, public_key, fingerprint = generate_server_keypair(server_name)
auth_token = secrets.token_hex(24)
enrollment_secret = secrets.token_hex(24)
config = {
"server_name": server_name,
"tcp_host": "0.0.0.0",
"tcp_port": 9443,
"hermes_host": "0.0.0.0",
"hermes_port": 8443,
"auth_token": auth_token,
"enrollment_secret": enrollment_secret,
"max_seats": 10,
"cert_dir": "certs",
"tls_enabled": True,
"db_path": DEFAULT_DB_FILE,
"evaluation_window_hours": EVALUATION_WINDOW_HOURS,
"min_persistence_runs": RUN_THRESHOLD,
"server_fingerprint": fingerprint,
"public_key": public_key,
"private_key": private_key
}
with open(config_path, "w", encoding="utf-8") as f:
json.dump(config, f, indent=2)
print(f"[+] Successfully generated new server config and OpenPGP keypair.")
print(f"[+] Server Encryption Fingerprint: {fingerprint}")
print(f"[+] Saved to: {os.path.abspath(config_path)}")
return config
def create_client_config(
server_host: str,
server_port: int,
output_path: str,
config_path: str = CONFIG_FILE_NAME,
hermes_host: Optional[str] = None,
hermes_port: Optional[int] = None
) -> Dict[str, Any]:
"""Creates a client configuration file containing the server address, auth token, and encryption-only key/fingerprint."""
server_conf = load_or_init_config(config_path)
client_conf = {
"server_host": server_host,
"server_port": server_port,
"hermes_host": hermes_host or server_conf.get("hermes_host", "127.0.0.1"),
"hermes_port": hermes_port or server_conf.get("hermes_port", 8443),
"enrollment_secret": server_conf.get("enrollment_secret"),
"cert_dir": "certs",
"server_fingerprint": server_conf["server_fingerprint"],
"server_public_key": server_conf["public_key"],
"auth_token": server_conf["auth_token"]
}
out_dir = os.path.dirname(os.path.abspath(output_path))
if out_dir and not os.path.exists(out_dir):
os.makedirs(out_dir, exist_ok=True)
with open(output_path, "w", encoding="utf-8") as f:
json.dump(client_conf, f, indent=2)
print(f"[+] Client configuration successfully written to: {os.path.abspath(output_path)}")
print(f" - Server Target: {server_host}:{server_port}")
print(f" - Encryption Fingerprint: {server_conf['server_fingerprint']}")
return client_conf
def init_db(db_path: str, enrollment_secret: Optional[str] = None, max_seats: int = 10):
"""Initializes the SQLite schema for multi-run temporal tracking, client tracking, and license quota."""
conn = sqlite3.connect(db_path)
conn.execute("""
CREATE TABLE IF NOT EXISTS active_issues (
fingerprint TEXT PRIMARY KEY,
site_name TEXT,
server TEXT,
signature TEXT,
severity TEXT,
message TEXT,
os_type TEXT,
first_seen TEXT,
last_seen TEXT,
run_count INTEGER,
status TEXT,
last_run_id TEXT
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS ingest_runs (
run_id TEXT PRIMARY KEY,
site_name TEXT,
server TEXT,
timestamp TEXT,
log_count INTEGER
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS license_config (
id INTEGER PRIMARY KEY CHECK (id = 1),
max_seats INTEGER NOT NULL DEFAULT 10,
enrollment_secret TEXT NOT NULL
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS clients (
client_id TEXT PRIMARY KEY,
hostname TEXT NOT NULL,
os_type TEXT NOT NULL,
cert_fingerprint TEXT NOT NULL,
status TEXT DEFAULT 'active',
first_seen TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
last_seen TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
""")
if enrollment_secret:
conn.execute("""
INSERT OR IGNORE INTO license_config (id, max_seats, enrollment_secret)
VALUES (1, ?, ?)
""", (max_seats, enrollment_secret))
conn.commit()
conn.close()
def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: int, min_runs: int) -> Dict[str, Any]:
"""
Evaluates candidate issues against the 12-hour evaluation window and 4-run rule.
Zero-state clients send raw candidate entries; this engine handles temporal state.
"""
client_server = payload.get("server", "unknown-host")
site_name = payload.get("site_name") or (client_server.split(".", 1)[1] if "." in client_server else "default")
logs = payload.get("logs", [])
run_id = str(uuid.uuid4())
now = datetime.now(timezone.utc)
now_iso = now.isoformat()
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
# Record the batch run
cursor.execute(
"INSERT INTO ingest_runs (run_id, site_name, server, timestamp, log_count) VALUES (?, ?, ?, ?, ?)",
(run_id, site_name, client_server, now_iso, len(logs))
)
processed_count = 0
promoted_to_verified = 0
for log in logs:
severity = str(log.get("severity", "WARNING")).upper()
# Edge forwarder filter safeguard: retain INFO to ERROR / CRITICAL; strip verbose debug noise
if severity in ["DEBUG", "TRACE"]:
continue
# Errors are always passed immediately; the 4-run rule only concerns warnings
is_error = severity in ["ERROR", "CRITICAL", "FATAL"]
signature = log.get("signature", "unknown")
server = log.get("server", client_server)
message = log.get("message", "")
os_type = log.get("os_type", "unknown")
fp = f"{site_name}:{server}:{signature}"
cursor.execute(
"SELECT run_count, first_seen, last_seen, status, last_run_id FROM active_issues WHERE fingerprint = ?",
(fp,)
)
row = cursor.fetchone()
if row:
run_count, first_seen_str, last_seen_str, current_status, last_run_id = row
try:
last_seen_dt = datetime.fromisoformat(last_seen_str)
except Exception:
last_seen_dt = now
# 12-hour evaluation window expiry check
if (now - last_seen_dt) > timedelta(hours=window_hours):
# Window elapsed: reset to new cycle
new_runs = 1
new_first_seen = now_iso
new_status = "VERIFIED" if is_error else "TRANSIENT"
else:
# Same run guard: only increment count once per distinct run batch
if last_run_id != run_id:
new_runs = run_count + 1
else:
new_runs = run_count
new_first_seen = first_seen_str
# 4-run rule applies to warnings; errors are always passed immediately as VERIFIED
new_status = "VERIFIED" if (is_error or new_runs >= min_runs) else "TRANSIENT"
if new_status == "VERIFIED" and current_status != "VERIFIED":
promoted_to_verified += 1
cursor.execute("""
UPDATE active_issues
SET run_count = ?, last_seen = ?, first_seen = ?, status = ?, last_run_id = ?, message = ?, severity = ?
WHERE fingerprint = ?
""", (new_runs, now_iso, new_first_seen, new_status, run_id, message, severity, fp))
else:
initial_status = "VERIFIED" if (is_error or 1 >= min_runs) else "TRANSIENT"
if initial_status == "VERIFIED":
promoted_to_verified += 1
cursor.execute("""
INSERT INTO active_issues
(fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status, last_run_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (fp, site_name, server, signature, severity, message, os_type, now_iso, now_iso, 1, initial_status, run_id))
processed_count += 1
conn.commit()
conn.close()
return {
"status": "success",
"run_id": run_id,
"processed": processed_count,
"promoted_verified": promoted_to_verified
}
def init_mtls_server_context(cert_dir: str = "certs") -> ssl.SSLContext:
"""Initializes TLS 1.3 server SSLContext with client certificate requirement (mTLS)."""
ca_file = os.path.join(cert_dir, "ca.crt")
srv_cert = os.path.join(cert_dir, "server.crt")
srv_key = os.path.join(cert_dir, "server.key")
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ctx.load_cert_chain(certfile=srv_cert, keyfile=srv_key)
ctx.load_verify_locations(cafile=ca_file)
ctx.verify_mode = ssl.CERT_REQUIRED
ctx.minimum_version = ssl.TLSVersion.TLSv1_3
return ctx
def reload_mtls_context(ssl_ctx: ssl.SSLContext, cert_dir: str = "certs"):
"""
Dynamically reloads server certificate chain and Root CA in an active SSLContext.
Allows in-flight TLS certificate rotation without dropping the listening socket.
"""
ca_file = os.path.join(cert_dir, "ca.crt")
srv_cert = os.path.join(cert_dir, "server.crt")
srv_key = os.path.join(cert_dir, "server.key")
ssl_ctx.load_cert_chain(certfile=srv_cert, keyfile=srv_key)
ssl_ctx.load_verify_locations(cafile=ca_file)
def check_and_rotate_server_certs(
cert_dir: str = "certs",
hostnames: Optional[List[str]] = None,
threshold_days: int = 30
) -> bool:
"""
Checks if Root CA or server TLS certificate are expiring within threshold_days.
If so, regenerates them, dynamically reloads the active SSLContext in-place,
and updates the server's in-memory CA reference so future enrollments use the new CA.
Returns True if renewed/reloaded, False otherwise.
"""
ca_renewed, srv_renewed = enrollment.check_and_renew_hub_pki(
cert_dir=cert_dir,
hostnames=hostnames,
threshold_days=threshold_days
)
if ca_renewed or srv_renewed:
print(f"[!] Server Hub PKI certificates renewed (CA renewed: {ca_renewed}, Server cert renewed: {srv_renewed}).")
ca_cert, ca_key, ca_pem, ca_key_pem = enrollment.generate_ca_if_needed(cert_dir=cert_dir, force_renew=False)
SERVER_STATE["ca_cert"] = ca_cert
SERVER_STATE["ca_key"] = ca_key
SERVER_STATE["ca_cert_pem"] = ca_pem
ssl_ctx = SERVER_STATE.get("ssl_ctx")
if ssl_ctx is not None:
try:
reload_mtls_context(ssl_ctx, cert_dir=cert_dir)
print("[+] In-flight mTLS SSLContext successfully reloaded with updated certificates.")
except Exception as e:
print(f"[!] Failed to reload in-flight SSLContext: {e}")
return True
return False
async def cert_validity_watchdog(interval_seconds: int = 43200, threshold_days: int = 30):
"""
Periodically checks the validity of Hub Root CA and Server TLS certificates (default every 12 hours).
Triggers in-flight renewal and dynamic context reloading if expiration is within threshold_days.
"""
config = SERVER_STATE.get("config", {})
cert_dir = config.get("cert_dir", "certs")
hostnames = [config.get("tcp_host", "0.0.0.0"), "127.0.0.1", "localhost"]
while True:
try:
await asyncio.sleep(interval_seconds)
check_and_rotate_server_certs(cert_dir=cert_dir, hostnames=hostnames, threshold_days=threshold_days)
except asyncio.CancelledError:
break
except Exception as e:
print(f"[!] Exception in cert_validity_watchdog: {e}")
async def handle_socket_client(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
"""
mTLS TCP socket handler.
Extracts client CN (client_id) from the TLS handshake,
validates active license status in SQLite, updates last_seen,
reads 4-byte big-endian length-prefixed JSON payload,
and ingests candidate logs into the temporal evaluation engine.
"""
client_id = None
ssl_obj = writer.get_extra_info("ssl_object")
if ssl_obj:
peercert = ssl_obj.getpeercert()
if peercert and "subject" in peercert:
for rdn in peercert["subject"]:
for key, val in rdn:
if key == "commonName":
client_id = val
break
# If mTLS is enforced, verify client in accounting database
if SERVER_STATE.get("tls_enabled", False):
if not client_id:
writer.close()
await writer.wait_closed()
return
db_path = SERVER_STATE["config"]["db_path"]
conn = sqlite3.connect(db_path)
c = conn.cursor()
c.execute("SELECT status FROM clients WHERE client_id = ?", (client_id,))
row = c.fetchone()
if not row or row[0] != "active":
conn.close()
writer.close()
await writer.wait_closed()
return
c.execute("UPDATE clients SET last_seen = CURRENT_TIMESTAMP WHERE client_id = ?", (client_id,))
conn.commit()
conn.close()
try:
# Read 4-byte length prefix
length_bytes = await reader.readexactly(4)
length = struct.unpack(">I", length_bytes)[0]
if length <= 0 or length > 10 * 1024 * 1024: # 10MB limit
raise ValueError(f"Invalid frame size: {length}")
payload_bytes = await reader.readexactly(length)
raw_payload = json.loads(payload_bytes.decode("utf-8"))
# Support both direct JSON payload over mTLS and legacy OpenPGP envelope
if "encrypted_payload" in raw_payload and SERVER_STATE.get("private_key_obj"):
pgp_msg = pgpy.PGPMessage.from_blob(raw_payload["encrypted_payload"])
priv_key = SERVER_STATE["private_key_obj"]
decrypted_obj = priv_key.decrypt(pgp_msg)
log_payload = json.loads(decrypted_obj.message)
else:
log_payload = raw_payload
# Attach authenticated client_id if not present
if client_id and "server" not in log_payload:
log_payload["server"] = client_id
# Ingest and apply 12h window / 4-run rule
res = process_ingested_logs(
log_payload,
db_path=SERVER_STATE["config"]["db_path"],
window_hours=SERVER_STATE["config"]["evaluation_window_hours"],
min_runs=SERVER_STATE["config"]["min_persistence_runs"]
)
resp_bytes = json.dumps(res).encode("utf-8")
writer.write(struct.pack(">I", len(resp_bytes)) + resp_bytes)
await writer.drain()
except Exception as e:
err = json.dumps({"status": "error", "message": str(e)}).encode("utf-8")
try:
writer.write(struct.pack(">I", len(err)) + err)
await writer.drain()
except Exception:
pass
finally:
writer.close()
try:
await writer.wait_closed()
except Exception:
pass
@app.post("/api/client/enroll")
def enroll_client(req: ClientEnrollRequest):
"""
Enrolls an edge client by validating the enrollment secret,
checking license seat limits, issuing a signed client certificate + key,
and recording the client in the SQLite accounting database.
"""
db_path = SERVER_STATE["config"]["db_path"]
conn = sqlite3.connect(db_path)
c = conn.cursor()
# 1. Validate enrollment secret against license_config
c.execute("SELECT enrollment_secret, max_seats FROM license_config WHERE id = 1")
row = c.fetchone()
if not row:
conn.close()
raise HTTPException(status_code=500, detail="License configuration not initialized")
expected_secret, max_seats = row
if not secrets.compare_digest(str(req.enrollment_secret), str(expected_secret)):
conn.close()
raise HTTPException(status_code=403, detail="Invalid enrollment secret")
ca_cert = SERVER_STATE.get("ca_cert")
ca_key = SERVER_STATE.get("ca_key")
ca_cert_pem = SERVER_STATE.get("ca_cert_pem")
if not ca_cert or not ca_key:
conn.close()
raise HTTPException(status_code=500, detail="Root CA not loaded on server")
# 2. Check if client_id already registered
c.execute("SELECT status FROM clients WHERE client_id = ?", (req.client_id,))
client_row = c.fetchone()
if client_row:
if client_row[0] == "revoked":
conn.close()
raise HTTPException(status_code=403, detail="Client certificate has been revoked")
# Re-issue for existing active client
client_cert_pem, client_key_pem = enrollment.issue_client_cert(req.client_id, ca_cert, ca_key)
fp = enrollment.calculate_cert_fingerprint(client_cert_pem)
c.execute("""
UPDATE clients
SET hostname = ?, os_type = ?, cert_fingerprint = ?, last_seen = CURRENT_TIMESTAMP
WHERE client_id = ?
""", (req.hostname, req.os, fp, req.client_id))
conn.commit()
conn.close()
print(f"[+] Re-enrolled active client: {req.client_id} ({req.hostname})")
return {
"ca_cert": ca_cert_pem,
"client_cert": client_cert_pem,
"client_key": client_key_pem
}
# 3. New client: check seat limits
c.execute("SELECT COUNT(*) FROM clients WHERE status = 'active'")
active_count = c.fetchone()[0]
if active_count >= max_seats:
conn.close()
raise HTTPException(status_code=403, detail="License seat limit reached")
# 4. Issue signed cert + key
client_cert_pem, client_key_pem = enrollment.issue_client_cert(req.client_id, ca_cert, ca_key)
fp = enrollment.calculate_cert_fingerprint(client_cert_pem)
c.execute("""
INSERT INTO clients (client_id, hostname, os_type, cert_fingerprint, status)
VALUES (?, ?, ?, ?, 'active')
""", (req.client_id, req.hostname, req.os, fp))
conn.commit()
conn.close()
print(f"[+] Successfully enrolled new client: {req.client_id} ({req.hostname}) [Seats: {active_count + 1}/{max_seats}]")
return {
"ca_cert": ca_cert_pem,
"client_cert": client_cert_pem,
"client_key": client_key_pem
}
@app.get("/api/clients")
def list_clients():
"""Returns all registered clients and license seat usage."""
db_path = SERVER_STATE["config"]["db_path"]
conn = sqlite3.connect(db_path)
c = conn.cursor()
c.execute("SELECT max_seats FROM license_config WHERE id = 1")
lic_row = c.fetchone()
max_seats = lic_row[0] if lic_row else 10
c.execute("SELECT client_id, hostname, os_type, cert_fingerprint, status, first_seen, last_seen FROM clients")
rows = c.fetchall()
conn.close()
clients = [
{
"client_id": r[0],
"hostname": r[1],
"os_type": r[2],
"cert_fingerprint": r[3],
"status": r[4],
"first_seen": r[5],
"last_seen": r[6]
}
for r in rows
]
active_count = sum(1 for cl in clients if cl["status"] == "active")
return {
"active_seats": active_count,
"max_seats": max_seats,
"clients": clients
}
@app.get("/api/hermes/report")
def get_verified_anomalies_for_hermes():
"""
Ingestion endpoint for Hermes agentic workflows.
Returns only verified anomalies that have satisfied the 4-run persistence rule
within the active 12-hour evaluation window. Transient blips (< 4 runs) are excluded.
"""
db_path = SERVER_STATE["config"]["db_path"]
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("""
SELECT fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status
FROM active_issues
WHERE status = 'VERIFIED'
ORDER BY last_seen DESC
""")
rows = cursor.fetchall()
conn.close()
report = []
for r in rows:
report.append({
"fingerprint": r[0],
"site": r[1],
"server": r[2],
"signature": r[3],
"severity": r[4],
"message": r[5],
"os_type": r[6],
"first_seen": r[7],
"last_seen": r[8],
"consecutive_runs": r[9],
"evaluation_window": f"{SERVER_STATE['config']['evaluation_window_hours']}h",
"verified": True,
"status": r[10]
})
return report
@app.get("/api/hermes/all")
def get_all_issues():
"""Diagnostic endpoint to inspect both transient candidate blips and verified anomalies."""
db_path = SERVER_STATE["config"]["db_path"]
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("""
SELECT fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status
FROM active_issues
""")
rows = cursor.fetchall()
conn.close()
return [
{
"fingerprint": r[0],
"site": r[1],
"server": r[2],
"signature": r[3],
"severity": r[4],
"message": r[5],
"os_type": r[6],
"first_seen": r[7],
"last_seen": r[8],
"run_count": r[9],
"status": r[10]
}
for r in rows
]
@app.get("/health")
def health_check():
return {
"status": "healthy",
"server_name": SERVER_STATE["config"]["server_name"],
"fingerprint": SERVER_STATE["config"]["server_fingerprint"],
"tcp_port": SERVER_STATE["config"]["tcp_port"],
"hermes_port": SERVER_STATE["config"]["hermes_port"],
"tls_enabled": SERVER_STATE.get("tls_enabled", False)
}
async def run_server():
"""Runs the mTLS TCP socket listener and the Hermes REST API concurrently."""
config = SERVER_STATE["config"]
tcp_host = config["tcp_host"]
tcp_port = int(config["tcp_port"])
hermes_host = config["hermes_host"]
hermes_port = int(config["hermes_port"])
ssl_ctx = SERVER_STATE.get("ssl_ctx")
# Start mTLS / TCP Socket Server
tcp_server = await asyncio.start_server(handle_socket_client, tcp_host, tcp_port, ssl=ssl_ctx)
mode_str = "mTLS TLSv1.3" if ssl_ctx else "Plain TCP"
print(f"[*] LOGAR {mode_str} Socket Server listening on {tcp_host}:{tcp_port}")
# Start FastAPI / Uvicorn server for Hermes & Enrollment
uv_config = uvicorn.Config(app, host=hermes_host, port=hermes_port, log_level="warning")
uv_server = uvicorn.Server(uv_config)
print(f"[*] Hermes Reporting API available at http://{hermes_host}:{hermes_port}/api/hermes/report")
print(f"[*] Client Enrollment API available at http://{hermes_host}:{hermes_port}/api/client/enroll")
watchdog_task = asyncio.create_task(cert_validity_watchdog())
try:
await asyncio.gather(
tcp_server.serve_forever(),
uv_server.serve(),
watchdog_task
)
finally:
watchdog_task.cancel()
try:
await watchdog_task
except asyncio.CancelledError:
pass
def main():
parser = argparse.ArgumentParser(description="LOGAR Cloud Hub & TCP Socket Ingestion Server")
parser.add_argument("--config", default=CONFIG_FILE_NAME, help="Path to server_config.json")
parser.add_argument("--create-client-config", action="store_true", help="Generate a client config with encryption-only fingerprint and server address")
parser.add_argument("--client-out", default="client_config.json", help="Output file path for generated client config")
parser.add_argument("--server-host", default="127.0.0.1", help="Server address to embed in client config")
parser.add_argument("--server-port", type=int, default=None, help="TCP port to embed in client config")
args = parser.parse_args()
config = load_or_init_config(args.config)
init_db(
config["db_path"],
enrollment_secret=config.get("enrollment_secret"),
max_seats=config.get("max_seats", 10)
)
# Initialize dynamic PKI (Root CA and Server TLS Certificate)
cert_dir = config.get("cert_dir", "certs")
ca_cert, ca_key, ca_pem, ca_key_pem = enrollment.generate_ca_if_needed(cert_dir=cert_dir)
srv_cert, srv_key, srv_pem, srv_key_pem = enrollment.generate_server_cert_if_needed(
ca_cert, ca_key,
hostnames=[config.get("tcp_host"), "127.0.0.1", "localhost"],
cert_dir=cert_dir
)
# Initialize mTLS SSLContext if enabled
ssl_ctx = None
if config.get("tls_enabled", True):
ssl_ctx = init_mtls_server_context(cert_dir=cert_dir)
# Load OpenPGP private key into memory (legacy fallback)
priv_key_obj, _ = pgpy.PGPKey.from_blob(config["private_key"])
SERVER_STATE["config"] = config
SERVER_STATE["private_key_obj"] = priv_key_obj
SERVER_STATE["ca_cert"] = ca_cert
SERVER_STATE["ca_key"] = ca_key
SERVER_STATE["ca_cert_pem"] = ca_pem
SERVER_STATE["ssl_ctx"] = ssl_ctx
SERVER_STATE["tls_enabled"] = config.get("tls_enabled", True)
if args.create_client_config:
port = args.server_port or config["tcp_port"]
create_client_config(
server_host=args.server_host,
server_port=port,
output_path=args.client_out,
config_path=args.config
)
sys.exit(0)
print("=" * 60)
print(f" LOGAR Server Hub: {config['server_name']}")
print(f" Transport Security: {'mTLS (TLS 1.3)' if ssl_ctx else 'Plain TCP'}")
print(f" License Quota: {config.get('max_seats', 10)} Active Seats")
print(f" Server Encryption Fingerprint: {config['server_fingerprint']}")
print(f" Evaluation Window: {config['evaluation_window_hours']} hours | 4-Run Rule: Warnings | Immediate Pass: Errors")
print("=" * 60)
try:
asyncio.run(run_server())
except KeyboardInterrupt:
print("\n[!] Server shutting down.")
if __name__ == "__main__":
main()
+457
View File
@@ -0,0 +1,457 @@
import os
import sys
import json
import socket
import ssl
import struct
import argparse
import urllib.request
import warnings
from datetime import datetime, timezone, timedelta
from typing import Optional, Dict, Any, List
# Suppress cryptography / pgpy deprecation notices
warnings.filterwarnings("ignore")
import pgpy
try:
import win32evtlog
except ImportError:
win32evtlog = None
CONFIG_FILE_NAME = "client_config.json"
STATE_FILE_NAME = "client_state.json"
def is_cert_expiring_soon(cert_path: str, threshold_days: int = 30) -> bool:
"""Checks if client certificate at cert_path is expiring within threshold_days."""
if not os.path.exists(cert_path):
return True
try:
from cryptography import x509
with open(cert_path, "r", encoding="utf-8") as f:
cert = x509.load_pem_x509_certificate(f.read().encode("utf-8"))
expiry = getattr(cert, "not_valid_after_utc", None)
if expiry is None:
expiry = cert.not_valid_after.replace(tzinfo=timezone.utc)
now = datetime.now(timezone.utc)
return expiry <= (now + timedelta(days=threshold_days))
except Exception:
return True
def enroll_client_if_needed(
hub_url: str,
enrollment_secret: str,
cert_dir: str,
client_id: str,
hostname: str,
os_type: str = "windows",
force_renew: bool = False,
threshold_days: int = 30
):
"""Bootstraps client enrollment if certificates are missing or expiring soon."""
os.makedirs(cert_dir, exist_ok=True)
ca_path = os.path.join(cert_dir, "ca.crt")
cert_path = os.path.join(cert_dir, "client.crt")
key_path = os.path.join(cert_dir, "client.key")
if not force_renew and os.path.exists(ca_path) and os.path.exists(cert_path) and os.path.exists(key_path):
if not is_cert_expiring_soon(cert_path, threshold_days=threshold_days):
return True
print(f"[*] Client certificate at {cert_path} is expiring within {threshold_days} days. Auto-renewing...")
action_name = "re-enrolling" if os.path.exists(cert_path) else "enrolling"
print(f"[*] Bootstrapping client {action_name} with LOGAR Hub at {hub_url}...")
enroll_endpoint = f"{hub_url.rstrip('/')}/api/client/enroll"
payload = {
"client_id": client_id,
"hostname": hostname,
"os": os_type,
"enrollment_secret": enrollment_secret
}
req = urllib.request.Request(
enroll_endpoint,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req, timeout=10) as resp:
if resp.status != 200:
raise RuntimeError(f"Enrollment failed with status code {resp.status}")
data = json.loads(resp.read().decode("utf-8"))
with open(ca_path, "w", encoding="utf-8") as f:
f.write(data["ca_cert"])
with open(cert_path, "w", encoding="utf-8") as f:
f.write(data["client_cert"])
with open(key_path, "w", encoding="utf-8") as f:
f.write(data["client_key"])
try:
os.chmod(key_path, 0o600)
except Exception:
pass
print(f"[+] Client certificates updated successfully in {os.path.abspath(cert_dir)}")
return True
def get_tls_socket(hub_host: str, hub_port: int, cert_dir: str):
"""Establishes an mTLS connection with the LOGAR hub using client certificates."""
ca_path = os.path.join(cert_dir, "ca.crt")
cert_path = os.path.join(cert_dir, "client.crt")
key_path = os.path.join(cert_dir, "client.key")
if not (os.path.exists(ca_path) and os.path.exists(cert_path) and os.path.exists(key_path)):
raise FileNotFoundError(f"mTLS certificates not found in '{cert_dir}'. Enroll client first.")
ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=ca_path)
ctx.load_cert_chain(certfile=cert_path, keyfile=key_path)
ctx.minimum_version = ssl.TLSVersion.TLSv1_3
ctx.check_hostname = False
raw_sock = socket.create_connection((hub_host, hub_port), timeout=15)
return ctx.wrap_socket(raw_sock, server_hostname=hub_host)
def get_state_path(config_path: str, custom_state_path: Optional[str] = None) -> str:
if custom_state_path:
return custom_state_path
config_dir = os.path.dirname(os.path.abspath(config_path))
return os.path.join(config_dir, STATE_FILE_NAME)
def load_state(state_path: str) -> dict:
if os.path.exists(state_path):
try:
with open(state_path, "r", encoding="utf-8") as f:
return json.load(f)
except Exception as e:
print(f"[!] Warning: Failed to read state file '{state_path}': {e}")
return {}
return {}
def save_state(state_path: str, state: dict):
try:
temp_path = f"{state_path}.tmp"
with open(temp_path, "w", encoding="utf-8") as f:
json.dump(state, f, indent=2)
os.replace(temp_path, state_path)
except Exception as e:
print(f"[!] Warning: Could not save client state to '{state_path}': {e}")
def commit_state(state: dict, state_path: str):
if "new_last_record_number" in state:
val = state.pop("new_last_record_number")
if val:
state["last_record_number"] = val
if "new_sent_record_ids" in state:
state["sent_record_ids"] = state.pop("new_sent_record_ids")
save_state(state_path, state)
def load_config(config_path: str = CONFIG_FILE_NAME):
if not os.path.exists(config_path):
raise FileNotFoundError(
f"Client configuration file not found at: {config_path}\n"
f"Generate one from the server using: python Server.py --create-client-config --client-out {config_path}"
)
with open(config_path, "r", encoding="utf-8") as f:
return json.load(f)
def get_machine_identifier() -> str:
"""
Returns the hostname of the machine sending the logs,
and appends the network/DNS domain if available.
"""
fqdn = socket.getfqdn()
if fqdn and "." in fqdn and not fqdn.startswith("localhost"):
return fqdn
hostname = socket.gethostname()
user_dns_domain = os.environ.get("USERDNSDOMAIN")
if user_dns_domain and user_dns_domain.lower() != hostname.lower():
return f"{hostname}.{user_dns_domain.lower()}"
try:
host_ip = socket.gethostbyname(hostname)
canonical_name = socket.gethostbyaddr(host_ip)[0]
if canonical_name and "." in canonical_name and not canonical_name.startswith("localhost"):
return canonical_name
except Exception:
pass
return hostname
def get_recent_windows_logs(hours: int = 24, state: Optional[dict] = None) -> list:
"""
Scans the Windows Application Event Log backwards for events within the window.
Edge Filtering: Retains INFO, WARNING, and ERROR. Drops Audit and Debug noise.
State Tracking: Skips events older than lookback window (default 24h) and events
that have already been sent in previous runs.
"""
if win32evtlog is None:
print("[!] pywin32 is not installed or not running on Windows. Returning mock/empty candidate list.")
return []
server = "localhost"
log_type = "Application"
flags = win32evtlog.EVENTLOG_BACKWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
try:
hand = win32evtlog.OpenEventLog(server, log_type)
except Exception as e:
print(f"[!] Error opening Windows event log: {e}")
return []
logs = []
cutoff_time = datetime.now() - timedelta(hours=hours)
machine_id = get_machine_identifier()
last_record_number = 0
sent_record_ids = set()
if state:
last_record_number = int(state.get("last_record_number", 0))
sent_record_ids = set(state.get("sent_record_ids", []))
# Windows Event Log EventTypes:
# 1: EVENTLOG_ERROR_TYPE -> ERROR
# 2: EVENTLOG_WARNING_TYPE -> WARNING
# 4: EVENTLOG_INFORMATION_TYPE -> INFO
# Excludes: 8 (Audit Success), 16 (Audit Failure), and other verbose noise
sev_map = {
1: "ERROR",
2: "WARNING",
4: "INFO"
}
newest_record_number = 0
collected_record_ids = []
while True:
events = win32evtlog.ReadEventLog(hand, flags, 0)
if not events:
break
for event in events:
rec_num = int(event.RecordNumber)
if newest_record_number == 0:
newest_record_number = rec_num
# 1. Skip entries older than lookback window (default: 24h)
if event.TimeGenerated < cutoff_time:
break
# 2. Skip already sent events if we've reached records <= last_record_number
# (unless the log was cleared and numbers wrapped, i.e. newest_record_number < last_record_number)
if last_record_number > 0 and newest_record_number >= last_record_number:
if rec_num <= last_record_number:
break
rec_id = f"{rec_num}:{event.TimeGenerated.isoformat()}"
if rec_id in sent_record_ids:
continue
# Filter: upload everything from INFO to ERROR only
if event.EventType in sev_map:
msg = " ".join(event.StringInserts) if event.StringInserts else "Event Log Entry"
logs.append({
"server": machine_id,
"os_type": "windows",
"signature": event.SourceName or "Windows-Event",
"severity": sev_map[event.EventType],
"message": msg[:2048] # Limit message length
})
collected_record_ids.append(rec_id)
if events[-1].TimeGenerated < cutoff_time:
break
if last_record_number > 0 and newest_record_number >= last_record_number and events[-1].RecordNumber <= last_record_number:
break
win32evtlog.CloseEventLog(hand)
if state is not None:
target_rec = max(newest_record_number, last_record_number)
state["new_last_record_number"] = target_rec
state["new_sent_record_ids"] = (list(sent_record_ids) + collected_record_ids)[-1000:]
state["last_run_timestamp"] = datetime.now(timezone.utc).isoformat()
return logs
def send_encrypted_logs_over_socket(config: dict, logs: list):
"""
Streams logs to the LOGAR hub.
Uses mutual TLS 1.3 (mTLS) with client certificates if available,
or falls back to OpenPGP encrypted envelope over TCP.
"""
server_host = config["server_host"]
server_port = int(config["server_port"])
cert_dir = config.get("cert_dir", "certs")
enrollment_secret = config.get("enrollment_secret")
machine_id = get_machine_identifier()
# Attempt automatic enrollment bootstrap if certs are missing and secret is provided
hub_url = None
if enrollment_secret:
hermes_host = config.get("hermes_host", server_host)
hermes_port = config.get("hermes_port", 8443)
hub_url = f"http://{hermes_host}:{hermes_port}"
try:
enroll_client_if_needed(hub_url, enrollment_secret, cert_dir, machine_id, machine_id, os_type="windows")
except Exception as e:
print(f"[!] Warning: Enrollment bootstrap failed: {e}")
ca_path = os.path.join(cert_dir, "ca.crt")
cert_path = os.path.join(cert_dir, "client.crt")
key_path = os.path.join(cert_dir, "client.key")
has_mtls_certs = os.path.exists(ca_path) and os.path.exists(cert_path) and os.path.exists(key_path)
if has_mtls_certs:
print(f"[*] Connecting to LOGAR server at {server_host}:{server_port} over mTLS (TLS 1.3)...")
sock = None
try:
sock = get_tls_socket(server_host, server_port, cert_dir)
except (ssl.SSLError, ssl.CertificateError, ConnectionResetError) as tls_err:
if enrollment_secret and hub_url:
print(f"[!] TLS handshake error ({tls_err}). Re-enrolling with LOGAR Hub...")
try:
enroll_client_if_needed(hub_url, enrollment_secret, cert_dir, machine_id, machine_id, os_type="windows", force_renew=True)
sock = get_tls_socket(server_host, server_port, cert_dir)
except Exception as retry_err:
print(f"[!] Re-enrollment or reconnection retry failed: {retry_err}")
raise
else:
raise
with sock:
payload = {
"server": machine_id,
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
payload_bytes = json.dumps(payload).encode("utf-8")
frame = struct.pack(">I", len(payload_bytes)) + payload_bytes
sock.sendall(frame)
resp_len_bytes = sock.recv(4)
if not resp_len_bytes:
raise ConnectionError("Server closed mTLS connection without response.")
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = bytearray()
while len(resp_bytes) < resp_len:
chunk = sock.recv(min(4096, resp_len - len(resp_bytes)))
if not chunk:
break
resp_bytes.extend(chunk)
response = json.loads(resp_bytes.decode("utf-8"))
print(f"[+] Server response: {response}")
return response
# Fallback to OpenPGP envelope over plain TCP socket
auth_token = config.get("auth_token", "")
pub_key_armored = config.get("server_public_key")
if not pub_key_armored:
raise ValueError("No server public key or mTLS certificates available for connection.")
expected_fp = config.get("server_fingerprint", "").replace(" ", "").upper()
pub_key, _ = pgpy.PGPKey.from_blob(pub_key_armored)
actual_fp = str(pub_key.fingerprint).replace(" ", "").upper()
if expected_fp and actual_fp != expected_fp:
raise ValueError(f"Server fingerprint mismatch! Expected {expected_fp}, but key has {actual_fp}.")
payload = {
"server": machine_id,
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
payload_json = json.dumps(payload)
pgp_msg = pgpy.PGPMessage.new(payload_json)
encrypted_msg = pub_key.encrypt(pgp_msg)
encrypted_armored = str(encrypted_msg)
envelope = {
"auth_token": auth_token,
"timestamp": datetime.now(timezone.utc).isoformat(),
"encrypted_payload": encrypted_armored
}
envelope_bytes = json.dumps(envelope).encode("utf-8")
print(f"[*] Connecting to LOGAR server at {server_host}:{server_port} over secure TCP socket...")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.settimeout(15.0)
sock.connect((server_host, server_port))
frame = struct.pack(">I", len(envelope_bytes)) + envelope_bytes
sock.sendall(frame)
resp_len_bytes = sock.recv(4)
if not resp_len_bytes:
raise ConnectionError("Server closed connection without response.")
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = bytearray()
while len(resp_bytes) < resp_len:
chunk = sock.recv(min(4096, resp_len - len(resp_bytes)))
if not chunk:
break
resp_bytes.extend(chunk)
response = json.loads(resp_bytes.decode("utf-8"))
print(f"[+] Server response: {response}")
return response
def main():
parser = argparse.ArgumentParser(description="LOGAR Windows Edge Log Forwarder with State Tracking")
parser.add_argument("--config", default=CONFIG_FILE_NAME, help="Path to client_config.json")
parser.add_argument("--hours", type=int, default=24, help="Lookback window in hours for event logs (default: 24)")
parser.add_argument("--state-file", default=None, help="Path to state tracking file (default: client_state.json next to config)")
parser.add_argument("--no-state", action="store_true", help="Disable state tracking and send all events matching lookback window")
args = parser.parse_args()
try:
config = load_config(args.config)
except Exception as e:
print(f"[!] Configuration error: {e}")
sys.exit(1)
state_path = get_state_path(args.config, args.state_file)
state = None if args.no_state else load_state(state_path)
machine_id = get_machine_identifier()
print(f"[*] Edge Forwarder Node: {machine_id}")
if state and "last_record_number" in state:
print(f"[*] State tracking active: resuming from record #{state['last_record_number']} (state file: {state_path})")
elif not args.no_state:
print(f"[*] State tracking initialized (state file: {state_path})")
print(f"[*] Scanning Windows Application event log for unsent entries (last {args.hours} hours)...")
candidate_logs = get_recent_windows_logs(hours=args.hours, state=state)
print(f"[*] Found {len(candidate_logs)} unsent candidate entries (INFO to ERROR, entries > {args.hours}h and already-sent skipped).")
if not candidate_logs:
print("[*] No new unsent events to transmit.")
if state is not None:
commit_state(state, state_path)
return
try:
resp = send_encrypted_logs_over_socket(config, candidate_logs)
if state is not None and resp and resp.get("status") == "success":
commit_state(state, state_path)
print(f"[+] State successfully committed to {state_path}")
except Exception as e:
print(f"[!] Failed to stream logs to server: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
View File
+380
View File
@@ -0,0 +1,380 @@
import os
import datetime
import ipaddress
from typing import Tuple, List, Optional
from cryptography import x509
from cryptography.x509.oid import NameOID, ExtendedKeyUsageOID
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
def calculate_cert_fingerprint(cert_pem: str) -> str:
"""Computes SHA-256 fingerprint for a PEM-encoded X.509 certificate."""
cert = x509.load_pem_x509_certificate(cert_pem.encode("utf-8"))
return cert.fingerprint(hashes.SHA256()).hex().upper()
def is_cert_expiring_soon(cert_pem: str, threshold_days: int = 30) -> bool:
"""
Checks if a PEM-encoded X.509 certificate expires within `threshold_days` (or is already expired).
Returns True if expiring soon or expired, False otherwise.
"""
try:
cert = x509.load_pem_x509_certificate(cert_pem.encode("utf-8"))
expiry = getattr(cert, "not_valid_after_utc", None)
if expiry is None:
expiry = cert.not_valid_after.replace(tzinfo=datetime.timezone.utc)
now = datetime.datetime.now(datetime.timezone.utc)
return expiry <= (now + datetime.timedelta(days=threshold_days))
except Exception:
return True
def generate_ca_if_needed(
cert_dir: str = "certs",
common_name: str = "LOGAR-Root-CA",
force_renew: bool = False,
threshold_days: int = 30
) -> Tuple[x509.Certificate, rsa.RSAPrivateKey, str, str]:
"""
Loads an existing Root CA or generates a self-signed Root CA certificate and private key.
If existing CA cert is expiring within threshold_days (or force_renew is True), regenerates it.
Returns (ca_cert_obj, ca_key_obj, ca_cert_pem, ca_key_pem).
"""
os.makedirs(cert_dir, exist_ok=True)
ca_cert_path = os.path.join(cert_dir, "ca.crt")
ca_key_path = os.path.join(cert_dir, "ca.key")
if not force_renew and os.path.exists(ca_cert_path) and os.path.exists(ca_key_path):
with open(ca_cert_path, "r", encoding="utf-8") as f:
ca_cert_pem = f.read()
with open(ca_key_path, "r", encoding="utf-8") as f:
ca_key_pem = f.read()
try:
ca_cert = x509.load_pem_x509_certificate(ca_cert_pem.encode("utf-8"))
ca_key = serialization.load_pem_private_key(ca_key_pem.encode("utf-8"), password=None)
if not is_cert_expiring_soon(ca_cert_pem, threshold_days=threshold_days):
return ca_cert, ca_key, ca_cert_pem, ca_key_pem
except Exception:
pass
# Create timestamped backup of previous CA if present
if os.path.exists(ca_cert_path):
try:
timestamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%d_%H%M%S")
os.replace(ca_cert_path, f"{ca_cert_path}.{timestamp}.bak")
if os.path.exists(ca_key_path):
os.replace(ca_key_path, f"{ca_key_path}.{timestamp}.bak")
except Exception:
pass
# Generate RSA 4096 private key for Root CA
ca_key = rsa.generate_private_key(public_exponent=65537, key_size=4096)
subject = issuer = x509.Name([
x509.NameAttribute(NameOID.COUNTRY_NAME, "AT"),
x509.NameAttribute(NameOID.ORGANIZATION_NAME, "LOGAR"),
x509.NameAttribute(NameOID.COMMON_NAME, common_name),
])
now = datetime.datetime.now(datetime.timezone.utc)
ca_cert = (
x509.CertificateBuilder()
.subject_name(subject)
.issuer_name(issuer)
.public_key(ca_key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(now - datetime.timedelta(minutes=5))
.not_valid_after(now + datetime.timedelta(days=3650))
.add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
.add_extension(
x509.KeyUsage(
digital_signature=True,
key_encipherment=False,
key_cert_sign=True,
crl_sign=True,
content_commitment=False,
data_encipherment=False,
key_agreement=False,
encipher_only=False,
decipher_only=False
),
critical=True
)
.add_extension(
x509.SubjectKeyIdentifier.from_public_key(ca_key.public_key()),
critical=False
)
.sign(ca_key, hashes.SHA256())
)
ca_cert_pem = ca_cert.public_bytes(serialization.Encoding.PEM).decode("utf-8")
ca_key_pem = ca_key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.NoEncryption()
).decode("utf-8")
with open(ca_cert_path, "w", encoding="utf-8") as f:
f.write(ca_cert_pem)
with open(ca_key_path, "w", encoding="utf-8") as f:
f.write(ca_key_pem)
try:
os.chmod(ca_key_path, 0o600)
except Exception:
pass
return ca_cert, ca_key, ca_cert_pem, ca_key_pem
def generate_server_cert_if_needed(
ca_cert: x509.Certificate,
ca_key: rsa.RSAPrivateKey,
hostnames: Optional[List[str]] = None,
cert_dir: str = "certs",
days_valid: int = 825,
force_renew: bool = False,
threshold_days: int = 30
) -> Tuple[x509.Certificate, rsa.RSAPrivateKey, str, str]:
"""
Loads an existing server certificate or generates a new server TLS certificate signed by the Root CA.
If existing server cert is expiring within threshold_days (or force_renew is True), regenerates it.
Includes SANs for localhost, 127.0.0.1, and specified hostnames.
"""
os.makedirs(cert_dir, exist_ok=True)
server_cert_path = os.path.join(cert_dir, "server.crt")
server_key_path = os.path.join(cert_dir, "server.key")
if not force_renew and os.path.exists(server_cert_path) and os.path.exists(server_key_path):
with open(server_cert_path, "r", encoding="utf-8") as f:
server_cert_pem = f.read()
with open(server_key_path, "r", encoding="utf-8") as f:
server_key_pem = f.read()
try:
srv_cert = x509.load_pem_x509_certificate(server_cert_pem.encode("utf-8"))
srv_key = serialization.load_pem_private_key(server_key_pem.encode("utf-8"), password=None)
if not is_cert_expiring_soon(server_cert_pem, threshold_days=threshold_days):
return srv_cert, srv_key, server_cert_pem, server_key_pem
except Exception:
pass
# Create timestamped backup of previous server cert if present
if os.path.exists(server_cert_path):
try:
timestamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%d_%H%M%S")
os.replace(server_cert_path, f"{server_cert_path}.{timestamp}.bak")
if os.path.exists(server_key_path):
os.replace(server_key_path, f"{server_key_path}.{timestamp}.bak")
except Exception:
pass
server_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
subject = x509.Name([
x509.NameAttribute(NameOID.COUNTRY_NAME, "AT"),
x509.NameAttribute(NameOID.ORGANIZATION_NAME, "LOGAR"),
x509.NameAttribute(NameOID.COMMON_NAME, "LOGAR-Server-Hub"),
])
san_list = [
x509.DNSName("localhost"),
x509.DNSName("LOGAR-Server-Hub"),
x509.IPAddress(ipaddress.IPv4Address("127.0.0.1")),
x509.IPAddress(ipaddress.IPv6Address("::1")),
]
if hostnames:
for host in hostnames:
if not host:
continue
try:
ip_obj = ipaddress.ip_address(host)
san_list.append(x509.IPAddress(ip_obj))
except ValueError:
san_list.append(x509.DNSName(host))
now = datetime.datetime.now(datetime.timezone.utc)
server_cert = (
x509.CertificateBuilder()
.subject_name(subject)
.issuer_name(ca_cert.subject)
.public_key(server_key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(now - datetime.timedelta(minutes=5))
.not_valid_after(now + datetime.timedelta(days=days_valid))
.add_extension(x509.BasicConstraints(ca=False, path_length=None), critical=True)
.add_extension(
x509.KeyUsage(
digital_signature=True,
key_encipherment=True,
key_cert_sign=False,
crl_sign=False,
content_commitment=False,
data_encipherment=False,
key_agreement=False,
encipher_only=False,
decipher_only=False
),
critical=True
)
.add_extension(
x509.ExtendedKeyUsage([ExtendedKeyUsageOID.SERVER_AUTH]),
critical=False
)
.add_extension(
x509.SubjectKeyIdentifier.from_public_key(server_key.public_key()),
critical=False
)
.add_extension(
x509.AuthorityKeyIdentifier.from_issuer_public_key(ca_key.public_key()),
critical=False
)
.add_extension(x509.SubjectAlternativeName(san_list), critical=False)
.sign(ca_key, hashes.SHA256())
)
server_cert_pem = server_cert.public_bytes(serialization.Encoding.PEM).decode("utf-8")
server_key_pem = server_key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.NoEncryption()
).decode("utf-8")
with open(server_cert_path, "w", encoding="utf-8") as f:
f.write(server_cert_pem)
with open(server_key_path, "w", encoding="utf-8") as f:
f.write(server_key_pem)
try:
os.chmod(server_key_path, 0o600)
except Exception:
pass
return server_cert, server_key, server_cert_pem, server_key_pem
def issue_client_cert(
client_id: str,
ca_cert: x509.Certificate,
ca_key: rsa.RSAPrivateKey,
days_valid: int = 365
) -> Tuple[str, str]:
"""
Generates a 2048-bit RSA private key and signs an X.509 client certificate
with Common Name set to client_id.
Returns (cert_pem, key_pem).
"""
client_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
subject = x509.Name([
x509.NameAttribute(NameOID.COUNTRY_NAME, "AT"),
x509.NameAttribute(NameOID.ORGANIZATION_NAME, "LOGAR"),
x509.NameAttribute(NameOID.COMMON_NAME, client_id),
])
now = datetime.datetime.now(datetime.timezone.utc)
cert = (
x509.CertificateBuilder()
.subject_name(subject)
.issuer_name(ca_cert.subject)
.public_key(client_key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(now - datetime.timedelta(minutes=5))
.not_valid_after(now + datetime.timedelta(days=days_valid))
.add_extension(x509.BasicConstraints(ca=False, path_length=None), critical=True)
.add_extension(
x509.KeyUsage(
digital_signature=True,
key_encipherment=True,
key_cert_sign=False,
crl_sign=False,
content_commitment=False,
data_encipherment=False,
key_agreement=False,
encipher_only=False,
decipher_only=False
),
critical=True
)
.add_extension(
x509.ExtendedKeyUsage([ExtendedKeyUsageOID.CLIENT_AUTH]),
critical=False
)
.add_extension(
x509.SubjectKeyIdentifier.from_public_key(client_key.public_key()),
critical=False
)
.add_extension(
x509.AuthorityKeyIdentifier.from_issuer_public_key(ca_key.public_key()),
critical=False
)
.sign(ca_key, hashes.SHA256())
)
cert_pem = cert.public_bytes(serialization.Encoding.PEM).decode("utf-8")
key_pem = client_key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.NoEncryption()
).decode("utf-8")
return cert_pem, key_pem
def check_and_renew_hub_pki(
cert_dir: str = "certs",
hostnames: Optional[List[str]] = None,
threshold_days: int = 30
) -> Tuple[bool, bool]:
"""
Evaluates expiration status of Root CA and Server TLS certificates.
If CA certificate is expiring within threshold_days (or missing):
- Regenerates Root CA.
- Automatically regenerates Server TLS certificate (since CA issuer changed).
- Returns (ca_renewed=True, server_renewed=True)
Else if Server TLS certificate is expiring within threshold_days (or missing):
- Regenerates Server TLS certificate signed by existing Root CA.
- Returns (ca_renewed=False, server_renewed=True)
Otherwise:
- Returns (False, False)
"""
os.makedirs(cert_dir, exist_ok=True)
ca_cert_path = os.path.join(cert_dir, "ca.crt")
server_cert_path = os.path.join(cert_dir, "server.crt")
renew_ca = False
renew_server = False
if not os.path.exists(ca_cert_path):
renew_ca = True
else:
try:
with open(ca_cert_path, "r", encoding="utf-8") as f:
ca_pem = f.read()
if is_cert_expiring_soon(ca_pem, threshold_days=threshold_days):
renew_ca = True
except Exception:
renew_ca = True
if renew_ca:
ca_cert, ca_key, _, _ = generate_ca_if_needed(cert_dir=cert_dir, force_renew=True)
generate_server_cert_if_needed(ca_cert, ca_key, hostnames=hostnames, cert_dir=cert_dir, force_renew=True)
return True, True
if not os.path.exists(server_cert_path):
renew_server = True
else:
try:
with open(server_cert_path, "r", encoding="utf-8") as f:
srv_pem = f.read()
if is_cert_expiring_soon(srv_pem, threshold_days=threshold_days):
renew_server = True
except Exception:
renew_server = True
if renew_server:
ca_cert, ca_key, _, _ = generate_ca_if_needed(cert_dir=cert_dir, force_renew=False)
generate_server_cert_if_needed(ca_cert, ca_key, hostnames=hostnames, cert_dir=cert_dir, force_renew=True)
return False, True
return False, False
-131
View File
@@ -1,131 +0,0 @@
import os
import sys
import json
import time
import socket
import struct
import sqlite3
import urllib.request
import warnings
from datetime import datetime, timezone, timedelta
warnings.filterwarnings("ignore")
import pgpy
# Test server endpoints
TCP_HOST = "127.0.0.1"
TCP_PORT = 9443
HERMES_HOST = "127.0.0.1"
HERMES_PORT = 8443
def run_tests():
print("=== [1] Verifying server_config.json & client_config.json ===")
assert os.path.exists("server_config.json"), "server_config.json must exist"
assert os.path.exists("client_config.json"), "client_config.json must exist"
with open("client_config.json", "r", encoding="utf-8") as f:
client_conf = json.load(f)
with open("server_config.json", "r", encoding="utf-8") as f:
server_conf = json.load(f)
assert "server_name" not in client_conf, "client_config.json must NOT contain server_name"
assert "name" not in client_conf, "client_config.json must NOT contain name"
assert "site_name" not in client_conf, "client_config.json must NOT contain site_name"
assert client_conf["server_fingerprint"] == server_conf["server_fingerprint"], "Fingerprints must match"
print(f"[OK] Verified client_config.json contains no machine/server/site name.")
print(f"[OK] Fingerprint verified: {client_conf['server_fingerprint']}")
# Load public key
pub_key, _ = pgpy.PGPKey.from_blob(client_conf["server_public_key"])
def send_socket_batch(logs, auth_token=client_conf["auth_token"]):
payload = {
"server": "test-edge-node.corp.internal",
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
pgp_msg = pgpy.PGPMessage.new(json.dumps(payload))
enc = pub_key.encrypt(pgp_msg)
envelope = {
"auth_token": auth_token,
"timestamp": datetime.now(timezone.utc).isoformat(),
"encrypted_payload": str(enc)
}
envelope_bytes = json.dumps(envelope).encode("utf-8")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.settimeout(5.0)
s.connect((TCP_HOST, TCP_PORT))
frame = struct.pack(">I", len(envelope_bytes)) + envelope_bytes
s.sendall(frame)
resp_len_bytes = s.recv(4)
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = s.recv(resp_len)
return json.loads(resp_bytes.decode("utf-8"))
print("\n=== [2] Testing Socket Authentication Failure ===")
bad_resp = send_socket_batch([], auth_token="invalid-token-12345")
assert bad_resp.get("status") == "error", f"Expected error, got: {bad_resp}"
print(f"[OK] Bad auth rejected correctly: {bad_resp['message']}")
test_signature = "TestServiceCrash"
candidate_log = [{
"server": "test-edge-node",
"os_type": "linux",
"signature": test_signature,
"severity": "ERROR",
"message": "Out of memory killer triggered"
}]
print("\n=== [3] Testing Temporal Persistence & 4-Run Rule ===")
for run_num in range(1, 5):
resp = send_socket_batch(candidate_log)
assert resp.get("status") == "success", f"Run {run_num} failed: {resp}"
print(f"[Run {run_num}/4] Ingested successfully. Promoted to verified: {resp.get('promoted_verified')}")
# Inspect SQLite database directly
conn = sqlite3.connect(server_conf.get("db_path", "logar_state.db"))
cursor = conn.cursor()
cursor.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", (test_signature,))
row = cursor.fetchone()
conn.close()
assert row is not None, "Issue not found in SQLite"
run_count, status = row
print(f"[DB Verification] Issue '{test_signature}' -> run_count: {run_count}, status: {status}")
assert run_count >= 4, f"Expected run_count >= 4, got {run_count}"
assert status == "VERIFIED", f"Expected status 'VERIFIED', got {status}"
print("[OK] 4-Run Rule verified: Transient issue promoted to VERIFIED anomaly!")
print("\n=== [4] Testing Hermes Reporting Endpoint (/api/hermes/report) ===")
req = urllib.request.Request(f"http://{HERMES_HOST}:{HERMES_PORT}/api/hermes/report")
with urllib.request.urlopen(req, timeout=5) as response:
assert response.status == 200, f"Expected 200, got {response.status}"
hermes_data = json.loads(response.read().decode("utf-8"))
print(f"[Hermes API] Returned {len(hermes_data)} verified anomalies:")
found_issue = False
for issue in hermes_data:
print(f" - Fingerprint: {issue['fingerprint']} | Consecutive Runs: {issue['consecutive_runs']} | Status: {issue['status']}")
if issue["signature"] == test_signature:
found_issue = True
assert issue["verified"] is True
assert issue["consecutive_runs"] >= 4
assert found_issue, f"Test issue {test_signature} should be in Hermes report"
print("[OK] Hermes reporting validated!")
print("\n=== [5] Testing Windows Client Script Integration ===")
from Win_Client import get_recent_windows_logs
win_logs = get_recent_windows_logs(hours=6)
print(f"[Win_Client] Successfully queried Windows logs: {len(win_logs)} candidate entries.")
print("\n==========================================")
print(" ALL VERIFICATION TESTS PASSED SUCCESSFULLY! ")
print("==========================================")
if __name__ == "__main__":
run_tests()
+249
View File
@@ -0,0 +1,249 @@
import os
import sys
import json
import struct
import unittest
import warnings
warnings.filterwarnings("ignore")
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "src")))
import Linux_Client
import pgpy
from pgpy.constants import PubKeyAlgorithm, KeyFlags, HashAlgorithm, SymmetricKeyAlgorithm, CompressionAlgorithm
class TestLinuxClientComponent(unittest.TestCase):
def setUp(self):
self.dummy_config = "test_linux_client_config.json"
# Generate dummy PGP key for testing
key = pgpy.PGPKey.new(PubKeyAlgorithm.RSAEncryptOrSign, 2048)
uid = pgpy.PGPUID.new("TestHub")
key.add_uid(
uid,
usage={KeyFlags.EncryptCommunications, KeyFlags.EncryptStorage},
hashes=[HashAlgorithm.SHA256],
ciphers=[SymmetricKeyAlgorithm.AES256],
compression=[CompressionAlgorithm.Uncompressed]
)
self.server_priv = key
self.server_pub = key.pubkey
self.fingerprint = str(key.pubkey.fingerprint)
with open(self.dummy_config, "w", encoding="utf-8") as f:
json.dump({
"server_host": "127.0.0.1",
"server_port": 9443,
"server_fingerprint": self.fingerprint,
"server_public_key": str(self.server_pub),
"auth_token": "secret-test-token"
}, f)
def tearDown(self):
if os.path.exists(self.dummy_config):
try:
os.remove(self.dummy_config)
except Exception:
pass
def test_client_config_anonymity(self):
config = Linux_Client.load_config(self.dummy_config)
self.assertNotIn("server_name", config)
self.assertNotIn("name", config)
self.assertNotIn("site_name", config)
self.assertEqual(config["server_fingerprint"], self.fingerprint)
def test_get_machine_identifier(self):
machine_id = Linux_Client.get_machine_identifier()
self.assertIsInstance(machine_id, str)
self.assertGreater(len(machine_id), 0)
self.assertNotEqual(machine_id, "localhost")
def test_journalctl_parsing_and_priority_filter(self):
sample_journal_lines = [
json.dumps({"PRIORITY": "3", "SYSLOG_IDENTIFIER": "sshd", "MESSAGE": "Failed password for root"}),
json.dumps({"PRIORITY": "4", "SYSLOG_IDENTIFIER": "systemd", "MESSAGE": "Unit entered failed state"}),
json.dumps({"PRIORITY": "6", "SYSLOG_IDENTIFIER": "cron", "MESSAGE": "Informational session opened"}),
json.dumps({"PRIORITY": "7", "SYSLOG_IDENTIFIER": "debugd", "MESSAGE": "Verbose debugging log"}),
]
logs = []
machine_id = Linux_Client.get_machine_identifier()
for line_str in sample_journal_lines:
entry = json.loads(line_str)
priority = int(entry.get("PRIORITY", "6"))
# Filter: Retain INFO to ERROR (<= 6), drop DEBUG (> 6)
if priority > 6:
continue
if priority <= 3:
sev = "ERROR"
elif priority in (4, 5):
sev = "WARNING"
else:
sev = "INFO"
logs.append({
"server": machine_id,
"os_type": "linux",
"signature": entry.get("SYSLOG_IDENTIFIER", "unknown"),
"severity": sev,
"message": entry.get("MESSAGE", "")
})
# Priority 7 (DEBUG) must be stripped, while 3 (ERROR), 4 (WARNING), 6 (INFO) are retained
self.assertEqual(len(logs), 3)
self.assertEqual(logs[0]["severity"], "ERROR")
self.assertEqual(logs[1]["severity"], "WARNING")
self.assertEqual(logs[2]["severity"], "INFO")
def test_encryption_and_decryption(self):
config = Linux_Client.load_config(self.dummy_config)
pub_key, _ = pgpy.PGPKey.from_blob(config["server_public_key"])
payload = {
"server": Linux_Client.get_machine_identifier(),
"logs": [{"signature": "kernel", "severity": "ERROR", "message": "Kernel panic - not syncing"}]
}
msg = pgpy.PGPMessage.new(json.dumps(payload))
enc = pub_key.encrypt(msg)
self.assertTrue(str(enc).startswith("-----BEGIN PGP MESSAGE-----"))
dec = self.server_priv.decrypt(enc)
restored = json.loads(dec.message)
self.assertEqual(restored["logs"][0]["signature"], "kernel")
def test_state_lifecycle(self):
state_path = "test_linux_state.json"
try:
# 1. Load non-existent returns empty dict
state = Linux_Client.load_state(state_path)
self.assertEqual(state, {})
# 2. Stage new cursor and timestamp
state["new_last_cursor"] = "s=abc;i=123"
state["new_last_timestamp_us"] = 1700000000000000
state["new_sent_cursors"] = ["s=abc;i=123"]
# 3. Commit state moves staged keys to permanent and writes atomically
Linux_Client.commit_state(state, state_path)
self.assertNotIn("new_last_cursor", state)
self.assertEqual(state.get("last_cursor"), "s=abc;i=123")
self.assertEqual(state.get("last_timestamp_us"), 1700000000000000)
self.assertEqual(state.get("sent_cursors"), ["s=abc;i=123"])
# 4. Reload from disk
reloaded = Linux_Client.load_state(state_path)
self.assertEqual(reloaded.get("last_cursor"), "s=abc;i=123")
self.assertEqual(reloaded.get("last_timestamp_us"), 1700000000000000)
finally:
if os.path.exists(state_path):
os.remove(state_path)
def test_duplicate_suppression_and_lookback_logic(self):
from datetime import datetime, timezone, timedelta
now_us = datetime.now(timezone.utc).timestamp() * 1_000_000
cutoff_epoch_us = (datetime.now(timezone.utc) - timedelta(hours=24)).timestamp() * 1_000_000
mock_entries = [
# 1. 26 hours old -> skip (> 24h)
{"__CURSOR": "c1", "__REALTIME_TIMESTAMP": str(int(now_us - 26 * 3600 * 1_000_000)), "PRIORITY": "3", "MESSAGE": "Old error"},
# 2. 2 hours old, already sent -> skip
{"__CURSOR": "c2", "__REALTIME_TIMESTAMP": str(int(now_us - 2 * 3600 * 1_000_000)), "PRIORITY": "4", "MESSAGE": "Already sent warning"},
# 3. 1 hour old, new entry -> retain
{"__CURSOR": "c3", "__REALTIME_TIMESTAMP": str(int(now_us - 1 * 3600 * 1_000_000)), "PRIORITY": "6", "MESSAGE": "New info"},
# 4. 30 mins old, debug -> skip priority
{"__CURSOR": "c4", "__REALTIME_TIMESTAMP": str(int(now_us - 1800 * 1_000_000)), "PRIORITY": "7", "MESSAGE": "Debug entry"}
]
state = {
"last_cursor": "c2",
"last_timestamp_us": int(now_us - 2 * 3600 * 1_000_000),
"sent_cursors": ["c2"]
}
# Simulate the filtering loop from get_recent_linux_logs
logs = []
last_cursor = state.get("last_cursor")
last_timestamp_us = float(state.get("last_timestamp_us", 0))
sent_cursors = set(state.get("sent_cursors", []))
newest_cursor = None
newest_timestamp_us = last_timestamp_us
collected_cursors = []
for entry in mock_entries:
entry_cursor = entry.get("__CURSOR")
entry_ts_us = float(entry.get("__REALTIME_TIMESTAMP"))
if entry_ts_us < cutoff_epoch_us:
continue
if entry_cursor and (entry_cursor in sent_cursors or entry_cursor == last_cursor):
continue
if last_timestamp_us > 0 and entry_ts_us < last_timestamp_us:
continue
if entry_cursor:
newest_cursor = entry_cursor
collected_cursors.append(entry_cursor)
if entry_ts_us > newest_timestamp_us:
newest_timestamp_us = entry_ts_us
priority = int(entry.get("PRIORITY", "6"))
if priority > 6:
continue
logs.append(entry)
self.assertEqual(len(logs), 1)
self.assertEqual(logs[0]["__CURSOR"], "c3")
self.assertEqual(newest_cursor, "c4")
def test_mtls_client_certificate_handling(self):
import shutil
test_dir = "test_linux_mtls_certs"
os.makedirs(test_dir, exist_ok=True)
try:
from src import server_enrollment as se
ca_cert, ca_key, ca_pem, _ = se.generate_ca_if_needed(test_dir)
client_cert_pem, client_key_pem = se.issue_client_cert("linux-client-test", ca_cert, ca_key)
with open(os.path.join(test_dir, "ca.crt"), "w") as f:
f.write(ca_pem)
with open(os.path.join(test_dir, "client.crt"), "w") as f:
f.write(client_cert_pem)
with open(os.path.join(test_dir, "client.key"), "w") as f:
f.write(client_key_pem)
# Test missing certs exception
empty_dir = "test_empty_linux_certs"
os.makedirs(empty_dir, exist_ok=True)
with self.assertRaises(FileNotFoundError):
Linux_Client.get_tls_socket("127.0.0.1", 9443, empty_dir)
shutil.rmtree(empty_dir, ignore_errors=True)
finally:
shutil.rmtree(test_dir, ignore_errors=True)
def test_client_certificate_validity_and_proactive_check(self):
import shutil
test_dir = "test_linux_client_validity"
os.makedirs(test_dir, exist_ok=True)
try:
from src import server_enrollment as se
ca_cert, ca_key, _, _ = se.generate_ca_if_needed(test_dir)
client_cert_pem, client_key_pem = se.issue_client_cert("linux-validity-test", ca_cert, ca_key, days_valid=365)
cert_path = os.path.join(test_dir, "client.crt")
with open(cert_path, "w", encoding="utf-8") as f:
f.write(client_cert_pem)
# Newly issued cert (365 days) is not expiring soon at 30 days
self.assertFalse(Linux_Client.is_cert_expiring_soon(cert_path, threshold_days=30))
# Large threshold (500 days) reports expiring soon
self.assertTrue(Linux_Client.is_cert_expiring_soon(cert_path, threshold_days=500))
# Non-existent file reports expiring / missing
self.assertTrue(Linux_Client.is_cert_expiring_soon(os.path.join(test_dir, "missing.crt")))
finally:
shutil.rmtree(test_dir, ignore_errors=True)
if __name__ == "__main__":
unittest.main()
+253
View File
@@ -0,0 +1,253 @@
import os
import sys
import json
import time
import socket
import ssl
import struct
import sqlite3
import urllib.request
import urllib.error
import warnings
from datetime import datetime, timezone, timedelta
# Ensure repository root and src/ directory are in sys.path
ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
SRC_DIR = os.path.join(ROOT_DIR, "src")
sys.path.insert(0, ROOT_DIR)
sys.path.insert(0, SRC_DIR)
warnings.filterwarnings("ignore")
import Win_Client
import Linux_Client
# Test server endpoints
TCP_HOST = "127.0.0.1"
TCP_PORT = 9443
HERMES_HOST = "127.0.0.1"
HERMES_PORT = 8443
def run_tests():
print("=== [1] Verifying server_config.json & client_config.json ===")
server_cfg_path = "server_config.json" if os.path.exists("server_config.json") else os.path.join(ROOT_DIR, "server_config.json")
client_cfg_path = "client_config.json" if os.path.exists("client_config.json") else os.path.join(ROOT_DIR, "client_config.json")
assert os.path.exists(server_cfg_path), f"{server_cfg_path} must exist"
assert os.path.exists(client_cfg_path), f"{client_cfg_path} must exist"
with open(client_cfg_path, "r", encoding="utf-8") as f:
client_conf = json.load(f)
with open(server_cfg_path, "r", encoding="utf-8") as f:
server_conf = json.load(f)
assert "server_name" not in client_conf, "client_config.json must NOT contain server_name"
assert "name" not in client_conf, "client_config.json must NOT contain name"
assert "site_name" not in client_conf, "client_config.json must NOT contain site_name"
assert client_conf["server_fingerprint"] == server_conf["server_fingerprint"], "Fingerprints must match"
print(f"[OK] Verified client_config.json contains no machine/server/site name.")
print(f"[OK] Fingerprint verified: {client_conf['server_fingerprint']}")
cert_dir = os.path.join(ROOT_DIR, "test_pipeline_certs")
os.makedirs(cert_dir, exist_ok=True)
client_id = "test-edge-node.corp.internal"
enrollment_secret = server_conf.get("enrollment_secret") or client_conf.get("enrollment_secret")
print("\n=== [2] Testing Client Dynamic PKI Enrollment API (/api/client/enroll) ===")
enroll_url = f"http://{HERMES_HOST}:{HERMES_PORT}/api/client/enroll"
# 2a. Test rejection on invalid enrollment secret
bad_enroll_payload = {
"client_id": client_id,
"hostname": client_id,
"os": "linux",
"enrollment_secret": "invalid-secret-xyz"
}
req_bad = urllib.request.Request(
enroll_url,
data=json.dumps(bad_enroll_payload).encode("utf-8"),
headers={"Content-Type": "application/json"}
)
try:
with urllib.request.urlopen(req_bad, timeout=5):
assert False, "Expected HTTP 403 on invalid secret"
except urllib.error.HTTPError as e:
assert e.code == 403, f"Expected HTTP 403, got {e.code}"
print("[OK] Invalid enrollment secret rejected with HTTP 403.")
# 2b. Test valid client enrollment
valid_enroll_payload = {
"client_id": client_id,
"hostname": client_id,
"os": "linux",
"enrollment_secret": enrollment_secret
}
req_valid = urllib.request.Request(
enroll_url,
data=json.dumps(valid_enroll_payload).encode("utf-8"),
headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req_valid, timeout=5) as resp:
assert resp.status == 200, f"Expected 200, got {resp.status}"
enroll_data = json.loads(resp.read().decode("utf-8"))
assert "ca_cert" in enroll_data
assert "client_cert" in enroll_data
assert "client_key" in enroll_data
ca_path = os.path.join(cert_dir, "ca.crt")
cert_path = os.path.join(cert_dir, "client.crt")
key_path = os.path.join(cert_dir, "client.key")
with open(ca_path, "w", encoding="utf-8") as f:
f.write(enroll_data["ca_cert"])
with open(cert_path, "w", encoding="utf-8") as f:
f.write(enroll_data["client_cert"])
with open(key_path, "w", encoding="utf-8") as f:
f.write(enroll_data["client_key"])
print(f"[OK] Client enrolled successfully. Certificates stored in {cert_dir}")
# 2c. Verify client shows in /api/clients
clients_req = urllib.request.Request(f"http://{HERMES_HOST}:{HERMES_PORT}/api/clients")
with urllib.request.urlopen(clients_req, timeout=5) as resp:
clients_data = json.loads(resp.read().decode("utf-8"))
assert clients_data["active_seats"] >= 1
found_c = any(c["client_id"] == client_id for c in clients_data["clients"])
assert found_c, f"Client {client_id} should be listed in /api/clients"
print(f"[OK] Verified client in /api/clients: Active Seats: {clients_data['active_seats']}/{clients_data['max_seats']}")
def send_mtls_batch(logs):
ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=ca_path)
ctx.load_cert_chain(certfile=cert_path, keyfile=key_path)
ctx.minimum_version = ssl.TLSVersion.TLSv1_3
ctx.check_hostname = False
raw_sock = socket.create_connection((TCP_HOST, TCP_PORT), timeout=10)
with ctx.wrap_socket(raw_sock, server_hostname=TCP_HOST) as s:
payload = {
"server": client_id,
"timestamp": datetime.now(timezone.utc).isoformat(),
"logs": logs
}
payload_bytes = json.dumps(payload).encode("utf-8")
frame = struct.pack(">I", len(payload_bytes)) + payload_bytes
s.sendall(frame)
resp_len_bytes = s.recv(4)
if not resp_len_bytes:
raise ConnectionError("Server closed connection without response.")
resp_len = struct.unpack(">I", resp_len_bytes)[0]
resp_bytes = bytearray()
while len(resp_bytes) < resp_len:
chunk = s.recv(min(4096, resp_len - len(resp_bytes)))
if not chunk:
break
resp_bytes.extend(chunk)
return json.loads(resp_bytes.decode("utf-8"))
print("\n=== [3] Testing Temporal Persistence & 4-Run Rule for Warnings over mTLS ===")
test_signature = "TestServiceDegraded"
candidate_log = [{
"server": client_id,
"os_type": "linux",
"signature": test_signature,
"severity": "WARNING",
"message": "Resource usage high warning"
}]
for run_num in range(1, 5):
resp = send_mtls_batch(candidate_log)
assert resp.get("status") == "success", f"Run {run_num} failed: {resp}"
promoted = resp.get("promoted_verified", 0)
print(f"[Run {run_num}/4] Ingested successfully via mTLS. Promoted to verified: {promoted}")
if run_num < 4:
assert promoted == 0, f"Expected 0 promoted on run {run_num} for warning, got {promoted}"
else:
assert promoted == 1, f"Expected 1 promoted on run 4 for warning, got {promoted}"
# Inspect SQLite database directly
conn = sqlite3.connect(server_conf.get("db_path", "logar_state.db"))
cursor = conn.cursor()
cursor.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", (test_signature,))
row = cursor.fetchone()
conn.close()
assert row is not None, "Issue not found in SQLite"
run_count, status = row
print(f"[DB Verification] Issue '{test_signature}' -> run_count: {run_count}, status: {status}")
assert run_count >= 4, f"Expected run_count >= 4, got {run_count}"
assert status == "VERIFIED", f"Expected status 'VERIFIED', got {status}"
print("[OK] 4-Run Rule verified: Warning promoted to VERIFIED anomaly on 4th run over mTLS!")
print("\n=== [4] Testing Immediate Pass for Errors over mTLS ===")
error_signature = "TestServiceCrashImmediate"
error_log = [{
"server": client_id,
"os_type": "linux",
"signature": error_signature,
"severity": "ERROR",
"message": "Fatal process crash occurred"
}]
err_resp = send_mtls_batch(error_log)
assert err_resp.get("status") == "success", f"Error run failed: {err_resp}"
print(f"[Run 1/1] Error ingested successfully. Promoted to verified: {err_resp.get('promoted_verified')}")
assert err_resp.get("promoted_verified") == 1, f"Expected error to be promoted to verified immediately, got {err_resp.get('promoted_verified')}"
conn = sqlite3.connect(server_conf.get("db_path", "logar_state.db"))
cursor = conn.cursor()
cursor.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", (error_signature,))
err_row = cursor.fetchone()
conn.close()
assert err_row is not None, "Error issue not found in SQLite"
err_run_count, err_status = err_row
print(f"[DB Verification] Issue '{error_signature}' -> run_count: {err_run_count}, status: {err_status}")
assert err_run_count == 1, f"Expected run_count == 1, got {err_run_count}"
assert err_status == "VERIFIED", f"Expected status 'VERIFIED', got {err_status}"
print("[OK] Immediate pass verified: Error promoted to VERIFIED anomaly immediately!")
print("\n=== [5] Testing Hermes Reporting Endpoint (/api/hermes/report) ===")
req = urllib.request.Request(f"http://{HERMES_HOST}:{HERMES_PORT}/api/hermes/report")
with urllib.request.urlopen(req, timeout=5) as response:
assert response.status == 200, f"Expected 200, got {response.status}"
hermes_data = json.loads(response.read().decode("utf-8"))
print(f"[Hermes API] Returned {len(hermes_data)} verified anomalies:")
found_warning = False
found_error = False
for issue in hermes_data:
print(f" - Fingerprint: {issue['fingerprint']} | Consecutive Runs: {issue['consecutive_runs']} | Status: {issue['status']}")
if issue["signature"] == test_signature:
found_warning = True
assert issue["verified"] is True
assert issue["consecutive_runs"] >= 4
if issue["signature"] == error_signature:
found_error = True
assert issue["verified"] is True
assert issue["consecutive_runs"] == 1
assert found_warning, f"Warning issue {test_signature} should be in Hermes report"
assert found_error, f"Error issue {error_signature} should be in Hermes report"
print("[OK] Hermes reporting validated!")
print("\n=== [6] Testing Windows Client Script Integration ===")
from Win_Client import get_recent_windows_logs
win_logs = get_recent_windows_logs(hours=24)
print(f"[Win_Client] Successfully queried Windows logs: {len(win_logs)} candidate entries.")
print("\n=== [7] Testing Linux Client Script Integration ===")
from Linux_Client import get_recent_linux_logs
linux_logs = get_recent_linux_logs(hours=24)
print(f"[Linux_Client] Successfully queried Linux logs: {len(linux_logs)} candidate entries.")
import shutil
if os.path.exists(cert_dir):
shutil.rmtree(cert_dir, ignore_errors=True)
print("\n=======================================================")
print(" ALL VERIFICATION TESTS (mTLS + PKI + PIPELINE) PASSED! ")
print("=======================================================")
if __name__ == "__main__":
run_tests()
+333
View File
@@ -0,0 +1,333 @@
import os
import sys
import json
import socket
import struct
import sqlite3
import unittest
import urllib.request
import warnings
from datetime import datetime, timezone, timedelta
warnings.filterwarnings("ignore")
# Ensure parent directory and src directory are in path to import Server
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "src")))
import Server
import pgpy
class TestServerComponent(unittest.TestCase):
def setUp(self):
self.test_db = "test_server_state.db"
self.test_config = "test_server_config.json"
if os.path.exists(self.test_db):
os.remove(self.test_db)
if os.path.exists(self.test_config):
os.remove(self.test_config)
def tearDown(self):
if os.path.exists(self.test_db):
try:
os.remove(self.test_db)
except Exception:
pass
if os.path.exists(self.test_config):
try:
os.remove(self.test_config)
except Exception:
pass
def test_first_run_config_and_keypair_generation(self):
config = Server.load_or_init_config(self.test_config)
self.assertTrue(os.path.exists(self.test_config))
self.assertIn("server_fingerprint", config)
self.assertIn("public_key", config)
self.assertIn("private_key", config)
self.assertIn("auth_token", config)
self.assertNotIn("site_name", config)
# Verify keypair
priv_key, _ = pgpy.PGPKey.from_blob(config["private_key"])
pub_key, _ = pgpy.PGPKey.from_blob(config["public_key"])
self.assertEqual(str(pub_key.fingerprint), config["server_fingerprint"])
def test_create_client_config(self):
Server.load_or_init_config(self.test_config)
client_out = "test_client_out.json"
try:
client_conf = Server.create_client_config(
server_host="10.0.0.1",
server_port=9443,
output_path=client_out,
config_path=self.test_config
)
self.assertTrue(os.path.exists(client_out))
self.assertEqual(client_conf["server_host"], "10.0.0.1")
self.assertEqual(client_conf["server_port"], 9443)
# Verify no machine name or site_name is included
self.assertNotIn("server_name", client_conf)
self.assertNotIn("name", client_conf)
self.assertNotIn("site_name", client_conf)
finally:
if os.path.exists(client_out):
os.remove(client_out)
def test_4_run_rule_and_12h_window(self):
Server.init_db(self.test_db)
log_entry = {
"server": "app-worker-01.corp.local",
"signature": "PostgresConnWarning",
"severity": "WARNING",
"message": "Connection to database pool near capacity: 85%",
"os_type": "linux"
}
payload = {
"server": "app-worker-01.corp.local",
"logs": [log_entry]
}
# Runs 1 to 3: WARNING should remain TRANSIENT
for run_idx in range(1, 4):
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
self.assertEqual(res["status"], "success")
self.assertEqual(res["promoted_verified"], 0)
conn = sqlite3.connect(self.test_db)
c = conn.cursor()
c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnWarning",))
row = c.fetchone()
conn.close()
self.assertEqual(row[0], 3)
self.assertEqual(row[1], "TRANSIENT")
# Run 4: promotes WARNING to VERIFIED!
res4 = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
self.assertEqual(res4["promoted_verified"], 1)
conn = sqlite3.connect(self.test_db)
c = conn.cursor()
c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnWarning",))
row = c.fetchone()
conn.close()
self.assertEqual(row[0], 4)
self.assertEqual(row[1], "VERIFIED")
def test_error_immediate_pass(self):
Server.init_db(self.test_db)
log_entry = {
"server": "app-worker-01.corp.local",
"signature": "KernelPanicCritical",
"severity": "ERROR",
"message": "Kernel panic - not syncing: Fatal hardware error",
"os_type": "linux"
}
payload = {
"server": "app-worker-01.corp.local",
"logs": [log_entry]
}
# Run 1: ERROR must immediately promote to VERIFIED
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
self.assertEqual(res["status"], "success")
self.assertEqual(res["promoted_verified"], 1)
conn = sqlite3.connect(self.test_db)
c = conn.cursor()
c.execute("SELECT run_count, status, severity FROM active_issues WHERE signature = ?", ("KernelPanicCritical",))
row = c.fetchone()
conn.close()
self.assertIsNotNone(row)
self.assertEqual(row[0], 1)
self.assertEqual(row[1], "VERIFIED")
self.assertEqual(row[2], "ERROR")
def test_server_severity_filtering(self):
Server.init_db(self.test_db)
payload = {
"server": "app-worker-01.corp.local",
"logs": [
{"server": "app-worker-01", "signature": "SigInfo", "severity": "INFO", "message": "Info msg", "os_type": "linux"},
{"server": "app-worker-01", "signature": "SigWarn", "severity": "WARNING", "message": "Warn msg", "os_type": "linux"},
{"server": "app-worker-01", "signature": "SigErr", "severity": "ERROR", "message": "Err msg", "os_type": "linux"},
{"server": "app-worker-01", "signature": "SigDebug", "severity": "DEBUG", "message": "Debug msg", "os_type": "linux"},
{"server": "app-worker-01", "signature": "SigTrace", "severity": "TRACE", "message": "Trace msg", "os_type": "linux"}
]
}
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
self.assertEqual(res["status"], "success")
conn = sqlite3.connect(self.test_db)
c = conn.cursor()
c.execute("SELECT signature, status FROM active_issues ORDER BY signature")
rows = dict(c.fetchall())
conn.close()
self.assertIn("SigInfo", rows)
self.assertIn("SigWarn", rows)
self.assertIn("SigErr", rows)
self.assertNotIn("SigDebug", rows)
self.assertNotIn("SigTrace", rows)
# SigErr is immediately VERIFIED; SigWarn and SigInfo are TRANSIENT on run 1
self.assertEqual(rows["SigErr"], "VERIFIED")
self.assertEqual(rows["SigWarn"], "TRANSIENT")
self.assertEqual(rows["SigInfo"], "TRANSIENT")
def test_license_schema_and_pki_generation(self):
secret = "test-secret-12345"
Server.init_db(self.test_db, enrollment_secret=secret, max_seats=5)
conn = sqlite3.connect(self.test_db)
c = conn.cursor()
c.execute("SELECT max_seats, enrollment_secret FROM license_config WHERE id = 1")
row = c.fetchone()
conn.close()
self.assertIsNotNone(row)
self.assertEqual(row[0], 5)
self.assertEqual(row[1], secret)
# Test Dynamic PKI
test_cert_dir = "test_certs_pki"
try:
ca_cert, ca_key, ca_pem, ca_key_pem = Server.enrollment.generate_ca_if_needed(cert_dir=test_cert_dir)
self.assertIn("BEGIN CERTIFICATE", ca_pem)
self.assertIn("BEGIN RSA PRIVATE KEY", ca_key_pem)
srv_cert, srv_key, srv_pem, srv_key_pem = Server.enrollment.generate_server_cert_if_needed(
ca_cert, ca_key, hostnames=["127.0.0.1", "localhost"], cert_dir=test_cert_dir
)
self.assertIn("BEGIN CERTIFICATE", srv_pem)
client_cert_pem, client_key_pem = Server.enrollment.issue_client_cert("node-test-1", ca_cert, ca_key)
self.assertIn("BEGIN CERTIFICATE", client_cert_pem)
self.assertIn("BEGIN RSA PRIVATE KEY", client_key_pem)
fp = Server.enrollment.calculate_cert_fingerprint(client_cert_pem)
self.assertEqual(len(fp), 64)
finally:
import shutil
if os.path.exists(test_cert_dir):
shutil.rmtree(test_cert_dir, ignore_errors=True)
def test_enrollment_endpoint_and_seat_quota(self):
from fastapi import HTTPException
secret = "super-secret-enrollment"
max_seats = 2
Server.init_db(self.test_db, enrollment_secret=secret, max_seats=max_seats)
test_cert_dir = "test_certs_enroll"
try:
ca_cert, ca_key, ca_pem, _ = Server.enrollment.generate_ca_if_needed(cert_dir=test_cert_dir)
Server.SERVER_STATE["config"] = {"db_path": self.test_db}
Server.SERVER_STATE["ca_cert"] = ca_cert
Server.SERVER_STATE["ca_key"] = ca_key
Server.SERVER_STATE["ca_cert_pem"] = ca_pem
# 1. Invalid secret should raise 403
bad_req = Server.ClientEnrollRequest(
client_id="client-1",
hostname="host-1",
os="linux",
enrollment_secret="wrong-secret"
)
with self.assertRaises(HTTPException) as cm:
Server.enroll_client(bad_req)
self.assertEqual(cm.exception.status_code, 403)
# 2. Valid enrollment for client 1
req1 = Server.ClientEnrollRequest(
client_id="client-1",
hostname="host-1",
os="linux",
enrollment_secret=secret
)
resp1 = Server.enroll_client(req1)
self.assertIn("client_cert", resp1)
self.assertIn("client_key", resp1)
self.assertEqual(resp1["ca_cert"], ca_pem)
# 3. Valid enrollment for client 2
req2 = Server.ClientEnrollRequest(
client_id="client-2",
hostname="host-2",
os="windows",
enrollment_secret=secret
)
resp2 = Server.enroll_client(req2)
self.assertIn("client_cert", resp2)
# 4. Seat quota exhausted: client 3 should raise 403
req3 = Server.ClientEnrollRequest(
client_id="client-3",
hostname="host-3",
os="linux",
enrollment_secret=secret
)
with self.assertRaises(HTTPException) as cm:
Server.enroll_client(req3)
self.assertEqual(cm.exception.status_code, 403)
self.assertIn("License seat limit reached", cm.exception.detail)
# 5. Re-enrollment for existing client 1 should succeed
resp1_re = Server.enroll_client(req1)
self.assertIn("client_cert", resp1_re)
# 6. Revoked client should be rejected
conn = sqlite3.connect(self.test_db)
conn.execute("UPDATE clients SET status = 'revoked' WHERE client_id = 'client-1'")
conn.commit()
conn.close()
with self.assertRaises(HTTPException) as cm:
Server.enroll_client(req1)
self.assertEqual(cm.exception.status_code, 403)
self.assertIn("revoked", cm.exception.detail)
finally:
import shutil
if os.path.exists(test_cert_dir):
shutil.rmtree(test_cert_dir, ignore_errors=True)
def test_cert_validity_and_hub_pki_renewal(self):
test_cert_dir = "test_certs_renew"
try:
ca_cert, ca_key, ca_pem, _ = Server.enrollment.generate_ca_if_needed(cert_dir=test_cert_dir)
srv_cert, srv_key, srv_pem, _ = Server.enrollment.generate_server_cert_if_needed(
ca_cert, ca_key, hostnames=["127.0.0.1"], cert_dir=test_cert_dir
)
# 1. Freshly generated certificates should NOT be expiring soon with standard 30-day threshold
self.assertFalse(Server.enrollment.is_cert_expiring_soon(ca_pem, threshold_days=30))
self.assertFalse(Server.enrollment.is_cert_expiring_soon(srv_pem, threshold_days=30))
# 2. Huge threshold (e.g. 5000 days) should flag expiration
self.assertTrue(Server.enrollment.is_cert_expiring_soon(srv_pem, threshold_days=5000))
# 3. check_and_renew_hub_pki with standard threshold should report no renewal needed
ca_renewed, srv_renewed = Server.enrollment.check_and_renew_hub_pki(cert_dir=test_cert_dir, threshold_days=30)
self.assertFalse(ca_renewed)
self.assertFalse(srv_renewed)
# 4. In-flight reload of SSLContext
ssl_ctx = Server.init_mtls_server_context(cert_dir=test_cert_dir)
Server.SERVER_STATE["ssl_ctx"] = ssl_ctx
Server.SERVER_STATE["config"] = {"db_path": self.test_db, "cert_dir": test_cert_dir, "tcp_host": "127.0.0.1"}
# Trigger rotation using high threshold
rotated = Server.check_and_rotate_server_certs(cert_dir=test_cert_dir, hostnames=["127.0.0.1"], threshold_days=5000)
self.assertTrue(rotated)
# Check that backup files were generated
bak_files = [f for f in os.listdir(test_cert_dir) if f.endswith(".bak")]
self.assertGreater(len(bak_files), 0)
finally:
import shutil
if os.path.exists(test_cert_dir):
shutil.rmtree(test_cert_dir, ignore_errors=True)
if __name__ == "__main__":
unittest.main()
+235
View File
@@ -0,0 +1,235 @@
import os
import sys
import json
import struct
import unittest
import warnings
warnings.filterwarnings("ignore")
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "src")))
import Win_Client
import pgpy
from pgpy.constants import PubKeyAlgorithm, KeyFlags, HashAlgorithm, SymmetricKeyAlgorithm, CompressionAlgorithm
class TestWinClientComponent(unittest.TestCase):
def setUp(self):
self.dummy_config = "test_win_client_config.json"
# Generate dummy PGP key for testing
key = pgpy.PGPKey.new(PubKeyAlgorithm.RSAEncryptOrSign, 2048)
uid = pgpy.PGPUID.new("TestHub")
key.add_uid(
uid,
usage={KeyFlags.EncryptCommunications, KeyFlags.EncryptStorage},
hashes=[HashAlgorithm.SHA256],
ciphers=[SymmetricKeyAlgorithm.AES256],
compression=[CompressionAlgorithm.Uncompressed]
)
self.server_priv = key
self.server_pub = key.pubkey
self.fingerprint = str(key.pubkey.fingerprint)
with open(self.dummy_config, "w", encoding="utf-8") as f:
json.dump({
"server_host": "127.0.0.1",
"server_port": 9443,
"server_fingerprint": self.fingerprint,
"server_public_key": str(self.server_pub),
"auth_token": "secret-test-token"
}, f)
def tearDown(self):
if os.path.exists(self.dummy_config):
try:
os.remove(self.dummy_config)
except Exception:
pass
def test_client_config_anonymity(self):
config = Win_Client.load_config(self.dummy_config)
self.assertNotIn("server_name", config)
self.assertNotIn("name", config)
self.assertNotIn("site_name", config)
self.assertEqual(config["server_fingerprint"], self.fingerprint)
def test_get_machine_identifier(self):
machine_id = Win_Client.get_machine_identifier()
self.assertIsInstance(machine_id, str)
self.assertGreater(len(machine_id), 0)
self.assertNotEqual(machine_id, "localhost")
def test_encryption_and_envelope_creation(self):
config = Win_Client.load_config(self.dummy_config)
logs = [{
"server": Win_Client.get_machine_identifier(),
"signature": "TestWinSignature",
"severity": "WARNING",
"message": "Disk space threshold warning"
}]
pub_key, _ = pgpy.PGPKey.from_blob(config["server_public_key"])
payload = {
"server": Win_Client.get_machine_identifier(),
"logs": logs
}
msg = pgpy.PGPMessage.new(json.dumps(payload))
enc = pub_key.encrypt(msg)
self.assertTrue(str(enc).startswith("-----BEGIN PGP MESSAGE-----"))
# Decrypt with private key to verify end-to-end payload integrity
dec = self.server_priv.decrypt(enc)
restored = json.loads(dec.message)
self.assertEqual(restored["logs"][0]["signature"], "TestWinSignature")
def test_framing_protocol(self):
envelope_data = json.dumps({"test": "data"}).encode("utf-8")
frame = struct.pack(">I", len(envelope_data)) + envelope_data
self.assertEqual(len(frame), 4 + len(envelope_data))
length = struct.unpack(">I", frame[:4])[0]
self.assertEqual(length, len(envelope_data))
def test_windows_event_filtering_and_severity_map(self):
# sev_map: 1 -> ERROR, 2 -> WARNING, 4 -> INFO
sev_map = {1: "ERROR", 2: "WARNING", 4: "INFO"}
raw_event_types = [1, 2, 4, 8, 16] # 8 is Audit Success, 16 is Audit Failure
filtered = [sev_map[et] for et in raw_event_types if et in sev_map]
self.assertEqual(filtered, ["ERROR", "WARNING", "INFO"])
def test_state_lifecycle(self):
state_path = "test_win_state.json"
try:
# 1. Load non-existent returns empty dict
state = Win_Client.load_state(state_path)
self.assertEqual(state, {})
# 2. Stage new record number and sent IDs
state["new_last_record_number"] = 42
state["new_sent_record_ids"] = ["42:2026-09-04T12:00:00"]
# 3. Commit state moves staged keys to permanent and writes atomically
Win_Client.commit_state(state, state_path)
self.assertNotIn("new_last_record_number", state)
self.assertEqual(state.get("last_record_number"), 42)
self.assertEqual(state.get("sent_record_ids"), ["42:2026-09-04T12:00:00"])
# 4. Reload from disk
reloaded = Win_Client.load_state(state_path)
self.assertEqual(reloaded.get("last_record_number"), 42)
self.assertEqual(reloaded.get("sent_record_ids"), ["42:2026-09-04T12:00:00"])
finally:
if os.path.exists(state_path):
os.remove(state_path)
def test_duplicate_suppression_and_lookback_logic(self):
from datetime import datetime, timezone, timedelta
now = datetime.now()
cutoff_time = now - timedelta(hours=24)
# Mock event object
class MockEvent:
def __init__(self, rec_num, time_gen, event_type, source="TestApp", inserts=None):
self.RecordNumber = rec_num
self.TimeGenerated = time_gen
self.EventType = event_type
self.SourceName = source
self.StringInserts = inserts or ["Test"]
# Events read backwards: newest (rec 103) down to older (rec 99)
mock_events = [
# 1. New error within last 24h
MockEvent(103, now - timedelta(hours=1), 1),
# 2. New warning within last 24h
MockEvent(102, now - timedelta(hours=2), 2),
# 3. Already sent event (rec 101)
MockEvent(101, now - timedelta(hours=3), 4),
# 4. Event at or before last_record_number (rec 100) -> should stop backwards scan
MockEvent(100, now - timedelta(hours=4), 1),
# 5. Old event (> 24h)
MockEvent(99, now - timedelta(hours=26), 1),
]
state = {
"last_record_number": 100,
"sent_record_ids": ["101:" + (now - timedelta(hours=3)).isoformat()]
}
sev_map = {1: "ERROR", 2: "WARNING", 4: "INFO"}
logs = []
last_record_number = int(state.get("last_record_number", 0))
sent_record_ids = set(state.get("sent_record_ids", []))
newest_record_number = 0
for event in mock_events:
rec_num = int(event.RecordNumber)
if newest_record_number == 0:
newest_record_number = rec_num
if event.TimeGenerated < cutoff_time:
break
if last_record_number > 0 and newest_record_number >= last_record_number:
if rec_num <= last_record_number:
break
rec_id = f"{rec_num}:{event.TimeGenerated.isoformat()}"
if rec_id in sent_record_ids:
continue
if event.EventType in sev_map:
logs.append(rec_num)
# Only rec 103 and 102 should be processed (101 is already sent, <= 100 breaks early)
self.assertEqual(logs, [103, 102])
def test_mtls_client_certificate_handling(self):
import shutil
test_dir = "test_win_mtls_certs"
os.makedirs(test_dir, exist_ok=True)
try:
from src import server_enrollment as se
ca_cert, ca_key, ca_pem, _ = se.generate_ca_if_needed(test_dir)
client_cert_pem, client_key_pem = se.issue_client_cert("win-client-test", ca_cert, ca_key)
with open(os.path.join(test_dir, "ca.crt"), "w") as f:
f.write(ca_pem)
with open(os.path.join(test_dir, "client.crt"), "w") as f:
f.write(client_cert_pem)
with open(os.path.join(test_dir, "client.key"), "w") as f:
f.write(client_key_pem)
# Test missing certs exception
empty_dir = "test_empty_certs"
os.makedirs(empty_dir, exist_ok=True)
with self.assertRaises(FileNotFoundError):
Win_Client.get_tls_socket("127.0.0.1", 9443, empty_dir)
shutil.rmtree(empty_dir, ignore_errors=True)
finally:
shutil.rmtree(test_dir, ignore_errors=True)
def test_client_certificate_validity_and_proactive_check(self):
import shutil
test_dir = "test_win_client_validity"
os.makedirs(test_dir, exist_ok=True)
try:
from src import server_enrollment as se
ca_cert, ca_key, _, _ = se.generate_ca_if_needed(test_dir)
client_cert_pem, client_key_pem = se.issue_client_cert("win-validity-test", ca_cert, ca_key, days_valid=365)
cert_path = os.path.join(test_dir, "client.crt")
with open(cert_path, "w", encoding="utf-8") as f:
f.write(client_cert_pem)
# Newly issued cert (365 days) is not expiring soon at 30 days
self.assertFalse(Win_Client.is_cert_expiring_soon(cert_path, threshold_days=30))
# Large threshold (500 days) reports expiring soon
self.assertTrue(Win_Client.is_cert_expiring_soon(cert_path, threshold_days=500))
# Non-existent file reports expiring / missing
self.assertTrue(Win_Client.is_cert_expiring_soon(os.path.join(test_dir, "missing.crt")))
finally:
shutil.rmtree(test_dir, ignore_errors=True)
if __name__ == "__main__":
unittest.main()
-135
View File
@@ -1,135 +0,0 @@
import os
import sys
import json
import argparse
import urllib.request
import urllib.parse
import mimetypes
import package_dist
DEFAULT_GITEA_URL = os.environ.get("GITEA_SERVER_URL", "https://gitea.eibl.tech")
DEFAULT_REPO = os.environ.get("GITEA_REPOSITORY", "me0nline/LOGAR")
def get_existing_assets(base_url, repo, release_id, token):
url = f"{base_url}/api/v1/repos/{repo}/releases/{release_id}/assets"
req = urllib.request.Request(url, headers={"Authorization": f"token {token}", "Accept": "application/json"})
try:
with urllib.request.urlopen(req) as resp:
return json.loads(resp.read().decode("utf-8"))
except Exception:
return []
def delete_asset(base_url, repo, release_id, asset_id, token):
url = f"{base_url}/api/v1/repos/{repo}/releases/{release_id}/assets/{asset_id}"
req = urllib.request.Request(url, method="DELETE", headers={"Authorization": f"token {token}"})
try:
with urllib.request.urlopen(req) as resp:
pass
except Exception:
pass
def upload_file_to_release(base_url, repo, release_id, token, file_path):
filename = os.path.basename(file_path)
# Clean up existing asset with same name if already present
existing_assets = get_existing_assets(base_url, repo, release_id, token)
for asset in existing_assets:
if asset.get("name") == filename:
print(f"[*] Removing existing asset '{filename}' (ID: {asset['id']})...")
delete_asset(base_url, repo, release_id, asset["id"], token)
url = f"{base_url}/api/v1/repos/{repo}/releases/{release_id}/assets?name={urllib.parse.quote(filename)}"
with open(file_path, "rb") as f:
file_bytes = f.read()
req = urllib.request.Request(url, data=file_bytes, method="POST")
req.add_header("Authorization", f"token {token}")
req.add_header("Content-Type", "application/octet-stream")
req.add_header("Accept", "application/json")
try:
with urllib.request.urlopen(req) as resp:
data = json.loads(resp.read().decode("utf-8"))
print(f"[+] Attached {filename} ({len(file_bytes)} bytes) to release.")
return data
except urllib.error.HTTPError as e:
err = e.read().decode("utf-8", errors="ignore")
print(f"[!] Error uploading {filename}: HTTP {e.code} - {err}")
return None
def create_or_get_release(base_url, repo, tag, token, title=None, notes=None):
url = f"{base_url}/api/v1/repos/{repo}/releases"
headers = {
"Authorization": f"token {token}",
"Content-Type": "application/json",
"Accept": "application/json"
}
# Check if release exists
check_url = f"{base_url}/api/v1/repos/{repo}/releases/tags/{urllib.parse.quote(tag)}"
check_req = urllib.request.Request(check_url, headers={"Authorization": f"token {token}"})
try:
with urllib.request.urlopen(check_req) as resp:
existing = json.loads(resp.read().decode("utf-8"))
print(f"[*] Found existing release for tag {tag} (ID: {existing['id']})")
return existing["id"]
except urllib.error.HTTPError:
pass
# Create new release
payload = {
"tag_name": tag,
"name": title or f"LOGAR Release {tag}",
"body": notes or f"Automated binary release for {tag}.",
"draft": False,
"prerelease": False
}
req = urllib.request.Request(url, data=json.dumps(payload).encode("utf-8"), headers=headers, method="POST")
with urllib.request.urlopen(req) as resp:
created = json.loads(resp.read().decode("utf-8"))
print(f"[+] Created release {tag} (ID: {created['id']})")
return created["id"]
def main():
parser = argparse.ArgumentParser(description="Upload LOGAR compiled binaries directly to Gitea Release")
parser.add_argument("--tag", default=os.environ.get("GITEA_REF_NAME"), help="Release tag name (e.g. v1.0.0)")
parser.add_argument("--token", default=os.environ.get("GITEA_TOKEN"), help="Gitea Personal Access Token (or set GITEA_TOKEN env var)")
parser.add_argument("--url", default=DEFAULT_GITEA_URL, help="Base Gitea instance URL")
parser.add_argument("--repo", default=DEFAULT_REPO, help="Repository owner/name")
parser.add_argument("--skip-build", action="store_true", help="Skip running package_dist.py before upload")
args = parser.parse_args()
tag = args.tag
if not tag:
tag = input("Enter tag name (e.g. v1.0.0): ").strip()
token = args.token
if not token:
token = input("Enter Gitea Token: ").strip()
if not tag or not token:
print("[!] Tag and Token are required.")
sys.exit(1)
if not args.skip_build:
print("[*] Assembling compiled binaries...")
package_dist.main()
dist_dir = os.path.abspath("dist")
if not os.path.exists(dist_dir) or not os.listdir(dist_dir):
print("[!] No binaries found in dist/. Run package_dist.py first.")
sys.exit(1)
print(f"[*] Connecting to Gitea: {args.url} (repo: {args.repo})...")
release_id = create_or_get_release(args.url, args.repo, tag, token)
print(f"[*] Uploading binary assets from '{dist_dir}'...")
for f in sorted(os.listdir(dist_dir)):
fpath = os.path.join(dist_dir, f)
if os.path.isfile(fpath):
upload_file_to_release(args.url, args.repo, release_id, token, fpath)
print(f"\n[+] Release successfully published with binary assets: {args.url}/{args.repo}/releases/tag/{tag}")
if __name__ == "__main__":
main()