Document Pipeline Orchestrator

What the measurements mean

Only what is unusual for this project. Every sentence names the field it came from; nothing here is asserted that cannot be checked against the data.

The 55 commits below come from Unified_Orchestrator/, not from this folder; the repository sits one directory down.see the evidencefrom git.found_in, git.commits
It serves 17 HTTP endpoints, so it runs as a service rather than a script.see the evidencefrom assets.routes, assets.ports
The code defines 478 functions, 592 methods and 126 classes.see the evidencefrom pythonAst.counts, pythonAst.coverage
It grades B, weakest on runnability at 28.6%. runnability coveragesee the evidencefrom grades

How it works

Two desktop tools doing neighbouring jobs meant two installations, two update paths and two sets of habits. Bringing them together in one web interface removed all of that. The larger gain is operational. Each pipeline starts as a tracked run with live progress, so a long document job can be watched, cancelled and inspected instead of waiting behind a frozen window. Statistics are recorded for each stage (CSV, PDF and cross-check counts), so a bad run can be diagnosed from its record rather than by running it again. The compression and clustering stages are exposed as steps that can be configured.

What kind of engineering this is

Six signals counted from the source: files that run models locally, that act without being asked each time, that check their own output, that survive failure, that touch parsers and protocols, and that were measured then improved. Every axis carries its file count.

What it is written in

Share of code bytes. Vendored bundles and generated output are set aside before this is computed, which is why it differs from a simple file count.

What stands out

Replaced two legacy desktop applications with one browser interface
Pipelines run as tracked jobs with live progress and can be cancelled
Statistics are recorded for each stage, so a bad run can be diagnosed afterwards
Compression and clustering are exposed as configurable steps

Capabilities

Unified LIA Process Runner
Browse...
Document Processing Toolkit
Clustering & Conversion
Line Removal
Process and Cluster
Convert PDFs to PNG Only
Load PNG Image...
Save Processed Image...
Error: Could not load image.

Grade

Every check names the field it measured. Checks that cannot apply are excluded, never counted as failures.

82.4% over 18 of 18 checks100% coverage

provenance100.0%
documentation100.0%
runnability28.6%
maintenance100.0%
integrity90.0%

Not met

entry_point none
atlas entry --list
environment none
python -m venv .venv (in the project folder

Architecture

Directories carrying real modules, largest first.

Unified_Orchestrator71 modules

HTTP surface

17 endpoints in total.

GET /drives
POST /open
POST /spawn
GET /{pipeline_id}
POST /{pipeline_id}/run
POST /{pipeline_id}/resume
POST /stop/{run_id}
GET /active

Data model

Typed structures and tables found in the source.

CsvStats
PdfStats
CrossCheckStats
runs
CompressOptions
FileResult
OpenFolderRequest
SpawnRequest
RunRequest
ResumeRequest
SettingsUpdate
TemplateBody

Vocabulary

Parsed 74 of 74 Python file(s). Counts are exact; the names below are a sample.

478functions

592methods

126classes

59documented modules

Classes

PipeReaderUnified_Orchestrator/LIA/main_lia.py:41__init__, run, stop
OrchestratorWorkerUnified_Orchestrator/LIA/main_lia.py:68__init__, stop, _run_command, _find_newest_folder, _get_db_counts, _audit_input_filenames,
MainWindowUnified_Orchestrator/LIA/main_lia.py:616__init__, load_settings, save_settings, browse_csv_folder, browse_header_excel, browse_db_
CollapsibleGroupBoxUnified_Orchestrator/Membership-Commuting/main.py:71__init__, content_layout
CsvStatsUnified_Orchestrator/Membership-Commuting/main.py:108
PdfStatsUnified_Orchestrator/Membership-Commuting/main.py:119
CrossCheckStatsUnified_Orchestrator/Membership-Commuting/main.py:130
PipeReaderUnified_Orchestrator/Membership-Commuting/main.py:460__init__, run, stop
OrchestratorWorkerUnified_Orchestrator/Membership-Commuting/main.py:490__init__, stop, _should_run, _run_command, _find_newest_folder, _find_newest_file, _dir_ha
MainWindowUnified_Orchestrator/Membership-Commuting/main.py:1468__init__, on_runmode_changed, _extract_templates_from_json, refresh_templates, _derive_typ
tkUnified_Orchestrator/LIA/DuckDB/create_database.py:52
MainAppUnified_Orchestrator/LIA/DuckDB/create_database.py:220__init__, create_widgets, browse_header_excel, browse_csv_files, browse_db_file, run_full_
MainAppUnified_Orchestrator/LIA/DuckDB/main_db_creator.py:272__init__, create_widgets, browse_header_excel, browse_csv_files, browse_db_file, run_full_
LiaRunnerUIUnified_Orchestrator/LIA/HTML_calc_LIA/gui_run_lia.py:67__init__, browse_input, browse_output_dir, on_run, _drain_log_queue, _disable_controls, _e
RunContextUnified_Orchestrator/unified_orchestrator/core/step.py:17log, get_step_output, to_dict, from_dict
StepResultUnified_Orchestrator/unified_orchestrator/core/step.py:86ok, fail
PipelineStepUnified_Orchestrator/unified_orchestrator/core/step.py:107validate, execute, get_required_inputs, get_required_settings, get_produced_outputs, execu
PipelineEventUnified_Orchestrator/unified_orchestrator/core/events.py:22to_sse
EventBusUnified_Orchestrator/unified_orchestrator/core/events.py:36__init__, set_loop, _get_queues, _broadcast, emit_log, emit_progress, emit_step_status, em
DataFixerAppUnified_Orchestrator/LIA/VisionLIA/JSON_allowance_add_2.py:727__init__, select_files, select_output, select_db, check_run, log_msg, poll_log, start_proc
RateLimitManagerUnified_Orchestrator/LIA/VisionLIA/OpenAI_LIA_tool_V03b.py:461__init__, wait_if_needed, handle_error, handle_success
SafeDictUnified_Orchestrator/LIA/VisionLIA/OpenAI_LIA_tool_V03b.py:576__missing__
TwoStageWorkerUnified_Orchestrator/LIA/VisionLIA/OpenAI_LIA_tool_V03b.py:829__init__, stop, _safe_emit, _api_call_with_retry_mm, _api_call_stageN, _score_prep_candida
ProcessorTabUnified_Orchestrator/LIA/VisionLIA/OpenAI_LIA_tool_V03b.py:1791__init__, _build, _update_file_info, _add_files, _add_folder, _clear_files, _load_stage1_t

Functions

setup_file_loggerUnified_Orchestrator/Membership-Commuting/main.py:35
_safe_list_csv_filesUnified_Orchestrator/Membership-Commuting/main.py:138
parse_csv_emplidsUnified_Orchestrator/Membership-Commuting/main.py:148
parse_pdf_folderUnified_Orchestrator/Membership-Commuting/main.py:263
compute_cross_checkUnified_Orchestrator/Membership-Commuting/main.py:311
format_csv_statsUnified_Orchestrator/Membership-Commuting/main.py:322
format_pdf_statsUnified_Orchestrator/Membership-Commuting/main.py:331
format_cross_checkUnified_Orchestrator/Membership-Commuting/main.py:339
_today_yyyy_mm_ddUnified_Orchestrator/Membership-Commuting/main.py:349
make_unique_run_rootUnified_Orchestrator/Membership-Commuting/main.py:353
ensure_empty_output_runUnified_Orchestrator/Membership-Commuting/main.py:376
safe_move_filesUnified_Orchestrator/Membership-Commuting/main.py:383
safe_copy_fileUnified_Orchestrator/Membership-Commuting/main.py:427
is_zero_likeUnified_Orchestrator/Membership-Commuting/main.py:840
note_defaultUnified_Orchestrator/Membership-Commuting/main.py:1819
_heartbeat_loopUnified_Orchestrator/unified_orchestrator/server.py:84
startupUnified_Orchestrator/unified_orchestrator/server.py:105
shutdownUnified_Orchestrator/unified_orchestrator/server.py:116
_is_port_in_useUnified_Orchestrator/unified_orchestrator/server.py:129
find_available_portUnified_Orchestrator/unified_orchestrator/server.py:139
_open_browserUnified_Orchestrator/unified_orchestrator/server.py:164
_openUnified_Orchestrator/unified_orchestrator/server.py:171
mainUnified_Orchestrator/unified_orchestrator/server.py:183
read_file_with_fallback_encodingUnified_Orchestrator/LIA/DuckDB/create_database.py:83

Command line

--csv-files
--header-excel
--db-name
--db-path
--run-pks
--run-indexes
--run-maint
--run-integrity
--input-file
--output-file

Depth

Signals counted from the code, not asserted.

55 commits across 7 separate working days over 1.1 months, 27 of them revising existing behaviour

Built from scratch

Parser / lexer
Concurrency primitives

Proves itself

Benchmarks
Reconciliation / audits
Golden / baseline files
Smoke tests

Survives failure

Logging / observability
Fallback paths
Timeouts
Recovery / restore
Retry / backoff
Rate limiting

Systems

Distributed systems

Worker orchestration
Health / heartbeat

Working within limits

Cost / token accounting
Timeout budgets
Context-window budgeting
Throttling / rate limits

Environment engineering

Dependency isolation
Virtual environments

GPU & memory management

Quantisation

Languages

Python31,670
JavaScript2,247
Batchfile300
Shell58

Built with

FastAPI
Uvicorn
PyQt
Tkinter
pandas
NumPy
Pillow
Pydantic
pip

Recurring themes in the commit history

lia
template
settings
salary
query
effdt
type
error
vision
browser

Evidence

Taken from the code itself: a module docstring or the opening of the README.

FastAPI application entry point. Run with: uvicorn unified_orchestrator.server:app --reload --port 8000 # or python -m unified_orchestrator.server

At a glance

Status: paused
Primary language: Python
First commit: 2026-03-09
Last activity: 2026-04-10
Domain: Desktop applications
Agent docs: CLAUDE.md, AGENTS.md