A backend platform for managing ESG (Environmental, Social & Governance) data. Businesses use it to enter, calculate, approve and report sustainability indicators, such as emissions, occupational safety and training, per operating facility and reporting period.
Excel Technologies · Completed June 2026
Project scale · counted from the repository, 2026-10-13
Commits
475
Lines of TypeScript source
~25,600
233 files in src, excluding specs
Route handlers
127
15 controllers
Entities
26
Migrations
22
Test blocks
233
15 spec files; static count, not run
Lines in the formula engine
~2,000
Excluding specs; 6 spec files in the same folder
// problem
Problem and goals
ESG data lived in scattered Excel files that were hard to control and trace. The platform brings it into one system with calculation, approval and reporting built in.
01
One central system, scoped by facility and period
Entries are held centrally and scoped by operating facility and reporting period, instead of in separate spreadsheets.
02
Formulas computed, not typed
Many indicators depend on others (emissions = activity × factor). A formula engine computes them, so users enter only base indicators.
03
A controlled approval workflow
A completed form goes to a manager for approval; approved data can be rejected or reopened for editing, with a recorded history.
04
Legacy data brought in
Older data sat in Excel. An import pipeline loads it, with skip reporting and clean-up of interrupted runs.
05
Excel in, Excel out
Master data and entries are imported from templates, and approved data is exported as a multi-sheet report.
// contribution
My contribution
Taken from my CV. The rest of this page describes the system, not who built each part.
>Designed scalable relational database schemas for complex ESG reporting and data-entry workflows.
>Developed RBAC-based modules: employee management, role and permission management, KPI, coefficient and target configuration, and operational unit management.
>Implemented backend APIs and business logic for hierarchical data management and reporting.
>Built Excel-based data import and migration features for bulk ESG data onboarding and operational processing.
>Developed notification and workflow backend services to support approval pipelines and operational coordination.
>Built APIs and backend logic for ESG reporting, aggregated data views, detailed entry tracking and enterprise data management workflows.
// architecture
System architecture
Request path
Controllers
HTTP entry point; a global validation pipe uses whitelisting.
Use cases
Application logic, kept out of the controllers.
Repository interfaces
Contracts the use cases depend on, injected through tokens.
Repository implementations
TypeORM implementations of those contracts.
Entities / PostgreSQL
Migrations, soft delete and audit columns.
Supporting services
Identity service
Separate service that issues tokens and resolves roles and facilities.
Formula engine
Lexer, parser, resolver and evaluator for the indicator DSL.
Excel engine
Template import, legacy import and a worker-thread report builder.
Notification scheduler
In-process cron, email and in-app notifications.
Redis
Soft editing sessions and job state for legacy imports.
File service
Separate service for attachment storage, called over HTTP.
Observability
Prometheus metrics, Grafana and Loki configuration.
Domain modules 12 Nest modules and 15 controllers in the main service; authentication, users and roles live in a separate identity service.
Simplified from the source. It shows layers and supporting capabilities, not a deployment topology.
// deep-dives
Engineering deep dives
Domain model and organizational data structure
How entries, indicators and operating facilities relate.
Documented design
Operating facilities (the operational unit) belong to a business unit, an industry sector and an activity type through required foreign keys. The organizational structure is flat: there is no parent column, recursive query or tree encoding.
A data entry belongs to one facility and one reporting period, and has a status. Its values are one row per entry and indicator, with a unique index on that pair, a unit and a JSON snapshot of the indicator at entry time.
The indicator catalogue is organized as pillar, topic, indicator group and indicator. Indicators carry input and standard units, a data cycle and an input type (manual or formula), and link to activity types.
Targets are indicator, unit, value and an effective date. Emission coefficients are versioned: a metric value belongs to an emission version, with partial unique indexes that ignore soft-deleted rows.
Every entity shares a base with soft delete and creator, updater and deleter audit columns. A database check constraint caps attachment size.
Schema synchronization is switched off in the data-source configuration, and changes ship as 22 migrations: an initial schema, mostly hand-written SQL migrations, and index changes.
The pillar-to-indicator grid shown to users is assembled in the use case as display headers; it is not a database hierarchy.
Design rationale
Storing an indicator snapshot with each value keeps an entry readable as it was entered, even after catalogue changes.
How an entry is anchored
Business unit, sector, activity type
Required references
Operating facility
The operational unit
Data entry
One facility, one period, one status
Entry values
One row per indicator; unique pair
RBAC and operational-unit access
Who can do what, and which facilities they can reach.
Documented design
A separate identity service issues tokens and holds users, roles, actions and resources. A permission key is a resource and an action, and a user's roles are resolved from the database, with a short-lived cache and explicit invalidation.
The main service does not verify tokens itself: it asks the identity service to validate each token.
A permission guard accepts a list of keys with OR semantics, or nested lists for AND. Permissions cover areas such as data entry, indicators, targets, coefficients, staff, settings and approval.
Facilities are assigned to a user as a list of ids. Every permission a user's roles grant applies to all of that user's facilities; there is no per-role facility difference.
Row-level scoping is applied to the data-entry list (a facility filter in the query, and an empty list returns nothing) and checked per record in create, update, detail, history, duplicate, delete, recalculate and the approval paths.
Facility dropdowns are limited to facilities where the user holds the view permission. Approval permissions are checked against the entry's facility.
Design rationale
Checking facility membership inside the use case, as well as the route permission, protects a record even when the route guard only knows about global permissions.
From token to permitted record
Token
Validated by the identity service
Permission guard
Resource and action; OR / AND lists
Use case check
Is the facility one of the user's?
Query filter
Facility ids in the list query
Excel import and migration pipeline
Three import paths with different guarantees.
Documented design
Master-data import (indicators, topics, targets, groups, facilities) reads the first sheet with SheetJS, maps columns by position or header, checks required fields and duplicates within the file, and resolves codes with batched lookups. Valid rows are written in one transaction, so the batch is all-or-nothing. The indicator import stops at the first validation error.
Entry import reads a fixed template: a facility code and period at the top, then indicator values and extra values. It collects every error across the sections and returns them together, each with a section, row, column, field and message.
The entry import writes nothing: it returns a preview, and the user saves through the normal workflow. It rejects a target entry that is not a draft.
Legacy history import is limited to admins, runs as a background job with a job id that is polled, and keeps job state in Redis with an expiry. Rows are skipped, with a count and capped samples per reason, for unknown facilities or indicators, formula indicators, data-cycle mismatches and non-numeric values.
Within a file, a repeated key keeps the last row, except for indicators that record an execution date, whose values are summed. An entry that already exists for a facility and period is skipped.
Each legacy entry is written in steps (create, bulk upsert of values, formula values, status update). A bulk upsert is one transaction, but the steps are not wrapped in one. A later run finds entries left incomplete by an interrupted run, soft-deletes them and reports them.
A command-line variant runs as a dry run by default. Supporting scripts verify imported values against the source and restore values from it. A value-scale fix is opt-in.
Prometheus counters record import duration, rows and failed rows.
Design rationale
Skip-existing plus clean-up of incomplete entries makes a rerun safe without a surrounding transaction around a long import.
Entry import path
Template upload
Fixed layout, first sheet
Validation
All errors, with section, row and column
Preview
Nothing is written yet
Normal save
Through the workflow
Legacy import, per entry
Create entry
As a draft
Bulk upsert values
One transaction
Compute formulas
Set status
Without notifications
Approval workflow and notifications
A single approval level, with edit requests and an audit trail.
Documented design
A data entry moves through seven states: DRAFT, PENDING_APPROVAL, APPROVED, MANAGER_REJECTED, ADMIN_REJECTED, EDIT_REQUESTED and EDIT_APPROVED. There is no separate publish state.
Approval is single-level: a user with the manager approval permission approves or rejects a pending entry, and one with the admin approval permission can reject an approved entry. A rejection needs a reason and the affected indicators.
A complete form moves from draft to pending approval automatically. An approved entry can have an edit requested, which an approver accepts, optionally narrowing the indicators, or rejects with a reason.
A rejected or edit-approved entry is corrected by duplicating it into a new draft, where editing is limited to the affected indicators.
Every transition writes a history record with an order, operator, operation, reason and status, and rejections and edit requests have their own records.
Notifications cover creation, reminders, overdue warnings, pending approval, rejections and edit-request outcomes. They go in-app through an external queue service and by email through SMTP or a mail queue.
Recipients are resolved by asking the identity service who holds a permission for a facility. A scheduled job reads per-setting day and hour values and notifies facilities with no entry or only a draft.
The recipient lookup returns an empty list if the identity service call fails, and notification dispatch runs after the database write and only logs failures.
Design rationale
Duplicating a rejected entry keeps the rejected version intact for the record, while correction happens in a fresh draft.
Core status path
DRAFT
Until the form is complete
PENDING_APPROVAL
Manager approves or rejects
APPROVED
Final state
Reporting and aggregation
A detail report built off the main thread.
Documented design
The report export loads approved entries for a period range, filters them by the user's export permission per facility, loads their values with a lean query, and groups indicators by data table into sheets.
Each sheet has one row per facility and period, with a column per indicator. The workbook is built by a worker thread so the main event loop stays free.
The report is a detail pivot, not a hierarchical roll-up: no SQL aggregation by business unit or sector was found. Year-to-date accumulation exists only inside the formula engine.
Other exports (indicators, targets and master data) build workbooks in memory with SheetJS. A Prometheus histogram records report export duration.
Report results are not cached.
Design rationale
A worker thread isolates CPU-heavy workbook building from request handling, which matters when a single export can reach hundreds of thousands of cells.
Report export
Approved entries
For the period range
Permission filter
Export permission per facility
Lean values query
Worker thread
Builds the sheets
A custom formula engine, without eval()
Interdependent sustainability indicators evaluated in dependency order, by an expression language the system controls.
Documented design
About 2,000 lines outside specs, in a lexer, parser, resolver and evaluator, with no use of eval() or the Function constructor in the source.
A domain-specific language: HS[] looks up emission factors, CT[] references an indicator of the same or an earlier period (with offsets), LK[] accumulates from the start of the year, and IF[] supports conditions. NCT, SUM_TNBC, DAY_TNBC and DAY_IN_MONTH are also documented.
Formulas run in dependency waves (buildFormulaWaves), so a formula that reads another formula runs after its inputs.
Lookups for referenced indicators are preloaded and cached per facility, which removed a per-formula query pattern.
6 spec files in the engine folder, plus specs for wave building and batch evaluation, cover the engine, resolver and caches.
Design rationale
Ordering evaluation in waves lets a formula safely read the results of other formulas.
Formula pipeline
Lexer
Expression processing
Parser
Expression processing
Resolver
Dependency resolution; preloaded lookups
Evaluator
Calculation, in dependency waves
DSL functionsAs documented in the repository.
Emission-factor lookup
HS[]
Indicator of the same or an earlier period
CT[]
Cumulative from the start of the year
LK[]
Conditions
IF[]
Days in a month
DAY_IN_MONTH
// performance
Measured performance
Figures are measurements recorded in the project's commit messages, taken with Grafana, Loki and EXPLAIN ANALYZE before and after each fix: measure, find the root cause, fix, measure again. The commit messages do not state the environment for every figure, so none of these figures should be read as production performance, and they were not re-measured for this page.
Indicator queries and the list API
Measured, as recorded in commits
Problem
The query that filters indicators is the hot path of every calculation, and the 9k+ row table was being full-scanned.
What changed
Add an index.
Before → after
Indicator filter query (~1,700×)
2,090 ms → 1.2 ms
GET /indicators, paginated
9.4 s → 0.2 ms
Recalculating a data entry
Measured, as recorded in commits
Problem
One recalculation issued 821 queries per request, and requests waited up to 28 s for a connection from the pool.
What changed
Batch the lookups. preloadCT() shares a cache per facility and prevents a race condition.
Remove the N+1 pattern elsewhere too: preloadLK and batched validation of activityType.
Cache indicators for the duration of an import run, and guard against connection pool exhaustion (pool of 20).
Before the fixAfter the fix the source records the work as batched, without a figure.
Queries per request
821
Wait for a pooled connection
up to 28 s
The report-export values query
Measured, as recorded in commits
Problem
Reading the values for a report-export took about 9.3 s because a JOIN and an ORDER BY forced an external-merge sort over roughly 166k rows.
What changed
Remove the JOIN and the ORDER BY that caused the sort.
Before → after
Values query (~40×)
~9.3 s → ~235 ms
Report export under concurrency
Measured, as recorded in commits
Problem
Exports timed out against the 30 s statement_timeout. The real cause was synchronous CPU work starving the event loop, so even unrelated database queries timed out.
What changed
First a setImmediate yield as a stopgap, then the full fix: build the Excel file (hundreds of thousands of cells) in a worker thread.
Before → after
Per request with 4 exports at once (before; timed out)
~68 s
Per request when run alone (after)
~16 s
Main thread stays responsiveIn one measurement of the worker-thread build.
Excel build time
~1.6 s
Of a 20 ms setInterval ran meanwhile
77 ticks
Loading a data entry
Measured, as recorded in commits
Problem
Opening one data entry spent almost all its time in a single query.
What changed
Fix a non-sargable filter and avoid a fan-out on activityTypes.
Before the fixThe source records no figure after the fix.
One query, out of a 16,056 ms request
14,984 ms
// trade-offs
Engineering trade-offs
A separate identity service
Why it matters
Users, roles and facilities have one source of truth that several services can share.
What it costs
Every token validation is a network call, so the main service caches the answer.
A flat facility structure
Why it matters
Access scope is a plain set of facility ids, which keeps filters and permission checks simple.
What it costs
There is no multi-level organization tree for roll-ups or inherited access.
Entry import returns a preview instead of writing
Why it matters
Users review validated data and save through the normal workflow, so imported entries follow the same rules as typed ones.
What it costs
An extra step for every import.
Legacy import without one wrapping transaction
Why it matters
A long multi-year import can run per entry and in parallel, and a rerun skips what exists and cleans up what was interrupted.
What it costs
A failure can leave an incomplete entry until the next run, and the HTTP endpoint has no dry run.
Single-level approval with edit requests
Why it matters
A short, understandable path, with an audit record for each transition.
What it costs
No approval chain, and corrections go through duplication.
Best-effort notifications after the write
Why it matters
A notification problem never rolls back a decision.
What it costs
A failed send is lost: no outbox and almost no retry.
A custom formula engine instead of eval()
Why it matters
Formulas run in a language the system controls, not as arbitrary code.
What it costs
Code to build and maintain, and a DSL that users and developers have to learn.
A worker thread for the Excel build
Why it matters
The main event loop stays free, so unrelated requests and queries are not starved.
What it costs
More moving parts than building the file inline.
// lessons
Lessons learned
01
Measure before you optimize
Each fix recorded in the commits started from Grafana, Loki or EXPLAIN ANALYZE and ended with a second measurement.
02
Look for the root cause outside the slow query
Synchronous CPU work starved the event loop, so unrelated queries timed out; the cure was moving work off the main thread, not tuning those queries.
03
Make bulk data operations rerunnable and checkable
Skip-existing, clean-up of interrupted entries, skip reports by reason and verification scripts make a large legacy import safer to repeat.
04
Enforce scope in more than one place
Route permissions know actions, not records; the facility check in the use case is what protects a single entry, so it has to be applied consistently and covered by tests.
05
Keep formulas out of eval()
A purpose-built DSL and evaluator keeps calculation logic inspectable and testable.
// limitations
Known limitations
!Organizational structure is flat; the source shows no multi-level tree and no roll-up reporting by business unit or sector.
!Values and targets use floating-point columns, and the source shows no database uniqueness rule for a facility and period pair.
!Facility scoping covers data entries; master data is global.
!Approval is single-level; the history record and the status update are separate writes.
!Notifications are fire-and-forget with no outbox or retry beyond one SMTP reconnect, and the scheduler has no distributed lock.
!Excel uploads are parsed in memory, and only the first sheet is read.
!The legacy import is not atomic, and its HTTP endpoint has no dry run.
!Automated tests focus on the formula engine, emissions, the scheduler and export; no tests cover authorization, the approval transitions, the shared Excel parser or the legacy pipeline.