ML Energy Forecasting

At Evonik Industries, energy was one of the largest and most controllable operating costs — but the organization was reacting to consumption after the fact rather than anticipating it. Without a reliable forecast, there was no way to schedule, procure, or optimize around demand.

The data itself was the first obstacle. Consumption signals were spread across redundant, overlapping sources with heavy manual overhead, which made any modeling effort slow and brittle before it even began.

My mandate as Senior Data Engineer was to own the full lifecycle: design the backend and pipelines that fed the data-science segment, then build a forecasting model accurate enough that the business could act on its predictions with confidence.

Procurement and plant operations, who turned forecasts into purchasing and scheduling decisions worth real money.

The data-science segment that built on the consolidated pipelines and backend architecture.

Junior data engineers I trained on the system — it had to be teachable, not just functional.

Site leadership accountable for energy cost as a controllable operating expense.

Fragmented, redundant source data with heavy manual overhead — the foundation had to be fixed before modeling was viable.

The forecast had to be trusted enough to spend against: interpretability was a requirement, not a preference.

Full-lifecycle ownership by a small team — the design had to be maintainable and hand-off-able.

Industrial energy data stayed in-house under company access controls.

1. Redundant, Fragmented Data: Energy signals lived in overlapping sources with duplicated records and manual hand-offs. Modeling was hopeless until the data foundation was consolidated.

2. Trend and Seasonality: Energy consumption carries strong trend and multi-scale seasonality (daily, weekly, seasonal). A forecast that ignored these would be systematically wrong at exactly the moments that mattered.

3. Actionable Accuracy: The bar was not a leaderboard metric — it was "accurate enough that operations will change decisions based on it." That demanded honest evaluation and calibrated expectations.

4. A Backend the DS Team Could Use: The model could not be a one-off script. It needed backend architecture and pipelines that data scientists and analysts could build on repeatedly.

5. Maintainability: As the person who also trained junior engineers, I needed the solution to be interpretable and teachable, not an opaque black box.

Foundation First, Then the Model: The win came from fixing the data foundation before modeling, then serving forecasts through backend architecture the whole segment could rely on.

**1. Data Consolidation Layer**

• Integrated redundant, overlapping sources into clean, canonical pipelines.

• Eliminated ~80% of data redundancy and cut project overhead ~50%, making every downstream effort faster.

**2. Feature & Backend Architecture**

• Designed the backend system architecture that data scientists and analysts built on.

• Engineered trend and seasonality features that made consumption predictable.

**3. Forecasting Model**

• A time-series model capturing level, trend, and seasonality to project energy consumption.

• Chosen for interpretability so the business could trust — and act on — its forecasts.

**4. Delivery**

• Forecasts served through the pipeline lifecycle rather than a throwaway script, so predictions reached the people making procurement and scheduling decisions.

**Decision 1: Fix the Data vs. Model Around the Mess**

✅ *Chose*: Consolidate the data foundation first

• *Rationale*: No model overcomes fragmented, redundant inputs; the 80%/50% cleanup was the real unlock

• *Trade-off*: Slower to a first model, but far faster and more reliable thereafter

**Decision 2: Interpretable Time-Series vs. Black-Box ML**

✅ *Chose*: Transparent, trend-aware time-series methods

• *Rationale*: The business had to trust the forecast enough to spend against it; interpretability drove adoption

• *Trade-off*: Might leave a little accuracy on the table versus heavier models, but earns the trust that creates value

**Decision 3: A Reusable Backend vs. a One-Off Script**

✅ *Chose*: Backend architecture for the whole data-science segment

• *Rationale*: A model only creates value if it is delivered and maintainable; the platform outlived the project

• *Trade-off*: More engineering up front, repaid in reuse and lower overhead

**Decision 4: Optimize a Metric vs. Optimize a Decision**

✅ *Chose*: Accuracy calibrated to the procurement/scheduling decision

• *Rationale*: The goal was $ saved, not a leaderboard score; evaluation targeted decision quality

• *Trade-off*: Less glamorous than chasing error metrics, but it is what produced the $2M

Consolidate, Then Model: The first deliverable was not a model — it was clean pipelines. Integrating redundant sources removed ~80% of duplication and halved project overhead, which is what made accurate forecasting possible.

Trend & Seasonality Features: Energy demand has structure. Encoding trend and daily/weekly/seasonal cycles gave the model the signal it needed to be right when it counted.

Interpretable Time-Series Methods: I favored transparent, trend-aware methods (exponential smoothing / seasonal decomposition) over opaque models, so stakeholders could understand and trust the forecast enough to change decisions.

Honest Evaluation: Back-testing on held-out periods kept the accuracy claims grounded and the forecasts calibrated for real decision-making.

Built to Be Handed Off: The backend and pipelines were designed for the data-science team to extend — part of why I could also raise junior engineers' readiness ~80%.

Canonical pipelines with validation at ingestion replaced fragile manual hand-offs — the 80% redundancy cut was also a reliability fix.

Forecasts were monitored against actuals, so drift showed up as a measured error trend rather than a surprise.

The serving path degraded gracefully: a late upstream source delayed a refresh instead of publishing a wrong forecast.

Consumption data was operational, not personal — but it was commercially sensitive, so it stayed inside company-controlled infrastructure.

Pipeline access followed the segment's existing role-based controls; no data left the governed environment for modeling.

Back-testing on held-out historical periods kept accuracy claims honest and calibrated to the decisions the forecast informed.

Data-quality checks on the consolidated pipelines caught upstream schema and completeness problems before they reached the model.

Evaluation targeted decision quality (cost saved at the procurement horizon) rather than leaderboard error metrics.

Business Impact:

• **$2M in energy-cost savings in a single year** — the forecast let the business optimize consumption instead of reacting to it.

• Predictions accurate and trusted enough to drive real procurement and scheduling decisions.

Engineering Impact:

• **80% reduction in data redundancy** and **50% reduction in project overhead** from consolidating the pipeline foundation.

• Backend architecture that the data-science segment reused well beyond this project.

Team Impact:

• Trained junior data engineers, raising their readiness ~80% by keeping the design interpretable and teachable.

• Established a repeatable, full-lifecycle pattern for future forecasting work.

**1. The Data Foundation Is the Model**

The biggest lever was not the algorithm — it was consolidating redundant sources (80% less redundancy, 50% less overhead). *Lesson: invest in clean pipelines first; the model is only as good as what feeds it.*

**2. Interpretability Drives Adoption**

A transparent, trend-aware model earned the trust that a black box never would, and trust is what turned forecasts into $2M of action. *Lesson: a model people act on beats a more accurate model they ignore.*

**3. Optimize the Decision, Not the Metric**

Framing success as "cost saved" rather than "error minimized" kept the work aimed at business value. *Lesson: tie model evaluation to the decision it informs.*

**4. Build It to Be Handed Off**

Designing the backend for the data-science team — and teaching it — multiplied the impact beyond what I could deliver alone. *Lesson: maintainable, teachable systems compound in value.*

**5. Seasonality Is Signal, Not Noise**

Explicitly modeling trend and seasonal cycles was what made the forecast trustworthy at the moments that mattered. *Lesson: in time series, structure you ignore becomes error you can't explain.*

Prediction intervals, not just point forecasts, so decisions could price in uncertainty explicitly.

Finer-grained seasonality (per-line, per-shift) as the consolidated data matured.

Automated retraining triggered by measured drift instead of a fixed calendar.