Shipping an AI Document Processing Platform Into a Locked-Down Enterprise Network

Problem

Ship a document-AI platform into an enterprise customer's own AKS environment — behind a corporate firewall, on a network we don't control, with security policies we can't see and mostly can't change. The defining property of that network: it never says no. Blocked egress isn't refused with an error; it's silently dropped. Idle connections aren't closed with a FIN or RST; they simply stop existing.

Which means every network policy violation surfaces as an application bug: a timeout here, a NoneType there, an email that just never arrives. The engineering work wasn't fighting the firewall — it was building a platform that stays diagnosable and reliable inside one.

Architecture

The platform itself is a Django backend with asynchronous document-processing workers, Redis for job state, and queue-based decoupling between stages — deployed via the standard enterprise promotion path (development → UAT → production) onto the customer's AKS clusters.

The interesting architecture is at the network boundary:

• Image supply chain: docker.io and public registries are black-holed inside the network. All images route through the customer's corporate Artifactory acting as a registry proxy — which becomes a hard dependency you have to design and document for, not an implementation detail.
• Outbound HTTP to partner APIs: every call crosses the egress firewall, inheriting its silent idle-connection policies and TLS interception on non-allowlisted hosts.
• Email: the corporate SMTP relay authenticates by source IP allowlist, not credentials — meaning the cluster's egress IP is effectively part of the application's identity and must be pinned static.

Technology Choices

• Work with the network's rules rather than tunneling around them. VPN sidecars or relay hops through friendlier networks would have "solved" several problems while violating the spirit of the customer's security model — and would eventually be found and cut. Every fix stays inside policy.
• Connection lifecycle management over retry-and-pray. For low-frequency outbound calls, the client creates fresh connections instead of trusting a pool, and every call uses split connect/read timeouts. Retries paper over a network like this; they don't make it diagnosable.
• Raw-protocol probes as first-class diagnostic tools. When the mail relay rejects you, Django's error is useless; a hand-driven smtplib conversation shows the actual SMTP dialogue. The deployment kit includes small, dependency-free probes for HTTP, SMTP, and registry pulls.

Challenges

Each of these arrived dressed as an application bug:

• Phantom ReadTimeouts to partner APIs. Random, unreproducible, all node probes healthy. Root cause: the egress firewall silently ages out idle keep-alive connections, and a module-level shared requests.Session kept handing dead sockets back out — each costing a 300-second hang. "Random" turned out to mean "time-dependent."
• Notifications silently stopping. A bare except had been swallowing send errors; the SMTP conversation underneath showed the sender accepted and every recipient refused with 554 — the relay allowlists by source IP, and the cluster's egress IP wasn't on the list. Diagnosis was ours; the fix (pin a static egress IP, request allowlisting) was necessarily the customer's.
• TLS interception on non-allowlisted hosts, which breaks certificate validation in ways that look like the remote service's fault.
• Proving which layer was lying, repeatedly. The recurring pattern of the whole engagement: the error message names the layer where failure surfaced, and the cause lives one or two layers below it.

Trade-offs

• You don't get to fix the firewall. Accepted fully: the application is designed to survive the network as-is — fresh connections, explicit timeouts, no silent failure paths — rather than depending on policy changes that may take months or never come.
• Fresh connections cost latency on every low-frequency call. Measurably slower than pooling, and worth it: a predictable few hundred milliseconds beats an unpredictable 300-second hang at a rate of roughly one incident per week.
• Static egress IP couples the platform to network configuration. Pinning the IP makes cluster networking changes more delicate — but on a network where your IP is your identity, that coupling already existed; pinning just makes it explicit and stable.

Results

• The platform runs across the customer's development, UAT, and production AKS environments, inside their security posture rather than despite it.
• Network validation is now part of deployment: a preflight suite proves egress, SMTP reachability, and registry access before rollout, so policy regressions surface as a failed check instead of a production mystery.
• The failure classes — silent idle-drop, IP-based SMTP identity, proxied registries, TLS interception — are documented as a playbook, turning each future occurrence from an investigation into a lookup.
• Two of the investigations were significant enough to write up in full: The Firewall That Never Says No and the SMTP 554 story both came out of this platform.