Inventory Parser and Cloud Ingestion Pipeline
PDF ingestion pipeline with presigned S3 uploads, ECS-based parsing, and an explicit parse-then-apply workflow guarded by DynamoDB transactional idempotency.
Problem
Manual invoice entry is slow and error-prone. Applying inventory changes directly from raw parsing output is risky. This system enforces a reviewable, two-step workflow where parsing is isolated from inventory mutation.
Constraints
- No Step Functions. Orchestration is handled by API + ECS RunTask + polling.
- Deterministic parsing scope. No OCR in MVP.
- Low-volume ingestion assumption with cost guardrails over throughput.
- No user authentication in MVP. Single-operator workflow.
Architecture
- API Gateway HTTP API + Lambda (container image) for job lifecycle endpoints
- Presigned S3 PUT for direct browser uploads without proxying files through the API
- ECS Fargate RunTask to isolate parsing workload from API execution
- DynamoDB: jobs table for state, inventory table for totals, apply-ledger table for idempotency guard
- CloudFront + OAC to serve the SPA from a private S3 origin (bucket is not public)
- Terraform-managed infrastructure with image digests pinned for deterministic deploys
Key decisions
Parse-Then-Apply Workflow
Kept parsing output separate from inventory mutation so inventory updates only happen through an explicit apply action.
Idempotent Apply with Transaction + Ledger
Used a DynamoDB transaction and a ledger record keyed by jobId to prevent double-apply and guarantee apply happens at most once.
ECS Fargate for Parsing
Moved parsing out of Lambda to avoid tight runtime limits and keep headroom for heavier extraction later.
Risks and mitigations
ECS task fails mid-run, leaving jobs stuck in queued/running
Job status is stored centrally. Operator can safely re-submit because apply is idempotent and parsing writes to versioned S3 keys.
S3 growth and log noise over time
Lifecycle rules expire raw/parsed/errors prefixes and CloudWatch log groups use retention limits.
Next steps
- Add an SQS-backed queue and worker pattern for burst handling
- Add automatic retry policy for transient ECS failures
- Add structured metrics (job latency, success rate) and basic dashboards
- Add authentication and move from single-operator to multi-tenant model