Inventory Ingestion and Review Pipeline
Invoice ingestion pipeline with presigned S3 uploads, ECS-based parsing, and a controlled parse-then-apply workflow protected by DynamoDB transactional idempotency.
Problem
Manual invoice entry is slow and error-prone, but applying inventory changes directly from raw parsing output is also risky. This system enforces a reviewable two-step workflow where parsing is isolated from inventory mutation.
Constraints
- No Step Functions. Orchestration is handled by API + ECS RunTask + polling.
- Deterministic parsing scope. No OCR in MVP.
- Low-volume ingestion assumption with cost guardrails over throughput.
- No user authentication in MVP. Single-operator workflow.
Architecture
- API Gateway HTTP API + Lambda (container image) for job lifecycle endpoints
- Presigned S3 PUT for direct browser uploads without proxying files through the API
- ECS Fargate RunTask to isolate parsing workload from API execution
- DynamoDB: jobs table for state, inventory table for totals, apply-ledger table for idempotency guard
- CloudFront + OAC to serve the SPA from a private S3 origin (bucket is not public)
- Terraform-managed infrastructure with image digests pinned for deterministic deploys
Key decisions
Parse-Then-Apply Workflow
Kept parsing output separate from inventory mutation so inventory updates only happen through an explicit apply action.
Idempotent Apply with Transaction + Ledger
Used a DynamoDB transaction and a ledger record keyed by jobId to prevent double-apply and guarantee apply happens at most once.
ECS Fargate for Parsing
Moved parsing out of Lambda to avoid tight runtime limits and keep headroom for heavier extraction later.
Risks and mitigations
ECS task fails mid-run, leaving jobs stuck in queued/running
Job status is stored centrally. Operator can safely re-submit because apply is idempotent and parsing writes to versioned S3 keys.
S3 growth and log noise over time
Lifecycle rules expire raw/parsed/errors prefixes and CloudWatch log groups use retention limits.
Next steps
- Add an SQS-backed queue and worker pattern for burst handling
- Add automatic retry handling for transient ECS task failures
- Add structured metrics and dashboards for job latency and success rate
- Add authentication and operator audit trails for apply actions
- Expand parsing scope with OCR and more complex document handling