Back to the blog

On-Demand Fargate Batch Jobs with One Pull Request per Run

An editorial worker moves from Lambda to on-demand Fargate tasks and from independent article pull requests to sequential commits in one batch pull request.

Introduction

I moved the editorial worker from a Lambda function to an on-demand ECS Fargate task and changed its output from one pull request per article to one pull request per batch. The two changes address different problems: allowing longer-running work without keeping an instance running, and putting updates to a shared manifest into one sequential history.

The previous workflow opened independent pull requests for generated articles. The evidence records a ten-article batch whose pull requests all modified the same generated.ts manifest and produced conflicts. It doesn't give a conflict count or the exact merge sequence, so I wouldn't describe this as ten conflicts requiring ten manual resolutions.

I implemented the replacement using a CDK Docker asset and individual article commits on one batch branch. That establishes the architecture and commit rule, but not a fully verified outcome. I haven't confirmed a real post-change multi-article run. The recorded networking description also needs qualification before it can be treated as a valid deployed Fargate configuration.

The Rule That Shipped

Each batch run creates one branch and one pull request, with a separate commit for each article. The PR Adapter creates branches, commits, and pull requests. It never merges a pull request or triggers deployment. It prepares the proposed changes; the decision to merge stays separate.

The worker runs as an on-demand ECS Fargate task that starts for a run and stops afterward, rather than as a continuously running service. A CDK Docker asset builds its image, replacing the earlier manual image-build-and-push step. The evidence doesn't specify exact task timeout, memory, or CPU values, so I can't report those as deployed settings.

The networking record says “no VPC attached,” matching its description of existing ECS tasks. I can't present that wording as a technically valid Fargate configuration: Fargate requires awsvpc networking, including a VPC and subnets. The actual networking details remain unverified. The record doesn't establish whether “no VPC attached” meant no newly created VPC, no need to connect to private resources, or something else. Those possibilities need checking, not silently substituting for the recorded claim.

Worked Example

The recorded input is a batch generating ten articles. Under the previous rule, it would create ten separate pull requests, each modifying generated.ts. This arrangement produced conflicts, but the evidence contains neither a conflict count nor the exact diffs. Editing the same file doesn't guarantee a conflict for every pull request; the edits and merge order matter.

Under the implemented replacement rule, the expected result is one branch and one pull request containing ten commits, one per article. The first commit includes the article changes and their manifest update. The second builds on that updated branch state, and the remaining commits follow in sequence. Each manifest update starts from the preceding commit's state instead of an independent copy of the original batch base.

This removes competition among the batch's separate pull requests. It doesn't establish that the batch branch will stay conflict-free against other work. Another branch can still change generated.ts before the batch merges.

I haven't confirmed that the full ten-article case was run after the change or that anyone inspected its resulting pull request. One pull request with ten sequential commits is the designed result, not an observed end-to-end test result. Partial failures are also unresolved in the evidence: it doesn't specify retry, resume, or cleanup behavior if a run stops after committing only some of the articles.

Challenges and Tradeoffs

Lambda offers a straightforward execution model for scheduled work, but its platform maximum is 15 minutes per invocation. That known limit is separate from the earlier function's configured timeout and the worker's measured duration, neither of which is documented here. I chose a different compute model to accommodate longer-running work. I can't claim that an observed Lambda timeout forced the change, and the evidence doesn't document the specific execution limit or timeout value that made Lambda unsuitable.

An always-on instance was another option, but it would retain capacity between runs. On-demand Fargate ties the worker's task lifecycle to a job instead. It avoids the need for a continuously running worker, but that doesn't establish a measured cost saving. Compared with a function invocation, it also adds container infrastructure, task configuration, permissions, networking, and startup work. Fargate doesn't promise that jobs can run indefinitely or succeed without operational limits.

The pull-request change has its own tradeoff. Separate article pull requests allow separate merge decisions. A batch pull request groups those decisions, though per-article commits still provide a useful division for inspection. Selecting only part of a batch may require restructuring commits or preparing separate changes. Sequential commits address the recorded intra-batch manifest problem; they don't replace validation of the manifest's contents or coordination with unrelated writers.

Verification Process

The recorded checks cover infrastructure and implementation steps. I deployed the CDK stack and confirmed that environment variables resolved correctly. I also built and pushed the worker image manually before moving image handling to a CDK Docker asset. That change removed the separate manual image-push step from the deployment workflow.

I implemented the PR Adapter's branch, commit, and pull-request operations, including its boundary against merging or triggering deployment. I also configured the Fargate role's access to the required GitHub credentials. These implementation details are supported, but they don't independently demonstrate a successful complete batch.

The next verification step is to inspect a real multi-article run: confirm that it creates one branch and one pull request, check the article-to-commit mapping, and verify that successive manifest updates preserve earlier entries. The same check should confirm that the adapter leaves the pull request unmerged and doesn't initiate deployment. These are still checks to perform, not results I've already observed.

Networking needs a separate inspection of the deployed task configuration, including its VPC and subnets. It's also unconfirmed whether the configuration was tested against a resource requiring VPC-only connectivity. Such a test wouldn't resolve the contradictory “no VPC” description on its own; the actual task networking must first be identified.

Conclusion

I replaced the Lambda-based worker design with an on-demand Fargate task built through a CDK Docker asset. I also replaced independent article pull requests with one batch pull request containing per-article commits. The adapter prepares changes without merging or deploying them. The intended result is a sequential batch history rather than competing manifest updates on independent branches.

Those implementation choices don't establish that every failure mode has been resolved. The post-change multi-article case remains unverified. Configured timeout, measured runtime, memory, and CPU details remain undocumented, and the actual Fargate networking still needs inspection. These limits distinguish the implemented rule from the operational behavior still to be demonstrated.