RemitTrack

1. Overview

RemitTrack is a real-time exchange rate comparison platform for the African diaspora. The core problem is simple: if you send money home regularly, the difference in rates between providers adds up over time, and finding the best rate means visiting each provider separately. RemitTrack pulls live rates from 15+ Money Transfer Providers, normalises them into a consistent format, and surfaces them in one place.

The platform serves two audiences. Consumers use it to find the best rate before sending. Money Transfer Providers (MTPs) use the aggregated rate data as a market intelligence and arbitrage risk signal across their corridor offerings. There is also an LLM-powered chat interface for users who find data tables harder to navigate.

The architecture is split into clean, decoupled tiers: a Next.js frontend served from CloudFront edge nodes, a Go modulith on a single EC2 host, a serverless Lambda extraction cluster, and an agentic pipeline running on ephemeral GitHub Actions runners that keeps the scrapers healthy without manual intervention.

2. System Architecture

The global topology isolates real-time data ingestion, public presentation caching, core business logic, and the agentic self-healing pipeline into distinct tiers. The feedback loop from the agentic pipeline back to the extraction cluster is intentional: when a merged patch activates a repaired scraper, it re-enters the EventBridge schedule and starts producing data again.

3. Frontend and Edge Caching

The frontend is a Next.js application that compiles to flat, static HTML and JSON files. Those files are hosted in S3 and served via CloudFront. For most requests, CloudFront resolves the page directly from edge cache with no compute running on the origin server at all.

When the ingestion pipeline finishes a scrape cycle, it fires an authenticated webhook that triggers an ISR rebuild. The order here matters: the build job writes updated files to S3 first, then calls the CloudFront Invalidation API. Invalidation alone never regenerates anything, it only purges. Regenerating the origin first ensures the next request fetches fresh output rather than stale cache. Only the changed corridor pages are rebuilt, so a page like /exchange-rates/gbp-to-ngn stays current every five minutes without a full site rebuild.

Static pages inject rate data as JSON-LD schemas: ExchangeRateSpecification for table rows and FAQPage for conversational queries. This gives web crawlers that skip client-side JavaScript pre-rendered, structured data without a live render server. The site also handles multiple locales through Next.js directory routing (/app/[locale]/exchange-rates/[corridor]/page.tsx), with canonical and hreflang tags generated at build time.

4. Backend and Alert Engine

The Go modulith runs on an EC2 t4g.micro under systemd. NGINX sits in front of it and handles ingress, configured with two upstream ports (8080 and 8081) for zero-downtime deployments. When a new binary ships, it starts on the idle port, NGINX rewrites its upstream block and reloads, and in-flight requests on the old port drain gracefully before that process stops.

Alert processing runs entirely in memory. When a scrape batch lands, a pool of background goroutines checks the new rates against an index keyed by currency corridor, such as USD:GHS. If a user's price target is hit, the goroutine dispatches an email via AWS SES. Running this in memory keeps latency near zero and avoids a round-trip to Redis for every alert check.

B2B consumers access rates through a separate token-gated endpoint (/api/v1/b2b/rates/latest). Requests go through Go middleware that validates an x-api-key header against hashed records in PostgreSQL, with rate limiting via a token-bucket algorithm held in instance memory.

Stripe billing webhooks go through signature verification and an idempotency check: the evt_id is hashed against a ledger table in PostgreSQL, so network retries never result in duplicate charges. A periodic reconciliation job re-reads Stripe subscription status and corrects any drift from missed webhooks.

The modulith also hosts an LLM chat package that answers plain-English rate questions, routing context to the configured model and streaming responses through worker goroutines. Its anonymised conversation logs feed the FAQ mining engine that publishes FAQPage schemas for long-tail search capture.

5. Extraction Pipeline

Scraping runs outside the core server entirely. Each Money Transfer Provider has a dedicated Lambda function written in Go, triggered on a five-minute EventBridge cron. The Lambda normalises the scraped rate into a unified JSON record, pushes it to SQS, and exits. The modulith long-polls the queue and writes records to RDS in batches.

Each provider entry in provider_map.json declares an ingestion strategy. The Lambda calls the provider's official partner API when one exists, and only falls back to browser-emulating scraping when no compliant API is available. API-first ingestion is more reliable and reduces how often the self-healing pipeline needs to fire.

For providers without a compliant API, the Lambda uses the utls library to emulate a standard browser TLS handshake rather than Go's default fingerprint. Outbound connections route through a rotating residential proxy service so requests appear to come from real user IPs rather than AWS data centre ranges. This defeats JA4 TLS fingerprinting checks used by Cloudflare and similar systems.

The database write uses an ON CONFLICT DO NOTHING insert, so duplicate SQS deliveries are silently discarded.

INSERT INTO historical_rates (provider, currency_pair, mid_rate, scrape_timestamp)
VALUES ($1, $2, $3, $4)
ON CONFLICT (provider, currency_pair, scrape_timestamp) DO NOTHING;

Two tables handle the rate data. historical_rates stores every scrape for trend analysis and alert evaluation. current_rates holds one row per provider and currency pair, updated on each insert. All hot user queries read from current_rates, so the growing history table is never scanned on a live request.

The RDS instance runs Multi-AZ active-passive replication with Point-In-Time Recovery. All infrastructure is managed through Terraform with a prevent_destroy = true lifecycle guard on stateful resources, and production applies require manual approval.

6. Agentic Pipeline

Provider APIs and page structures change without notice. For a rates comparison platform, the worst failure mode is silent: a wrong number looks exactly like a right one. The agentic pipeline is the mechanism that keeps data accurate long-term without someone manually monitoring every provider. Every resolved failure strengthens the system: it gets recorded in the failure registry, turned into a replay test, and added to the regression corpus. The next time a similar break happens, the fix is faster and the regression is permanently guarded against.

Three agents run on ephemeral GitHub Actions runners, each triggered independently.

The Trust Boundary

No single failure or injected instruction can reach production. Every candidate patch climbs a sequential ladder of independent gates, each of which can only reject. The blast radius of any compromised input is a rejected pull request, not a production change.

• Scope check: the agent may only edit JSON field-mapping transformers. If the problem is a broken HTML DOM structure, it stops immediately and alerts a human engineer. That class of change is always out of bounds.
• AST allowlist gate: the generated Go is parsed by Go's own go/parser. Only four pure data-mapping packages are permitted: strings, strconv, encoding/json, and errors. Anything else, including net/http or //go:linkname compiler directives, is discarded before the code ever compiles.
• Chaos sandbox: the patched scraper runs inside an isolated Docker Compose mock-provider network. It never touches production at this stage. The chaos environment can serve any historical payload shape on demand, including broken ones, so the full scope of known failures can be validated offline.
• Statistical canary: the rate produced by the patched scraper must fall within three standard deviations of the historical mean for that corridor, and must align with a cross-provider variance check. For a brand-new corridor with no history, the check is skipped and the candidate is held for mandatory human review rather than auto-passed.
• Regression corpus: the patch must keep every historical failure case in failure_registry.json green. No candidate merges unless the entire failure history replays cleanly.
• Human PR: nothing merges without a human reviewing and approving the pull request. The agent can only propose; it cannot ship.

The agent attempts at most three repair loops per failure. If no candidate clears every gate within three attempts, it stops and escalates to a human rather than looping indefinitely.

func VerifyASTSafety(goSource string) error {
    if strings.Contains(goSource, "//go:linkname") {
        return errors.New("security alert: //go:linkname directive is not permitted")
    }
    fset := token.NewFileSet()
    file, err := parser.ParseFile(fset, "patch.go", goSource, parser.ImportsOnly)
    if err != nil {
        return fmt.Errorf("patch rejected: unparseable Go source: %w", err)
    }
    for _, imp := range file.Imports {
        path, _ := strconv.Unquote(imp.Path.Value)
        if !allowedImports[path] {
            return fmt.Errorf("security alert: forbidden import %q", path)
        }
    }
    return nil
}

The Three Agents

Self-Healing Agent: when CloudWatch detects a scraper crash or empty payload, it fires a repository dispatch webhook into GitHub Actions. The runner downloads the ChromaDB vector store archive from S3, initialises a local session, and runs a semantic search to retrieve the top matches for similar past failures and their successful fixes. An LLM generates a repair patch, which then climbs the full validation ladder above before a PR is opened.

Compliance Sentinel: runs daily. Downloads the robots.txt and Terms of Service for every monitored provider, checksums them against yesterday's snapshot stored in S3, and runs an LLM evaluation if anything changed. If the updated terms restrict scraping, the agent opens a PR to deactivate that provider in provider_map.json.

Onboarding Agent: triggered via make onboard URL=.... Compliance rules are checked first. If the target is compliant, the agent launches a headless browser using playwright-stealth, listens to background network traffic for 45 seconds, and if it finds rate data in a JSON response, generates a new Go Lambda scraper and opens a PR.

Build Order

Because the gates are what make autonomous changes safe, they have to be in place before the code generator exists. The correct build sequence is: observability first (CloudWatch, metric filters, the dispatch webhook), then the failure registry and chaos sandbox, then the standalone gates (AST linter, statistical canary), then the self-healing generator, then the Compliance Sentinel, and finally the Onboarding Agent last since it generates entirely new scrapers and carries the highest blast radius. Guardrails and corpus come before code generation, not after.

7. CI/CD and DevOps

The platform uses a GitOps model. Everything that touches the cloud is managed through Terraform in /infra/terraform/. When a change merges, GitHub Actions boots an ephemeral Ubuntu runner and executes two parallel tracks.

The validation track runs the full test suite, boots the Docker Compose chaos environment and replays the entire failure registry corpus, then cross-compiles the Go modulith binary and Lambda zip packages. The infrastructure track runs terraform plan, posts the plan output to the PR for human review, and only applies after manual approval on the protected production environment.

If both tracks pass, the runner ships the compiled artifacts to AWS via CLI, SSHes into the EC2 host, and executes the zero-downtime binary swap: the new binary starts on the idle NGINX upstream port, NGINX reloads its config, and in-flight requests on the old port drain before the old process stops.

Stateful resources, the RDS instance above all, carry a prevent_destroy = true Terraform lifecycle guard so a bad plan can never replace or delete them. The chaos regression replay running in the validation track means no deployment can ship a change that breaks historical scraper behaviour.

8. Engineering Trade-offs

• Lambda over scraping on the host: decouples extraction from the application server and prevents connection exhaustion on the EC2 instance. Lambda cold starts on a 5-minute schedule are not a concern. The trade-off is more moving parts and cross-service IAM configuration to manage.
• In-memory alert cache over Redis: zero cost, near-zero latency, no network hop. The trade-off is that alert state is tied to a single host. Moving to multi-node horizontal scaling would require migrating the alert pool to ElastiCache. The REMITTRACK_INFRA_PROFILE environment variable is the planned switch: LEAN keeps the single EC2 modulith, ENTERPRISE repoints CloudFront to an Application Load Balancer fronting ECS Fargate containers.
• Standard SQS over FIFO: removes throughput caps and simplifies scaling. Idempotent SQL constraints handle out-of-order and duplicate deliveries at the database level, so strict queue ordering is not needed.
• CloudFront edge caching over live SSR: eliminates almost all host compute for read traffic. The trade-off is losing real-time per-user personalisation based on request headers. If true per-user rendering becomes necessary, that is the point to introduce SSR or a Lambda@Edge render tier.
• Strict AST scope gate over broader agent autonomy: restricting the self-healing agent to JSON field-mapping changes means it cannot fix structural HTML changes on legacy platforms. But it materially reduces the attack surface against compiler bypass vectors and resource-exhaustion loops. The blast radius of any compromised input remains a rejected PR.
• GitHub Actions runners over dedicated testing infrastructure: offloading chaos testing and compilation to GitHub Actions introduces queue latency compared to a dedicated cloud container node. However, it eliminates recurring testing infrastructure costs and keeps operational overhead at zero.