Seamless Data Integration & Lifecycle
Tracking for Meridian Port Services
Overview
Meridian Port Services (MPS), one of West Africa’s most advanced container terminals, partnered with Scanport Ltd to build a robust backend data pipeline that could ingest, process, and track container operations in real-time — integrating deeply with their Terminal Operating System (TOS).
The goal: unlock end-to-end container lifecycle visibility, power regulatory reporting, and fuel real-time analytics — all while enabling data exchange with Scanport’s inspection ERP and centralized Power BI dashboards.
Project Scope & Key Deliverables
This integration-focused project was architected around five core pillars:
1. EDI/XML File Transmission from MPS TOS
2. Backend Data Processing Pipeline
3. Full Container Lifecycle Tracking
4. MPS Integration Layer (Shadow DB + Sync Logic)
5. Power BI Dashboard Enablement
1. EDI/XML File Transmission from MPS TOS
Supported File Types
- EDI (EDIFACT) — shipping line data (e.g., container movements, port codes, BL references)
- XML - RSE — scan event results (RED/GREEN/ORANGE verdicts)
- XML - CR — customs release notices
Processing System
- Tracks scanning equipment and detection units
- Weekly checklist logs and radiation readings recorded digitally
- Service visit reports maintained and linked to each asset
- Preventive maintenance alerts based on usage thresholds
Data Transport
- Files received via S3-bucket–backed SFTP simulation
- Celery worker polls every 3 minutes for new arrivals
- Each file triggers a real-time processing job and is archived post-processing
2. Backend Data
Processing Pipeline
-
EDI Processor
Parses EDIFACT segments using pydifact
Extracts container data, BL references, drivers, vehicle IDs
Automatically handles twin 20ft containers
Links or creates driver and container records in PostgreSQL -
XML Processor
Parses both RSE (Risk Scan Events) and CR (Customs Releases)
Updates verdicts, scan timestamps, and release statuses
Triggers downstream processes (e.g., inspection allocation or exit clearance) -
XML Processor
Parses both RSE (Risk Scan Events) and CR (Customs Releases)
Updates verdicts, scan timestamps, and release statuses
Triggers downstream processes (e.g., inspection allocation or exit clearance) -
SFTP Engine
Drives the full pipeline: monitors input folders, triggers parsing jobs, and moves files to audit-safe storage
Secure Network Connectivity &
VPN Setup
To enable secure, high-availability data exchange between Scanport and MPS, a Site-to-Site VPN was implemented as a foundational requirement for backend system communication.
Site-to-Site VPN Integration
MPS provisioned and managed the VPN infrastructure on their end.
On Scanport’s side, we handled:
- Network compatibility checks
- VPN endpoint configuration
- Firewall rule adjustments
- End-to-end testing to confirm tunnel integrity and encryption
NAT Gateway Setup for
Outbound Stability
To facilitate stable outbound connectivity to MPS’s SQL database:
- A NAT Gateway was provisioned within our cloud environment
- It provides a dedicated static IP, which is the only IP whitelisted by MPS for database access
- All outbound traffic to MPS is routed through this gateway to:
- Enforce outbound traffic isolation
- Guarantee consistent routing and auditability
- Prevent unauthorized data access attempts
Security & Monitoring
- Continuous monitoring of VPN status and NAT Gateway health is in place
- Alerts are triggered for latency spikes or connection drops
- Logs and metrics ensure visibility into all outbound traffic related to MPS
3. Full Container
Lifecycle Tracking
Each step is
- Timestamped
- Tracked with audit logs
- Updated via file ingestion or real-time updates
- Exposed through an API layer and WebSocket channels for client UI and ERP sync
Key Entities
- Container
- Container Activity
- Driver Activity
- Scan Verdict + Customs Status
- Driver Records
- Inspection System
- Bay Allocations
Conflict Resolution
- If data mismatches are detected (e.g., outdated allocations), the system flags for manual intervention
- Validated updates are reflected in both the internal container model and the ERP system
Real-Time Architecture
Containers
Bay Allocations
Scan Events
Customs Status
Agent Assignments
File & Data Processing
EDI & XML Workflow
Security & Compliance
- Local replica of critical MPS records is maintained for safety and sync efficiency
- Read-only sync logic ensures no accidental overwrites of terminal data
- Uses Celery beat tasks every 6 hours for full refreshes, with partial sync triggers as needed
Responsive
Design
Fully adaptive layout optimized for all device types and screen sizes
Stakeholder
Feedback
“The integration has made it possible for us to track every container in real-time — from scan to release. It has improved visibility, reduced manual checks, and allowed our stakeholders to access accurate data across terminals, security systems, and regulatory units.”
Loretta Arhin, Agent
Solution
Architecture
Backend API
Django 4.2.6 Django REST Framework
Real Time Engine
Django Channels
Redis
Frontend
Next.js
(PWA-enabled for mobile use)
Backend API
PostgreSQL (Primary)
MS SQL (MPS Sync)
- Engineering Highlights
- Bulk processing logic with bulk_create and bulk_update_or_create
- Modular Interchange architecture with reusable processors
- Celery beat orchestration for recurring syncs and cleanups
- Extensive retry & fallback mechanisms
- Flexible webhook and manual triggers for urgent updates
- Engineering Highlights
- Streaming file parser for real-time ingestion of high-volume drops
- Kafka-based pipeline with distributed processing
- Smart anomaly detection (e.g., mismatch in release vs scan verdict)
- REST + GraphQL API exposure for third-party data access
- Enabling scan-to-dashboard latency < 5 minutes
Conclusion
This MPS–Scanport collaboration showcases how high-throughput port operations can be transformed through robust backend engineering, secure data flows, and thoughtful lifecycle tracking. By seamlessly integrating terminal systems with inspection ERP workflows and dashboard analytics, the system now delivers real-time control, regulatory confidence, and operational clarity — container by container.