arrow_back Back to freelances
F

Sales Data Pipeline Development

Freelancer

Share:
placeGH home_workRemote assignmentContract publicAggregated job · GH

eventPublished on Oct 18, 2026 · verifiedWe confirmed on Oct 18, 2026 that it's still live

Is this your business?

US$ 30 – US$ 250 per project

About the job

I need a small sales data pipeline (proof of concept) that runs directly on Windows 10/11. NO Docker. Delivery in 4 days, $100-$300 fixed, 2 milestones. DATA FLOW C++ simulator -> MQTT (Mosquitto) -> C# .NET collector -> SQLite -> Python loader -> Hadoop HDFS -> sales reports WHAT TO BUILD 1) C++ sales simulator - Simulates 10 stores and 50 products, sends each sale as JSON to MQTT (QoS 1) - Each sale: message_id, store_id, product_id, quantity, price, timestamp (UTC) - Settings: number of messages and speed. Stops after N messages (default 50,000) - Sends a few invalid messages on purpose (for testing) 2) C# .NET 8 collector - Subscribes to MQTT (e.g. MQTTnet) with a persistent session, so no data is lost when it restarts - Checks every message: good rows -> "sales" table, bad rows -> "rejected" table - No duplicates (unique message_id) 3) SQLite database - Tables: sales, rejected, load_log 4) Python loader -> Hadoop HDFS - Copies new SQLite rows to HDFS as CSV, in folders by date: /sales/dt=YYYY-MM-DD/ (UTC) - Safe to run twice: no duplicates, no missing rows - Lock, so two loaders cannot run at the same time 5) Hadoop sales reports - Hadoop Streaming jobs with Python mapper/reducer. Output 5 CSV files: 1. Total sales per day 2. Sales per store per day 3. Top 10 products by revenue 4. Sales by hour (UTC) 5. Average basket value per store 6) Verify script - One command that checks: sent = SQLite = HDFS = report totals. Prints OK or FAIL. TECHNICAL REQUIREMENTS - NO Docker, NO containers of any kind - On Windows: Mosquitto (Windows installer), .NET 8, Python 3.11+, C++ with CMake (+ vcpkg) - Hadoop 3.3.x single node in WSL2 Ubuntu (Hadoop does not run well on Windows directly; WSL2 is OK) - All ports bound to 127.0.0.1 only (MQTT 1883, Hadoop web UI 9870) - All times in UTC - All settings in one config file. No hard-coded paths or passwords. - Setup scripts (PowerShell + bash) and a clear step-by-step README - Unit tests for the collector and the loader MILESTONES Milestone 1 (40%) - Day 2: C++ simulator + Mosquitto + C# collector + SQLite working. 50,000 messages sent = 50,000 rows stored (sales + rejected), 0 duplicates. Milestone 2 (60%) - Day 4: Hadoop setup, Python loader, 5 reports, verify script, final README, short screen recording of a full run. ACCEPTANCE - I will run everything on my own Windows machine before I release payment - Fresh install works by following only the README, without Docker - Collector stopped for 60 seconds, then started again: 0 messages lost - Loader run twice: 0 duplicates in HDFS - Verify script prints OK, and the 5 report CSVs are correct DELIVERY - One ZIP per milestone: full source code, build and setup scripts, README, sample config - Full ownership of the code goes to me HOW TO BID - Start your bid with the word PIPELINE so I know you read this - Tell me your experience with MQTT and Hadoop (links to similar work if possible) - Confirm you can deliver in 4 days without Docker - Your fixed price

Keep reading for free

Create a free account to see the full job and apply.

  • badgePortfolio visible to companies
  • notificationsNew job alert by email
  • favoriteFree subscription, cancel at any time