Python AI Data Cleaning Automation

I need a set of well-structured Python scripts that automatically clean and validate raw data so it is ready for downstream analysis. The workflow should use AI-driven techniques where they add value—for example, leveraging an LLM to detect anomalous patterns or flag inconsistent entries more intelligently than simple rule-based checks. Scope • Read data from common cloud sources (CSV, JSON or database dump on S3/GCS/Azure Blob). • Perform thorough cleaning: handle missing values, normalise formats, de-duplicate records, and validate ranges or schema consistency. • Produce a concise, human-readable report that summarises the cleaning actions taken and highlights rows that still need manual review. • Package everything into modular, reusable functions (pandas, PySpark or Polars are fine) with clear docstrings and typing. • Optionally expose the pipeline as a lightweight API or CLI so it can be scheduled in an orchestration tool such as Airflow or executed in a Jupyter notebook. AI/LLM Angle If you have experience with LangChain, OpenAI functions or similar, feel free to integrate a step where the model reviews suspicious rows and suggests corrections; however, deterministic rule-based logic must remain the primary validator to keep the pipeline auditable. Deliverables 1. Clean, commented Python code in a Git-ready repo. 2. A README explaining how to configure cloud credentials and run the pipeline locally or in CI/CD. 3. Sample test cases (pytest) proving the validation rules work. 4. The generated cleaning report in HTML/Markdown for the sample dataset I’ll provide after kickoff. Acceptance criteria: the test suite passes, the report matches the expected row counts and error rates, and the code deploys without modification on my cloud instance. I’m aiming for maintainable, production-grade code, so please highlight any best-practice choices you make regarding logging, config management or environment reproducibility.