ETL and ELT for clinical research, hospital systems, and enterprise IoT, built in SQL and Python with dbt, Spark, Airflow, and Databricks, on Azure and GCP plus some AWS. Every pipeline ships with data-quality controls, lineage, and audit evidence.
Certified: Databricks Data Engineer · Apache Spark Developer · dbt Fundamentals · Azure Data Fundamentals
Design the data models and schemas, and agree the KPI definitions before any pipeline is built.
Engineer SQL and Python pipelines with dbt, Spark, and Databricks that ingest, clean, and standardise source data.
Build data-quality monitoring and validation checks into the SQL workflows themselves, so failures surface before reports do.
Run independent code reviews and surface data-quality metrics in dashboards (SQL, Power BI, DAX).
from pyspark.sql import functions as F
def standardise_events(df):
"""Clean and standardise raw events
before loading to Databricks."""
key = F.col("event_id")
ts = F.current_timestamp()
return (
df.dropDuplicates(["event_id"])
.filter(key.isNotNull())
.withColumn("ingested_at", ts)
)-- Data-quality gate: fail the run if
-- silver holds dupes or null keys
SELECT event_id, COUNT(*) AS occurrences
FROM silver.events
GROUP BY event_id
HAVING COUNT(*) > 1 OR event_id IS NULL;