Data Engineer vocab

13 flashcards · Programming Languages · by Thang Vo

  1. Monitoring pipelines

    (IPA): /ˈmɒnɪtərɪŋ ˈpaɪplaɪnz/ Part of Speech: Noun phrase Usage & 5 Examples: “We implemented monitoring pipelines to detect Airflow task failures before they impact downstream tables.” “Effective monitoring pipelines alert the data engineering team via Slack whenever a data freshness threshold is breached.” “By integrating Prometheus and Grafana, we established robust monitoring pipelines for our streaming ETL jobs.” “The new monitoring pipelines dashboard gives us real-time visibility into j…

  2. Orchestration

    (IPA): /ˌɔːrkəˈstreɪʃən/ Definition: The act of arranging, coordinating, and managing multiple elements or tasks so they work together efficiently and in the correct order. Part of Speech: Noun Usage & 5 Examples: “We rely on Kubernetes for the orchestration of our containerized applications across the cluster.” “Effective orchestration of ETL jobs in Airflow ensures that data arrives in the warehouse on schedule.” “The orchestration of the product launch involved marketing, sales, logistics, a…

  3. Troubleshooting failures

    (IPA): /ˈtrʌbəlˌʃuːtɪŋ ˈfeɪljərz/ Definition: The process of identifying, diagnosing, and resolving faults or errors in a system or workflow. Part of Speech: Noun phrase Usage & 5 Examples: “Our on-call rotation includes troubleshooting failures in production ETL jobs to minimize downstream data lag.” “We documented a runbook for troubleshooting failures in the Airflow DAG, covering common error codes and recovery steps.” “Effective troubleshooting failures requires clear log aggregation and al…

  4. Writing & updating ETL code

    (IPA): /ˈraɪ.tɪŋ ænd əpˌdeɪ.tɪŋ ˌiːˌtiːˈɛl koʊd/ Definition: The task of authoring, modifying, and maintaining the scripts or programs that extract, transform, and load data between systems. Part of Speech: Noun phrase Usage & 5 Examples: “Our team spends most mornings writing & updating ETL code to accommodate new data sources.” “When the source schema changed, I had to refactor several DAGs, focusing on writing & updating ETL code to prevent pipeline failures.” “A major part of the job is wri…

  5. Optimizing performance

    (IPA): /ˈɑptəˌmaɪzɪŋ pərˈfɔrməns/ Definition: The process of improving the speed, efficiency, and reliability of a system, process, or component. Part of Speech: Noun phrase Usage & 5 Examples: “We focused on optimizing performance of our SQL queries, cutting report generation time by 80%.” “Our DevOps team spent last sprint optimizing performance of the microservices cluster under peak load.” “By optimizing performance in the React front end—lazy loading and code splitting—we reduced initial p…

  6. Designing schemas

    (IPA): /dɪˈzaɪnɪŋ ˈskiːməz/ Definition: The process of planning and creating the structure (tables, fields, relationships) of a database or data store to organize and enforce data integrity. Part of Speech: Gerund phrase (verb + noun) Usage & 5 Examples: “We spent this sprint designing schemas that enforce referential integrity and minimize redundancy.” “Good designing schemas practices include normalizing tables to at least 3NF and indexing foreign keys.” “When designing schemas for a multi-te…

  7. Implementing data quality checks

    (IPA): /ˌɪmplɪˈmɛntɪŋ ˈdeɪtə ˈkwɒlɪti tʃɛks/ Definition: The process of designing, executing, and maintaining procedures or tests to ensure that data is accurate, complete, consistent, and valid across systems. Part of Speech: Gerund phrase (noun phrase) Usage & 5 Examples: “We began implementing data quality checks in our Airflow DAGs to catch missing timestamps before they propagate downstream.” “By implementing data quality checks with dbt tests, we prevented schema drift and maintained reli…

  8. Code review & CI/CD

    (IPA): /ˈkoʊd rɪˈvjuː ænd siː aɪ siː ˈdiː/ Definition: The combined practices of systematically examining source code for quality and correctness (code review) and automating building, testing, and deployment processes (CI/CD). Part of Speech: Noun phrase Usage & 5 Examples: “Our team enforces code review & CI/CD on every merge request to catch bugs early and deploy reliably.” “Integrating code review & CI/CD with GitHub Actions reduced our manual testing time by 50%.” “We standardized code rev…

  9. Collaborating with stakeholders

    (IPA): /kəˈlæbəreɪtɪŋ wɪð ˈsteɪkˌhoʊldərz/ Definition: The act of working jointly with parties who have an interest or stake in a project to achieve shared objectives. Part of Speech: Gerund phrase (verb + prepositional phrase) Usage & 5 Examples: “We spent the kickoff week collaborating with stakeholders from marketing, finance, and IT to define project requirements.” “Effective collaborating with stakeholders ensures alignment on timelines and reduces scope creep.” “She leads workshops collab…

  10. Documenting workflows

    (IPA): /ˌdɒkjʊˈmɛntɪŋ ˈwɜːrkfloʊz/ Definition: The process of creating formal records—written or visual—detailing the sequence of steps, roles, inputs, and outputs within a business or technical process. Part of Speech: Gerund phrase Usage & 5 Examples: “We began documenting workflows in Confluence to ensure seamless handoffs between the data ingestion and transformation teams.” “Before automating the pipeline, documenting workflows helped us identify redundant steps and optimize resource usage…

  11. Resource planning

    (IPA): /rɪˈsɔːrs ˈplænɪŋ/ Definition: The process of identifying, allocating, and managing the people, tools, time, and budget needed to complete projects or operations efficiently. Part of Speech: Noun phrase Usage & 5 Examples: “Effective resource planning allowed us to assign the right engineers to the ETL migration without causing delays.” “During the Q4 kickoff, we reviewed our resource planning to ensure we had enough test environments for load testing.” “Automating resource planning with…

  12. Data profiling & anomaly detection

    (IPA): /ˈdeɪtə ˈproʊfaɪlɪŋ ænd əˌnɒməli dɪˈtɛkʃən/ Definition: The combined processes of examining a dataset’s structure, content, and quality (data profiling) and automatically identifying outliers or unexpected patterns (anomaly detection). Part of Speech: Noun phrase Usage & 5 Examples: “Before loading into the warehouse, we run data profiling & anomaly detection to catch schema changes and outliers.” “Our ETL framework includes data profiling & anomaly detection steps that raise alerts on u…

  13. downstream tables

    (IPA): /ˌdaʊnˈstriːm ˈteɪbəlz/ Definition: Tables in a database or data warehouse that receive, consume, or depend on data produced by upstream processes or tables. Part of Speech: Noun phrase Usage & 5 Examples: “After the staging layer refresh, we rebuild all downstream tables to ensure reports use the latest data.” “A schema change in the source triggered failures across several downstream tables, breaking our dashboards.” “We added a validation step upstream to prevent bad records from poll…