AWS Data Pipelines

AWS Data Pipeline is a web service that makes it easy to schedule regular data movement and data processing activities in the AWS cloud. AWS Data Pipeline integrates with on-premise and cloud-based storage systems.

11 flashcards · Programming Languages · by lakesare

  1. What does AWS stand for?

    Amazon Web Services

  2. What is AWS Data Pipeline used for?

    To move data between: storage services on the AWS public cloud, and on-premises resources.

  3. Name 4 ways to access Data Pipeline.

    Access to the Data Pipeline is available through: AWS Management Console, AWS CLI - the command-line interface, or AWS SDKs - language specific APIs Query API - low-level APIs that you call using HTTPS requests. Links: https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/what-is-datapipeline.html#accessing-datapipeline

  4. What is AWS SDK for Python called?

    Boto3. Links: https://aws.amazon.com/sdk-for-python

  5. What's a pipeline? response = client.create_pipeline( name='string', uniqueId='string', description='string', tags=[ { 'key': 'string', 'value': 'string' }, ] )

    A pipeline is the AWS Data Pipeline resource that contains the definition of the dependent chain of data sources, destinations, and predefined or custom data processing activities required to execute your business logic.

  6. What is a data node?

    Defines the location and type of data that a pipeline activity uses as input or output. AWS Data Pipeline supports the following types of data nodes: DynamoDB table - that contains data for HiveActivity or EmrActivity to use. SqlDataNode - An SQL table and database query that represent data for a pipeline activity to use. RedshiftDataNode - An Amazon Redshift table that contains data for RedshiftCopyActivity to use. S3DataNode - An Amazon S3 location that contains one or more files for a pipeli…

  7. What does Amazon S3 stand for?

    Amazon Simple Storage Service

  8. What is bucket VS object in Amazon S3?

    Amazon S3 is cloud storage for the internet. To upload your data (photos, videos, documents etc.), you first create a bucket in one of the AWS Regions. You can then upload any number of objects to the bucket.

  9. What is an activity?

    Action that AWS Data Pipeline initiates on your behalf as part of a pipeline. Example activities are EMR or Hive jobs, copies, SQL queries, or command-line scripts.

  10. What is a precondition: for data source VS for activity?

    data source - precondition check must complete successfully before any activities consuming the data source are launched. activity - precondition check must complete successfully before the activity is run. A precondition is a readiness check that can be optionally associated with a data source or activity. This can be useful if you are running an activity that is expensive to compute, and should not run until specific criteria are met.

  11. What is a schedule?

    Schedules define when your pipeline activities run and the frequency with which the service expects your data to be available. All schedules must have a start date and a frequency; for example, every day starting Jan 1, 2013, at 3pm. Schedules can optionally have an end date, after which time the AWS Data Pipeline service does not execute any activities. When you associate a schedule with an activity, the activity runs on it. When you associate a schedule with a data source, you are telling the…