Call Rest API from Airflow

Apache Airflow is a powerful tool for managing workflows and tasks. One of the key features of Airflow is its ability to interact with external APIs, which can be used to trigger DAGs, check the status of tasks, and more. However, as we will discuss in this article, Amazon Managed Workflows for Apache Airflow (MWAA) does not support the classic Airflow REST API endpoints like `/dags` or `/variables`.

Instead, MWAA provides a CLI endpoint that allows you to run Airflow CLI commands remotely. This can be useful for tasks that require interacting with your MWAA environment, such as triggering DAGs or checking the status of tasks.

In this article, we will discuss how to call rest API from Airflow. We will cover the basics of using the Airflow CLI and demonstrate how to use it to interact with an external API.

To get started, you will need to have Apache Airflow installed on your machine. You can install Airflow by following the instructions on the official Airflow website.

Once you have Airflow installed, you will need to configure the Airflow CLI. This involves setting up a connection to your MWAA environment and specifying the endpoint that you want to use.

The first step in configuring the Airflow CLI is to define an HTTP connection. To do this, go to the Airflow homepage and click on the 'Admin' button. Then, select 'Connections' from the dropdown menu and click on the 'Add Connection' button.

In the 'Add Connection' dialog box, specify the connection parameters as shown in Image 1 below:

'Image 1 - Define the HTTP connection in Airflow'

Once you have defined the HTTP connection, you can use it to run Airflow CLI commands. For example, you can use the `airflow` command to trigger a DAG or check the status of a task.

To do this, open a terminal window and navigate to your Airflow installation directory. Then, type the following command:

```

airflow dags trigger

```

Replace `` with the ID of the DAG that you want to trigger.

Alternatively, you can use the `airflow` command to check the status of a task. For example:

```

airflow tasks status

```

Replace `` with the ID of the task that you want to check.

In addition to using the Airflow CLI, you can also use Python code to interact with an external API. This involves creating a Python script that uses the `requests` library to make HTTP requests to the API.

For example, you can create a Python script that makes a GET request to an API and stores the response in a variable:

```

import requests

response = requests.get('https://example.com/api/data')

data = response.json()

```

You can then use this data in your Airflow DAG by creating a PythonOperator that calls the script.

In conclusion, calling rest API from Airflow is a powerful way to interact with external APIs and automate tasks. By using the Airflow CLI or writing Python code, you can easily trigger DAGs, check the status of tasks, and more. In this article, we have covered the basics of using the Airflow CLI and demonstrated how to use it to interact with an external API.

As always, good practice when working with external APIs is to check if they are available first. This involves creating a separate task that checks if the API is running before attempting to make requests to it.

In the next article, we will discuss Airflow and Amazon S3 connection in more detail. Stay tuned!

Call rest api from airflow