The proposed CLI tool is authored to make creating and deployment of airflow projects faster and smoother. As of now, there is no tool out there that can empower the user to create a boilerplate code structure for airflow projects and make development deployment of projects seamless.
Python 3.5
Docker
Getting Started
1. Installation
Create a new python virtualenv. You can use the following command.
(source / path_to_venv / bin / activate 2. Initialize a new afctl project.
The project is created in your present working directory. Along with this a configuration file with the same name is generated in / home / .afctl_configs directory.
afctl init (>
Eg.
The following directory structure will be generated
├── deployments │ └── project_demo-docker-compose.yml Rations migrations ├── plugins ├── project_demo ├── ├── commons Ags ags dags ├── requirements.txt └── tests
If you already have a git repository and want to turn it into an afctl project. Run the following command: –
3. Add a new module in the project.
afctl generate module -n (
The following directory structure will be generated:
afctl generate module -n first_module afctl generate module -n second_module ├── deployments │ └── project_demo-docker-compose.yml Rations migrations ├── plugins ├── project_demo ├── ├── commons Ags ags dags _ ├── first_module └── └── second_module ├── requirements.txt └── tests _ First_module _ Second_module 4. Generate dag
afctl generate dag -n (> -m
) ( The following directory structure will be generate:
afctl generate dag -n new -m first_module ├── deployments │ └── project_demo-docker-compose.yml Rations migrations ├── plugins ├── project_demo ├── ├── commons Ags ags dags _ ├── first_module │ │ └── new_dag.py └── └── second_module ├── requirements.txt └── tests _ First_module _ Second_module The dag file will look like this:
from airflow (import DAG from datetime (import datetime, timedelta default_args
=
{
' owner ' : ' project_demo
,
# 'depends_on_past':, # 'start_date':, # 'email':, # 'email_on_failure':, # 'email_on_retry':, # 'retries': 0 } dag =(DAG) dag_id
(') new
(default_args
= default_args,
schedule_interval
' @ once '
5. Deploy project locally
You can add python packages that will be required by your dags in requirements.txt . They will automatically get installed.
To deploy your project, run the following command (make sure docker is running):
If you do not want to see the logs, you can run
This will run it in detached mode and won't print the logs on the console.
You can access your airflow webserver on browser at (localhost:
6. Deploy project on production
Here we will be deploying our project to (Qubole) . Sign up at us.qubole.com.
add git-origin and access-token (if want to keep the project as private repo on Github) to the configs. See how
Push the project once completed to Github.
Deploying to Qubole will require adding deployment configurations.
afctl config add -d qubole -n -e ( env> -c () This command will modify your config file. You can see your config file with the following command:
For example -
afctl config add -d qubole -n demo -e https://api.qubole.com -c airflow _ -t khd (djs3) To deploy run the following command afctl deploy qubole -n
name The following video also contains all the steps of deploying project using afctl -
https: //www.youtube.com/watch?v=A4rcZDGtJME&feature=youtu.be
Manage configurations
The configuration file is used for deployment contains the following information.
global : - airflow_version : - git : - - origin : - - access-token :
deployment : - qubole
: - - local : --- compose:
(airflow_version) can be added to the project when you initialize the project.
afctl init (>
global configs (airflow_version, origin, access-token) can all be added / updated with the following command:
afctl config global -o
git-origin > (-t) ( -v ( Usage
Commands right now supported are
init
config
deploy
list
generate
To learn more, run
Not yet ported for Windows.
Docker-compose file: https://github.com/puckel/docker-airflow
- Python 3.5
- Docker
Getting Started
1. Installation
Create a new python virtualenv. You can use the following command.
(source / path_to_venv / bin / activate 2. Initialize a new afctl project.
The project is created in your present working directory. Along with this a configuration file with the same name is generated in / home / .afctl_configs directory.
afctl init (>
Eg.
The following directory structure will be generated
├── deployments │ └── project_demo-docker-compose.yml Rations migrations ├── plugins ├── project_demo ├── ├── commons Ags ags dags ├── requirements.txt └── tests
If you already have a git repository and want to turn it into an afctl project. Run the following command: –
3. Add a new module in the project.
afctl generate module -n (
The following directory structure will be generated:
afctl generate module -n first_module afctl generate module -n second_module ├── deployments │ └── project_demo-docker-compose.yml Rations migrations ├── plugins ├── project_demo ├── ├── commons Ags ags dags _ ├── first_module └── └── second_module ├── requirements.txt └── tests _ First_module _ Second_module 4. Generate dag
afctl generate dag -n (> -m
) ( The following directory structure will be generate:
afctl generate dag -n new -m first_module ├── deployments │ └── project_demo-docker-compose.yml Rations migrations ├── plugins ├── project_demo ├── ├── commons Ags ags dags _ ├── first_module │ │ └── new_dag.py └── └── second_module ├── requirements.txt └── tests _ First_module _ Second_module The dag file will look like this:
from airflow (import DAG from datetime (import datetime, timedelta default_args
=
{
' owner ' : ' project_demo
,
# 'depends_on_past':, # 'start_date':, # 'email':, # 'email_on_failure':, # 'email_on_retry':, # 'retries': 0 } dag =(DAG) dag_id
(') new
(default_args
= default_args,
schedule_interval
' @ once '
5. Deploy project locally
You can add python packages that will be required by your dags in requirements.txt . They will automatically get installed.
To deploy your project, run the following command (make sure docker is running):
If you do not want to see the logs, you can run
This will run it in detached mode and won't print the logs on the console.
You can access your airflow webserver on browser at (localhost:
6. Deploy project on production
Here we will be deploying our project to (Qubole) . Sign up at us.qubole.com.
add git-origin and access-token (if want to keep the project as private repo on Github) to the configs. See how
Push the project once completed to Github.
Deploying to Qubole will require adding deployment configurations.
afctl config add -d qubole -n -e ( env> -c () This command will modify your config file. You can see your config file with the following command:
For example -
afctl config add -d qubole -n demo -e https://api.qubole.com -c airflow _ -t khd (djs3) To deploy run the following command afctl deploy qubole -n
name The following video also contains all the steps of deploying project using afctl -
https: //www.youtube.com/watch?v=A4rcZDGtJME&feature=youtu.be
Manage configurations
The configuration file is used for deployment contains the following information.
global : - airflow_version : - git : - - origin : - - access-token :
deployment : - qubole
: - - local : --- compose:
(airflow_version) can be added to the project when you initialize the project.
afctl init (>
global configs (airflow_version, origin, access-token) can all be added / updated with the following command:
afctl config global -o
git-origin > (-t) ( -v ( Usage
Commands right now supported are
init
config
deploy
list
generate
To learn more, run
Not yet ported for Windows.
Docker-compose file: https://github.com/puckel/docker-airflow
Create a new python virtualenv. You can use the following command.
The project is created in your present working directory. Along with this a configuration file with the same name is generated in / home / .afctl_configs directory.
Eg.
If you already have a git repository and want to turn it into an afctl project. Run the following command: –
The following directory structure will be generated:
afctl generate module -n first_module afctl generate module -n second_module ├── deployments │ └── project_demo-docker-compose.yml Rations migrations ├── plugins ├── project_demo ├── ├── commons Ags ags dags _ ├── first_module └── └── second_module ├── requirements.txt └── tests _ First_module _ Second_module4. Generate dag
afctl generate dag -n (> -m
) (The following directory structure will be generate:
afctl generate dag -n new -m first_module ├── deployments │ └── project_demo-docker-compose.yml Rations migrations ├── plugins ├── project_demo ├── ├── commons Ags ags dags _ ├── first_module │ │ └── new_dag.py └── └── second_module ├── requirements.txt └── tests _ First_module _ Second_module
The dag file will look like this:
from airflow (import DAGfrom datetime (importdatetime, timedelta default_args
={'owner ': ' project_demo
,#'depends_on_past':,#'start_date':,#'email':,#'email_on_failure':,#'email_on_retry':,#'retries': 0 } dag=(DAG)dag_id(') new
(default_args= default_args,schedule_interval
'@ once'5. Deploy project locally You can add python packages that will be required by your dags in requirements.txt . They will automatically get installed.
To deploy your project, run the following command (make sure docker is running):
If you do not want to see the logs, you can run
This will run it in detached mode and won't print the logs on the console.
You can access your airflow webserver on browser at (localhost:
6. Deploy project on production
Here we will be deploying our project to (Qubole) . Sign up at us.qubole.com.
add git-origin and access-token (if want to keep the project as private repo on Github) to the configs. See how
Push the project once completed to Github.
Deploying to Qubole will require adding deployment configurations.
afctl config add -d qubole -n -e ( env>-c () This command will modify your config file. You can see your config file with the following command:
For example -
afctl config add -d qubole -n demo -e https://api.qubole.com -c airflow _ -t khd (djs3)To deploy run the following command
afctl deploy qubole -n
nameThe following video also contains all the steps of deploying project using afctl -
https: //www.youtube.com/watch?v=A4rcZDGtJME&feature=youtu.be
Manage configurations The configuration file is used for deployment contains the following information.
global: - airflow_version : - git : - - origin : - - access-token :deployment : - qubole
: - - local : --- compose:
(airflow_version) can be added to the project when you initialize the project.
afctl init (>
global configs (airflow_version, origin, access-token) can all be added / updated with the following command:
afctl config global -o
git-origin> (-t) (-v (
Usage Commands right now supported are
init
config
deploy
list
generate
To learn more, run
Not yet ported for Windows.
Docker-compose file: https://github.com/puckel/docker-airflow
GIPHY App Key not set. Please check settings