Database

This is the user interface of the pipeline. It integrates all functions into a class to navigate the database and update it.

• Connecting to the relational database management system
• Creating the database with all necessary tables and relations
• Filling the static information
• Updating the dynamic data continuously

Examples:

Establish a connection and create the SQL database if it does not exist

>>> from pipeline import Database
>>> db = Database(**config)

Add cities (if not already present) and their airports

>>> db.add_cities(['Berlin', 'Hamburg', 'Munich'])

Fetch dynamic data and update it in the database

>>> db.fetch_weather()
>>> db.fetch_flights()

For more information, see the usage documentation.

Database(weather_api_key, rapid_api_key, reset=False, timezone='Europe/Berlin', **connection)

Class to maintain and update the database

Initialize the database with the necessary credentials

The SQL database will be created if it does not exist yet.

Parameters:

• weather_api_key (string) –

  API key for the weather API

• rapid_api_key (string) –

  API key for the rapid API

• reset (bool, default: False ) –

  If True, the database is recreated from scratch and all data is lost. Default is False

• timezone (string, default: 'Europe/Berlin' ) –

  Timezone to use for determining the date of the following day. This should be the local timezone of the base of operations from where the data is maintained. This setting is used for fetching the flight data. Default is 'Europe/Berlin'

• connection (dict, default: {} ) –

  Connection parameters for the MySQL database

Notes

The connection parameters should contain the following keys: host, port, user, password, database. All times are in UTC. To relate weather and flights data to local time, use the timezone information in the 'geo' table.

setup(reset=False)

Setup the database and tables

Parameters:

• reset (bool, default: False ) –

  If True, the database is recreated from scratch and all data is lost

add_cities(city_list)

Add cities to the database if they do not exist yet

Parameters:

• city_list (list) –

  List of cities to add to the database

See also

cities.scrape : Web scraping city data

add_airports()

Add airports for all cities to the database if they do not exist yet

fetch_population()

Fetch the population data for the cities in the database

Notes

Only add rows that contain new data

fetch_weather()

Fetch the weather data for the cities in the database

fetch_flights()

Fetch the flight data for the airports in the database