🛰 Train an AI model with geospatial data and use it to predict spatial occurrences

In this tutorial, we will learn how to handle raster images annotated with a vector (shapefile) file within a deep learning framework. We will train a deep learning model and use it to predict new annotations on unlabeled rasters.

🎬 Scenario

Imagine you are a research scientist and you have framed boats on satellite images with QGIS. At the end on the annotation process, you have GeoJSON files of annotations and your satellite images. As part of your project, you know that the expensive work you did annotating satellite images was only the tip of the iceberg because you now have dozens of satellite images from other years left to annotate (😩). Seeing the colossal work you have yet to accomplish, you sweat drops (😓). Fortunately, in a useful breath, you discover artus (😱)! But unfortunately you have no knowledge of artificial intelligence. That’s good, it’s not necessary, this tutorial will guide you to your holy grail! 😎

Input data needed

As explained previously, you must have a raster. The desired format is the tif format. There is no obligation to have a stallite image, you can absolutely work on an orthomosaic made using drone images, from photogrammetric models.

To train a supervised deep learning model, you must have annotations. You can either use annotations in a geospatial format (ideally geojson) or in a specialized deep learning format like the COCO format.

1 Prepare data for training

1. 1. Tile a heavy raster file

Depending on machine resources (such as GPU cache storage) available to user, the process to prepare the data for model trainings can be adapted. In most of the case, a raster file is large (more than 500 Mo) and cannot be supported as is in GPU memory for training a deep learning model. In this tutorial, we provide a technique to handle large raster by tiling large raster into smaller pieces. The size of the tiles (defined in pixels or in meters) is defined by the user according to his constraints (available machine resources, size of the annotated objects or sampled surface). Both wide geospatial raster data and related vector annotation data are split into a large numberof raster tiles (for instance, 500 x 500 pixels) along with smaller vector files sharing the exact same boundaries as the raster tiles (converted in GeoJSON files).

Tiling process on annotated orthomosaics (a). Tiles are cut according to a regular grid (b) and produces georeferenced tiles (c) with their matching annotations (d).

To clip our large raster into smaller tiles, we can choose different clipping grids. To train a deep learning model, we can choose to add an overlap between the tiles. This overlap makes it possible to generate additional tiles, to increase the number of annotations per class and is part of a data augmentation process.

Tiling strategies for georeferenced rasters : a regular grid (left) and an overlapping grid (right). Here, the overlapping grid is 50% vertical overlapping and 50% horizontal overlapping.

import artus.prepare.tile as tustile

annotation_path = '/path/to/your/vector/file.shp/OR/file.geojson'
raster_path = '/path/to/your/raster/file.tif'

tustile.clip_annotated_ortho(
    annotation_path,
    raster_path,
    matching_crs='4326', #set your EPSG code that matches both raster and vector,
    dest_dir='/path/to/the/export/dir/', #set the directory where tiles and annotations will be saved
    tuple_tile_size=(500,500), #set the tuple size (in pixels or meters)
    h_shift=0.5, #50% horizontal overlapping (as shown on the grids figure above)
    v_shift=0.5, #50% vertical overlapping (as shown on the grids figure above)
    annot_type='bbox' # example where annotations were bounding boxes, can be set to "segm" if you annotated polygons
)

1. 2. Convert annotations files into COCO format

To train a computer vision model, we need a standard format. For this, we have chosen the COCO format. The next step is therefore to convert the annotation tiles (geojson files created previously) into a single COCO file containing all the annotations.

import artus.prepare.transform_geojson_to_coco as tustransform
import glob

tiled_geojsons = glob.glob('/path/to/the/export/dir/geojsons/*.geojson')
tiled_ortho = glob.glob('/path/to/the/export/dir/*.tif')

tustransform.geojson_to_coco(
    tiled_geojsons=tiled_geojsons,
    tiled_ortho=tiled_ortho,
    coco_dest_path='/path/to/coco/export/directory/coco_annotations.json',
    feature=None)

1. 3. Data splitting

Data splitting in an important step when training a deep learning model. We will split the COCO file created into 3 coco files : a train dataset, a validation dataset and a test dataset to further evaluate the model.

import artus.prepare.coco_splitting as tusplit
import artus.evaluate_model.coco_stats as tustats

coco_path = '/path/to/coco/export/directory/coco_annotations.json'

If you have under represented classes in your annotations, you can set a minimal number of occurrences (min_nb_occurences) to remove classes that do not reach the threshold.

min_nb_occurences=50

You can also export some statistics on the classes distribution before the training process. This is optional but useful to get an idea of the annotation composition.

stats = tustats.COCOStats(coco_path, min_nb_occurences)
stats.get_class_stats()
stats.export_stats(export_path='/path/to/export/stats.csv') #optionnal : export the classes distribution in csv format

splitter = tusplit.COCOSplitter(
    coco_path=coco_path,
    export_dir='/path/to/coco/export/directory/',
    coco_train_name='coco_train.json',
    coco_test_name='coco_test.json',
    coco_val_name='coco_val.json',
    min_nb_occurrences=min_nb_occurences,
    train_pct=.8,
    val_pct=.1,
    test_pct=.1,
    batch_size=8
)

splitter.split_coco()

2 Train a deep learning model

2. 1. Configure config file and train

To configure the deep learning model that you will train, you must write a model configuration file. Examples are available in the ../models_config/ folder.

import artus.train.train as tustrain

config_path = '../../configs/x101_allsites_species_overlapping25_tiles1500_ITER3000.yml'

In the cell below, you will train a deep learning model. Depending on you data and on your machine ressources, this step can take several hours.

tustrain.train_model(config_path)

2. 2. Evaluate model

By running the code below, you open tensorboard which will help you to analyze training metrics and detect you have, for instance, overfitted training data.

import os
%load_ext tensorboard
%tensorboard --logdir='/path/to/logs/directory/'

When evaluating the model trained, you will get a csv that reports what is the performance of the trained model on your test dataset.

import artus.evaluate_model.evaluate as tuseval

tuseval.evaluate_model(
    config_path,
    csv_metrics_name='/path/to/export/models_metrics.csv')

You can also plot the evaluation results with artus. You will get several interactive barplots that can be useful to compare models when you tried different model configuration.

import plotly.express as px
import pandas as pd
import artus.evaluate_model.write_eval_results as tusevalplot

plots = tusevalplot.ModelsMetricsPlots(
    csv_metrics_path = '/home/justine/Documents/G2OI/collaborations/brianna/logs/models_metrics.csv', #the results from evaluate_model.ipynb
    export_dir = '/path/to/export/plots/',
    plot_name = 'metrics.html',
    title = 'Average precision'
)

fig = plots.plot_metrics()
fig.show()

Data type cannot be displayed: application/vnd.plotly.v1+json

plots.export_plots()

3 Use the model you trained to predict new data!

Now that you have a trained deep learning model, you can use it to predict new annotations on unlabeled rasters and export the results into a spatial format!

import yaml
import torch
import os
import artus.inference as tusinf
import artus.spatialize as tuspal
import fiftyone as fo
import fiftyone.utils.coco as fouc

raster_or_tiles_path = '/path/to/raster/or/tiles/directory/'

3. 1. Tile heavy raster

If your raster exceeds the cache memory then you should tile it first (overlapping is not needed in this step). If the raster you want to automatically annotate fits into memory, you can skip the next cell.

bounds = tustile.tile_ortho(
    ortho_path=raster_path,
    dest_dir='/path/to/tiles/directory/',
    tuple_tile_size=(500,500),
    h_shift=0.0,
    v_shift=0.0
)

3. 2. Deploy an unlabeled fiftytone dataset

In artus package, we choose to work with fiftyone datasets to handle images and annotations formats. Fiftyone is an open source python package very relevant in deep learning frameworks. You can find all the features on their website.

dataset = tusinf.deploy_unlabeled_dataset.create_or_load_dataset(
    dataset_name=os.path.basename(config_path), #add a name for your fiftyone dataset
    dataset_type='unlabeled',
    images_path=raster_or_tiles_path,
    label_type='detections')

dataset.persistent = True #optional : save dataset on your machine for further exploration

dataset.save()

print(dataset)

dataset.compute_metadata()

3. 3. Predict new annotations thanks to AI!

We first call the trained model an load it (this is the predictor), then we predict new labels on unlabeled rasters with the predictor. In a yaml file, we stored the classes that the model can predict. You can write a yaml file or add a python list directly.

device = ("cuda" if torch.cuda.is_available() else "cpu")

#Load model's classes
model_classes = '../../configs/model_classes_species.yml'

with open(model_classes) as f:
    model_classes = yaml.load(f, Loader=yaml.FullLoader)

predictor = tusinf.predict.build_predictor(config_path, device)

dataset = tusinf.predict.add_predictions_to_dataset(
    dataset=dataset,
    predictor=predictor,
    device=device,
    classes=model_classes['species_classes'], #the list of a class names or config file containing the list
    predictions_field='predictions',
    nms_threshold=0.5)

3. 4. Export the results into a spatial format! 🎉

geojson_exporter = tuspal.GeoFiftyoneExporter(
    export_dir='/path/to/export/dir',
    label_type='polylines', #can be 'detections' for bbox or 'polylines' for segmentation masks
    epsg_code='4326', #set the destination CRS adapted to your data
    dest_name='geospatial_predictions.geojson'
)

dataset.export(
    dataset_exporter=geojson_exporter,
    label_field='predictions',
    export_media=False
    )