artus.evaluate_model package

Submodules

artus.evaluate_model.coco_stats module

The module provides basic descriptive statisitics on a coco file (coco.json).

The COCOStats class can be used to get basic statistics on a coco file after deleting images without annotations in a dataset and removing under-representated classes with the min_nb_occurrences argument.

Typical usage examples:

cocostats = COCOStats(coco_path=’/a/path/to/coco.json’, min_nb_occurrences=50) nb_images = cocostats.get_nb_images() class_occ = cocostats.get_class_stats() cocostats.export_stats(export_path=’/path/to/dir/export.csv)

class artus.evaluate_model.coco_stats.COCOStats(coco_path, min_nb_occurrences=None)

Bases: object

A class to explore the basic stats of a coco file.

The COCOStats class can be used to get basic statistics on a coco file after deleting images without annotations in a dataset and removing under-representated classes with the min_nb_occurrences argument. Statistics can be printed or exported in a csv file.

coco_path

a path to a coco file

Type:

str

min_nb_occurrences

an integer that is the minimum number of occurrences of a

Type:

int

class to be kept in the dataframe.
export_stats(export_path)

Export the number of occurrences per classin csv format.

Statistics are exported after removing images without annotations and (optional) under-representated classes.

Parameters:

export_path (str) – csv path to export the class counts.

get_class_stats()

Print the number of occurrences per classes and number of classes in dataframe after removing images without annotations and under-representated classes.

get_nb_images()

Print the number of images in dataframe after removing images without annotations and under-representated classes.

process_coco(coco_path, min_nb_occurrences)

Read coco file and removes the samples without annotations and underrepresetned classes if needed.

Parameters:

  • coco_path (str) – a path to a coco file

  • min_nb_occurrences (int) – an integer that is the minimum number of occurrences of a

  • dataframe (class to be kept in the) –

  • classes. (remove under representated) –

Returns:

A dataframe where one row is one label.

artus.evaluate_model.coco_stats.rm_min_classes(dataset, min_nb_occurrences)

Remove classes that do not reach the minimum number of occurences per class.

Allows to get rid of under-representated in a dataset before splitting a coco file for deep learning purposes.

Parameters:

  • dataset (of a class to be kept in the) – a coco file read with importer.ImportCoco() from pylabel. This function

  • label (returns a dataframe where one row is one) –

  • min_nb_occurrences (int) – an integer that is the minimum number of occurrences

  • dataset –

Returns:

The same dataframe format but classes that have less than the minimum number of occurrences are removed.

artus.evaluate_model.coco_stats.rm_tiles_without_annot(dataset)

Remove tiles without annotations.

Parameters:

  • dataset – a coco file read with importer.ImportCoco() from pylabel. This function

  • label (returns a dataframe where one row is one) –

Returns:

The same dataframe format but rows with empty column β€˜cat_name’ are removed.

artus.evaluate_model.evaluate module

Performs COCO-style evaluation on a trained deep learning model.

To evaluate the model, a config file (yaml format) must be provided to set the model parameters. Model is tested on the test_dataset indicated. The test set must be new to the model evaluated (not used during training process). Results are exported in csv format at the desired location.

Typical usage examples:

evaluate_model(config_path=’/path/to/config.yml’, csv_metrics_name=’models_metrics.csv’)

artus.evaluate_model.evaluate.evaluate_model(config_path, csv_metrics_name)

A function that reads a config file and execute a COCO evaluation on a trained model.

Config file must be written in yaml format and must follow the example here : https://github.com/6tronl/artus-examples/blob/main/configs/x101_allsites_species_overlapping25_tiles5000_ITER3000.yml The config file contains information needed to set the detectron2 evaluator based on the model trained. The model will be evaluated using the COCOEvaluator (https://detectron2.readthedocs.io/en/latest/modules/evaluation.html?highlight=COCOEvaluator#detectron2.evaluation.COCOEvaluator). A csv will be exported containing metrics about the model (AP50, AP, AP75, APl and AP per class). If a csv already exsists in the log directory mentionned in the config file, then results will be appended to it.

Parameters:

config_path – the path to a config file in yaml format

artus.evaluate_model.write_eval_results module

Export plots or csv after COCO-style evaluation of a trained deep learning model.

A csv file can be exported or merged to an already existed one. This csv contains AP, AP50, AP75, APm, APl and AP-perclass. Plots can also be exported after having exporting this csv. The plots created are multiple histograms where each bar stands for a model.

class artus.evaluate_model.write_eval_results.ModelsMetricsFormat(session, eval_results, export_dir, csv_name)

Bases: object

A class that writes the evaluation results in a CSV format.

If the CSV already exists, it will merge the results at the last row creating a dataframe where each row is one model evaluated along with their metrics.

session

the name of the directory containing a model_final.pth to evaluate

Type:

str

eval_results

the evaluation results from detectron2.trainer.test()

Type:

dict

export_dir

the path to the directory where metrics will be exported or where previous metrics.csv are saved.

Type:

str

csv_name

a csv name to save the results of evaluation

Type:

str

check_csv_exists()

Check if a csv containing model’s metrics exists or not.

If csv already exists,results of this session will be appended to this csv. Otherwise, a csv will be created.

to_pandas()

Take results from COCOevaluator and format it into a pandas dataframe.

write_to_csv()

Export results to csv format at the csv_path

class artus.evaluate_model.write_eval_results.ModelsMetricsPlots(csv_metrics_path, export_dir, plot_name, title)

Bases: object

A class that plots the results of different models trained.

The plots created are multiple histograms where each bar stands for a model. Plots are more relevant if the csv contains multiple models metrics.

csv_metrics_path

the path to a csv dataframe.

Type:

str

export_dir

the directory where the interactive plots will be exported

Type:

str

plot_name

the name of the plot file with .html extension

Type:

str

title

title to display on plot

Type:

str

export_plots()

Export the plots in html format. Plots are interactives.

plot_metrics()

Plot the metrics for all the models described in the csv file.

Plots created are facet bar plots. One facet is created per variables (AP50, Ap-class1 etc…) and each model precision is described by one bar.

Returns:

A plotly interactive barplot that can be seen with Python(fig.show())

process_csv()

Read csv and pivot the dataframe.

Create a dataframe with one row for each variable and each model.

Module contents