Commit 20a18d7f by Arne de Wall

initial commit

parent f7e6972c
# BigEarthNet-S1 Tools
This repository contains tools for processing the BigEarthNet Sentinel-1 (denoted as BigEarthNet-S1) benchmark archive. For the detailed explanation about the archive and downloading the archive, you can visit the website of BigEarthNet, [http://bigearth.net/](http://bigearth.net/).
* Tools are written in Python at the moment.
* All the codes are tested in Python 2.7 and 3.5 on Ubuntu 16.04 at the moment.
* To use the tools, you may need to first download the archive and/or the lists of image patches with seasonal snow and cloud&shadow from the website.
List of Tools and How to Use Them
-----
#### Python script to read Sentinel-1 image patches in the BigEarthNet-S1 ([read_patch.py](https://gitlab.tubit.tu-berlin.de/rsim/BigEarthNet-S1_tools/blob/master/scripts/read_patch.py))
This example script reads GeoTIFF files for each radar band of a Sentinel-1 image patch in the BigEarthNet-S1 archive. The script is capable of reading either all spectral bands of one patch folder (-p option) or all bands for all patches (-r option). After reading files, Sentinel-1 image values can be used as numpy array for further purposes.
* Examples:
```
python read_patch.py -p BigEarthNet-S1-v1.0/S2A_MSIL2A_20171104T095201_5_43/
python read_patch.py -r BigEarthNet-S1-v1.0/
```
* Requirements for running:
The BigEarthNet-S1 archive should be downloaded first. Either [GDAL](https://pypi.org/project/GDAL/) or [rasterio](https://github.com/mapbox/rasterio/) package should be installed. To use the resulting pixel values, [NumPy](http://www.numpy.org/) package can also be installed.
#### Python script to extract names and download links of the Sentinel-1 Level-1C GRD scenes that were used to create the BigEarthNet-S1 image patches ([extract_tile_names_and_links.py](https://gitlab.tubit.tu-berlin.de/rsim/BigEarthNet-S1_tools/blob/master/scripts/extract_tile_names_and_links.py))
This script creates a csv file whose each row contains the patch name, corresponding Sentinel-1 GRD scene filename and tile download link. You can directly use the resulting file from [files/tile_names_and_links.csv](https://gitlab.tubit.tu-berlin.de/rsim/BigEarthNet-S2_tools/blob/master/files/tile_names_and_links.csv) without running the script.
* Examples:
```
python extract_tile_names_and_links.py -r BigEarthNet-S1-v1.0/ -u <COPERNICUS_HUB_USERNAME> -p <COPERNICUS_HUB_PASSWORD> -o tile_names_links.csv
```
* Requirements for running:
The BigEarthNet-S1 archive should be downloaded first. [sentinelsat](https://github.com/sentinelsat/sentinelsat) package should be installed to search tiles within Copernicus Open Access Hub via sentinelsat API.
# License
The BigEarthNet Archive is licensed under the **Community Data License Agreement – Permissive, Version 1.0** ([Text](https://cdla.io/permissive-1-0/)).
The code in this repository to facilitate the use of the archive is licensed under the **MIT License**:
```
MIT License
Copyright (c) 2020 The BigEarthNet Authors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```
# -*- coding: utf-8 -*-
# This script creates a csv file that contains the Sentinel-2 1C level tile
# names and their download links of the BigEarthNet patches.
#
# The script is capable of searching corresponding tiles of the BigEarthNet
# patches in Copernicus Open Access Hub via sentinelsat API.
#
# python extract_tile_names_and_links.py --help can be used to learn
# how to use this script.
#
# Date: 14 May 2019
# Version: 1.0.2
# Usage: extract_tile_names_and_links.py -u USERNAME -p PASSWORD
# -r ROOT_FOLDER -o OUT_PATH
from __future__ import print_function
from datetime import datetime, timedelta
import json
import os
import argparse
import csv
parser = argparse.ArgumentParser(
description='This script extracts the tile names and ' +
'their download links for the BigEarthNet image patches')
parser.add_argument('-u', '--username', dest='username', help='username ' +
'of a Copernicus Open Access Hub account')
parser.add_argument('-p', '--password', dest='password', help='password ' +
'of a Copernicus Open Access Hub account')
parser.add_argument('-r', '--root_folder', dest='root_folder',
help='root folder path contains multiple patch folders')
parser.add_argument('-o', '--out_path', dest='out_path',
help='file path of the output csv file whose each ' +
'row contains patch name, tile name and download link')
args = parser.parse_args()
if (not args.username) or (not args.password) or (not args.out_path):
print('ERROR: need to specify -u, -p, -u and -o arguments')
print('INFO: For help, python extract_tile_names_and_links.py --help')
exit()
elif args.root_folder:
if not os.path.exists(args.root_folder):
print('ERROR: folder', args.root_folder, 'does not exist')
print('INFO: need to download archive first')
exit()
root_folder = os.path.realpath(args.root_folder)
# Checks the existence of required python packages
try:
from sentinelsat import SentinelAPI
except ImportError:
print('ERROR: please install sentinelsat')
print('INFO: for how to install sentinelsat, see ' +
'https://github.com/sentinelsat/sentinelsat')
exit()
try:
from os import scandir
except ImportError:
try:
from scandir import scandir
except ImportError:
print('ERROR: Either use Python 3 or ' +
'install scandir package for Python 2')
exit()
# Creates SentinelAPI object with username and password
api = SentinelAPI(args.username, args.password)
dict_iter_items = (lambda d: d.iteritems()) if hasattr(
dict, 'iteritems') else (lambda d: iter(d.items()))
search_res = {}
# Creates resulting csv file
with open(os.path.realpath(args.out_path), 'w') as fp:
csv_writer = csv.writer(fp)
# Iterates over patch folders
for folder in scandir(root_folder):
send_query = False
patch_folder_path = os.path.realpath(folder.path)
patch_name = os.path.basename(patch_folder_path)
labels_metadata_path = os.path.join(patch_folder_path,
patch_name + '_labels_metadata.json')
# Reads labels_metadata json file
with open(labels_metadata_path, 'r') as f:
labels_metadata = json.load(f)
#S1A_IW_GRDH_1SDV_20180327T064301_20180327T064326_021197_024730_016F
scene_identifier = labels_metadata["scene_source"]
scene_filename = scene_identifier + ".SAFE"
if not scene_filename in search_res:
query_kwargs = {
'filename': scene_filename,
'platformname': 'Sentinel-1'
}
results = api.query(**query_kwargs)
link = results.popitem()[1]["link"]
search_res[scene_filename] = link
else:
link = search_res[scene_filename]
csv_writer.writerow([patch_name, scene_filename, link])
\ No newline at end of file
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# This script reads GeoTIFF files each of which is for one spectral
# band of a Sentinel-21 image patch in the BigEarthNet Archive.
#
# The script is capable of reading either all spectral bands of one patch
# folder (-p option) or all bands for all patches (-r option).
#
# After reading files, Sentinel-1 image patch values can be used as numpy array
# for further purposes.
#
# read_patch --help can be used to learn how to use this script.
#
# Date: 22 Dec 2020
# Version: 1.0.1
# Usage: read_patch.py [-h] [-p PATCH_FOLDER] [-r ROOT_FOLDER]
from __future__ import print_function
import argparse
import os
parser = argparse.ArgumentParser(
description='This script reads the BigEarthNet image patches')
parser.add_argument('-p', '--patch_folder',
dest='patch_folder', help='patch folder path')
parser.add_argument('-r', '--root_folder', dest='root_folder',
help='root folder path contains multiple patch folders')
args = parser.parse_args()
# Checks the existence of patch folders and populate the list of patch folder paths
folder_path_list = []
if args.root_folder:
print('INFO: -p argument will not be considered since -r argument is defined')
if not os.path.exists(args.root_folder):
print('ERROR: folder', args.root_folder, 'does not exist')
exit()
else:
folder_path_list = [i[0] for i in os.walk(args.root_folder)][1:-1]
if len(folder_path_list) == 0:
print('ERROR: there is no patch directories in the root folder')
exit()
elif not args.patch_folder:
print('ERROR: at least one of -p and -r arguments is required')
exit()
else:
if not os.path.exists(args.patch_folder):
print('ERROR: folder', args.patch_folder, 'does not exist')
exit()
else:
folder_path_list = [args.patch_folder]
# Checks the existence of required python packages
gdal_existed = rasterio_existed = georasters_existed = False
try:
import gdal
gdal_existed = True
print('INFO: GDAL package will be used to read GeoTIFF files')
except ImportError:
try:
import rasterio
rasterio_existed = True
print('INFO: rasterio package will be used to read GeoTIFF files')
except ImportError:
print('ERROR: please install either GDAL or rasterio package to read GeoTIFF files')
exit()
# Spectral band names to read related GeoTIFF files
band_names = ['VV', 'VH']
# Reads spectral bands of all patches whose folder names are populated before
for patch_folder_path in folder_path_list:
patch_folder_path = os.path.realpath(patch_folder_path)
patch_name = os.path.basename(patch_folder_path)
for band_name in band_names:
# First finds related GeoTIFF path and reads values as an array
band_path = os.path.join(
patch_folder_path, patch_name + '_' + band_name + '.tif')
if gdal_existed:
band_ds = gdal.Open(band_path, gdal.GA_ReadOnly)
raster_band = band_ds.GetRasterBand(1)
band_data = raster_band.ReadAsArray()
elif rasterio_existed:
band_ds = rasterio.open(band_path)
band_data = band_ds.read(1)
# band_data keeps the values of band band_name for the patch patch_name
print('INFO: band', band_name, 'of patch', patch_name,
'is ready with size', band_data.shape)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment