Table of Contents
UPORTO DATA ARCHIVE (UDA)
Perdigão's datasets repository using the THREDDS Data Server (TDS)
- most recent info at the docs.google.com/document
- [2020.05.23] TDS password removed
- [2020.05.23] cron jobs removed and the mirroring process stopped DTU account closing status
1. Using the UDA
The UPORTO Data Archive (UDA) may be accessed using the THREDDS Data Server (TDS) by providing the same credentials as in the UCAR ftp site (perdigao / Bxxxxx!).
WindsP App users may explore the UPORTO Data Archive (UDA) but, when they request access to data or meta-data that is in the Data Archive, they have to provide the TDS credentials (during the embargoed period of 12 months).
uporto_data_archive_20171120_.pdf | uporto_data_archive_2020-05-23.pdf
2. UDA contents
Export | Size | Last 24 hours |
---|---|---|
DLR | 1.2 TiB | 2.32553 GiB |
DTU | 2.0 TiB | 0.000537872 GiB |
INEGI | 301 MiB | 0 GiB |
UCAR | 1.6 TiB | 6.24173 GiB |
WINDFORS | 3.2 GiB | 0 GiB |
Summary on 20-03-2018 00:00. For more info see the datalogs details.
perdigao@windsptds:/data$ tree -d -L 3 /data/perdigao data/perdigao ├── dlr │ ├── HATPRO_level-1 │ │ ├── 201704 │ │ ├── 201704_quicklook │ │ ... │ ├── HATPRO_level-2 │ │ ├── 201704 │ │ ├── 201704_quicklook │ │ ... │ ├── HATPRO_surface-met │ │ ├── 201704 │ │ ├── 201704_quicklook │ │ ... │ ├── mcs_data │ │ ├── 20170430095732 │ │ ... │ ├── netcdf_lidar │ │ ├── DLR85 │ │ ├── DLR86 │ │ ├── DLR89 │ │ └── readme.tx │ │── raw_data │ │ ├── DLR85 │ │ ├── DLR86 │ │ └── DLR89 │ └── sound │ ├── mic1 │ ├── mic2 │ ├── mic3 │ ├── mic4 │ ├── mic5 │ └── microphone_position.txt ├── dtu │ ├── data │ │ ├── DTU_Leica_Scanning │ │ ├── DTU_Mast_Data │ │ └── DTU_WindScanner │ ├── docs │ ├── landscape │ ├── photos │ └── plots │ └── DTU_WindScanner ├── inegi │ ├── EnerconWindTurbine │ ├── LeosphereWindcube │ │ └── 01_RawData │ └── LidarAerialSurvey_RawData │ ├── Images │ ├── PerdigaoTurbineTopView.pdf │ ├── PointCloud │ └── Portugal Laserscanning Report.pdf ├── ucar │ ├── arl │ │ ├── ARL_Scanning_Lidar_George_Site │ │ ├── ARL_Scanning_Lidar_Lionstail_Site │ │ └── ARL_Scintillometer │ ├── colorado │ │ └── CU_Lidar │ ├── eol │ │ └── WV-DIAL │ ├── isfs │ │ ├── hr_noqc_geo │ │ └── noqc_geo_notiltcor │ ├── ncas │ │ └── NCAS_profiler │ ├── notredame │ │ ├── UND_Ceilometer │ │ ├── UND_Radiosonde │ │ ├── UND_Scanning_Lidar_Lionshead_Site │ │ ├── UND_Scanning_Lidar_MI6_Site │ │ ├── UND_Scanning_Lidar_Orange_Site │ │ └── UND_SODAR_RASS │ └── oklahoma │ ├── CLAMPS_AERI │ ├── CLAMPS_MWR │ └── CLAMPS_Scanning_Lidar └── windfors ├── 2017 │ ├── 201704 │ ├── 201705 │ └── 201706 └── cross └── 2017 441 directories, 8 files
3. Building the UDA
Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in their catalogue in UDA (rsync exports).
Available exports:
nejoco@VIND-pNEWA04:~> rsync -rdt rsync://windsptds.fe.up.pt test RSYNC test archive RSYNC UDA FILES (read only) ucar RSYNC UCAR FILES dtu RSYNC DTU FILES inegi RSYNC INEGI FILES dlr RSYNC DLR FILES windfors RSYNC WindForS FILES
4. Upload DTU data
UPORTO (as nejoco@login.neweuropeanwindatlas.eu
) uses the UDA export dtu@windsptds.fe.up.pt::dtu
to sync data collected by DTU.
First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job:
/usr/bin/rsync -az –delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu
.
Later the –delete
option was removed and some directories excluded to achieve the Perdigão Data Archive at UDA.
- crons
# DTU data sync to UDA, At minute 31 past every 4th hour #31 */4 * * * /usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1 31 */4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1
cat ~nejoco/sync-exclude-list archive/ data/DLR_WindScanner/
5. Upload UCAR data
UCAR uses the UDA export ucar@windsptds.fe.up.pt::ucar
to copy NCAR/EOL ISFS data.
6. Upload DLR data
DLR uses the UDA export dlr@windsptds.fe.up.pt::dlr
to maintain the DLR data.
7. Upload INEGI data
INEGI uses the UDA export inegi@windsptds.fe.up.pt::inegi
to maintain the ENERCON data and "Lidar Aerial Survey Data".
8. Upload WindsForS data
WindsForS uses the UDA export windfors@windsptds.fe.up.pt::windfors
to maintain the WindForS data.
9. Mirror UCAR ftp site
Preliminary data at the ftp site (ARL, Notre dame, …) uploaded with:
#! /bin/sh dir=arl source=ftp://ftp.eol.ucar.edu/pub/data/incoming/perdigao/uda/$dir destination=/data/perdigao/ucar nohup wget -m -nH --cut-dirs=5 -P $destination $source >| /dev/null 2>&1 &
Afterwards it is verified by running in the ftp site:
#! /bin/sh dir=arl cd <ftp-root>/incoming/perdigao/uda export RSYNC_PASSWORD=t****YLa**** archive=ucar@windsptds.fe.up.pt::ucar/$dir rsync -avz --delete --dry-run $dir $archive
10. Mirror UDA to DTU
The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: uda@windsptds.fe.up.pt::archive/
, using a cron job.
- crons
# UDA archive to DTU, At midnight every day 0 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1
- sync-uda.sh
#! /bin/sh # the Perdigao root at NEWA storage perdigao=/newa/WP2/PERDIGAO # the archive root archive=$perdigao/archive # the actual size of the archive echo "Total du of $archive:" du -ks $archive # the UDA readonly password export RSYNC_PASSWORD=-password- # catalogues to sync CATALOGS="dlr inegi ucar windfors" for c in $CATALOGS; do # mirror catalog from the version at UDA (UPORTO) echo; echo "$(tr [a-z] [A-Z] <<< "$c"):" #cmd="rsync -avz uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/" cmd="rsync -avz --delete uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/" echo "$cmd..." # do it $cmd done # catalog structure echo tree -L 2 $archive # total space usage for each archive echo du -khs $archive/* # the final size of the archive echo echo "Total du of $archive:" du -ks $archive # the end echo echo "Done."
The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to avoid using a duplication 2.0 TiB of storage).
/newa/WP2/PERDIGAO/archive ├── dlr │ ├── HATPRO_level-1 │ ├── HATPRO_level-2 │ ├── HATPRO_surface-met │ ├── mcs_data │ ├── netcdf_lidar │ └── raw_data ├── dtu │ ├── DTU_Leica_Scanning -> /newa/WP2/PERDIGAO/data/DTU_Leica_Scanning │ ├── DTU_Mast_Data -> /newa/WP2/PERDIGAO/data/DTU_Mast_Data │ └── DTU_WindScanner -> /newa/WP2/PERDIGAO/data/DTU_WindScanner ├── inegi │ ├── EnerconWindTurbine │ ├── LeosphereWindcube │ └── LidarAerialSurvey_RawData ├── ucar │ ├── isfs │ ├── iss │ └── ncas └── windfors ├── 2017 └── cross
11. Current status
snap at 22/12/2017 | snap at 20/03/2018 | snap at 23/05/2020
— Correia Lopes 2017/11/17 11:10