# Explanation

- Gitub repository with all the code that produces these files https://github.com/crrel/mammoth-a-tls
- [Scripts for livox registration](https://github.com/CRREL/Mammoth-A-TLS/tree/master/Mammoth/Scripts/livox)
- [Scripts for VZ scans](https://github.com/CRREL/Mammoth-A-TLS/tree/master/Mammoth/Scripts/Scans)
- The livox transformation were completed on 2025-10-13 

- `/laz/` (LIVOX and Riegl VZ data converted to laz file format and georegistered.)
    - `*.frame.laz` (full frame of the riegl)
    - `*.line.laz` line scan 
    - `*.spectralon-gondola.line.laz` line scan aiming at spectralon at McCoy
    - `*.spectralon-line.laz` line aimed at spectralon behind pillow
    - `*.extended.line.laz`
    - `*.LIVOX1/2/3.REG.laz` scans of LIVOX 1/2/3 (the `REG` meaning: aligned / registered / same CRS as (RIEGL))    
- `/processed/` (produced by Sven Sorhus)
    - sesame vs cues
    - contains dtms and classified
- `/original_scans/`
    - output unregisterd livox (LIVOX1/2/3.laz)
    - Raw output of the VZ500 (zipped rxp. rxp is riegl proprietary file format)
- `/390i/`
    - `/transformed/`: data collected from 2011 to 2020 with the VZ390i in laz
    - `/las/`: data collected from 2011 to 2020 with the VZ390i as laz
    - `/2011/` to `/2020/` data collected from 2011 to 2020 with the VZ390i in 3dd files
- `/pipeline/`
    - configs for the pipelines

# Riegl VZ-400i locations
- Deployed at the Sesame site until the afternoon of 10/21/2020 -> data files prior to that date relate to Sesame
- Moved to the Dozier Snow Site on 10/21/2020
- Period of missing data from 02/10/2022 to 01/21/2023 - lidar was temporarily removed to be used in field campaign



# Footprints

![Footprints](lidar_foo.png)


# Data Sync between CRREL S3 and UCSB:
## Step 1. Configure  S3 remote

```bash
rclone config
```

named it `crrel_lidar_s3`. This is what you refer to in the rclone command later

## Step 2: Copy
```bash
rclone copy  -P --checksum crrel_lidar_s3:crrel-mammoth-lidarscans/processed/ /home/cues/data/lidar/processed --include "*"
rclone copy  -P --checksum crrel_lidar_s3:crrel-mammoth-lidarscans/laz/ /home/cues/data/lidar/laz --include "*"
rclone copy  -P --checksum crrel_lidar_s3:crrel-mammoth-lidarscans/original_scans/ /home/cues/data/lidar/original_scans --include "*"
```

- Copies only new or updated files.
- Never deletes local files.
- Ideal if you just want to “top up” your local directory with missing or newer files.

## Step 3: Sync (with cron)
`/home/cues/bin/crrel_lidar/crrel_s3_sync.sh`

... which syncs `processed/`, `lidar/`, and `original_scans/`.

Note: Since this scans all files for their checksums, this takes about 30 minutes for each folder (1.5 h in total). Should verify how much egress this costs at CRREL.


```bash
rclone sync -P --checksum s3:mybucket /local/data/ --include "*"
```

- Compares size, modification time, and hashes (MD5/ETag).
- Copies only files that are new or changed.
- Deletes local files not present on S3 (unless you add --ignore-existing or --update).

Flags:
```bash
-P	                        Show progress
--dry-run	                Preview what would happen
--checksum	                Compare files by checksum instead of size+mtime (more accurate, slightly slower)
--update	                Skip files that are newer on the destination
--create-empty-src-dirs	    Preserve empty directories from S3 (if needed)
--max-age 7d	            Only copy files modified in the last 7 days
```