Release v0.2.0

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.1.0
+current_version = 0.2.0
 commit = False
 allow_dirty = True
 

.flake8

@@ -4,7 +4,7 @@ import-order-style=pycharm
 application-import-names=pyflysight,tests
 extend-ignore=
     # pycodestyle
-    E226,
+    E203,E226,
     # flake8-annotations
     ANN002,ANN003,ANN101,ANN102,ANN204,ANN206,
 extend-exclude=

.github/workflows/pypi_release.yml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
     - name: Set up Python
       uses: actions/setup-python@v4
       with:
@@ -22,8 +22,14 @@ jobs:
         pip install poetry
         poetry build
 
-    - name: Publish package
+    - name: Publish to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1.5
       with:
         user: __token__
         password: ${{ secrets.pypi_api_token }}
+
+    - name: Upload wheel to release
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      run:
+        gh release upload ${{ github.event.release.tag_name }} ./dist/matplotlib_window-*.whl

.pre-commit-config.yaml

@@ -12,7 +12,7 @@ repos:
       - id: isort
         name: isort
   - repo: https://github.com/pycqa/flake8
-    rev: 7.0.0
+    rev: 7.1.0
     hooks:
       - id: flake8
         additional_dependencies:
@@ -35,6 +35,6 @@ repos:
       -  id: python-check-blanket-type-ignore
       -  id: python-use-type-annotations
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.4.8
+    rev: v0.5.0
     hooks:
       - id: ruff

CHANGELOG.md

@@ -1,2 +1,12 @@
 # Changelog
 Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html) (`<major>`.`<minor>`.`<patch>`)
+
+## [v0.2.0]
+### Added
+* Add parsing pipelines for Flysight V1 track data files
+* #8 Add parsing pipelines for Flysight V2 track & sensor data files
+* #11 Add trimming for Flysight V1 & V2 data files
+* #12 Add simplified CSV log export for Flysight V2 data files
+
+## [v0.1.0]
+Initial release - yay!

README.md

@@ -1,13 +1,10 @@
-# PyFlySight
-[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pyflysight/0.1.0?logo=python&logoColor=FFD43B)](https://pypi.org/project/pyflysight/)
+# pyflysight
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pyflysight/0.2.0?logo=python&logoColor=FFD43B)](https://pypi.org/project/pyflysight/)
 [![PyPI](https://img.shields.io/pypi/v/pyflysight?logo=Python&logoColor=FFD43B)](https://pypi.org/project/pyflysight/)
 [![PyPI - License](https://img.shields.io/pypi/l/pyflysight?color=magenta)](https://github.com/sco1/pyflysight/blob/main/LICENSE)
 [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/sco1/pyflysight/main.svg)](https://results.pre-commit.ci/latest/github/sco1/pyflysight/main)
-[![Code style: black](https://img.shields.io/badge/code%20style-black-black)](https://github.com/psf/black)
 
-Helper library for processing [FlySight GPS](https://www.flysight.ca/) flight logs.
-
-🚨 This is an alpha project. User-facing functionality is still under development 🚨
+Helper library for processing [FlySight GPS](https://www.flysight.ca/) flight logs. Support is provided for both the V1 and V2 hardware revisions, see: [Hardware Revisions](#hardware-revisions) for a description of the differences between the two.
 
 ## Installation
 Install from PyPi with your favorite `pip` invocation:
@@ -38,7 +35,9 @@ Commands:
 ```
 <!-- [[[end]]] -->
 
-## Usage
+## CLI Usage
+🚨 **NOTE:** CLI functionality is currently still under development. 🚨
+
 ### Environment Variables
 The following environment variables are provided to help customize pipeline behaviors.
 
@@ -64,6 +63,63 @@ Batch process a directory of FlySight log files.
 1. Case sensitivity is deferred to the host OS
 2. Recursive globbing requires manual specification (e.g. `**/*.CSV`)
 
+## Internal Data Representation
+See: [Data Format](/doc/data_format.md) for a more detailed breakdown of the expected data file formats.
+
+`pyflysight` exposes helpers for parsing your flight data from their CSV representation into a [Polars](https://docs.pola.rs/) dataframe. All dataframes derive an `elapsed_time` column, calculated as the delta of the row timestamp from the first seen timestamp of the data file. All GPS dataframes calculate a `groundspeed` column from the northing and easting GPS velocity components.
+
+### Flysight V1
+`pyflysight.flysight_proc.load_flysight` parses the GPS track log into a dataframe, inferring column names from the first row of the CSV; unit information is discarded.
+
+`pyflysight.flysight_proc.batch_load_flysight` wraps `pyflysight.flysight_proc.load_flysight` to batch load a directory of logs into a dictionary of dataframes. Because the FlySight V1 hardware groups logs by date & the log CSV name does not contain date information, the date is inferred from the log's parent directory name & the output dictionary is of the form `{log date: {log_time: DataFrame}}`.
+
+### Flysight V2
+Both the `SENSOR.CSV` and `TRACK.CSV` files share a similar data format: a series of header rows followed by a series of data rows. As the filenames suggest, `SENSOR.CSV` contains all of the onboard sensor information and `TRACK.CSV` contains the GPS track. `RAW.UBX` is the raw binary data stream from the onboard u-blox hardware; at this time this file is currently ignored.
+
+`pyflysight.flysight_proc.parse_v2_log_directory` is intended to be the main user interface, which wraps the data parsing pipelines and outputs an instance of the `pyflysight.flysight_proc.FlysightV2FlightLog` container class.
+
+## Interactive Log Trimming
+🚨 **NOTE:** Log trimming is currently only implemented for Flysight V2 data. 🚨
+
+`pyflysight.trim_app.windowtrim_flight_log` allows the user to interactively trim your Flysight log data using a plot of the elapsed time vs. pressure altitude. Trimmed data may be optionally written to disk into a directory of files, named by the current flight session. A CSV file is created for each sensor present, along with the GPS track and device information:
+
+```
+.
+└── device/
+    └── session/
+        ├── BARO.CSV
+        ├── device_info.json
+        ├── IMU.CSV
+        ├── TRACK.CSV
+        └── ...
+```
+
+Data trimming & CSV export is handled programmatically by methods of `pyflysight.flysight_proc.FlysightV2FlightLog`. The CSV output is designed to be round-trippable.
+
+## Hardware Revisions
+![hardware comparison](/doc/hardware_revs.png)
+
+Flysight released a new hardware revision in Summer 2023 with many improvements over the original, including the addition of additional sensors. In addition to the GPS information logged by the Flysight V1, the Flysight V2 adds IMU & environmental data streams to your flight logs. The main user-facing change is a difference in log output.
+
+Where the Flysight V1 log output looks something like:
+
+```
+.
+└── 24-04-20/
+    └── 04-20-00.CSV
+```
+
+The Flysight V2 log output looks something like:
+
+```
+.
+└── 24-04-20/
+    └── 04-20-00/
+        ├── RAW.UBX
+        ├── SENSOR.CSV
+        └── TRACK.CSV
+```
+
 ## Contributing
 ### Development Environment
 This project uses [Poetry](https://python-poetry.org/) to manage dependencies. With your fork cloned to your local machine, you can install the project and its dependencies to create a development environment using:

doc/data_format.md

@@ -0,0 +1,125 @@
+# Flysight Data Format
+## Flysight V1
+### File Structure
+The Flysight V1 hardware revision outputs a single CSV file containing GPS track information. Flight logs are grouped into a directory by GPS date, and each flight session CSV is named based on the initial GPS time.
+
+```
+.
+└── 24-04-20/
+    └── 04-20-00.CSV
+```
+
+### Data Structure
+```
+time,lat,lon,hMSL,velN,velE,velD,hAcc,vAcc,sAcc,heading,cAcc,gpsFix,numSV
+,(deg),(deg),(m),(m/s),(m/s),(m/s),(m),(m),(m/s),(deg),(deg),,
+2021-04-20T04:20:00.00Z,33.6568828,-117.7466357,15.060,-0.63,0.98,0.29,207.635,481.468,7.15,0.00000,180.00000,3,4
+```
+
+The track file CSV contains 2 header lines, one with column name and one with data units, followed by one or more rows of track data.
+
+## Flysight V2
+### File Structure
+The Flysight V2 hardware revision outputs a collection of files containing flight log information. Flight logs are grouped into directories by GPS date, then GPS time, based on the initial GPS time.
+
+```
+.
+└── 24-04-20/
+    └── 04-20-00/
+        ├── RAW.UBX
+        ├── SENSOR.CSV
+        └── TRACK.CSV
+```
+
+Currently output files are:
+  * `RAW.UBX` - The raw binary data stream from the onboard u-blox hardware; the protocol used has not been investigated
+  * `SENSOR.CSV` - Data records from the onboard sensors (minus GPS)
+  * `TRACK.CSV` - Data records from the GPS sensor
+
+### Data Structure
+Both the sensor and track data CSV files share a similar format: a series of header lines, followed by a delimiter (`$DATA`), and then one or more rows of data records.
+
+Currently encountered information contained in the header file is as follows:
+  * `$FLYS` - Unsure what this is :)
+  * `$VAR` - Device information
+  * Sensor information is provided as one or more pairs of rows:
+    * `$COL` - Sensor ID & measured quantities; the sensor ID is used in the first column of a data row to identify the information source for each row
+    * `$UNIT` - Measured quantity units
+
+#### Sensor data
+The sensor data stream is comingled, where the source sensor for each row is provided in the first column of each row & corresponds to the sensor information provided in the file's header.
+
+I believe the timestamps are provided as GPS time of day, in seconds.
+
+```
+$FLYS,1
+$VAR,FIRMWARE_VER,v2024.05.25.pairing_request
+$VAR,DEVICE_ID,abc123
+$VAR,SESSION_ID,abc123
+$COL,BARO,time,pressure,temperature
+$UNIT,BARO,s,Pa,deg C
+$COL,HUM,time,humidity,temperature
+$UNIT,HUM,s,percent,deg C
+$COL,MAG,time,x,y,z,temperature
+$UNIT,MAG,s,gauss,gauss,gauss,deg C
+$COL,IMU,time,wx,wy,wz,ax,ay,az,temperature
+$UNIT,IMU,s,deg/s,deg/s,deg/s,g,g,g,deg C
+$COL,TIME,time,tow,week
+$UNIT,TIME,s,s,
+$COL,VBAT,time,voltage
+$UNIT,VBAT,s,volt
+$DATA
+$IMU,59970.376,-0.427,1.770,1.953,-0.01464,-0.00732,0.94287,25.64
+```
+
+Currently seen sensors are:
+  * `$BARO`
+    * Time
+    * Pressure
+    * Temperature
+  * `$HUM`
+    * Time
+    * Humidity
+    * Temperature
+  * `$IMU`
+    * Time
+    * Gyro XYZ
+    * Accel XYC
+    * Temperature
+  * `$MAG`
+    * Time
+    * Magnetic field XYZ
+  * `$TIME`
+    * Time
+    * Time of week
+    * Week
+  * `$VBAT`
+    * Time
+    * Voltage
+
+#### Track data
+
+```
+$FLYS,1
+$VAR,FIRMWARE_VER,v2024.05.25.pairing_request
+$VAR,DEVICE_ID,abc123
+$VAR,SESSION_ID,abc123
+$COL,GNSS,time,lat,lon,hMSL,velN,velE,velD,hAcc,vAcc,sAcc,numSV
+$UNIT,GNSS,,deg,deg,m,m/s,m/s,m/s,m,m,m/s,
+$DATA
+$GNSS,2024-04-20T04:20:00.000Z,33.6568828,-117.7466357,630.077,-31.92,48.42,-34.93,136.117,170.718,4.74,4
+```
+
+Currently seen sensors are:
+  * `$GNSS`
+    * Time (ISO format)
+    * Lat
+    * Lon
+    * Altitude MSL
+    * Northing velocity
+    * Easting velocity
+    * Vertical velocity
+    * Horzontal accuracy
+    * Vertical accuracy
+    * Speed accuracy
+    * Satellites in view