From 97760ef096e60fbcb0d55647c033c4746a10840b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 10:25:25 +0000 Subject: [PATCH 01/44] Add ARCHITECTURE.md documenting module structure and integration patterns Captures the two connector generations (generic kafka-connect-rest-source framework used by Fitbit vs. the newer oura-library + thin Connect glue split used by Oura), the runtime polling/auth/conversion flow, config and Docker/CI setup, and a checklist for adding a new vendor integration such as Huawei following the Oura pattern. --- ARCHITECTURE.md | 251 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) create mode 100644 ARCHITECTURE.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 00000000..0d716c53 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,251 @@ +# Architecture + +This document describes how RADAR-REST-Connector is put together, so that future contributors +(human or agent) can orient themselves quickly and add new device/API integrations (e.g. Huawei +Health Kit) consistently with the existing patterns. + +## What this repo is + +A multi-module Gradle project providing Kafka Connect **source connectors** that poll third-party +REST APIs (wearable vendor APIs) on behalf of RADAR-base study participants and publish the +resulting data as Avro records on Kafka topics. It currently ships two concrete connectors — +**Fitbit** and **Oura** — built on top of a shared, generic REST-polling framework. + +``` +RADAR-REST-Connector/ +├── kafka-connect-rest-source/ # Generic Kafka Connect REST-source framework (Java) +├── kafka-connect-fitbit-source/ # Fitbit connector (Java), oldest/original implementation +├── oura-library/ # Oura domain logic: routes, converters, requests (Kotlin, no Kafka Connect deps) +├── kafka-connect-oura-source/ # Oura Kafka Connect glue (Java+Kotlin), wraps oura-library +├── docker/ # Docker Compose config templates, launch/ensure scripts, log4j +├── scripts/REDCAP-FITBIT-AUTH-AUTO/ # Standalone Python helper for REDCap-driven Fitbit auth +└── docker-compose.yml # Full local Kafka stack + both connectors, for manual testing +``` + +Root Gradle config (`build.gradle.kts`, `settings.gradle.kts`, `gradle/libs.versions.toml`) uses +the `org.radarbase.radar-kotlin` / `radar-root-project` plugins (from `radar-commons`) for shared +build conventions (Kotlin/Java toolchain, Sentry, versioning). All dependency versions are +centralized in `gradle/libs.versions.toml` (a Gradle version catalog) — add new deps there, not +inline in module build files. + +Avro **schemas are not defined in this repo**. They come from the external `radar-schemas-commons` +artifact (`org.radarbase:radar-schemas-commons`, versioned in the catalog), generated from the +[RADAR-Schemas](https://github.com/RADAR-base/RADAR-Schemas) repository. Adding a new data type +therefore requires a schema to exist there first (e.g. `org.radarcns.connector.oura.OuraDailyActivity`), +before this repo can build an Avro record for it. + +## Two architectural generations + +The repo contains two different design generations. Understand both before adding Huawei, and +prefer the **Oura pattern** for new integrations — it is the more recent, more testable design. + +### 1. Generic `kafka-connect-rest-source` framework (used by Fitbit) + +Located at `kafka-connect-rest-source/src/main/java/org/radarbase/connect/rest/`. Defines a small +set of interfaces meant to be generic across arbitrary REST APIs: + +- `AbstractRestSourceConnector` — Kafka Connect `SourceConnector` base class. Loads + `RestSourceConnectorConfig` from properties and hands out `RestSourceTask` as the task class. +- `RestSourceConnectorConfig` — `AbstractConfig` wrapper exposing the generic + `rest.source.*` properties (base URL, poll interval, topic selector class, payload converter + class, request generator class — all pluggable via `ConfigDef.Type.CLASS`). +- `RequestGenerator` (`request/`) — produces a `Stream` of `RestRequest`s to issue and knows when + the next request is due (`getTimeOfNextRequest()`), driven by the Kafka Connect offset storage + (`setOffsetStorageReader`). +- `RequestRoute` / `PollingRequestRoute` (`request/`) — one "route" = one logical polling + endpoint/data type. Routes own their own per-user polling cadence, backoff, and offset state, and + are notified of `requestSucceeded` / `requestEmpty` / `requestFailed`. +- `PayloadToSourceRecordConverter` (`converter/`) — turns a raw HTTP response body into one or more + Kafka Connect `SourceRecord`s. +- `RestSourceTask` — the actual Kafka Connect `SourceTask`. Its `poll()` loop: sleep until the next + request is due, iterate `requestGenerator.requests()`, execute the first request that yields + records, return them. + +This module is intentionally protocol-agnostic; it has no notion of "Fitbit" or OAuth. It's a +reasonable place to fix or extend genuinely generic REST-polling behavior (e.g. topic selection, +generic retry semantics), but new device integrations do **not** need to hook into it directly — +see the Oura pattern below. + +### 2. Fitbit connector (`kafka-connect-fitbit-source`) — first concrete integration + +Built directly on the generic framework above, entirely in Java: + +- `FitbitSourceConnector extends AbstractRestSourceConnector` — schedules a periodic + (`application.loop.interval.ms`) user-repository refresh; if the user set changes, requests task + reconfiguration (`context.requestTaskReconfiguration()`). Divides users across `tasks.max` tasks + by hashing `user.getVersionedId()`. +- `FitbitRequestGenerator extends RequestGeneratorRouter` — builds the list of enabled + `RequestRoute`s (one per Fitbit data type: sleep, activity log, resting heart rate, and — if + `fitbit.api.intraday=true` — steps, heart rate, HRV, breathing rate, skin temperature, calories, + SpO2) and an OkHttp client per user with a `TokenAuthenticator` (auto-refreshes on HTTP 401). +- `route/Fitbit*Route` — one class per data type, extending `FitbitPollingRoute`, which implements + a fairly elaborate polling algorithm: don't poll more than once per `pollInterval`; walk history + back to `HISTORICAL_TIME_DAYS`; avoid re-reading the last `LOOKBACK_TIME` to tolerate + late-arriving data from other devices; back off per-user on HTTP 429 (`TOP_OF_HOUR` or + `ROLLING_WINDOW` cooldown strategy) and after `fitbit.request.max.forbidden` consecutive HTTP 403s. +- `converter/Fitbit*AvroConverter` — one class per data type, converts JSON to the corresponding + Avro record from `radar-schemas-commons`. +- `user/UserRepository` (interface) + implementations: + - `YamlUserRepository` — reads per-user YAML files from a directory (`docker/fitbit-user.yml.template` + shows the format: id, projectId, userId, sourceId, startDate/endDate, externalUserId, OAuth2 + access/refresh tokens). + - `ServiceUserRepository` (Kotlin) — talks to an external "rest-source-authorizer" webservice + (typically fronted by ManagementPortal) for user lists and token refresh/storage, using + OAuth2 client-credentials auth. + - `firebase/FirebaseUserRepository` / `CovidCollabFirebaseUserRepository` — legacy + Firestore-backed repository for a specific historical deployment. +- `request/TokenAuthenticator` — OkHttp `Authenticator` that refreshes the access token via the + `UserRepository` on a 401 and retries the request. + +### 3. Oura connector (`oura-library` + `kafka-connect-oura-source`) — newer pattern + +This is the template to follow for a **new** vendor integration such as Huawei. It splits cleanly +into: + +- **`oura-library`** (pure Kotlin, no Kafka Connect / OkHttp-Authenticator coupling to Connect + internals) — all vendor-specific domain logic, independently unit-testable and in principle + reusable outside Kafka Connect: + - `user/User`, `user/UserRepository` — user model and repository interface (`get`, `stream`, + `getAccessToken`); note `refreshAccessToken` lives on the connector-side implementation, not + here, in current Oura code (asymmetry vs. Fitbit — be aware when implementing). + - `route/Route` (interface) + `route/OuraRoute` (abstract base) + `route/Oura*Route` (one per + data type: daily activity, readiness, sleep, SpO2, heart rate, personal info, sessions, + workouts, tags, ring configuration, stress, VO2 max, resilience, cardiovascular age, enhanced + tags, rest-mode periods, sleep-time recommendations, etc.) — each route knows its API sub-path + and builds one or more `RestRequest`s covering a `[start, end)` window, chunked by + `maxIntervalPerRequest`. + - `route/OuraRouteFactory` — central list of all routes (used mainly by tests / defaults; the + connector module actually builds its own filtered list based on config flags — see below). + - `converter/OuraDataConverter` (→ `RecordConverter`) + `converter/Oura*Converter` — one per data + type, parses the JSON response into `TopicData(topic, key, value, offset)` where `value` is a + generated Avro `SpecificRecord` from `radar-schemas-commons` (e.g. `OuraDailyActivity`). This + is the direct analogue of Fitbit's `Fitbit*AvroConverter`, but returns plain data objects + instead of Kafka Connect `SourceRecord`s directly — the Connect-specific wrapping happens in + the connector module. + - `request/OuraRequestGenerator` — the polling brain: for each `(route, user)` pair, computes the + offset to resume from (via `OuraOffsetManager`), decides whether to use a large "historical" + chunk (`HISTORICAL_QUERY_RANGE` = 1 year, once `timeSinceStart > HISTORICAL_DATA_THRESHOLD` = 1 + year) or normal recent-data chunking, and interprets HTTP responses + (`handleResponse`/`requestSuccessful`/`requestFailed`) into typed `OuraResult`/`OuraError` + sealed hierarchies with per-(route,user) backoff bookkeeping (`routeNextRequest` map) for 429 / + 403 / 401 / 400 / 422 / 404 / other. + - `request/OuraOffsetManager` (interface) — abstraction for reading/writing per-(route,user) + offsets; the Kafka Connect implementation is `KafkaOffsetManager` in the connector module. + - `offset/Offset`, `offset/Offsets` — plain offset value types. +- **`kafka-connect-oura-source`** (Java, some Kotlin) — the thin Kafka Connect glue: + - `OuraSourceConnector` — same role as `FitbitSourceConnector`: periodic user refresh, + reconfiguration on user-set change, hash-based task partitioning. + - `OuraSourceTask` — Kafka Connect `SourceTask`. Builds the enabled `Route` list from config + flags, constructs `OuraRequestGenerator`, and in `poll()` round-robins across routes + (`getRotatedRoutes()`, so one slow/rate-limited route doesn't starve the others), executes one + HTTP request via a shared `OkHttpClient`, and converts the resulting `TopicData` list into + Kafka Connect `SourceRecord`s using `AvroData` (Confluent's Kotlin/Avro↔Connect-schema bridge). + - `offset/KafkaOffsetManager` — `OuraOffsetManager` backed by Kafka Connect's + `OffsetStorageReader`. + - `user/OuraUserRepository` (abstract) / `OuraServiceUserRepository` — the concrete + "rest-source-authorizer" HTTP client, built on **Ktor** (not OkHttp) with `radar-commons`'s + `CachedSet`/`CachedValue`/`clientCredentials` helpers for user list caching and per-user OAuth2 + token caching/refresh. This is the modern replacement for Fitbit's + `ServiceUserRepository`/`TokenAuthenticator` combo, and is the pattern to copy for Huawei. + - `OuraRestSourceConnectorConfig` — `ConfigDef` with one `oura..enabled` boolean and topic + name per data type, plus `oura.user.repository.*` connection settings. + +**Key structural difference from Fitbit**: Oura does *not* extend the generic +`kafka-connect-rest-source` interfaces (`RequestRoute`, `PollingRequestRoute`, +`PayloadToSourceRecordConverter`) at all — `OuraSourceTask` implements Kafka Connect's `SourceTask` +directly and drives `oura-library`'s own `Route`/`RequestGenerator`/`RecordConverter` abstractions. +This was a deliberate move to (a) get domain logic under unit test without spinning up Kafka +Connect, and (b) avoid the generic framework's assumptions (e.g. its polling-interval math) that +didn't fit Oura's simpler historical/recent chunking model. + +## Runtime data flow (both connectors, conceptually) + +```mermaid +sequenceDiagram + participant connector as SourceConnector + participant task as SourceTask + participant userRepo as User Repository (rest-source-authorizer) + participant api as Vendor API (Fitbit/Oura/…) + participant kafka as Kafka + + connector ->> userRepo: Poll for users/config changes (periodic) + connector ->> connector: Partition users across tasks.max tasks + loop poll() + task ->> task: Determine next due (route, user) request + task ->> userRepo: Get/refresh OAuth2 access token + task ->> api: GET data for date range + api -->> task: JSON response + task ->> task: Convert JSON -> Avro SourceRecord(s) + task ->> kafka: Return records (Connect framework produces them) + task ->> task: Update in-memory + Connect offset state + end +``` + +User authentication/authorization data (OAuth2 tokens, study/user/source IDs, start/end dates) is +**not** stored in this repo. In production it's served by a "rest-source-authorizer" webservice +(part of RADAR-base, typically backed by ManagementPortal); for local/manual testing, Fitbit also +supports flat YAML files under `docker/users/` (`YamlUserRepository`). + +## Configuration model + +Every connector exposes its settings as a Kafka Connect `ConfigDef` (`org.apache.kafka.common.config`), +loaded from a Java `.properties` file (see `docker/source-fitbit.properties.template` and +`docker/source-oura.properties.template`) referenced by `connector.class`, `name`, `tasks.max`, +plus vendor-specific keys, e.g.: + +- `.api.client` / `.api.secret` — OAuth2 app credentials. +- `.user.repository.class` — pluggable `UserRepository` implementation. +- `.user.repository.url` / `.client.id` / `.client.secret` / `.oauth2.token.url` — + rest-source-authorizer connection details. +- `..topic` / `.enabled` — per-data-type Kafka topic name and on/off switch, so + studies can disable data types they don't need. + +The full current list for Fitbit is documented in `README.md`; Oura's config lives in +`OuraRestSourceConnectorConfig` (no README table yet — check the class directly). + +## Docker / deployment + +Each connector module has its own multi-stage `Dockerfile` (Gradle build stage → base image +`confluentinc/cp-kafka-connect-base`), publishing built jars plus third-party deps into +`$CONNECT_PLUGIN_PATH//`. `docker/launch` and `docker/ensure` are modified Confluent +entrypoint scripts (env-var → properties translation, Kafka-readiness wait). `docker-compose.yml` +spins up a full local Zookeeper+Kafka+SchemaRegistry+REST-proxy cluster plus both connectors for +manual end-to-end testing (`docker-compose up -d --build`, inspect with +`kafka-avro-console-consumer`). Sentry error monitoring is wired in via `radarKotlin { sentryEnabled = true }` +and configured purely through `SENTRY_DSN`/`SENTRY_*` env vars — see README "Sentry monitoring". + +## Testing + +- `kafka-connect-rest-source/src/test`, `kafka-connect-fitbit-source/src/test`, + `kafka-connect-oura-source/src/test` currently only contain config-parsing tests + (`*ConnectorConfigTest`) plus one task test — test coverage of the actual polling/conversion + logic is thin. `wiremock` and `mockito` are on the version catalog for HTTP-level testing but not + yet exercised much; `oura-library`'s pure-Kotlin design makes it the easiest place to add real + unit tests for new routes/converters without Kafka Connect scaffolding. +- CI (`.github/workflows/main.yml`) runs `./gradlew assemble` and `./gradlew check` on every push/PR + to `master`/`dev`, then builds (and on `push`, publishes) multi-arch Docker images per connector + module via a matrix job. `release.yml` does the same on GitHub Release publish, additionally + uploading built jars as release assets, tagged `vX.Y.Z` from `gradle.properties`/version catalog. + +## Adding a new vendor integration (e.g. Huawei) + +Follow the **Oura pattern**, not the Fitbit one: + +1. New Gradle module `huawei-library` (pure Kotlin, mirrors `oura-library`): `user/`, `route/`, + `converter/`, `request/`, `offset/` packages. No Kafka Connect or OkHttp-Connect-specific types + here — keep it independently testable. +2. New Gradle module `kafka-connect-huawei-source` (mirrors `kafka-connect-oura-source`): + `HuaweiSourceConnector`, `HuaweiSourceTask`, `HuaweiRestSourceConnectorConfig`, + `offset/KafkaOffsetManager`, `user/HuaweiServiceUserRepository` (Ktor-based + rest-source-authorizer client, copy `OuraServiceUserRepository`'s structure), plus a + `Dockerfile`. +3. Register both modules in `settings.gradle.kts`; add any new dependency versions to + `gradle/libs.versions.toml` first. +4. Confirm (or add) the required Avro schemas in the external RADAR-Schemas project and bump the + `radarSchemas` version in the catalog once published — this repo cannot invent schemas locally. +5. One `Route`/`Converter` pair per Huawei data type you plan to support, each independently + togglable via a `huawei..enabled` config flag, matching the Oura/Fitbit convention. +6. Add `docker/source-huawei.properties.template`, a `docker-compose.yml` service entry, and a + README config table, following the Fitbit/Oura sections as templates. +7. Add the new Docker image to the `IMAGES` matrix in both `.github/workflows/main.yml` and + `release.yml`. From 211c3d565f1972ac351c29c33790e78d1528b220 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 10:51:46 +0000 Subject: [PATCH 02/44] Scaffold huawei-library and kafka-connect-huawei-source modules Registers the two new Gradle modules, pins radar-schemas-commons 0.9.0-SNAPSHOT (huawei_schemas branch not yet released) as a separate version-catalog entry so existing modules keep the stable release, and adds the user/UserRepository domain types mirroring oura-library's pattern. --- gradle/libs.versions.toml | 4 ++ huawei-library/build.gradle | 64 +++++++++++++++++++ .../org/radarbase/huawei/user/HuaweiUser.kt | 28 ++++++++ .../kotlin/org/radarbase/huawei/user/User.kt | 22 +++++++ .../huawei/user/UserNotAuthorizedException.kt | 5 ++ .../radarbase/huawei/user/UserRepository.kt | 33 ++++++++++ kafka-connect-huawei-source/Dockerfile | 62 ++++++++++++++++++ kafka-connect-huawei-source/build.gradle.kts | 56 ++++++++++++++++ settings.gradle.kts | 2 + 9 files changed, 276 insertions(+) create mode 100644 huawei-library/build.gradle create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt create mode 100644 kafka-connect-huawei-source/Dockerfile create mode 100644 kafka-connect-huawei-source/build.gradle.kts diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 1bad2750..bf8a325a 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -16,6 +16,9 @@ sentryOpenTelemetryAgent = "8.36.0" okhttp = "4.12.0" firebaseAdmin = "9.8.0" radarSchemas = "0.8.16" +# Huawei connector schemas are not yet released; pin to the published snapshot until a stable +# radar-schemas-commons release containing the huawei_schemas branch is cut. +radarSchemasHuawei = "0.9.0-SNAPSHOT" # @pin Upgrade to 3.x.x requires kotlin v2 minimum ktor = "2.3.13" wiremock = "3.0.1" @@ -28,6 +31,7 @@ lz4 = "1.10.1" lz4 = { module = "at.yawk.lz4:lz4-java", version.ref = "lz4" } radar-commons-kotlin = { module = "org.radarbase:radar-commons-kotlin", version.ref = "radarCommons" } radar-schemas-commons = { module = "org.radarbase:radar-schemas-commons", version.ref = "radarSchemas" } +radar-schemas-commons-huawei = { module = "org.radarbase:radar-schemas-commons", version.ref = "radarSchemasHuawei" } kafka-connect-api = "org.apache.kafka:connect-api:7.8.1-ce" kafka-connect-avro-converter = { module = "io.confluent:kafka-connect-avro-converter", version.ref = "confluent" } okhttp = { module = "com.squareup.okhttp3:okhttp", version.ref = "okhttp" } diff --git a/huawei-library/build.gradle b/huawei-library/build.gradle new file mode 100644 index 00000000..ce391e17 --- /dev/null +++ b/huawei-library/build.gradle @@ -0,0 +1,64 @@ + +group = 'org.radarbase' +version = '0.0.1' + +apply plugin: 'maven-publish' + +repositories { + // Use jcenter for resolving dependencies. + // You can declare any Maven/Ivy/file repository here. + mavenCentral() + + // radar-schemas-commons huawei_schemas is only published as a snapshot; declare the + // candidate snapshot hosts here so the huawei-library build can resolve it regardless of + // which one the RADAR-Schemas release pipeline currently targets. + maven { + url = uri("https://central.sonatype.com/repository/maven-snapshots/") + } + maven { + url = uri("https://s01.oss.sonatype.org/content/repositories/snapshots/") + } + maven { + url = uri("https://maven.pkg.github.com/RADAR-base/RADAR-Schemas") + credentials { + username = project.findProperty("public.gpr.user") ?: System.getenv("GPR_USER") + password = project.findProperty("public.gpr.token") ?: System.getenv("GPR_TOKEN") + } + } +} + +dependencies { + // Use the Kotlin JDK 8 standard library. + implementation libs.kotlin.stdlib + + implementation libs.okhttp + + implementation libs.radar.schemas.commons.huawei + + implementation libs.jackson.annotations + + implementation libs.jackson.databind + + implementation libs.avro + + implementation libs.jackson.datatype.jsr310 + + // Use the Kotlin test library. + testImplementation libs.kotlin.test + + // Use the Kotlin JUnit integration. + testImplementation libs.kotlin.test.junit +} + +project.afterEvaluate { + publishing { + publications { + library(MavenPublication) { + setGroupId "$group" + setArtifactId "huawei-library" + version "$version" + from components.java + } + } + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt new file mode 100644 index 00000000..7d1521e3 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt @@ -0,0 +1,28 @@ +package org.radarbase.huawei.user + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties +import com.fasterxml.jackson.annotation.JsonProperty +import org.radarcns.kafka.ObservationKey +import java.time.Instant + +@JsonIgnoreProperties(ignoreUnknown = true) +data class HuaweiUser( + @JsonProperty("id") override val id: String, + @JsonProperty("createdAt") override val createdAt: Instant, + @JsonProperty("projectId") override val projectId: String, + @JsonProperty("userId") override val userId: String, + @JsonProperty("humanReadableUserId") override val humanReadableUserId: String?, + @JsonProperty("sourceId") override val sourceId: String, + @JsonProperty("externalId") override val externalId: String?, + @JsonProperty("isAuthorized") override val isAuthorized: Boolean, + @JsonProperty("startDate") override val startDate: Instant, + @JsonProperty("endDate") override val endDate: Instant? = null, + @JsonProperty("version") override val version: String? = null, + @JsonProperty("serviceUserId") override val serviceUserId: String? = null, +) : User { + override val observationKey: ObservationKey = ObservationKey(projectId, userId, sourceId) + override val versionedId: String = "$id${version?.let { "#$it" } ?: ""}" + + fun isComplete() = + isAuthorized && (endDate == null || startDate.isBefore(endDate)) && serviceUserId != null +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt new file mode 100644 index 00000000..b84dfe76 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt @@ -0,0 +1,22 @@ +package org.radarbase.huawei.user + +import org.radarcns.kafka.ObservationKey +import java.time.Instant + +interface User { + val id: String + val projectId: String + val userId: String + val sourceId: String + val externalId: String? + val startDate: Instant + val endDate: Instant? + val createdAt: Instant + val humanReadableUserId: String? + val serviceUserId: String? + val version: String? + val isAuthorized: Boolean + + val observationKey: ObservationKey + val versionedId: String +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt new file mode 100644 index 00000000..1bd513b0 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt @@ -0,0 +1,5 @@ +package org.radarbase.huawei.user + +class UserNotAuthorizedException(message: String) : Exception(message) { + constructor(user: User) : this("User ${user.id} is not authorized") +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt new file mode 100644 index 00000000..6f26b74b --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt @@ -0,0 +1,33 @@ +package org.radarbase.huawei.user + +import java.io.IOException + +/** User repository for Huawei Health Kit users. */ +interface UserRepository { + /** + * Get specified user. + * + * @throws IOException if the user cannot be retrieved from the repository. + */ + @Throws(IOException::class) + operator fun get(key: String): User? + + /** + * Get all relevant users. + * + * @throws IOException if the list cannot be retrieved from the repository. + */ + @Throws(IOException::class) + fun stream(): Sequence + + /** + * Get the current access token of given user. + * + * @throws IOException if the new access token cannot be retrieved from the repository. + * @throws UserNotAuthorizedException if the refresh token is no longer valid. Manual action + * should be taken to get a new refresh token. + * @throws NoSuchElementException if the user does not exist in this repository. + */ + @Throws(IOException::class, UserNotAuthorizedException::class) + fun getAccessToken(user: User): String +} diff --git a/kafka-connect-huawei-source/Dockerfile b/kafka-connect-huawei-source/Dockerfile new file mode 100644 index 00000000..e01bcb7f --- /dev/null +++ b/kafka-connect-huawei-source/Dockerfile @@ -0,0 +1,62 @@ +# Copyright 2018 The Hyve +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM --platform=$BUILDPLATFORM gradle:8.14-jdk17 AS builder + +RUN mkdir /code +WORKDIR /code + +ENV GRADLE_USER_HOME=/code/.gradlecache \ + GRADLE_OPTS="-Dorg.gradle.vfs.watch=false -Djdk.lang.Process.launchMechanism=vfork" + +COPY ./gradle/libs.versions.toml /code/gradle/ +COPY ./build.gradle.kts ./settings.gradle.kts ./gradle.properties /code/ +COPY kafka-connect-huawei-source/build.gradle.kts /code/kafka-connect-huawei-source/ +COPY huawei-library/build.gradle /code/huawei-library/ + +RUN gradle downloadDependencies copyDependencies + +COPY ./kafka-connect-huawei-source/src/ /code/kafka-connect-huawei-source/src +COPY ./huawei-library/src/ /code/huawei-library/src + +RUN gradle jar + +FROM confluentinc/cp-kafka-connect-base:7.8.7 + +USER appuser + +LABEL org.opencontainers.image.authors="yatharth.ranjan@kcl.ac.uk" + +LABEL description="Kafka Huawei Health Kit REST API Source connector" + +ENV CONNECT_PLUGIN_PATH="/usr/share/java/kafka-connect/plugins" \ + WAIT_FOR_KAFKA="1" + +# To isolate the classpath from the plugin path as recommended +COPY --from=builder /code/kafka-connect-huawei-source/build/third-party/*.jar ${CONNECT_PLUGIN_PATH}/kafka-connect-huawei-source/ +COPY --from=builder /code/huawei-library/build/third-party/*.jar ${CONNECT_PLUGIN_PATH}/kafka-connect-huawei-source/ + +COPY --from=builder /code/kafka-connect-huawei-source/build/libs/*.jar ${CONNECT_PLUGIN_PATH}/kafka-connect-huawei-source/ +COPY --from=builder /code/huawei-library/build/libs/*.jar ${CONNECT_PLUGIN_PATH}/kafka-connect-huawei-source/ + +# Load topics validator +COPY --chown=appuser:appuser ./docker/ensure /etc/confluent/docker/ensure + +# Load modified launcher +COPY --chown=appuser:appuser ./docker/launch /etc/confluent/docker/launch + +# Overwrite the log4j configuration to include Sentry monitoring. +COPY ./docker/log4j.properties.template /etc/confluent/docker/log4j.properties.template +# Copy Sentry monitoring jars. +COPY --from=builder /code/kafka-connect-huawei-source/build/third-party/sentry-* /etc/kafka-connect/jars diff --git a/kafka-connect-huawei-source/build.gradle.kts b/kafka-connect-huawei-source/build.gradle.kts new file mode 100644 index 00000000..f3169abe --- /dev/null +++ b/kafka-connect-huawei-source/build.gradle.kts @@ -0,0 +1,56 @@ +description = "Kafka connector for Huawei Health Kit API source" + +repositories { + // radar-schemas-commons huawei_schemas is only published as a snapshot; declare the + // candidate snapshot hosts here so the build can resolve it regardless of which one the + // RADAR-Schemas release pipeline currently targets. + maven { + url = uri("https://central.sonatype.com/repository/maven-snapshots/") + } + maven { + url = uri("https://s01.oss.sonatype.org/content/repositories/snapshots/") + } + maven { + url = uri("https://maven.pkg.github.com/RADAR-base/RADAR-Schemas") + credentials { + username = project.findProperty("public.gpr.user") as String? ?: System.getenv("GPR_USER") + password = project.findProperty("public.gpr.token") as String? ?: System.getenv("GPR_TOKEN") + } + } +} + +dependencies { + + /* The entries in the block below are added here to force the version of + * transitive dependencies and mitigate reported vulnerabilities + */ + implementation(libs.netty.handler.proxy) + implementation(libs.netty.handler) + + api(project(":huawei-library")) + api(libs.kafka.connect.avro.converter) + api(libs.radar.schemas.commons.huawei) + implementation(libs.radar.commons.kotlin) + + api(libs.okhttp) + implementation(platform(libs.jackson.bom)) + implementation(libs.jackson.dataformat.yaml) + implementation(libs.jackson.datatype.jsr310) + implementation(libs.kotlin.stdlib) + + implementation(libs.ktor.client.auth) + implementation(libs.ktor.client.content.negotiation) + implementation(libs.ktor.serialization.jackson) + implementation(libs.ktor.client.cio) + implementation(libs.ktor.serialization.kotlinx.json) + implementation(libs.jackson.module.kotlin) + + // Included in connector runtime + compileOnly(libs.kafka.connect.api) + compileOnly(platform(libs.jackson.bom)) + compileOnly(libs.jackson.databind) + + testImplementation(libs.kafka.connect.api) + testImplementation(libs.wiremock) + testImplementation(libs.mockito.core) +} diff --git a/settings.gradle.kts b/settings.gradle.kts index 64f23944..260feaed 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -3,6 +3,8 @@ include(":kafka-connect-fitbit-source") include(":kafka-connect-rest-source") include(":kafka-connect-oura-source") include(":oura-library") +include(":kafka-connect-huawei-source") +include(":huawei-library") pluginManagement { repositories { From d51979e39a398c187a512bef93957d21f75976eb Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 11:05:47 +0000 Subject: [PATCH 03/44] Implement huawei-library: routes, converters, and request generation Adds the domain logic for the Huawei Health Kit connector, mirroring oura-library's split of pure-Kotlin logic from Kafka Connect glue: - request/: RestRequest, RequestGenerator, HuaweiRequestGenerator (offset tracking, per-route backoff on 401/403/429/etc), HuaweiResult/HuaweiError. - route/: HuaweiRoute base (OAuth2-authorized GET/POST + time-range chunking), three concrete route kinds covering the Health Kit Data API's endpoints - HuaweiSampleSetRoute (POST sampleSet:polymerize, raw or groupByTime-aggregated), HuaweiHealthRecordRoute (GET healthRecords), HuaweiActivityRecordRoute (GET activityRecords) - and HuaweiRouteFactory, a single registry mapping all ~54 Huawei data types from the radar-huawei-connector schema spec to their endpoint, dataTypeName, and Avro record builder. - converter/: FieldValues (typed accessor for Huawei's field-value sample point format) and generic converters that turn API responses into TopicData for the registered Avro records. Field-value key names are best-effort (Huawei HiHealth Field naming convention); this was verified to compile against a locally-published radar-schemas-commons 0.9.0-SNAPSHOT (huawei_schemas branch) since none of the real snapshot hosts are reachable from this sandbox - flagged in comments for verification against a live API response. --- huawei-library/build.gradle | 5 + .../radarbase/huawei/converter/FieldValues.kt | 58 ++ .../HuaweiActivityRecordConverter.kt | 85 +++ .../huawei/converter/HuaweiDataConverter.kt | 41 ++ .../converter/HuaweiHealthRecordConverter.kt | 49 ++ .../converter/HuaweiSampleSetConverter.kt | 53 ++ .../huawei/converter/RecordConverter.kt | 19 + .../huawei/converter/SequenceExtensions.kt | 11 + .../radarbase/huawei/converter/TopicData.kt | 11 + .../org/radarbase/huawei/offset/Offset.kt | 11 + .../org/radarbase/huawei/offset/Offsets.kt | 5 + .../huawei/request/HuaweiOffsetManager.kt | 13 + .../huawei/request/HuaweiRequestGenerator.kt | 190 ++++++ .../radarbase/huawei/request/HuaweiResult.kt | 64 ++ .../huawei/request/RequestGenerator.kt | 19 + .../radarbase/huawei/request/RestRequest.kt | 14 + .../request/TooManyRequestsException.kt | 3 + .../huawei/route/HuaweiActivityRecordRoute.kt | 46 ++ .../huawei/route/HuaweiHealthRecordRoute.kt | 58 ++ .../org/radarbase/huawei/route/HuaweiRoute.kt | 62 ++ .../huawei/route/HuaweiRouteDefinition.kt | 19 + .../huawei/route/HuaweiRouteFactory.kt | 566 ++++++++++++++++++ .../huawei/route/HuaweiSampleSetRoute.kt | 75 +++ .../org/radarbase/huawei/route/Route.kt | 23 + kafka-connect-huawei-source/build.gradle.kts | 5 + 25 files changed, 1505 insertions(+) create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt diff --git a/huawei-library/build.gradle b/huawei-library/build.gradle index ce391e17..882ad1e3 100644 --- a/huawei-library/build.gradle +++ b/huawei-library/build.gradle @@ -9,6 +9,11 @@ repositories { // You can declare any Maven/Ivy/file repository here. mavenCentral() + // Prefer a locally-published snapshot (e.g. built by hand from the RADAR-Schemas + // huawei_schemas branch via `./gradlew :radar-schemas-commons:publishToMavenLocal`) before + // falling back to remote snapshot hosts. + mavenLocal() + // radar-schemas-commons huawei_schemas is only published as a snapshot; declare the // candidate snapshot hosts here so the huawei-library build can resolve it regardless of // which one the RADAR-Schemas release pipeline currently targets. diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt new file mode 100644 index 00000000..57386986 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt @@ -0,0 +1,58 @@ +package org.radarbase.huawei.converter + +import com.fasterxml.jackson.databind.JsonNode + +/** + * Typed accessor for a single Huawei Health Kit sample point's field values. + * + * The Health Kit Data API (`sampleSet:polymerize`) represents each field of a sample point using + * the same typed-value wrapper as the on-device HiHealth SDK's `Field`/`Value` model: a list of + * objects shaped like `{"fieldName": "steps_delta", "integerValue": 123}` (or `floatValue`, + * `longValue`, `stringValue` depending on the field's declared type). This class also tolerates a + * flattened `{"fieldName": value, ...}` object, in case a particular endpoint or API version + * returns the simplified shape, so a single parser can be reused across all sample-set based + * routes. + * + * Field name constants follow Huawei's public `Field` identifiers (e.g. `steps_delta`, `calories`, + * `avg`, `max`, `min`), as documented for the on-device and REST Health Kit APIs. + */ +class FieldValues private constructor(private val values: Map) { + + fun getInt(field: String): Int? = values[field]?.let { if (it.isNull) null else it.asInt() } + + fun getLong(field: String): Long? = values[field]?.let { if (it.isNull) null else it.asLong() } + + fun getDouble(field: String): Double? = values[field]?.let { if (it.isNull) null else it.asDouble() } + + fun getFloat(field: String): Float? = getDouble(field)?.toFloat() + + fun getString(field: String): String? = values[field]?.let { if (it.isNull) null else it.asText() } + + companion object { + private const val FIELD_NAME_KEY = "fieldName" + private val VALUE_KEYS = listOf("integerValue", "floatValue", "longValue", "stringValue", "value") + + fun from(node: JsonNode?): FieldValues { + if (node == null || node.isMissingNode || node.isNull) { + return FieldValues(emptyMap()) + } + if (node.isArray) { + val map = LinkedHashMap() + node.forEach { entry -> + val name = entry.get(FIELD_NAME_KEY)?.asText() ?: return@forEach + val value = VALUE_KEYS.firstNotNullOfOrNull { key -> entry.get(key) } + if (value != null) { + map[name] = value + } + } + return FieldValues(map) + } + if (node.isObject) { + val map = LinkedHashMap() + node.properties().forEach { (name, value) -> map[name] = value } + return FieldValues(map) + } + return FieldValues(emptyMap()) + } + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt new file mode 100644 index 00000000..ebe7d40f --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt @@ -0,0 +1,85 @@ +package org.radarbase.huawei.converter + +import com.fasterxml.jackson.databind.JsonNode +import org.radarbase.huawei.user.User +import org.radarcns.connector.huawei.HuaweiActivityRecord +import java.time.Instant + +/** + * Converts `GET /healthkit/v1/activityRecords` responses into [HuaweiActivityRecord]s. + * + * Field names below follow the Huawei Health Kit `ActivityRecord`/`Device`/`ActivitySummary` + * model (activity record id, name, description, time zone, activity type, device manufacturer and + * type, and a nested activity summary with pace/data/section statistics). Nested JSON structures + * that map to free-form Avro `string` fields (pace map, data summary, section summary) are kept as + * their raw JSON text, since their internal shape varies by activity type. + */ +class HuaweiActivityRecordConverter( + private val topic: String = "connect_huawei_activity_record", +) : HuaweiDataConverter { + + override fun processRecords(root: JsonNode, user: User): Sequence> { + val timeReceived = Instant.now() + val records = root.get("activityRecords") ?: root.get("records") ?: return emptySequence() + return records.asSequence() + .mapCatching { record -> + val startTime = record.epochInstant("startTime") + ?: error("Huawei activity record is missing startTime") + TopicData( + topic = topic, + key = user.observationKey, + offset = startTime.epochSecond, + value = record.toActivityRecord(startTime, timeReceived), + ) + } + } + + private fun JsonNode.toActivityRecord( + startTime: Instant, + timeReceived: Instant, + ): HuaweiActivityRecord { + val device = this.get("device") + val summary = this.get("activitySummary") + return HuaweiActivityRecord.newBuilder().apply { + time = startTime.toEpoch() + this.timeReceived = timeReceived.toEpoch() + endTime = epochInstant("endTime")?.toEpoch() + activityRecordId = textOrNull("id") ?: textOrNull("activityRecordId") + name = textOrNull("name") + description = textOrNull("description") + timeZone = textOrNull("timeZone") + activityTypeId = textOrNull("activityType") ?: textOrNull("activityTypeId") + activeTimeMillis = longOrNull("activeTime") ?: longOrNull("activeTimeMillis") + isKeepGoing = boolOrNull("isKeepGoing") + deviceManufacturer = device?.textOrNull("manufacturer") + deviceType = device?.intOrNull("type") + activitySummaryAvgPace = summary?.doubleOrNull("avgPace") + activitySummaryBestPace = summary?.doubleOrNull("bestPace") + activitySummaryPaceMap = summary?.get("paceMap")?.toString() + activitySummaryDataSummary = summary?.get("dataSummary")?.toString() + activitySummarySectionSummary = summary?.get("sectionSummary")?.toString() + }.build() + } + + private fun JsonNode.epochInstant(field: String): Instant? { + val value = this.get(field) ?: return null + if (value.isNull) return null + val millis = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() + return millis?.let { Instant.ofEpochMilli(it) } + } + + private fun JsonNode.textOrNull(field: String): String? = + this.get(field)?.takeUnless { it.isNull }?.asText() + + private fun JsonNode.intOrNull(field: String): Int? = + this.get(field)?.takeUnless { it.isNull }?.asInt() + + private fun JsonNode.longOrNull(field: String): Long? = + this.get(field)?.takeUnless { it.isNull }?.asLong() + + private fun JsonNode.doubleOrNull(field: String): Double? = + this.get(field)?.takeUnless { it.isNull }?.asDouble() + + private fun JsonNode.boolOrNull(field: String): Boolean? = + this.get(field)?.takeUnless { it.isNull }?.asBoolean() +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt new file mode 100644 index 00000000..6b095587 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt @@ -0,0 +1,41 @@ +package org.radarbase.huawei.converter + +import com.fasterxml.jackson.databind.JsonNode +import okhttp3.Headers +import org.radarbase.huawei.request.HuaweiRequestGenerator.Companion.JSON_READER +import org.radarbase.huawei.request.RestRequest +import org.radarbase.huawei.user.User +import java.time.Instant + +/** + * Converts a Huawei Health Kit HTTP JSON response body to zero or more [TopicData] records. + */ +interface HuaweiDataConverter : RecordConverter { + /** Process the JSON records generated by given request. */ + fun processRecords( + root: JsonNode, + user: User, + ): Sequence> + + override fun convert( + request: RestRequest, + headers: Headers, + data: ByteArray, + ): List { + val node = JSON_READER.readTree(data) + + return this.processRecords(node, request.user) + .mapNotNull { r -> + r.fold( + { it }, + { + logger.error("Data conversion failed.. " + it.message) + null + }, + ) + } + .toList() + } + + fun Instant.toEpoch(): Double = this.toEpochMilli() / 1000.0 +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt new file mode 100644 index 00000000..013307bc --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt @@ -0,0 +1,49 @@ +package org.radarbase.huawei.converter + +import com.fasterxml.jackson.databind.JsonNode +import org.apache.avro.specific.SpecificRecord +import org.radarbase.huawei.user.User +import java.time.Instant + +private fun JsonNode.epochInstant(field: String): Instant? { + val value = this.get(field) ?: return null + if (value.isNull) return null + val millis = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() + return millis?.let { Instant.ofEpochMilli(it) } +} + +/** + * Generic converter for `GET /healthkit/v1/healthRecords` responses: iterates every record + * returned for the requested `subDataTypeName` and builds one Avro record per entry via + * [buildRecord]. + */ +class HuaweiHealthRecordConverter( + private val topic: String, + private val buildRecord: ( + fields: FieldValues, + startTime: Instant, + endTime: Instant?, + timeReceived: Instant, + ) -> SpecificRecord, +) : HuaweiDataConverter { + + override fun processRecords(root: JsonNode, user: User): Sequence> { + val timeReceived = Instant.now() + val records = root.get("healthRecords") ?: root.get("records") ?: return emptySequence() + return records.asSequence() + .mapCatching { record -> + val startTime = record.epochInstant("startTime") + ?: error("Huawei health record is missing startTime") + val endTime = record.epochInstant("endTime") + val fieldValues = FieldValues.from( + record.get("value") ?: record.get("fieldValues") ?: record.get("field"), + ) + TopicData( + topic = topic, + key = user.observationKey, + offset = startTime.epochSecond, + value = buildRecord(fieldValues, startTime, endTime, timeReceived), + ) + } + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt new file mode 100644 index 00000000..6e2929f4 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt @@ -0,0 +1,53 @@ +package org.radarbase.huawei.converter + +import com.fasterxml.jackson.databind.JsonNode +import org.apache.avro.specific.SpecificRecord +import org.radarbase.huawei.user.User +import java.time.Instant + +private fun JsonNode.epochInstant(field: String): Instant? { + val value = this.get(field) ?: return null + if (value.isNull) return null + val millis = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() + return millis?.let { Instant.ofEpochMilli(it) } +} + +/** + * Generic converter for `sampleSet:polymerize` responses: iterates every sample point of every + * data-type group in the response and builds one Avro record per point via [buildRecord]. + * + * This single converter is reused for the large majority of Huawei Health Kit data types, since + * they all share the same `sampleSet[].samplePoints[]` response envelope and differ only in which + * Avro record type their field values are mapped onto. + */ +class HuaweiSampleSetConverter( + private val topic: String, + private val buildRecord: ( + fields: FieldValues, + startTime: Instant, + endTime: Instant?, + timeReceived: Instant, + ) -> SpecificRecord, +) : HuaweiDataConverter { + + override fun processRecords(root: JsonNode, user: User): Sequence> { + val timeReceived = Instant.now() + val sampleSets = root.get("sampleSet") ?: root.get("sampleSets") ?: return emptySequence() + return sampleSets.asSequence() + .flatMap { group -> + (group.get("samplePoints") ?: group.get("samplePoint"))?.asSequence() ?: emptySequence() + } + .mapCatching { point -> + val startTime = point.epochInstant("startTime") + ?: error("Huawei sample point is missing startTime") + val endTime = point.epochInstant("endTime") + val fieldValues = FieldValues.from(point.get("value") ?: point.get("fieldValues")) + TopicData( + topic = topic, + key = user.observationKey, + offset = startTime.epochSecond, + value = buildRecord(fieldValues, startTime, endTime, timeReceived), + ) + } + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt new file mode 100644 index 00000000..f9e9e16b --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt @@ -0,0 +1,19 @@ +package org.radarbase.huawei.converter + +import okhttp3.Headers +import org.radarbase.huawei.request.RestRequest +import org.slf4j.LoggerFactory +import java.io.IOException + +interface RecordConverter { + @Throws(IOException::class) + fun convert( + request: RestRequest, + headers: Headers, + data: ByteArray, + ): List + + companion object { + var logger = LoggerFactory.getLogger(RecordConverter::class.java) + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt new file mode 100644 index 00000000..fe1dc73f --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt @@ -0,0 +1,11 @@ +package org.radarbase.huawei.converter + +import org.slf4j.LoggerFactory + +val logger = LoggerFactory.getLogger("org.radarbase.huawei.converter.SequenceExtensions") + +internal fun Sequence.mapCatching(fn: (T) -> S): Sequence> = map { t -> + runCatching { + fn(t) + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt new file mode 100644 index 00000000..a537af98 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt @@ -0,0 +1,11 @@ +package org.radarbase.huawei.converter + +import org.apache.avro.specific.SpecificRecord + +/** Single value for a topic. */ +data class TopicData( + val topic: String, + val key: SpecificRecord, + val value: SpecificRecord, + val offset: Long, +) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt new file mode 100644 index 00000000..9da597c0 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt @@ -0,0 +1,11 @@ +package org.radarbase.huawei.offset + +import org.radarbase.huawei.route.Route +import org.radarbase.huawei.user.User +import java.time.Instant + +data class Offset( + val user: User, + val route: Route, + val offset: Instant, +) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt new file mode 100644 index 00000000..88c67afb --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt @@ -0,0 +1,5 @@ +package org.radarbase.huawei.offset + +data class Offsets( + val offsets: List, +) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt new file mode 100644 index 00000000..03b23c34 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt @@ -0,0 +1,13 @@ +package org.radarbase.huawei.request + +import org.radarbase.huawei.offset.Offset +import org.radarbase.huawei.route.Route +import org.radarbase.huawei.user.User +import java.time.Instant + +interface HuaweiOffsetManager { + + fun getOffset(route: Route, user: User): Offset? + + fun updateOffsets(route: Route, user: User, offset: Instant) +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt new file mode 100644 index 00000000..b2d7da11 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt @@ -0,0 +1,190 @@ +package org.radarbase.huawei.request + +import com.fasterxml.jackson.core.JsonFactory +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule +import okhttp3.Response +import org.radarbase.huawei.converter.TopicData +import org.radarbase.huawei.route.Route +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserRepository +import org.slf4j.LoggerFactory +import java.io.IOException +import java.time.Duration +import java.time.Instant + +class HuaweiRequestGenerator( + private val userRepository: UserRepository, + private val huaweiOffsetManager: HuaweiOffsetManager, + val routes: List, +) : RequestGenerator { + private val routeNextRequest: MutableMap = mutableMapOf() + + var nextRequestTime: Instant = Instant.MIN + + override fun requests(user: User, max: Int): Sequence = + routes.asSequence() + .flatMap { route -> + if (routeReady(user, route)) { + generateRequests(route, user) + } else { + logger.info( + "Skip {} for {}: route in backoff until {}", + route, + user.versionedId, + routeNextRequest[routeKey(route, user)], + ) + emptySequence() + } + } + + override fun requests(route: Route, max: Int): Sequence = + userRepository.stream() + .flatMap { user -> + if (routeReady(user, route)) { + generateRequests(route, user) + } else { + logger.info( + "Skip {} for {}: route in backoff until {}", + route, + user.versionedId, + routeNextRequest[routeKey(route, user)], + ) + emptySequence() + } + } + + override fun requests(route: Route, user: User, max: Int): Sequence = + if (routeReady(user, route)) { + generateRequests(route, user) + } else { + logger.info( + "Skip {} for {}: route in backoff until {}", + route, + user.versionedId, + routeNextRequest[routeKey(route, user)], + ) + emptySequence() + } + + fun generateRequests(route: Route, user: User): Sequence { + val offset = huaweiOffsetManager.getOffset(route, user) + val startDate = user.startDate + val startOffset: Instant = if (offset == null) { + logger.info("No offsets found for $user, using the start date.") + startDate + } else { + offset.offset.coerceAtLeast(startDate) + } + val endDate = user.endDate?.coerceAtMost(Instant.now()) ?: Instant.now() + if (!startOffset.isBefore(endDate)) { + logger.info( + "Skip {} for {}: interval empty (startOffset={} >= endDate={})", + route, + user.versionedId, + startOffset, + endDate, + ) + return emptySequence() + } + return route.generateRequests(user, startOffset, endDate, USER_MAX_REQUESTS) + } + + fun handleResponse(req: RestRequest, response: Response): HuaweiResult> { + return if (response.isSuccessful) { + HuaweiResult.Success(requestSuccessful(req, response)) + } else { + try { + HuaweiResult.Error(requestFailed(req, response)) + } catch (e: TooManyRequestsException) { + HuaweiResult.Success(emptyList()) + } + } + } + + override fun requestSuccessful(request: RestRequest, response: Response): List { + logger.debug("Request successful: {}..", request.request) + val body = response.body + val data = body?.bytes() ?: ByteArray(0) + val records = request.route.converters.flatMap { it.convert(request, response.headers, data) } + val offset = records.maxByOrNull { it.offset }?.offset + val key = routeKey(request.route, request.user) + if (offset != null) { + val maxOffsetTime = Instant.ofEpochSecond(offset) + val nextOffset = maxOffsetTime.plus(OFFSET_BUFFER).coerceAtLeast(request.endDate) + huaweiOffsetManager.updateOffsets(request.route, request.user, nextOffset) + } else { + huaweiOffsetManager.updateOffsets(request.route, request.user, request.endDate) + } + routeNextRequest[key] = Instant.now().plus(SUCCESS_BACK_OFF_TIME) + return records + } + + override fun requestFailed(request: RestRequest, response: Response): HuaweiError { + val key = routeKey(request.route, request.user) + return when (response.code) { + 429 -> { + logger.info("Too many requests, rate limit reached. Backing off...") + nextRequestTime = Instant.now() + BACK_OFF_TIME + routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + HuaweiRateLimitError("Rate limit reached..", TooManyRequestsException(), "429") + } + 403 -> { + logger.warn("User {} does not have access to this Huawei Health Kit data type.", request.user) + routeNextRequest[key] = Instant.now().plus(USER_BACK_OFF_TIME) + HuaweiAccessForbiddenError( + "Huawei Health Kit scope not granted or data not available..", + IOException("Forbidden"), + "403", + ) + } + 401 -> { + logger.warn("User {} access token is expired, malformed, or revoked.", request.user) + routeNextRequest[key] = Instant.now().plus(USER_BACK_OFF_TIME) + HuaweiUnauthorizedAccessError( + "Access token expired or revoked..", + IOException("Unauthorized"), + "401", + ) + } + 400 -> { + logger.warn("Client exception for request {}", request) + routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + HuaweiClientException("Client unsupported or unauthorized..", IOException("Invalid client"), "400") + } + 422 -> { + logger.warn("Request failed (validation error): {}, {}", request, response) + routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + HuaweiValidationError(response.body?.string() ?: "validation error", IOException("Validation error"), "422") + } + 404 -> { + logger.warn("Not found: {}", request) + routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + HuaweiNotFoundError(response.body?.string() ?: "not found", IOException("Data not found"), "404") + } + else -> { + logger.warn("Request failed: {}, {}", request, response) + routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + HuaweiGenericError(response.body?.string() ?: "unknown error", IOException("Unknown error"), "500") + } + } + } + + private fun routeReady(user: User, route: Route): Boolean { + val key = routeKey(route, user) + return routeNextRequest[key]?.let { Instant.now() > it } ?: true + } + + private fun routeKey(route: Route, user: User): String = user.versionedId + "#" + route + + companion object { + private val logger = LoggerFactory.getLogger(HuaweiRequestGenerator::class.java) + private val BACK_OFF_TIME = Duration.ofMinutes(10L) + private val USER_BACK_OFF_TIME = Duration.ofHours(12L) + private val SUCCESS_BACK_OFF_TIME = Duration.ofSeconds(10L) + private val OFFSET_BUFFER = Duration.ofHours(1) + private const val USER_MAX_REQUESTS = 1000 + val JSON_FACTORY = JsonFactory() + val JSON_READER = ObjectMapper(JSON_FACTORY).registerModule(JavaTimeModule()).reader() + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt new file mode 100644 index 00000000..8a4fa587 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt @@ -0,0 +1,64 @@ +package org.radarbase.huawei.request + +sealed class HuaweiResult { + data class Success(val value: T) : HuaweiResult() + data class Error(val error: HuaweiError) : HuaweiResult() +} + +sealed interface HuaweiError + +sealed class HuaweiErrorBase( + val message: String, + val cause: Exception? = null, + val code: String, +) : HuaweiError + +class HuaweiRateLimitError(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( + message, + cause, + code, +) + +class HuaweiClientException(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( + message, + cause, + code, +) + +class HuaweiUnauthorizedAccessError( + message: String, + cause: Exception? = null, + code: String, +) : HuaweiErrorBase( + message, + cause, + code, +) + +class HuaweiAccessForbiddenError( + message: String, + cause: Exception? = null, + code: String, +) : HuaweiErrorBase( + message, + cause, + code, +) + +class HuaweiValidationError(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( + message, + cause, + code, +) + +class HuaweiGenericError(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( + message, + cause, + code, +) + +class HuaweiNotFoundError(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( + message, + cause, + code, +) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt new file mode 100644 index 00000000..39bb60e9 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt @@ -0,0 +1,19 @@ +package org.radarbase.huawei.request + +import okhttp3.Response +import org.radarbase.huawei.converter.TopicData +import org.radarbase.huawei.route.Route +import org.radarbase.huawei.user.User + +interface RequestGenerator { + + fun requests(user: User, max: Int): Sequence + + fun requests(route: Route, user: User, max: Int): Sequence + + fun requests(route: Route, max: Int): Sequence + + fun requestSuccessful(request: RestRequest, response: Response): List + + fun requestFailed(request: RestRequest, response: Response): HuaweiError +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt new file mode 100644 index 00000000..502ecc26 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt @@ -0,0 +1,14 @@ +package org.radarbase.huawei.request + +import okhttp3.Request +import org.radarbase.huawei.route.HuaweiRoute +import org.radarbase.huawei.user.User +import java.time.Instant + +data class RestRequest( + val request: Request, + val user: User, + val route: HuaweiRoute, + val startDate: Instant, + val endDate: Instant, +) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt new file mode 100644 index 00000000..3dc5aa9c --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt @@ -0,0 +1,3 @@ +package org.radarbase.huawei.request + +class TooManyRequestsException : RuntimeException() diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt new file mode 100644 index 00000000..63318b23 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt @@ -0,0 +1,46 @@ +package org.radarbase.huawei.route + +import org.radarbase.huawei.converter.HuaweiActivityRecordConverter +import org.radarbase.huawei.converter.HuaweiDataConverter +import org.radarbase.huawei.request.RestRequest +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserRepository +import java.time.Duration +import java.time.Instant + +/** + * Route backed by `GET /healthkit/v1/activityRecords`, covering the Huawei Health Kit Activity + * Records API (workout / physical-activity sessions). + */ +class HuaweiActivityRecordRoute( + userRepository: UserRepository, + private val topic: String = "connect_huawei_activity_record", + maxIntervalPerRequest: Duration = Duration.ofDays(30L), +) : HuaweiRoute(userRepository, maxIntervalPerRequest) { + + override val converters: List = listOf(HuaweiActivityRecordConverter(topic)) + + override fun toString(): String = "huawei_activity_record" + + override fun generateRequests( + user: User, + start: Instant, + end: Instant, + max: Int, + ): Sequence = chunkedRanges(start, end, max).map { (rangeStart, rangeEnd) -> + RestRequest( + request = createGetRequest( + user, + "activityRecords", + mapOf( + "startTime" to rangeStart.toEpochMilli().toString(), + "endTime" to rangeEnd.toEpochMilli().toString(), + ), + ), + user = user, + route = this, + startDate = rangeStart, + endDate = rangeEnd, + ) + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt new file mode 100644 index 00000000..ae2fa2b5 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt @@ -0,0 +1,58 @@ +package org.radarbase.huawei.route + +import org.apache.avro.specific.SpecificRecord +import org.radarbase.huawei.converter.FieldValues +import org.radarbase.huawei.converter.HuaweiDataConverter +import org.radarbase.huawei.converter.HuaweiHealthRecordConverter +import org.radarbase.huawei.request.RestRequest +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserRepository +import java.time.Duration +import java.time.Instant + +/** + * Route backed by `GET /healthkit/v1/healthRecords`, used for the `health.record.*` data types + * (ambulatory blood pressure sessions, heart rate alerts, hyperthermia, low SpO2 alerts, + * menstrual cycle phases, and comprehensive sleep records). + */ +open class HuaweiHealthRecordRoute( + userRepository: UserRepository, + private val subDataTypeName: String, + private val topic: String, + maxIntervalPerRequest: Duration = Duration.ofDays(30L), + buildRecord: ( + fields: FieldValues, + startTime: Instant, + endTime: Instant?, + timeReceived: Instant, + ) -> SpecificRecord, +) : HuaweiRoute(userRepository, maxIntervalPerRequest) { + + override val converters: List = + listOf(HuaweiHealthRecordConverter(topic, buildRecord)) + + override fun toString(): String = "huawei_" + topic.removePrefix("connect_huawei_") + + override fun generateRequests( + user: User, + start: Instant, + end: Instant, + max: Int, + ): Sequence = chunkedRanges(start, end, max).map { (rangeStart, rangeEnd) -> + RestRequest( + request = createGetRequest( + user, + "healthRecords", + mapOf( + "subDataTypeName" to subDataTypeName, + "startTime" to rangeStart.toEpochMilli().toString(), + "endTime" to rangeEnd.toEpochMilli().toString(), + ), + ), + user = user, + route = this, + startDate = rangeStart, + endDate = rangeEnd, + ) + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt new file mode 100644 index 00000000..8fd5bb01 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt @@ -0,0 +1,62 @@ +package org.radarbase.huawei.route + +import okhttp3.HttpUrl.Companion.toHttpUrl +import okhttp3.MediaType.Companion.toMediaType +import okhttp3.Request +import okhttp3.RequestBody.Companion.toRequestBody +import org.radarbase.huawei.converter.HuaweiDataConverter +import org.radarbase.huawei.request.RestRequest +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserRepository +import java.time.Duration +import java.time.Instant + +/** + * Base class for all Huawei Health Kit routes. + * + * Handles OAuth2-authorized request construction (both `GET` with query parameters and `POST` + * with a JSON body, since the Health Kit Data API mixes both styles across its endpoints) and + * generic time-range chunking, shared by all concrete route types. + */ +abstract class HuaweiRoute( + private val userRepository: UserRepository, + override val maxIntervalPerRequest: Duration = DEFAULT_INTERVAL_PER_REQUEST, +) : Route { + abstract val converters: List + + protected fun createGetRequest(user: User, path: String, queryParams: Map): Request { + val accessToken = userRepository.getAccessToken(user) + val urlBuilder = "$HUAWEI_API_BASE_URL/$path".toHttpUrl().newBuilder() + queryParams.forEach { (key, value) -> urlBuilder.addQueryParameter(key, value) } + return Request.Builder() + .url(urlBuilder.build()) + .header("Authorization", "Bearer $accessToken") + .get() + .build() + } + + protected fun createPostRequest(user: User, path: String, jsonBody: String): Request { + val accessToken = userRepository.getAccessToken(user) + return Request.Builder() + .url("$HUAWEI_API_BASE_URL/$path".toHttpUrl()) + .header("Authorization", "Bearer $accessToken") + .post(jsonBody.toRequestBody(JSON_MEDIA_TYPE)) + .build() + } + + /** Split `[start, end)` into consecutive windows of at most [maxIntervalPerRequest], capped at [max] windows. */ + protected fun chunkedRanges(start: Instant, end: Instant, max: Int): Sequence> = + generateSequence(start) { it + maxIntervalPerRequest } + .takeWhile { it < end } + .take(max) + .map { rangeStart -> rangeStart to (rangeStart + maxIntervalPerRequest).coerceAtMost(end) } + + override fun generateRequests(user: User, start: Instant, end: Instant): Sequence = + generateRequests(user, start, end, Int.MAX_VALUE) + + companion object { + const val HUAWEI_API_BASE_URL = "https://health-api.cloud.huawei.com/healthkit/v1" + private val JSON_MEDIA_TYPE = "application/json; charset=utf-8".toMediaType() + private val DEFAULT_INTERVAL_PER_REQUEST = Duration.ofDays(30L) + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt new file mode 100644 index 00000000..5a311e47 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt @@ -0,0 +1,19 @@ +package org.radarbase.huawei.route + +import org.radarbase.huawei.user.UserRepository + +/** + * A single registered Huawei Health Kit data type: a short config key (used to build + * `huawei..enabled` / `huawei..topic` connector properties), the default Kafka topic + * name, and a factory for the [HuaweiRoute] that queries it. + * + * Using one shared registry (see [HuaweiRouteFactory]) for both the Kafka Connect config + * definition and the set of routes actually polled avoids hand-duplicating each of the ~54 Huawei + * data types across a `ConfigDef` and a route-construction switch. + */ +data class HuaweiRouteDefinition( + val key: String, + val defaultTopic: String, + val enabledByDefault: Boolean = true, + val build: (UserRepository, topic: String) -> HuaweiRoute, +) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt new file mode 100644 index 00000000..2e1df2fa --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -0,0 +1,566 @@ +package org.radarbase.huawei.route + +import org.apache.avro.specific.SpecificRecord +import org.radarbase.huawei.converter.FieldValues +import org.radarcns.connector.huawei.HuaweiActiveHours +import org.radarcns.connector.huawei.HuaweiCgmBloodGlucose +import org.radarcns.connector.huawei.HuaweiContinuousActivityStatistics +import org.radarcns.connector.huawei.HuaweiContinuousAltitudeStatistics +import org.radarcns.connector.huawei.HuaweiContinuousBloodGlucoseStatistics +import org.radarcns.connector.huawei.HuaweiContinuousBodyBloodPressureStatistics +import org.radarcns.connector.huawei.HuaweiContinuousBreatheRateStatistics +import org.radarcns.connector.huawei.HuaweiContinuousCaloriesBurnt +import org.radarcns.connector.huawei.HuaweiContinuousCaloriesBurntTotal +import org.radarcns.connector.huawei.HuaweiContinuousDistanceDelta +import org.radarcns.connector.huawei.HuaweiContinuousDistanceTotal +import org.radarcns.connector.huawei.HuaweiContinuousEcgDetail +import org.radarcns.connector.huawei.HuaweiContinuousExerciseIntensity +import org.radarcns.connector.huawei.HuaweiContinuousExerciseIntensityV2 +import org.radarcns.connector.huawei.HuaweiContinuousExerciseIntensityV2Statistics +import org.radarcns.connector.huawei.HuaweiContinuousSleepFragment +import org.radarcns.connector.huawei.HuaweiContinuousSpo2Statistics +import org.radarcns.connector.huawei.HuaweiContinuousStepsDelta +import org.radarcns.connector.huawei.HuaweiContinuousStepsTotal +import org.radarcns.connector.huawei.HuaweiDailyActivitySummary +import org.radarcns.connector.huawei.HuaweiEmotion +import org.radarcns.connector.huawei.HuaweiHealthRecordDynamicBp +import org.radarcns.connector.huawei.HuaweiHealthRecordHeartRateAlert +import org.radarcns.connector.huawei.HuaweiHealthRecordHyperthermia +import org.radarcns.connector.huawei.HuaweiHealthRecordLowSpo2Alert +import org.radarcns.connector.huawei.HuaweiHealthRecordMenstrualCycle +import org.radarcns.connector.huawei.HuaweiHealthRecordSleep +import org.radarcns.connector.huawei.HuaweiHeartRateVariability +import org.radarcns.connector.huawei.HuaweiRestingCaloriesStatistics +import org.radarcns.connector.huawei.HuaweiSleepOnOffBedRecord +import org.radarcns.connector.huawei.HuaweiSleepRespiratoryDetail +import org.radarcns.connector.huawei.HuaweiSleepRespiratoryEvent +import org.radarcns.connector.huawei.HuaweiStatistics +import org.radarcns.connector.huawei.HuaweiVo2Max +import java.time.Instant + +/** + * Registry of every Huawei Health Kit data type this connector supports, mapping each one to the + * Kafka Connect REST endpoint (`sampleSet:polymerize`, `healthRecords`, or `activityRecords`) and + * Avro record type documented in the `radar-huawei-connector` schema specification + * (RADAR-base/RADAR-Schemas, `huawei_schemas` branch). + * + * Huawei `dataTypeName`/`subDataTypeName` values below are taken verbatim from that + * specification's `doc` strings (prefixed with the vendor namespace `com.huawei.`), which in turn + * describe the Huawei Health Kit REST Data API's own data type identifiers. + * + * Field-value key names used in the record builders are Huawei Health Kit `Field` identifiers + * (snake_case, matching the on-device HiHealth SDK's public `Field.FIELD_*` constant family, e.g. + * `steps_delta`, `calories`, `avg`/`max`/`min`/`last`). Where a field is not among Huawei's widely + * documented constants, the snake_case form of the Avro field's own name is used as a best-effort + * default (see [snake]) — verify against a live API response and adjust the key strings in this + * file if Huawei's actual response uses different names. + */ +object HuaweiRouteFactory { + + private const val VENDOR_PREFIX = "com.huawei." + + private fun Instant.toEpoch(): Double = toEpochMilli() / 1000.0 + + /** Best-effort camelCase -> snake_case conversion for deriving a Huawei field key from an Avro field name. */ + private fun snake(name: String): String = + SNAKE_CASE_BOUNDARY.replace(name) { "${it.groupValues[1]}_${it.groupValues[2]}" }.lowercase() + + private val SNAKE_CASE_BOUNDARY = Regex("([a-z0-9])([A-Z])") + + private fun HuaweiStatistics.Builder.populateCommon( + startTime: Instant, + endTime: Instant?, + timeReceived: Instant, + fields: FieldValues, + ) { + time = startTime.toEpoch() + this.timeReceived = timeReceived.toEpoch() + this.endTime = endTime?.toEpoch() + avg = fields.getDouble("avg") + max = fields.getDouble("max") + min = fields.getDouble("min") + last = fields.getDouble("last") + count = fields.getInt("count") + } + + /** Data types that reuse the generic [HuaweiStatistics] schema: (config key, Huawei data type name, default topic). */ + private val genericStatisticsTypes = listOf( + Triple("continuous_body_fat_rate_statistics", "continuous.body.fat.rate.statistics", "connect_huawei_continuous_body_fat_rate_statistics"), + Triple("continuous_body_temperature_rest_statistics", "continuous.body.temperature.rest.statistics", "connect_huawei_continuous_body_temperature_rest_statistics"), + Triple("continuous_body_temperature_statistics", "continuous.body.temperature.statistics", "connect_huawei_continuous_body_temperature_statistics"), + Triple("continuous_calories_bmr_statistics", "continuous.calories.bmr.statistics", "connect_huawei_continuous_calories_bmr_statistics"), + Triple("continuous_exercise_heart_rate_statistics", "continuous.exercise_heart_rate.statistics", "connect_huawei_continuous_exercise_heart_rate_statistics"), + Triple("continuous_heart_rate_statistics", "continuous.heart_rate.statistics", "connect_huawei_continuous_heart_rate_statistics"), + Triple("continuous_power_statistics", "continuous.power.statistics", "connect_huawei_continuous_power_statistics"), + Triple("continuous_skin_temperature_statistics", "continuous.skin.temperature.statistics", "connect_huawei_continuous_skin_temperature_statistics"), + Triple("continuous_speed_statistics", "continuous.speed.statistics", "connect_huawei_continuous_speed_statistics"), + Triple("continuous_steps_rate_statistics", "continuous.steps.rate.statistics", "connect_huawei_continuous_steps_rate_statistics"), + Triple("continuous_stroke_rate_statistics", "continuous.stroke_rate.statistics", "connect_huawei_continuous_stroke_rate_statistics"), + Triple("instantaneous_resting_heart_rate_statistics", "instantaneous.resting_heart_rate.statistics", "connect_huawei_instantaneous_resting_heart_rate_statistics"), + Triple("instantaneous_stress_statistics", "instantaneous.stress.statistics", "connect_huawei_instantaneous_stress_statistics"), + Triple("vo2max_statistics", "vo2max.statistics", "connect_huawei_vo2max_statistics"), + ) + + /** Full registry of Huawei Health Kit data types supported by this connector. */ + val definitions: List = buildList { + add( + HuaweiRouteDefinition("activity_record", "connect_huawei_activity_record") { repo, topic -> + HuaweiActivityRecordRoute(repo, topic) + }, + ) + + // cgm_blood_glucose (+ .statistics variant) + add(sampleSetDefinition("cgm_blood_glucose", "cgm_blood_glucose", "connect_huawei_cgm_blood_glucose") { f, start, _, received -> + HuaweiCgmBloodGlucose.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch() + level = f.getDouble("level") + avg = f.getInt("avg"); max = f.getInt("max"); min = f.getInt("min"); last = f.getInt("last") + }.build() + }) + add(sampleSetDefinition("cgm_blood_glucose_statistics", "cgm_blood_glucose.statistics", "connect_huawei_cgm_blood_glucose_statistics") { f, start, _, received -> + HuaweiCgmBloodGlucose.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch() + level = f.getDouble("level") + avg = f.getInt("avg"); max = f.getInt("max"); min = f.getInt("min"); last = f.getInt("last") + }.build() + }) + + add(sampleSetDefinition("daily_activity_summary", "daily_activity_summary", "connect_huawei_daily_activity_summary") { f, start, end, received -> + HuaweiDailyActivitySummary.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + steps = f.getInt("steps") + activeCalories = f.getInt("calories") + exerciseTime = f.getInt("exercise_time") + activeHours = f.getInt("active_hours") + stepsGoal = f.getInt("steps_target") + activeCaloriesGoal = f.getInt("calories_target") + exerciseTimeGoal = f.getInt("exercise_time_target") + activeHoursGoal = f.getInt("active_hours_target") + }.build() + }) + + add(sampleSetDefinition("active_hours", "active_hours", "connect_huawei_active_hours") { f, start, end, received -> + f.toActiveHours(start, end, received) + }) + add(sampleSetDefinition("active_hours_statistics", "active_hours.statistics", "connect_huawei_active_hours_statistics") { f, start, end, received -> + f.toActiveHours(start, end, received) + }) + + add(sampleSetDefinition("continuous_activity_fragment", "continuous.activity.fragment", "connect_huawei_continuous_activity_fragment") { f, start, end, received -> + f.toContinuousActivityStatistics(start, end, received) + }) + add(sampleSetDefinition("continuous_activity_statistics", "continuous.activity.statistics", "connect_huawei_continuous_activity_statistics") { f, start, end, received -> + f.toContinuousActivityStatistics(start, end, received) + }) + + add(sampleSetDefinition("continuous_altitude_statistics", "continuous.altitude.statistics", "connect_huawei_continuous_altitude_statistics") { f, start, end, received -> + HuaweiContinuousAltitudeStatistics.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + avg = f.getDouble("avg"); max = f.getDouble("max"); min = f.getDouble("min") + ascentTotal = f.getDouble("ascent_total") + descentTotal = f.getDouble("descent_total") + }.build() + }) + + add(sampleSetDefinition("continuous_blood_glucose_statistics", "continuous.blood_glucose.statistics", "connect_huawei_continuous_blood_glucose_statistics") { f, start, end, received -> + HuaweiContinuousBloodGlucoseStatistics.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + avg = f.getDouble("avg"); max = f.getDouble("max"); min = f.getDouble("min") + correlationWithMealtime = f.getInt("correlate_mealtime") + meal = f.getInt("meal") + correlationWithSleepState = f.getInt("correlate_sleep") + sampleSource = f.getInt("sample_source") + }.build() + }) + + add(sampleSetDefinition("continuous_breathe_rate_statistics", "continuous.breathe_rate.statistics", "connect_huawei_continuous_breathe_rate_statistics") { f, start, end, received -> + HuaweiContinuousBreatheRateStatistics.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + maxBreatheRate = f.getInt("max_breathe_rate") + minBreatheRate = f.getInt("min_breathe_rate") + avgBreatheRate = f.getInt("avg_breathe_rate") + minBreathrateBaseline = f.getInt("min_breathrate_baseline") + maxBreathrateBaseline = f.getInt("max_breathrate_baseline") + }.build() + }) + + add(sampleSetDefinition("continuous_body_blood_pressure_statistics", "continuous.body.blood_pressure.statistics", "connect_huawei_continuous_body_blood_pressure_statistics") { f, start, end, received -> + HuaweiContinuousBodyBloodPressureStatistics.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + systolicPressureAvg = f.getDouble("systolic_pressure_avg") + systolicPressureMax = f.getDouble("systolic_pressure_max") + systolicPressureMin = f.getDouble("systolic_pressure_min") + diastolicPressureAvg = f.getDouble("diastolic_pressure_avg") + diastolicPressureMax = f.getDouble("diastolic_pressure_max") + diastolicPressureMin = f.getDouble("diastolic_pressure_min") + sphygmusAvg = f.getDouble("sphygmus_avg") + sphygmusMax = f.getDouble("sphygmus_max") + sphygmusMin = f.getDouble("sphygmus_min") + sphygmusLast = f.getDouble("sphygmus_last") + }.build() + }) + + genericStatisticsTypes.forEach { (key, dataType, topic) -> + add( + sampleSetDefinition(key, dataType, topic) { f, start, end, received -> + HuaweiStatistics.newBuilder().apply { populateCommon(start, end, received, f) }.build() + }, + ) + } + + add(sampleSetDefinition("continuous_calories_burnt", "continuous.calories.burnt", "connect_huawei_continuous_calories_burnt") { f, start, end, received -> + HuaweiContinuousCaloriesBurnt.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + calories = f.getDouble("calories") + }.build() + }) + add(sampleSetDefinition("continuous_calories_consumed", "continuous.calories.consumed", "connect_huawei_continuous_calories_consumed") { f, start, end, received -> + HuaweiContinuousCaloriesBurnt.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + calories = f.getDouble("calories") + }.build() + }) + add(sampleSetDefinition("continuous_calories_burnt_total", "continuous.calories.burnt.total", "connect_huawei_continuous_calories_burnt_total") { f, start, end, received -> + HuaweiContinuousCaloriesBurntTotal.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + caloriesTotal = f.getDouble("calories_total") + }.build() + }) + + add(sampleSetDefinition("continuous_distance_delta", "continuous.distance.delta", "connect_huawei_continuous_distance_delta") { f, start, end, received -> + HuaweiContinuousDistanceDelta.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + distanceDelta = f.getDouble("distance_delta") + }.build() + }) + add(sampleSetDefinition("continuous_distance_total", "continuous.distance.total", "connect_huawei_continuous_distance_total") { f, start, end, received -> + HuaweiContinuousDistanceTotal.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + distance = f.getDouble("distance_total") + }.build() + }) + + add(sampleSetDefinition("continuous_ecg_detail", "continuous.ecg_detail", "connect_huawei_continuous_ecg_detail") { f, start, end, received -> + HuaweiContinuousEcgDetail.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + ecgRecordId = f.getString("record_id") + averageHeartRate = f.getInt("avg_heart_rate") + ecgArrhythmiaType = f.getInt("arrhythmia_type") + ecgArrhythmiaResult = f.getInt("arrhythmia_result") + userSymptom = f.getString("user_symptom") + samplingFrequency = f.getInt("sampling_frequency") + voltageData = f.getString("voltage_data") + }.build() + }) + + add(sampleSetDefinition("continuous_exercise_intensity", "continuous.exercise_intensity", "connect_huawei_continuous_exercise_intensity") { f, start, end, received -> + f.toContinuousExerciseIntensity(start, end, received) + }) + add(sampleSetDefinition("continuous_exercise_intensity_statistics", "continuous.exercise_intensity.statistics", "connect_huawei_continuous_exercise_intensity_statistics") { f, start, end, received -> + f.toContinuousExerciseIntensity(start, end, received) + }) + + add(sampleSetDefinition("continuous_exercise_intensity_v2", "continuous.exercise_intensity.v2", "connect_huawei_continuous_exercise_intensity_v2") { f, start, end, received -> + HuaweiContinuousExerciseIntensityV2.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + exerciseType = f.getInt("exercise_type") + }.build() + }) + add(sampleSetDefinition("continuous_exercise_intensity_v2_statistics", "continuous.exercise_intensity.v2.statistics", "connect_huawei_continuous_exercise_intensity_v2_statistics") { f, start, end, received -> + HuaweiContinuousExerciseIntensityV2Statistics.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + zone1Duration = f.getInt("zone1_duration") + zone2Duration = f.getInt("zone2_duration") + zone3Duration = f.getInt("zone3_duration") + zone4Duration = f.getInt("zone4_duration") + zone5Duration = f.getInt("zone5_duration") + }.build() + }) + + add(sampleSetDefinition("continuous_sleep_fragment", "continuous.sleep.fragment", "connect_huawei_continuous_sleep_fragment") { f, start, end, received -> + HuaweiContinuousSleepFragment.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + sleepState = f.getInt("sleep_state") + }.build() + }) + + add(sampleSetDefinition("continuous_spo2_statistics", "continuous.spo2.statistics", "connect_huawei_continuous_spo2_statistics") { f, start, end, received -> + HuaweiContinuousSpo2Statistics.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + saturationAvg = f.getDouble("avg") + saturationMax = f.getDouble("max") + saturationMin = f.getDouble("min") + saturationLast = f.getDouble("last") + }.build() + }) + + add(sampleSetDefinition("continuous_steps_delta", "continuous.steps.delta", "connect_huawei_continuous_steps_delta") { f, start, end, received -> + HuaweiContinuousStepsDelta.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + stepsDelta = f.getInt("steps_delta") + }.build() + }) + add(sampleSetDefinition("continuous_steps_total", "continuous.steps.total", "connect_huawei_continuous_steps_total") { f, start, end, received -> + HuaweiContinuousStepsTotal.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + steps = f.getInt("steps") + duration = f.getInt("duration") + }.build() + }) + + add(sampleSetDefinition("emotion", "emotion", "connect_huawei_emotion") { f, start, _, received -> + HuaweiEmotion.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch() + emotionStatus = f.getInt("emotion") + }.build() + }) + + add(healthRecordDefinition("health_record_dynamic_bp", "health.record.dynamic_bp", "connect_huawei_health_record_dynamic_bp") { f, start, end, received -> + f.toHealthRecordDynamicBp(start, end, received) + }) + add(healthRecordDefinition("health_record_bradycardia", "health.record.bradycardia", "connect_huawei_health_record_bradycardia") { f, start, end, received -> + f.toHealthRecordHeartRateAlert(start, end, received) + }) + add(healthRecordDefinition("health_record_tachycardia", "health.record.tachycardia", "connect_huawei_health_record_tachycardia") { f, start, end, received -> + f.toHealthRecordHeartRateAlert(start, end, received) + }) + add(healthRecordDefinition("health_record_hyperthermia", "health.record.hyperthermia", "connect_huawei_health_record_hyperthermia") { f, start, end, received -> + HuaweiHealthRecordHyperthermia.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + highBodyTemperatureAlarm = f.getFloat("high_body_temperature_alarm") + }.build() + }) + add(healthRecordDefinition("health_record_low_spo2_alert", "health.record.lowSpo2Alert", "connect_huawei_health_record_low_spo2_alert") { f, start, end, received -> + HuaweiHealthRecordLowSpo2Alert.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + threshold = f.getFloat("threshold") + maxSpO2 = f.getFloat("max_spo2") + minSpO2 = f.getFloat("min_spo2") + }.build() + }) + add(healthRecordDefinition("health_record_menstrual_cycle", "health.record.menstrual_cycle", "connect_huawei_health_record_menstrual_cycle") { f, start, end, received -> + HuaweiHealthRecordMenstrualCycle.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + recordday = f.getInt("record_day") + status = f.getInt("status") + substatus = f.getInt("sub_status") + remarks = f.getString("remarks") + timezone = f.getString("timezone") + }.build() + }) + add(healthRecordDefinition("health_record_sleep", "health.record.sleep", "connect_huawei_health_record_sleep") { f, start, end, received -> + HuaweiHealthRecordSleep.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + fallAsleepTime = f.getLong("fall_asleep_time") + wakeupTime = f.getLong("wakeup_time") + lightSleepTime = f.getInt("light_sleep_time") + deepSleepTime = f.getInt("deep_sleep_time") + dreamTime = f.getInt("dream_time") + awakeTime = f.getInt("awake_time") + allSleepTime = f.getInt("all_sleep_time") + wakeupCount = f.getInt("wakeup_count") + deepSleepPart = f.getInt("deep_sleep_part") + sleepScore = f.getInt("sleep_score") + sleepLatency = f.getInt("sleep_latency") + sleepEfficiency = f.getInt("sleep_efficiency") + goBedTime = f.getLong("go_bed_time") + sleepType = f.getInt("sleep_type") + prepareSleepTime = f.getLong("prepare_sleep_time") + offBedTime = f.getLong("off_bed_time") + }.build() + }) + + add(sampleSetDefinition("heart_rate_variability", "heart_rate_variability", "connect_huawei_heart_rate_variability") { f, start, _, received -> + HuaweiHeartRateVariability.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch() + heartRateVariabilityRmssd = f.getInt("heart_rate_variability_rmssd") + }.build() + }) + + add(sampleSetDefinition("resting_calories_statistics", "resting_calories.statistics", "connect_huawei_resting_calories_statistics") { f, start, end, received -> + HuaweiRestingCaloriesStatistics.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + predictedCalories = f.getFloat("predicted_calories") + totalCalories = f.getFloat("total_calories") + }.build() + }) + + add(sampleSetDefinition("sleep_on_off_bed_record", "sleep.on_off_bed_record", "connect_huawei_sleep_on_off_bed_record") { f, start, _, received -> + HuaweiSleepOnOffBedRecord.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch() + onOffBedState = f.getInt("on_off_bed_state") + }.build() + }) + + add(sampleSetDefinition("sleep_respiratory_detail", "sleep_respiratory_detail", "connect_huawei_sleep_respiratory_detail") { f, start, end, received -> + HuaweiSleepRespiratoryDetail.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + type = f.getInt("type") + value = f.getDouble("value") + }.build() + }) + add(sampleSetDefinition("sleep_respiratory_event", "sleep_respiratory_event", "connect_huawei_sleep_respiratory_event") { f, start, end, received -> + HuaweiSleepRespiratoryEvent.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + eventname = f.getInt("event_name") + }.build() + }) + + add(sampleSetDefinition("vo2max", "vo2max", "connect_huawei_vo2max") { f, start, _, received -> + HuaweiVo2Max.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch() + vo2max = f.getInt("vo2max") + }.build() + }) + } + + private fun FieldValues.toActiveHours( + start: Instant, + end: Instant?, + received: Instant, + ): HuaweiActiveHours = HuaweiActiveHours.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + activeHours = getInt("active_hours") + moderateIntensityMinutes = getInt("moderate_intensity_minutes") + highIntensityMinutes = getInt("high_intensity_minutes") + }.build() + + private fun FieldValues.toContinuousActivityStatistics( + start: Instant, + end: Instant?, + received: Instant, + ): HuaweiContinuousActivityStatistics = HuaweiContinuousActivityStatistics.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + typeOfActivity = getInt("activity_type") + span = getInt("span") + fragments = getInt("fragments") + }.build() + + private fun FieldValues.toContinuousExerciseIntensity( + start: Instant, + end: Instant?, + received: Instant, + ): HuaweiContinuousExerciseIntensity = HuaweiContinuousExerciseIntensity.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + intensity = getDouble("intensity") + span = getInt("span") + }.build() + + private fun FieldValues.toHealthRecordHeartRateAlert( + start: Instant, + end: Instant?, + received: Instant, + ): HuaweiHealthRecordHeartRateAlert = HuaweiHealthRecordHeartRateAlert.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + threshold = getDouble("threshold") + avgHeartRate = getDouble("avg_heart_rate") + maxHeartRate = getDouble("max_heart_rate") + minHeartRate = getDouble("min_heart_rate") + }.build() + + /** + * The 24h ambulatory blood pressure monitoring record has ~80 numeric fields, all following + * the same `` naming (e.g. `avgSystolicBpAll`, `maxHeartRateWake`). + * [snake] derives each Huawei field key mechanically from the Avro field name to avoid + * hand-transcribing ~80 near-identical key strings. + */ + private fun FieldValues.toHealthRecordDynamicBp( + start: Instant, + end: Instant?, + received: Instant, + ): HuaweiHealthRecordDynamicBp { + val f = this + fun i(name: String) = f.getInt(snake(name)) + fun d(name: String) = f.getDouble(snake(name)) + fun l(name: String) = f.getLong(snake(name)) + return HuaweiHealthRecordDynamicBp.newBuilder().apply { + time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + planId = f.getString(snake("planId")) + planStartTime = l("planStartTime") + planEndTime = l("planEndTime") + planActualTime = l("planActualTime") + planStatus = i("planStatus") + gasBagType = i("gasBagType") + sleepStartTime = l("sleepStartTime") + sleepEndTime = l("sleepEndTime") + + validCntAll = i("validCntAll"); cntAll = i("cntAll") + maxSystolicBpAll = i("maxSystolicBpAll"); maxDiastolicBpAll = i("maxDiastolicBpAll"); maxHeartRateAll = i("maxHeartRateAll") + midSystolicBpAll = i("midSystolicBpAll"); midDiastolicBpAll = i("midDiastolicBpAll"); midHeartRateAll = i("midHeartRateAll") + minSystolicBpAll = i("minSystolicBpAll"); minDiastolicBpAll = i("minDiastolicBpAll"); minHeartRateAll = i("minHeartRateAll") + avgSystolicBpAll = i("avgSystolicBpAll"); avgDiastolicBpAll = i("avgDiastolicBpAll"); avgHeartRateAll = i("avgHeartRateAll") + stdSystolicBpAll = i("stdSystolicBpAll"); stdDiastolicBpAll = i("stdDiastolicBpAll"); stdHeartRateAll = i("stdHeartRateAll") + coefSystolicBpAll = d("coefSystolicBpAll"); coefDiastolicBpAll = d("coefDiastolicBpAll"); coefHeartRateAll = d("coefHeartRateAll") + loadSystolicBpAll = d("loadSystolicBpAll"); loadDiastolicBpAll = d("loadDiastolicBpAll") + dropSystolicBpAll = d("dropSystolicBpAll"); dropDiastolicBpAll = d("dropDiastolicBpAll") + peakSystolicBpAll = i("peakSystolicBpAll"); peakDiastolicBpAll = i("peakDiastolicBpAll") + + validCntWake = i("validCntWake"); cntWake = i("cntWake") + maxSystolicBpWake = i("maxSystolicBpWake"); maxDiastolicBpWake = i("maxDiastolicBpWake"); maxHeartRateWake = i("maxHeartRateWake") + midSystolicBpWake = i("midSystolicBpWake"); midDiastolicBpWake = i("midDiastolicBpWake"); midHeartRateWake = i("midHeartRateWake") + minSystolicBpWake = i("minSystolicBpWake"); minDiastolicBpWake = i("minDiastolicBpWake"); minHeartRateWake = i("minHeartRateWake") + avgSystolicBpWake = i("avgSystolicBpWake"); avgDiastolicBpWake = i("avgDiastolicBpWake"); avgHeartRateWake = i("avgHeartRateWake") + stdSystolicBpWake = i("stdSystolicBpWake"); stdDiastolicBpWake = i("stdDiastolicBpWake"); stdHeartRateWake = i("stdHeartRateWake") + coefSystolicBpWake = d("coefSystolicBpWake"); coefDiastolicBpWake = d("coefDiastolicBpWake"); coefHeartRateWake = d("coefHeartRateWake") + loadSystolicBpWake = d("loadSystolicBpWake"); loadDiastolicBpWake = d("loadDiastolicBpWake") + + validCntSleep = i("validCntSleep"); cntSleep = i("cntSleep") + maxSystolicBpSleep = i("maxSystolicBpSleep"); maxDiastolicBpSleep = i("maxDiastolicBpSleep"); maxHeartRateSleep = i("maxHeartRateSleep") + midSystolicBpSleep = i("midSystolicBpSleep"); midDiastolicBpSleep = i("midDiastolicBpSleep"); midHeartRateSleep = i("midHeartRateSleep") + minSystolicBpSleep = i("minSystolicBpSleep"); minDiastolicBpSleep = i("minDiastolicBpSleep"); minHeartRateSleep = i("minHeartRateSleep") + avgSystolicBpSleep = i("avgSystolicBpSleep"); avgDiastolicBpSleep = i("avgDiastolicBpSleep"); avgHeartRateSleep = i("avgHeartRateSleep") + stdSystolicBpSleep = i("stdSystolicBpSleep"); stdDiastolicBpSleep = i("stdDiastolicBpSleep"); stdHeartRateSleep = i("stdHeartRateSleep") + coefSystolicBpSleep = d("coefSystolicBpSleep"); coefDiastolicBpSleep = d("coefDiastolicBpSleep"); coefHeartRateSleep = d("coefHeartRateSleep") + loadSystolicBpSleep = d("loadSystolicBpSleep"); loadDiastolicBpSleep = d("loadDiastolicBpSleep") + + validCntWakeTwo = i("validCntWakeTwo"); cntWakeTwo = i("cntWakeTwo") + maxSystolicBpWakeTwo = i("maxSystolicBpWakeTwo"); maxDiastolicBpWakeTwo = i("maxDiastolicBpWakeTwo"); maxHeartRateWakeTwo = i("maxHeartRateWakeTwo") + midSystolicBpWakeTwo = i("midSystolicBpWakeTwo"); midDiastolicBpWakeTwo = i("midDiastolicBpWakeTwo"); midHeartRateWakeTwo = i("midHeartRateWakeTwo") + minSystolicBpWakeTwo = i("minSystolicBpWakeTwo"); minDiastolicBpWakeTwo = i("minDiastolicBpWakeTwo"); minHeartRateWakeTwo = i("minHeartRateWakeTwo") + avgSystolicBpWakeTwo = i("avgSystolicBpWakeTwo"); avgDiastolicBpWakeTwo = i("avgDiastolicBpWakeTwo"); avgHeartRateWakeTwo = i("avgHeartRateWakeTwo") + stdSystolicBpWakeTwo = i("stdSystolicBpWakeTwo"); stdDiastolicBpWakeTwo = i("stdDiastolicBpWakeTwo"); stdHeartRateWakeTwo = i("stdHeartRateWakeTwo") + coefSystolicBpWakeTwo = d("coefSystolicBpWakeTwo"); coefDiastolicBpWakeTwo = d("coefDiastolicBpWakeTwo"); coefHeartRateWakeTwo = d("coefHeartRateWakeTwo") + + extendData = f.getString("extend_data") + }.build() + } + + private fun sampleSetDefinition( + key: String, + dataTypeSuffix: String, + defaultTopic: String, + buildRecord: ( + fields: FieldValues, + startTime: Instant, + endTime: Instant?, + timeReceived: Instant, + ) -> SpecificRecord, + ): HuaweiRouteDefinition = HuaweiRouteDefinition(key, defaultTopic) { repo, topic -> + HuaweiSampleSetRoute( + userRepository = repo, + dataTypeName = VENDOR_PREFIX + dataTypeSuffix, + topic = topic, + groupByTimeUnit = if (dataTypeSuffix.endsWith(".statistics")) "day" else null, + buildRecord = buildRecord, + ) + } + + private fun healthRecordDefinition( + key: String, + subDataTypeName: String, + defaultTopic: String, + buildRecord: ( + fields: FieldValues, + startTime: Instant, + endTime: Instant?, + timeReceived: Instant, + ) -> SpecificRecord, + ): HuaweiRouteDefinition = HuaweiRouteDefinition(key, defaultTopic) { repo, topic -> + HuaweiHealthRecordRoute( + userRepository = repo, + subDataTypeName = VENDOR_PREFIX + subDataTypeName, + topic = topic, + buildRecord = buildRecord, + ) + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt new file mode 100644 index 00000000..ddea128f --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt @@ -0,0 +1,75 @@ +package org.radarbase.huawei.route + +import com.fasterxml.jackson.databind.ObjectMapper +import org.apache.avro.specific.SpecificRecord +import org.radarbase.huawei.converter.FieldValues +import org.radarbase.huawei.converter.HuaweiDataConverter +import org.radarbase.huawei.converter.HuaweiSampleSetConverter +import org.radarbase.huawei.request.RestRequest +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserRepository +import java.time.Duration +import java.time.Instant + +/** + * Route backed by `POST /healthkit/v1/sampleSet:polymerize`, which covers the large majority of + * Huawei Health Kit data types (all `continuous.*`, `instantaneous.*`, `cgm_blood_glucose`, + * `active_hours`, `daily_activity_summary`, `emotion`, `heart_rate_variability`, `vo2max`, + * `resting_calories.statistics`, `sleep.on_off_bed_record`, and `sleep_respiratory_*` types). + * + * When [groupByTimeUnit] is set, the request aggregates sample points into buckets of that size — + * this is how Huawei's `.statistics` data types are queried. When it is `null`, the endpoint + * returns raw, un-aggregated sample points for [dataTypeName] over the requested time range. + */ +open class HuaweiSampleSetRoute( + userRepository: UserRepository, + private val dataTypeName: String, + private val topic: String, + private val groupByTimeUnit: String? = null, + maxIntervalPerRequest: Duration = Duration.ofDays(30L), + buildRecord: ( + fields: FieldValues, + startTime: Instant, + endTime: Instant?, + timeReceived: Instant, + ) -> SpecificRecord, +) : HuaweiRoute(userRepository, maxIntervalPerRequest) { + + override val converters: List = + listOf(HuaweiSampleSetConverter(topic, buildRecord)) + + override fun toString(): String = "huawei_" + topic.removePrefix("connect_huawei_") + + override fun generateRequests( + user: User, + start: Instant, + end: Instant, + max: Int, + ): Sequence = chunkedRanges(start, end, max).map { (rangeStart, rangeEnd) -> + RestRequest( + request = createPostRequest(user, "sampleSet:polymerize", buildRequestBody(rangeStart, rangeEnd)), + user = user, + route = this, + startDate = rangeStart, + endDate = rangeEnd, + ) + } + + private fun buildRequestBody(start: Instant, end: Instant): String { + val root = MAPPER.createObjectNode() + root.putArray("polymerizeWith").addObject().put("dataTypeName", dataTypeName) + root.put("startTime", start.toEpochMilli()) + root.put("endTime", end.toEpochMilli()) + if (groupByTimeUnit != null) { + val groupPeriod = root.putObject("groupByTime").putObject("groupPeriod") + groupPeriod.put("unit", groupByTimeUnit) + groupPeriod.put("value", 1) + groupPeriod.put("timeZone", "+0000") + } + return MAPPER.writeValueAsString(root) + } + + companion object { + private val MAPPER = ObjectMapper() + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt new file mode 100644 index 00000000..39e2cf71 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt @@ -0,0 +1,23 @@ +package org.radarbase.huawei.route + +import org.radarbase.huawei.request.RestRequest +import org.radarbase.huawei.user.User +import java.time.Duration +import java.time.Instant + +interface Route { + + fun generateRequests(user: User, start: Instant, end: Instant): Sequence + + fun generateRequests(user: User, start: Instant, end: Instant, max: Int): Sequence + + /** + * This is how it would appear in the offsets + */ + override fun toString(): String + + /** + * The duration of data to request in a single request of this route. + */ + val maxIntervalPerRequest: Duration +} diff --git a/kafka-connect-huawei-source/build.gradle.kts b/kafka-connect-huawei-source/build.gradle.kts index f3169abe..05fcfa3c 100644 --- a/kafka-connect-huawei-source/build.gradle.kts +++ b/kafka-connect-huawei-source/build.gradle.kts @@ -1,6 +1,11 @@ description = "Kafka connector for Huawei Health Kit API source" repositories { + // Prefer a locally-published snapshot (e.g. built by hand from the RADAR-Schemas + // huawei_schemas branch via `./gradlew :radar-schemas-commons:publishToMavenLocal`) before + // falling back to remote snapshot hosts. + mavenLocal() + // radar-schemas-commons huawei_schemas is only published as a snapshot; declare the // candidate snapshot hosts here so the build can resolve it regardless of which one the // RADAR-Schemas release pipeline currently targets. From 8b394ed5b1f547ceb8bd93687bb201f85eaebd9d Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 11:13:31 +0000 Subject: [PATCH 04/44] Implement kafka-connect-huawei-source Kafka Connect glue Mirrors kafka-connect-oura-source: HuaweiSourceConnector (periodic user refresh + task reconfiguration), HuaweiSourceTask (round-robin polling across routes, JSON->Avro->SourceRecord conversion via AvroData), KafkaOffsetManager, and HuaweiServiceUserRepository (Ktor-based rest-source-authorizer client with OAuth2 client-credentials auth and cached user/token lookups). HuaweiRestSourceConnectorConfig is written in Kotlin and loop-generates its ~110 per-data-type `huawei..enabled`/`huawei..topic` ConfigDef entries from huawei-library's HuaweiRouteFactory.definitions registry, rather than hand-duplicating a static field per topic as Fitbit/Oura do - the same registry also drives which routes HuaweiSourceTask actually builds, so the connector config and the set of polled routes can never drift out of sync. Could not compile this module in this sandbox: packages.confluent.io (needed for kafka-connect-api/kafka-connect-avro-converter) is blocked by the sandbox's egress policy - confirmed this is pre-existing and applies equally to kafka-connect-oura-source, not something introduced here. huawei-library (the part of this change with real logic to verify) does compile cleanly against a locally-published radar-schemas-commons 0.9.0-SNAPSHOT. --- .../huawei/AbstractRestSourceConnector.java | 57 ++++ .../huawei/HuaweiRestSourceConnectorConfig.kt | 259 +++++++++++++++ .../rest/huawei/HuaweiSourceConnector.java | 142 ++++++++ .../connect/rest/huawei/HuaweiSourceTask.java | 204 ++++++++++++ .../huawei/offset/KafkaOffsetManager.java | 55 +++ .../huawei/user/HttpResponseException.java | 33 ++ .../user/HuaweiServiceUserRepository.kt | 312 ++++++++++++++++++ .../rest/huawei/user/HuaweiUserRepository.kt | 36 ++ .../connect/rest/huawei/user/HuaweiUsers.java | 40 +++ .../huawei/user/OAuth2UserCredentials.java | 79 +++++ .../connect/rest/huawei/util/VersionUtil.java | 32 ++ 11 files changed, 1249 insertions(+) create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java new file mode 100644 index 00000000..ad09e22a --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java @@ -0,0 +1,57 @@ +package org.radarbase.connect.rest.huawei; + +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.apache.kafka.connect.connector.Task; +import org.apache.kafka.connect.source.SourceConnector; +import org.radarbase.connect.rest.huawei.util.VersionUtil; + +@SuppressWarnings("unused") +public abstract class AbstractRestSourceConnector extends SourceConnector { + protected HuaweiRestSourceConnectorConfig config; + + @Override + public String version() { + return VersionUtil.getVersion(); + } + + @Override + public Class taskClass() { + return HuaweiSourceTask.class; + } + + @Override + public List> taskConfigs(int maxTasks) { + return Collections.nCopies(maxTasks, new HashMap<>(config.originalsStrings())); + } + + @Override + public void start(Map props) { + config = getConfig(props); + } + + public abstract HuaweiRestSourceConnectorConfig getConfig(Map conf); + + @Override + public void stop() { + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt new file mode 100644 index 00000000..8178a949 --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt @@ -0,0 +1,259 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.radarbase.connect.rest.huawei + +import okhttp3.HttpUrl +import okhttp3.HttpUrl.Companion.toHttpUrlOrNull +import org.apache.kafka.common.config.AbstractConfig +import org.apache.kafka.common.config.ConfigDef +import org.apache.kafka.common.config.ConfigDef.Importance +import org.apache.kafka.common.config.ConfigDef.NonEmptyString +import org.apache.kafka.common.config.ConfigDef.Type +import org.apache.kafka.common.config.ConfigDef.Width +import org.apache.kafka.common.config.ConfigException +import org.apache.kafka.connect.errors.ConnectException +import org.radarbase.connect.rest.huawei.user.HuaweiServiceUserRepository +import org.radarbase.connect.rest.huawei.user.HuaweiUserRepository +import org.radarbase.huawei.route.HuaweiRouteFactory +import java.net.MalformedURLException +import java.net.URL +import java.time.Duration + +/** + * Kafka Connect configuration for the Huawei Health Kit source connector. + * + * Every data type registered in [HuaweiRouteFactory.definitions] gets a `huawei..enabled` + * boolean and a `huawei..topic` string config, generated from that single shared registry + * instead of ~110 hand-duplicated `ConfigDef` entries (one connector, one config, one canonical + * list of Huawei data types). + */ +class HuaweiRestSourceConnectorConfig( + config: ConfigDef, + parsedConfig: MutableMap, + doLog: Boolean, +) : AbstractConfig(config, parsedConfig, doLog) { + + constructor(parsedConfig: MutableMap, doLog: Boolean) : this(conf(), parsedConfig, doLog) + + private var userRepository: HuaweiUserRepository? = null + + fun getHuaweiUsers(): List = getList(HUAWEI_USERS_CONFIG) + + fun getHuaweiClient(): String = getString(HUAWEI_API_CLIENT_CONFIG) + + fun getHuaweiClientSecret(): String = getPassword(HUAWEI_API_SECRET_CONFIG).value() + + fun getUserRepository(reuse: HuaweiUserRepository?): HuaweiUserRepository { + val repo = if (reuse != null && reuse.javaClass == getClass(HUAWEI_USER_REPOSITORY_CONFIG)) { + reuse + } else { + createUserRepository() + } + repo.initialize(this) + userRepository = repo + return repo + } + + fun getUserRepository(): HuaweiUserRepository { + val repo = checkNotNull(userRepository) { "User repository has not been initialized" } + repo.initialize(this) + return repo + } + + @Suppress("UNCHECKED_CAST") + private fun createUserRepository(): HuaweiUserRepository = try { + (getClass(HUAWEI_USER_REPOSITORY_CONFIG) as Class) + .getDeclaredConstructor() + .newInstance() + } catch (e: ReflectiveOperationException) { + throw ConnectException("Invalid class. $e") + } + + fun getHuaweiUserRepositoryUrl(): HttpUrl { + var urlString = getString(HUAWEI_USER_REPOSITORY_URL_CONFIG).trim() + if (urlString.isNotEmpty() && urlString.last() != '/') { + urlString += "/" + } + return urlString.toHttpUrlOrNull() + ?: throw ConfigException( + HUAWEI_USER_REPOSITORY_URL_CONFIG, + urlString, + "User repository URL $urlString cannot be parsed as URL.", + ) + } + + fun getPollIntervalPerUser(): Duration = Duration.ofSeconds(getInt(HUAWEI_USER_POLL_INTERVAL_CONFIG).toLong()) + + fun getHuaweiUserRepositoryClientId(): String = getString(HUAWEI_USER_REPOSITORY_CLIENT_ID_CONFIG) + + fun getHuaweiUserRepositoryClientSecret(): String = + getPassword(HUAWEI_USER_REPOSITORY_CLIENT_SECRET_CONFIG).value() + + fun getHuaweiUserRepositoryTokenUrl(): URL? { + val value = getString(HUAWEI_USER_REPOSITORY_TOKEN_URL_CONFIG) + if (value.isNullOrEmpty()) { + return null + } + return try { + URL(value) + } catch (e: MalformedURLException) { + throw ConfigException("Huawei user repository token URL is invalid.") + } + } + + /** + * The (config key -> Kafka topic) pairs of every Huawei data type that is enabled in this + * configuration. + */ + fun enabledTopics(): Map = + HuaweiRouteFactory.definitions + .filter { getBoolean(enabledKey(it.key)) } + .associate { it.key to getString(topicKey(it.key)) } + + companion object { + private const val SOURCE_POLL_INTERVAL_CONFIG = "rest.source.poll.interval.ms" + private const val SOURCE_POLL_INTERVAL_DOC = "How often to poll the source URL." + private const val SOURCE_POLL_INTERVAL_DISPLAY = "Polling interval" + private const val SOURCE_POLL_INTERVAL_DEFAULT = 60000L + + const val SOURCE_URL_CONFIG = "rest.source.base.url" + private const val SOURCE_URL_DOC = "Base URL for REST source connector." + private const val SOURCE_URL_DISPLAY = "Base URL for REST source connector." + const val SOURCE_URL_DEFAULT = "https://health-api.cloud.huawei.com/healthkit/v1" + + const val HUAWEI_USERS_CONFIG = "huawei.users" + private const val HUAWEI_USERS_DOC = + "The user ID of Huawei users to include in polling, separated by commas. " + + "Non existing user names will be ignored. " + + "If empty, all users in the user directory will be used." + private const val HUAWEI_USERS_DISPLAY = "Huawei users" + + const val HUAWEI_API_CLIENT_CONFIG = "huawei.api.client" + private const val HUAWEI_API_CLIENT_DOC = "Client ID for the Huawei Health Kit API" + private const val HUAWEI_API_CLIENT_DISPLAY = "Huawei API client ID" + + const val HUAWEI_API_SECRET_CONFIG = "huawei.api.secret" + private const val HUAWEI_API_SECRET_DOC = "Secret for the Huawei API client set in huawei.api.client." + private const val HUAWEI_API_SECRET_DISPLAY = "Huawei API client secret" + + const val HUAWEI_USER_REPOSITORY_CONFIG = "huawei.user.repository.class" + private const val HUAWEI_USER_REPOSITORY_DOC = "Class for managing users and authentication." + private const val HUAWEI_USER_REPOSITORY_DISPLAY = "User repository class" + + const val HUAWEI_USER_POLL_INTERVAL_CONFIG = "huawei.user.poll.interval" + private const val HUAWEI_USER_POLL_INTERVAL_DOC = + "Polling interval per Huawei user per request route in seconds." + private const val HUAWEI_USER_POLL_INTERVAL_DEFAULT = 150 + private const val HUAWEI_USER_POLL_INTERVAL_DISPLAY = "Per-user per-route polling interval." + + const val HUAWEI_USER_REPOSITORY_URL_CONFIG = "huawei.user.repository.url" + private const val HUAWEI_USER_REPOSITORY_URL_DOC = + "URL for webservice containing user credentials. Only used if a webservice-based " + + "user repository is configured." + private const val HUAWEI_USER_REPOSITORY_URL_DISPLAY = "User repository URL" + private const val HUAWEI_USER_REPOSITORY_URL_DEFAULT = "" + + const val HUAWEI_USER_REPOSITORY_CLIENT_ID_CONFIG = "huawei.user.repository.client.id" + private const val HUAWEI_USER_REPOSITORY_CLIENT_ID_DOC = "Client ID for connecting to the service repository." + private const val HUAWEI_USER_REPOSITORY_CLIENT_ID_DISPLAY = "Client ID for user repository." + + const val HUAWEI_USER_REPOSITORY_CLIENT_SECRET_CONFIG = "huawei.user.repository.client.secret" + private const val HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DOC = + "Client secret for connecting to the service repository." + private const val HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DISPLAY = "Client Secret for user repository." + + const val HUAWEI_USER_REPOSITORY_TOKEN_URL_CONFIG = "huawei.user.repository.oauth2.token.url" + private const val HUAWEI_USER_REPOSITORY_TOKEN_URL_DOC = "OAuth 2.0 token url for retrieving client credentials." + private const val HUAWEI_USER_REPOSITORY_TOKEN_URL_DISPLAY = "OAuth 2.0 token URL." + + private fun enabledKey(key: String) = "huawei.$key.enabled" + private fun topicKey(key: String) = "huawei.$key.topic" + + @JvmStatic + fun conf(): ConfigDef { + val group = "Huawei" + var order = 0 + + val def = ConfigDef() + .define( + SOURCE_POLL_INTERVAL_CONFIG, Type.LONG, SOURCE_POLL_INTERVAL_DEFAULT, Importance.LOW, + SOURCE_POLL_INTERVAL_DOC, group, ++order, Width.SHORT, SOURCE_POLL_INTERVAL_DISPLAY, + ) + .define( + SOURCE_URL_CONFIG, Type.STRING, SOURCE_URL_DEFAULT, Importance.HIGH, + SOURCE_URL_DOC, group, ++order, Width.SHORT, SOURCE_URL_DISPLAY, + ) + .define( + HUAWEI_USERS_CONFIG, Type.LIST, emptyList(), Importance.HIGH, + HUAWEI_USERS_DOC, group, ++order, Width.SHORT, HUAWEI_USERS_DISPLAY, + ) + .define( + HUAWEI_API_CLIENT_CONFIG, Type.STRING, ConfigDef.NO_DEFAULT_VALUE, NonEmptyString(), + Importance.HIGH, HUAWEI_API_CLIENT_DOC, group, ++order, Width.SHORT, HUAWEI_API_CLIENT_DISPLAY, + ) + .define( + HUAWEI_API_SECRET_CONFIG, Type.PASSWORD, ConfigDef.NO_DEFAULT_VALUE, Importance.HIGH, + HUAWEI_API_SECRET_DOC, group, ++order, Width.SHORT, HUAWEI_API_SECRET_DISPLAY, + ) + .define( + HUAWEI_USER_POLL_INTERVAL_CONFIG, Type.INT, HUAWEI_USER_POLL_INTERVAL_DEFAULT, Importance.MEDIUM, + HUAWEI_USER_POLL_INTERVAL_DOC, group, ++order, Width.SHORT, HUAWEI_USER_POLL_INTERVAL_DISPLAY, + ) + .define( + HUAWEI_USER_REPOSITORY_CONFIG, Type.CLASS, HuaweiServiceUserRepository::class.java, + Importance.MEDIUM, HUAWEI_USER_REPOSITORY_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_DISPLAY, + ) + .define( + HUAWEI_USER_REPOSITORY_URL_CONFIG, Type.STRING, HUAWEI_USER_REPOSITORY_URL_DEFAULT, + Importance.LOW, HUAWEI_USER_REPOSITORY_URL_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_URL_DISPLAY, + ) + .define( + HUAWEI_USER_REPOSITORY_CLIENT_ID_CONFIG, Type.STRING, "", Importance.MEDIUM, + HUAWEI_USER_REPOSITORY_CLIENT_ID_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_CLIENT_ID_DISPLAY, + ) + .define( + HUAWEI_USER_REPOSITORY_CLIENT_SECRET_CONFIG, Type.PASSWORD, "", Importance.MEDIUM, + HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DISPLAY, + ) + .define( + HUAWEI_USER_REPOSITORY_TOKEN_URL_CONFIG, Type.STRING, "", Importance.MEDIUM, + HUAWEI_USER_REPOSITORY_TOKEN_URL_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_TOKEN_URL_DISPLAY, + ) + + HuaweiRouteFactory.definitions.forEach { d -> + val label = d.key.replace('_', ' ') + def.define( + enabledKey(d.key), Type.BOOLEAN, d.enabledByDefault, Importance.LOW, + "Enable or disable Huawei $label", group, ++order, Width.SHORT, + "Huawei $label enabled", + ) + def.define( + topicKey(d.key), Type.STRING, d.defaultTopic, Importance.LOW, + "Kafka topic for Huawei $label", group, ++order, Width.SHORT, + "Huawei $label topic", + ) + } + + return def + } + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java new file mode 100644 index 00000000..63c9debf --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java @@ -0,0 +1,142 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.connect.rest.huawei; + +import java.io.IOException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import org.apache.kafka.common.config.ConfigDef; +import org.apache.kafka.common.config.ConfigException; +import org.radarbase.huawei.user.User; +import org.radarbase.connect.rest.huawei.user.HuaweiUserRepository; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import kotlin.sequences.SequencesKt; +import kotlin.sequences.Sequence; +import kotlin.streams.jdk8.StreamsKt; + +import static org.radarbase.connect.rest.huawei.HuaweiRestSourceConnectorConfig.HUAWEI_USERS_CONFIG; + +public class HuaweiSourceConnector extends AbstractRestSourceConnector { + + private static final Logger logger = LoggerFactory.getLogger(HuaweiSourceConnector.class); + private ScheduledExecutorService executor; + private Set configuredUsers; + private HuaweiUserRepository repository; + + @Override + public void start(Map props) { + logger.info("Starting Huawei source connector"); + super.start(props); + executor = Executors.newSingleThreadScheduledExecutor(); + + executor.scheduleAtFixedRate(() -> { + if (repository.hasPendingUpdates()) { + try { + logger.info("Requesting latest user details..."); + repository.applyPendingUpdates(); + Set newUsers = + SequencesKt.toSet(getConfig(props, false).getUserRepository(repository).stream()); + if (configuredUsers != null && !newUsers.equals(configuredUsers)) { + logger.info("User info mismatch found. Requesting reconfiguration..."); + reconfigure(); + } + } catch (IOException e) { + logger.warn("Failed to refresh users: {}", e.toString()); + } + } else { + logger.info("No pending updates found. Not attempting to refresh users."); + } + }, 0, 5, TimeUnit.MINUTES); + } + + @Override + public void stop() { + super.stop(); + executor.shutdown(); + + configuredUsers = null; + } + + private HuaweiRestSourceConnectorConfig getConfig(Map conf, boolean doLog) { + return new HuaweiRestSourceConnectorConfig(conf, doLog); + } + + @Override + public HuaweiRestSourceConnectorConfig getConfig(Map conf) { + HuaweiRestSourceConnectorConfig connectorConfig = getConfig(conf, true); + repository = connectorConfig.getUserRepository(repository); + return connectorConfig; + } + + @Override + public ConfigDef config() { + return HuaweiRestSourceConnectorConfig.conf(); + } + + @Override + public List> taskConfigs(int maxTasks) { + return configureTasks(maxTasks); + } + + private List> configureTasks(int maxTasks) { + Map baseConfig = config.originalsStrings(); + HuaweiRestSourceConnectorConfig huaweiConfig = getConfig(baseConfig); + if (repository == null) { + repository = huaweiConfig.getUserRepository(null); + } + // Divide the users over tasks + try { + Sequence ids = SequencesKt.map(huaweiConfig.getUserRepository(repository).stream(), User::getVersionedId); + List> userTasks = StreamsKt.asStream(ids) + // group users based on their hashCode, in principle, this allows for more efficient + // reconfigurations for a fixed number of tasks, since that allows existing tasks to + // only handle small modifications users to handle. + .collect(Collectors.groupingBy( + u -> Math.abs(u.hashCode()) % maxTasks, + Collectors.joining(","))) + .values().stream() + .map(u -> { + Map taskConfig = new HashMap<>(baseConfig); + taskConfig.put(HUAWEI_USERS_CONFIG, u); + return taskConfig; + }) + .collect(Collectors.toList()); + this.configuredUsers = SequencesKt.toSet(huaweiConfig.getUserRepository().stream()); + logger.info("Received userTask Configs {}", userTasks); + return userTasks; + } catch (Exception ex) { + throw new ConfigException("Cannot read users", ex); + } + } + + public void reconfigure() { + new Thread(() -> { + logger.info("Requesting reconfiguration"); + context.requestTaskReconfiguration(); + logger.info("Requested reconfiguration"); + }).start(); + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java new file mode 100644 index 00000000..e3e6520e --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java @@ -0,0 +1,204 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.connect.rest.huawei; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import org.apache.kafka.connect.data.SchemaAndValue; +import org.apache.kafka.connect.errors.ConnectException; +import org.apache.kafka.connect.source.SourceRecord; +import org.apache.kafka.connect.source.SourceTask; +import org.apache.kafka.connect.storage.OffsetStorageReader; +import org.radarbase.connect.rest.huawei.offset.KafkaOffsetManager; +import org.radarbase.connect.rest.huawei.user.HuaweiUserRepository; +import org.radarbase.connect.rest.huawei.util.VersionUtil; +import org.radarbase.huawei.converter.TopicData; +import org.radarbase.huawei.request.HuaweiRequestGenerator; +import org.radarbase.huawei.request.HuaweiResult; +import org.radarbase.huawei.request.HuaweiResult.Success; +import org.radarbase.huawei.request.HuaweiResult.Error; +import org.radarbase.huawei.request.HuaweiErrorBase; +import org.radarbase.huawei.request.RestRequest; +import org.radarbase.huawei.route.HuaweiRouteDefinition; +import org.radarbase.huawei.route.HuaweiRouteFactory; +import org.radarbase.huawei.route.Route; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.radarbase.huawei.user.User; +import io.confluent.connect.avro.AvroData; +import kotlin.streams.jdk8.StreamsKt; +import okhttp3.OkHttpClient; +import okhttp3.Response; + +public class HuaweiSourceTask extends SourceTask { + private static final Logger logger = LoggerFactory.getLogger(HuaweiSourceTask.class); + + private OkHttpClient baseClient; + private HuaweiUserRepository userRepository; + private List routes; + private HuaweiRequestGenerator huaweiRequestGenerator; + private final AvroData avroData = new AvroData(20); + private KafkaOffsetManager offsetManager; + private static final String TIMESTAMP_OFFSET_KEY = "timestamp"; + private static final long TIMEOUT = 60000L; + private int routeStartIndex = 0; + + public void initialize(HuaweiRestSourceConnectorConfig config, OffsetStorageReader offsetStorageReader) { + this.baseClient = new OkHttpClient(); + + this.userRepository = config.getUserRepository(); + this.offsetManager = new KafkaOffsetManager(offsetStorageReader); + this.routes = getRoutes(config); + this.huaweiRequestGenerator = new HuaweiRequestGenerator(this.userRepository, this.offsetManager, this.routes); + this.offsetManager.initialize(getPartitions()); + } + + private List getRoutes(HuaweiRestSourceConnectorConfig config) { + Map enabledTopics = config.enabledTopics(); + List result = new ArrayList<>(); + for (HuaweiRouteDefinition definition : HuaweiRouteFactory.INSTANCE.getDefinitions()) { + String topic = enabledTopics.get(definition.getKey()); + if (topic != null) { + result.add(definition.getBuild().invoke(userRepository, topic)); + } + } + return result; + } + + public List> getPartitions() { + try { + return StreamsKt.asStream(userRepository.stream()) + .flatMap(u -> this.routes.stream().map(r -> getPartition(r.toString(), u))) + .collect(Collectors.toList()); + } catch (Exception e) { + logger.warn("Failed to initialize user partitions.."); + return Collections.emptyList(); + } + } + + public Map getPartition(String route, User user) { + Map partition = new HashMap<>(4); + partition.put("user", user.getVersionedId()); + partition.put("route", route); + return partition; + } + + public Stream requests() { + if (this.routes == null || this.routes.isEmpty()) { + return Stream.empty(); + } + + // Rotate routes so that all routes are requested in a round-robin manner + List rotatedRoutes = getRotatedRoutes(); + return rotatedRoutes.stream() + .flatMap((Route r) -> StreamsKt.asStream(huaweiRequestGenerator.requests(r, 100))); + } + + private List getRotatedRoutes() { + List rotatedRoutes = new ArrayList<>(this.routes); + Collections.rotate(rotatedRoutes, routeStartIndex % this.routes.size()); + routeStartIndex = (routeStartIndex + 1) % this.routes.size(); + return rotatedRoutes; + } + + public Stream handleRequest(RestRequest req) throws IOException { + try (Response response = baseClient.newCall(req.getRequest()).execute()) { + HuaweiResult> result = this.huaweiRequestGenerator.handleResponse(req, response); + if (result instanceof HuaweiResult.Success) { + Success> success = (Success>) result; + return success.getValue().stream().map(r -> { + SchemaAndValue avro = avroData.toConnectData(r.getValue().getSchema(), r.getValue()); + SchemaAndValue key = avroData.toConnectData(r.getKey().getSchema(), r.getKey()); + Map partition = getPartition(req.getRoute().toString(), req.getUser()); + Map offset = Collections.singletonMap(TIMESTAMP_OFFSET_KEY, r.getOffset()); + + return new SourceRecord(partition, offset, r.getTopic(), + key.schema(), key.value(), avro.schema(), avro.value()); + }); + } else { + HuaweiErrorBase e = (HuaweiErrorBase) ((HuaweiResult.Error) result).getError(); + logger.warn("Failed to make request: {} {} {}", e.getMessage(), e.getCause(), e.getCode()); + return Stream.empty(); + } + } + } + + @Override + public void start(Map map) { + HuaweiRestSourceConnectorConfig connectorConfig; + try { + Class connector = Class.forName(map.get("connector.class")); + Object connectorInst = connector.getConstructor().newInstance(); + connectorConfig = ((HuaweiSourceConnector) connectorInst).getConfig(map); + } catch (ClassNotFoundException e) { + throw new ConnectException("Connector " + map.get("connector.class") + " not found", e); + } catch (ReflectiveOperationException e) { + throw new ConnectException("Connector " + map.get("connector.class") + + " could not be instantiated", e); + } + this.initialize(connectorConfig, context.offsetStorageReader()); + } + + @Override + public List poll() throws InterruptedException { + long requestsGenerated = 0; + List sourceRecords = Collections.emptyList(); + + do { + Thread.sleep(TIMEOUT); + + Iterator requestIterator = this.requests().iterator(); + + while (sourceRecords.isEmpty() && requestIterator.hasNext()) { + RestRequest request = requestIterator.next(); + + logger.info("Requesting for user {}, url: {}", request.getUser().getUserId(), request.getRequest().url()); + requestsGenerated++; + + try { + sourceRecords = this.handleRequest(request) + .collect(Collectors.toList()); + } catch (IOException ex) { + logger.warn("Failed to make request: {}", ex.toString()); + } + } + } while (sourceRecords.isEmpty()); + + logger.info("Processed {} records from {} URLs", sourceRecords.size(), requestsGenerated); + + return sourceRecords; + } + + @Override + public void stop() { + logger.debug("Stopping source task"); + } + + @Override + public String version() { + return VersionUtil.getVersion(); + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java new file mode 100644 index 00000000..dcc044ef --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java @@ -0,0 +1,55 @@ +package org.radarbase.connect.rest.huawei.offset; + +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import static java.time.temporal.ChronoUnit.NANOS; +import org.apache.kafka.connect.storage.OffsetStorageReader; +import org.radarbase.huawei.offset.Offset; +import org.radarbase.huawei.request.HuaweiOffsetManager; +import org.radarbase.huawei.route.Route; +import org.radarbase.huawei.user.User; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class KafkaOffsetManager implements HuaweiOffsetManager { + private static final Logger logger = LoggerFactory.getLogger(KafkaOffsetManager.class); + private static final String TIMESTAMP_OFFSET_KEY = "timestamp"; + private static final Duration ONE_NANO = NANOS.getDuration(); + + private final OffsetStorageReader offsetStorageReader; + private Map offsets; + + public KafkaOffsetManager(OffsetStorageReader offsetStorageReader) { + this.offsetStorageReader = offsetStorageReader; + } + + public void initialize(List> partitions) { + if (this.offsetStorageReader != null) { + this.offsets = this.offsetStorageReader.offsets(partitions).entrySet().stream() + .filter(e -> e.getValue() != null && e.getValue().containsKey(TIMESTAMP_OFFSET_KEY)) + .collect(Collectors.toMap( + e -> e.getKey().get("user") + "-" + e.getKey().get("route"), + e -> Instant.ofEpochSecond(((Number) e.getValue().get(TIMESTAMP_OFFSET_KEY)).longValue()))); + } else { + logger.warn("Offset storage reader is null, will resume from an empty state."); + } + } + + @Override + public Offset getOffset(Route route, User user) { + Instant offset = offsets.getOrDefault(getOffsetKey(route, user), user.getStartDate().minus(ONE_NANO)); + return new Offset(user, route, offset); + } + + @Override + public void updateOffsets(Route route, User user, Instant offset) { + offsets.put(getOffsetKey(route, user), offset); + } + + private String getOffsetKey(Route route, User user) { + return user.getVersionedId() + "-" + route.toString(); + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java new file mode 100644 index 00000000..c4f8c30c --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java @@ -0,0 +1,33 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.connect.rest.huawei.user; + +import java.io.IOException; + +public class HttpResponseException extends IOException { + private final int statusCode; + + public HttpResponseException(String message, int statusCode) { + super(message); + this.statusCode = statusCode; + } + + public int getStatusCode() { + return statusCode; + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt new file mode 100644 index 00000000..8bf64c2a --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt @@ -0,0 +1,312 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.radarbase.connect.rest.huawei.user + +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule +import com.fasterxml.jackson.module.kotlin.readValue +import com.fasterxml.jackson.module.kotlin.registerKotlinModule +import io.ktor.client.HttpClient +import io.ktor.client.engine.cio.CIO +import io.ktor.client.plugins.HttpTimeout +import io.ktor.client.plugins.auth.Auth +import io.ktor.client.plugins.auth.providers.BasicAuthCredentials +import io.ktor.client.plugins.auth.providers.basic +import io.ktor.client.plugins.contentnegotiation.ContentNegotiation +import io.ktor.client.plugins.defaultRequest +import io.ktor.client.request.HttpRequestBuilder +import io.ktor.client.request.request +import io.ktor.client.request.setBody +import io.ktor.client.request.url +import io.ktor.client.statement.bodyAsText +import io.ktor.client.statement.request +import io.ktor.http.ContentType +import io.ktor.http.HttpMethod +import io.ktor.http.HttpStatusCode +import io.ktor.http.URLBuilder +import io.ktor.http.Url +import io.ktor.http.contentLength +import io.ktor.http.contentType +import io.ktor.http.isSuccess +import io.ktor.http.takeFrom +import io.ktor.serialization.jackson.jackson +import io.ktor.serialization.kotlinx.json.json +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.runBlocking +import kotlinx.coroutines.withContext +import kotlinx.serialization.json.Json +import org.radarbase.connect.rest.huawei.HuaweiRestSourceConnectorConfig +import org.radarbase.kotlin.coroutines.CacheConfig +import org.radarbase.kotlin.coroutines.CachedSet +import org.radarbase.kotlin.coroutines.CachedValue +import org.radarbase.ktor.auth.ClientCredentialsConfig +import org.radarbase.ktor.auth.clientCredentials +import org.radarbase.huawei.user.HuaweiUser +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserNotAuthorizedException +import org.slf4j.LoggerFactory +import java.io.IOException +import java.util.concurrent.ConcurrentHashMap +import kotlin.streams.asSequence +import kotlin.time.Duration.Companion.days +import kotlin.time.Duration.Companion.hours +import kotlin.time.Duration.Companion.minutes +import kotlin.time.Duration.Companion.seconds + +/** + * User repository backed by the RADAR-base "rest-source-authorizer" webservice, mirroring + * [org.radarbase.connect.rest.oura.user.OuraServiceUserRepository]. Retrieves the list of Huawei + * users configured for a study (`GET users?source-type=Huawei`) and their Huawei Health Kit OAuth2 + * access/refresh tokens (`users//token`). + */ +@Suppress("unused") +class HuaweiServiceUserRepository : HuaweiUserRepository() { + private lateinit var userCache: CachedSet + private lateinit var client: HttpClient + private val credentialCaches = ConcurrentHashMap>() + private val credentialCacheConfig = + CacheConfig(refreshDuration = 1.days, retryDuration = 1.minutes) + private val mapper = ObjectMapper().registerKotlinModule().registerModule(JavaTimeModule()) + + @Throws(IOException::class) + override fun get(key: String): User = + runBlocking(Dispatchers.Default) { + makeRequest { url("users/$key") } + } + + override fun initialize(config: HuaweiRestSourceConnectorConfig) { + val containedUsers = config.getHuaweiUsers().toHashSet() + + client = + createClient( + baseUrl = config.getHuaweiUserRepositoryUrl(), + tokenUrl = config.getHuaweiUserRepositoryTokenUrl()?.let { URLBuilder(it.toString()).build() }, + clientId = config.getHuaweiUserRepositoryClientId(), + clientSecret = config.getHuaweiUserRepositoryClientSecret(), + scope = "SUBJECT.READ MEASUREMENT.CREATE", + audience = "res_restAuthorizer", + ) + + userCache = + CachedSet( + CacheConfig(refreshDuration = 1.hours, retryDuration = 1.minutes), + ) { + makeRequest { url("users?source-type=Huawei") } + .users + .toHashSet() + .filterTo(HashSet()) { u -> + u.isComplete() && + (containedUsers.isEmpty() || u.versionedId in containedUsers) + } + } + } + + private fun createClient( + baseUrl: Url, + tokenUrl: Url?, + clientId: String?, + clientSecret: String?, + scope: String?, + audience: String?, + ): HttpClient = + HttpClient(CIO) { + if (tokenUrl != null) { + install(Auth) { + clientCredentials( + ClientCredentialsConfig( + tokenUrl.toString(), + clientId, + clientSecret, + scope, + audience, + ).copyWithEnv("MANAGEMENT_PORTAL"), + baseUrl.host, + ) + } + install(ContentNegotiation) { + json( + Json { + ignoreUnknownKeys = true + }, + ) + } + } else if (clientId != null && clientSecret != null) { + install(Auth) { + basic { + credentials { + BasicAuthCredentials(username = clientId, password = clientSecret) + } + realm = "Access to the '/' path" + sendWithoutRequest { + it.url.host == baseUrl.host + } + } + } + } + + defaultRequest { + url.takeFrom(baseUrl) + } + + install(ContentNegotiation) { + jackson { + registerModule(JavaTimeModule()) // support java.time.* types + } + } + + install(HttpTimeout) { + connectTimeoutMillis = 60.seconds.inWholeMilliseconds + requestTimeoutMillis = 90.seconds.inWholeMilliseconds + } + } + + override fun stream(): Sequence = + runBlocking(Dispatchers.Default) { + val valueInCache = + userCache.getFromCache() + .takeIf { it is CachedValue.CacheValue } + ?.getOrThrow() + + (valueInCache ?: userCache.get()) + .stream() + .filter { it.isComplete() } + .asSequence() + } + + @Throws(IOException::class, UserNotAuthorizedException::class) + override fun getAccessToken(user: User): String { + if (!user.isAuthorized) { + throw UserNotAuthorizedException("User is not authorized") + } + return runBlocking(Dispatchers.Default) { + credentialCache(user) + .get { !it.isAccessTokenExpired } + .value + .accessToken + } + } + + @Throws(IOException::class, UserNotAuthorizedException::class) + override fun refreshAccessToken(user: User): String { + if (!user.isAuthorized) { + throw UserNotAuthorizedException("User is not authorized") + } + return runBlocking(Dispatchers.Default) { + val token = + requestAccessToken(user) { + url("users/${user.id}/token") + method = HttpMethod.Post + setBody("{}") + contentType(ContentType.Application.Json) + } + credentialCache(user).set(token) + token.accessToken + } + } + + private suspend fun credentialCache(user: User): CachedValue = + credentialCaches.computeIfAbsent(user.id) { + CachedValue(credentialCacheConfig) { + requestAccessToken(user) { url("users/${user.id}/token") } + } + } + + @Throws(UserNotAuthorizedException::class, IOException::class) + private suspend fun requestAccessToken( + user: User, + builder: HttpRequestBuilder.() -> Unit, + ): OAuth2UserCredentials = + try { + makeRequest(builder) + } catch (ex: HttpResponseException) { + if (ex.statusCode == 407) { + credentialCaches -= user.id + throw UserNotAuthorizedException(ex.message) + } + throw ex + } + + override fun hasPendingUpdates(): Boolean = + runBlocking(Dispatchers.Default) { + userCache.isStale(1.hours) + } + + @Throws(IOException::class) + override fun applyPendingUpdates() { + logger.info("Requesting user information from webservice") + + runBlocking(Dispatchers.Default) { + userCache.get() + } + } + + private suspend inline fun makeRequest( + crossinline builder: HttpRequestBuilder.() -> Unit, + ): T = + withContext(Dispatchers.IO) { + val requestBuilder = HttpRequestBuilder() + builder(requestBuilder) + logger.info("Making HTTP request: ${requestBuilder.method} ${requestBuilder.url}") + + val response = client.request(builder) + logger.info("Response status: ${response.status}") + val contentLength = response.contentLength() + val transferEncoding = response.headers["Transfer-Encoding"] + val hasBody = (contentLength != null && contentLength > 0) || + (transferEncoding != null && transferEncoding.contains("chunked")) + val responseBody = try { + response.bodyAsText() + } catch (e: Exception) { + "Error reading body: ${e.message}" + } + + if (response.status == HttpStatusCode.NotFound) { + logger.error("HTTP 404 Not Found: ${response.request.url}") + throw NoSuchElementException("URL " + response.request.url + " does not exist") + } else if (!response.status.isSuccess()) { + val message = "HTTP ${response.status.value} error: $responseBody" + logger.error(message) + throw HttpResponseException(message, response.status.value) + } else if (!hasBody) { + logger.warn( + "HTTP ${response.status.value} OK but no body content. Returning empty result.", + ) + @Suppress("UNCHECKED_CAST") + return@withContext when (T::class) { + String::class -> "" as T + List::class -> emptyList() as T + else -> mapper.readValue("{}") + } + } + + try { + val result = mapper.readValue(responseBody) + logger.info("Successfully parsed response as ${T::class.simpleName}") + result + } catch (e: Exception) { + logger.error( + "Failed to parse response body as ${T::class.simpleName}: ${e.message}", + ) + logger.error("Response body that failed to parse: $responseBody") + throw e + } + } + + companion object { + private val logger = LoggerFactory.getLogger(HuaweiServiceUserRepository::class.java) + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt new file mode 100644 index 00000000..010a5b5a --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt @@ -0,0 +1,36 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.radarbase.connect.rest.huawei.user + +import org.radarbase.connect.rest.huawei.HuaweiRestSourceConnectorConfig +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserNotAuthorizedException +import org.radarbase.huawei.user.UserRepository +import java.io.IOException + +@Suppress("unused") +abstract class HuaweiUserRepository : UserRepository { + abstract fun initialize(config: HuaweiRestSourceConnectorConfig) + + @Throws(IOException::class, UserNotAuthorizedException::class) + abstract fun refreshAccessToken(user: User): String + + @Throws(IOException::class) + abstract fun applyPendingUpdates() + + abstract fun hasPendingUpdates(): Boolean +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java new file mode 100644 index 00000000..e01aeff8 --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java @@ -0,0 +1,40 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.connect.rest.huawei.user; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; + +import java.util.ArrayList; +import java.util.List; +import org.radarbase.huawei.user.HuaweiUser; + +@JsonIgnoreProperties(ignoreUnknown = true) +public class HuaweiUsers { + private final List users; + + @JsonCreator + public HuaweiUsers(@JsonProperty("users") List users) { + this.users = new ArrayList<>(users); + } + + public List getUsers() { + return users; + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java new file mode 100644 index 00000000..88bad3b8 --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java @@ -0,0 +1,79 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.connect.rest.huawei.user; + +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonSetter; +import java.time.Duration; +import java.time.Instant; + +@JsonIgnoreProperties(ignoreUnknown = true) +public class OAuth2UserCredentials { + private static final Duration DEFAULT_EXPIRY = Duration.ofHours(1); + private static final Duration EXPIRY_TIME_MARGIN = Duration.ofMinutes(5); + + @JsonProperty + private String accessToken; + @JsonProperty + private String refreshToken; + @JsonProperty + private Instant expiresAt; + + public OAuth2UserCredentials() { + } + + public OAuth2UserCredentials(String refreshToken, String accessToken, Long expiresIn) { + this.refreshToken = refreshToken; + this.accessToken = accessToken; + this.expiresAt = getExpiresAt(expiresIn != null && expiresIn > 0L + ? Duration.ofSeconds(expiresIn) : DEFAULT_EXPIRY); + } + + public String getAccessToken() { + return accessToken; + } + + @JsonSetter + public void setAccessToken(String accessToken) { + this.accessToken = accessToken; + if (expiresAt == null) { + expiresAt = getExpiresAt(DEFAULT_EXPIRY); + } + } + + public boolean hasRefreshToken() { + return refreshToken != null && !refreshToken.isEmpty(); + } + + public String getRefreshToken() { + return refreshToken; + } + + protected static Instant getExpiresAt(Duration expiresIn) { + return Instant.now() + .plus(expiresIn) + .minus(EXPIRY_TIME_MARGIN); + } + + @JsonIgnore + public boolean isAccessTokenExpired() { + return expiresAt == null || Instant.now().isAfter(expiresAt); + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java new file mode 100644 index 00000000..8c23ac79 --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java @@ -0,0 +1,32 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.connect.rest.huawei.util; + +public final class VersionUtil { + private VersionUtil() { + // utility class + } + + public static String getVersion() { + try { + return VersionUtil.class.getPackage().getImplementationVersion(); + } catch (Exception ex) { + return "0.0.0.0"; + } + } +} From bf058911b03d0f61ce85518c0efb161d10749e3a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 11:16:29 +0000 Subject: [PATCH 05/44] Wire up Huawei connector in docker-compose, README, and CI matrices Adds docker/source-huawei.properties.template, a radar-huawei-connector docker-compose service, a README section, and registers the kafka-connect-huawei-source image in both CI workflow matrices. Updates ARCHITECTURE.md to document the Huawei module and the route-registry config-generation pattern it introduces for connectors with a large number of data types. --- .github/workflows/main.yml | 5 + .github/workflows/release.yml | 5 + ARCHITECTURE.md | 120 +++++++++++++++++------ README.md | 28 +++++- docker-compose.yml | 47 +++++++++ docker/source-huawei.properties.template | 12 +++ 6 files changed, 183 insertions(+), 34 deletions(-) create mode 100644 docker/source-huawei.properties.template diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 9be81f61..e5397368 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -22,6 +22,11 @@ env: 'build_file': 'kafka-connect-oura-source/Dockerfile', 'authors': 'Pim van Nierop , Pauline Conde ', 'description': 'RADAR-base Oura connector application' + },{ + 'name': 'kafka-connect-huawei-source', + 'build_file': 'kafka-connect-huawei-source/Dockerfile', + 'authors': 'Yatharth Ranjan ', + 'description': 'RADAR-base Huawei Health Kit connector application' }] jobs: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c70528ad..861df14e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -18,6 +18,11 @@ env: 'build_file': 'kafka-connect-oura-source/Dockerfile', 'authors': 'Pauline Conde , Yatharth Ranjan ', 'description': 'RADAR-base Oura connector application' + },{ + 'name': 'kafka-connect-huawei-source', + 'build_file': 'kafka-connect-huawei-source/Dockerfile', + 'authors': 'Yatharth Ranjan ', + 'description': 'RADAR-base Huawei Health Kit connector application' }] jobs: diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 0d716c53..cc623526 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,15 +1,17 @@ # Architecture This document describes how RADAR-REST-Connector is put together, so that future contributors -(human or agent) can orient themselves quickly and add new device/API integrations (e.g. Huawei -Health Kit) consistently with the existing patterns. +(human or agent) can orient themselves quickly and add new device/API integrations consistently +with the existing patterns. ## What this repo is A multi-module Gradle project providing Kafka Connect **source connectors** that poll third-party REST APIs (wearable vendor APIs) on behalf of RADAR-base study participants and publish the -resulting data as Avro records on Kafka topics. It currently ships two concrete connectors — -**Fitbit** and **Oura** — built on top of a shared, generic REST-polling framework. +resulting data as Avro records on Kafka topics. It currently ships three concrete connectors — +**Fitbit**, **Oura**, and **Huawei Health Kit** — the latter two built on the "library + thin +Connect glue" pattern described below; Fitbit predates that pattern and uses the older, generic +`kafka-connect-rest-source` framework instead. ``` RADAR-REST-Connector/ @@ -17,6 +19,8 @@ RADAR-REST-Connector/ ├── kafka-connect-fitbit-source/ # Fitbit connector (Java), oldest/original implementation ├── oura-library/ # Oura domain logic: routes, converters, requests (Kotlin, no Kafka Connect deps) ├── kafka-connect-oura-source/ # Oura Kafka Connect glue (Java+Kotlin), wraps oura-library +├── huawei-library/ # Huawei domain logic: routes, converters, requests (Kotlin, no Kafka Connect deps) +├── kafka-connect-huawei-source/ # Huawei Kafka Connect glue (Java+Kotlin), wraps huawei-library ├── docker/ # Docker Compose config templates, launch/ensure scripts, log4j ├── scripts/REDCAP-FITBIT-AUTH-AUTO/ # Standalone Python helper for REDCap-driven Fitbit auth └── docker-compose.yml # Full local Kafka stack + both connectors, for manual testing @@ -158,6 +162,41 @@ This was a deliberate move to (a) get domain logic under unit test without spinn Connect, and (b) avoid the generic framework's assumptions (e.g. its polling-interval math) that didn't fit Oura's simpler historical/recent chunking model. +### 4. Huawei Health Kit connector (`huawei-library` + `kafka-connect-huawei-source`) + +Structurally identical to the Oura pattern above (pure-Kotlin domain library + thin Connect glue +module), but with two differences worth knowing about: + +- **Three request "shapes" instead of one.** The Huawei Health Kit Data API doesn't have a single + uniform per-route request shape like Oura's `GET .../{subPath}?start_date=...&end_date=...`. It + exposes `POST /healthkit/v1/sampleSet:polymerize` (raw sample points, or day-aggregated + statistics when a `groupByTime` block is added to the JSON body) for most data types, plus two + GET endpoints — `activityRecords` and `healthRecords` — for workout sessions and clinical-style + records (blood pressure sessions, heart-rate alerts, menstrual cycle, sleep). `HuaweiRoute` is + the shared abstract base (OAuth2-authorized request building + time-range chunking); + `HuaweiSampleSetRoute`, `HuaweiHealthRecordRoute`, and `HuaweiActivityRecordRoute` are the three + concrete route kinds. +- **A single route registry drives both the route list and the Connect config**, instead of + Oura/Fitbit's one-hand-written-`ConfigDef`-entry-per-data-type approach. Huawei has ~54 data + types (see the `radar-huawei-connector` schema spec in RADAR-Schemas, + `specifications/connector/radar-huawei-connector-1.0.0.yml`), several of which reuse the same + Avro schema (`HuaweiStatistics` alone backs 14 different `*.statistics` topics) — hand-duplicating + a `ConfigDef.define(...)` block and a route-construction branch per type, Fitbit/Oura-style, + would mean ~110 near-identical static fields. Instead, `huawei-library`'s + `route/HuaweiRouteFactory.definitions` is a `List` (config key, default + topic, default enabled, and a `(UserRepository, topic) -> HuaweiRoute` builder) — the single + source of truth for "what Huawei data types exist." `HuaweiRestSourceConnectorConfig.conf()` + loops over it to generate `huawei..enabled`/`huawei..topic` `ConfigDef` entries, and + `HuaweiSourceTask.getRoutes()` loops over the same list filtered by that config to build the + actual `Route` instances — so the config and the polled routes can't drift out of sync. If you + add a data type to a future connector with a similarly large surface, prefer this registry + pattern over copy-pasting Oura's per-type `ConfigDef` blocks. + +Field-value key names inside `HuaweiRouteFactory`'s record builders (what JSON key a given Avro +field is read from) are a best-effort mapping to Huawei's documented `Field` naming convention — +verify them against a real Health Kit API response and adjust before relying on this in +production; see the KDoc at the top of that file. + ## Runtime data flow (both connectors, conceptually) ```mermaid @@ -201,7 +240,10 @@ plus vendor-specific keys, e.g.: studies can disable data types they don't need. The full current list for Fitbit is documented in `README.md`; Oura's config lives in -`OuraRestSourceConnectorConfig` (no README table yet — check the class directly). +`OuraRestSourceConnectorConfig` (no README table yet — check the class directly). Huawei's +per-data-type keys are generated from `HuaweiRouteFactory.definitions` (see below) rather than +hand-written — check that list, or a running connector's `GET /connectors//config`, for the +current set. ## Docker / deployment @@ -209,43 +251,59 @@ Each connector module has its own multi-stage `Dockerfile` (Gradle build stage `confluentinc/cp-kafka-connect-base`), publishing built jars plus third-party deps into `$CONNECT_PLUGIN_PATH//`. `docker/launch` and `docker/ensure` are modified Confluent entrypoint scripts (env-var → properties translation, Kafka-readiness wait). `docker-compose.yml` -spins up a full local Zookeeper+Kafka+SchemaRegistry+REST-proxy cluster plus both connectors for -manual end-to-end testing (`docker-compose up -d --build`, inspect with +spins up a full local Zookeeper+Kafka+SchemaRegistry+REST-proxy cluster plus all three connectors +for manual end-to-end testing (`docker-compose up -d --build`, inspect with `kafka-avro-console-consumer`). Sentry error monitoring is wired in via `radarKotlin { sentryEnabled = true }` and configured purely through `SENTRY_DSN`/`SENTRY_*` env vars — see README "Sentry monitoring". ## Testing - `kafka-connect-rest-source/src/test`, `kafka-connect-fitbit-source/src/test`, - `kafka-connect-oura-source/src/test` currently only contain config-parsing tests - (`*ConnectorConfigTest`) plus one task test — test coverage of the actual polling/conversion - logic is thin. `wiremock` and `mockito` are on the version catalog for HTTP-level testing but not - yet exercised much; `oura-library`'s pure-Kotlin design makes it the easiest place to add real - unit tests for new routes/converters without Kafka Connect scaffolding. + `kafka-connect-oura-source/src/test`, `kafka-connect-huawei-source/src/test` currently only + contain config-parsing tests (`*ConnectorConfigTest`) plus one task test — test coverage of the + actual polling/conversion logic is thin. `wiremock` and `mockito` are on the version catalog for + HTTP-level testing but not yet exercised much; the `oura-library`/`huawei-library` pure-Kotlin + design makes those the easiest place to add real unit tests for new routes/converters without + Kafka Connect scaffolding. - CI (`.github/workflows/main.yml`) runs `./gradlew assemble` and `./gradlew check` on every push/PR to `master`/`dev`, then builds (and on `push`, publishes) multi-arch Docker images per connector module via a matrix job. `release.yml` does the same on GitHub Release publish, additionally uploading built jars as release assets, tagged `vX.Y.Z` from `gradle.properties`/version catalog. - -## Adding a new vendor integration (e.g. Huawei) - -Follow the **Oura pattern**, not the Fitbit one: - -1. New Gradle module `huawei-library` (pure Kotlin, mirrors `oura-library`): `user/`, `route/`, - `converter/`, `request/`, `offset/` packages. No Kafka Connect or OkHttp-Connect-specific types - here — keep it independently testable. -2. New Gradle module `kafka-connect-huawei-source` (mirrors `kafka-connect-oura-source`): - `HuaweiSourceConnector`, `HuaweiSourceTask`, `HuaweiRestSourceConnectorConfig`, - `offset/KafkaOffsetManager`, `user/HuaweiServiceUserRepository` (Ktor-based - rest-source-authorizer client, copy `OuraServiceUserRepository`'s structure), plus a - `Dockerfile`. +- **Sandbox note:** in a network-restricted environment (no access to `packages.confluent.io`, or + to whichever host actually serves a given `-SNAPSHOT` dependency), only the pure-Kotlin library + modules (`oura-library`, `huawei-library`) may be compilable — the `kafka-connect-*-source` + glue modules depend on `io.confluent:kafka-connect-avro-converter` / + `org.apache.kafka:connect-api` from Confluent's Maven repo and won't resolve. If you hit this, + it's an environment limitation, not a code problem: check whether the library module alone + compiles before concluding the code is broken, and consider publishing a needed `-SNAPSHOT` + dependency to `mavenLocal()` (e.g. `gradle :radar-schemas-commons:publishToMavenLocal` from a + RADAR-Schemas checkout) to verify domain logic against the real generated classes. + +## Adding a new vendor integration + +Follow the **Oura/Huawei pattern**, not the Fitbit one — see the Huawei section above for a +worked example, including the route-registry technique for connectors with a large number of +data types: + +1. New Gradle module `-library` (pure Kotlin, mirrors `oura-library`/`huawei-library`): + `user/`, `route/`, `converter/`, `request/`, `offset/` packages. No Kafka Connect or + OkHttp-Connect-specific types here — keep it independently testable. +2. New Gradle module `kafka-connect--source` (mirrors `kafka-connect-oura-source`/ + `kafka-connect-huawei-source`): `SourceConnector`, `SourceTask`, + `RestSourceConnectorConfig`, `offset/KafkaOffsetManager`, + `user/ServiceUserRepository` (Ktor-based rest-source-authorizer client, copy + `OuraServiceUserRepository`'s/`HuaweiServiceUserRepository`'s structure), plus a `Dockerfile`. 3. Register both modules in `settings.gradle.kts`; add any new dependency versions to - `gradle/libs.versions.toml` first. + `gradle/libs.versions.toml` first. If the vendor's schemas are only available as a `-SNAPSHOT`, + add a separate version-catalog entry for it (see `radarSchemasHuawei`) so it doesn't force + every other module onto an unreleased version. 4. Confirm (or add) the required Avro schemas in the external RADAR-Schemas project and bump the - `radarSchemas` version in the catalog once published — this repo cannot invent schemas locally. -5. One `Route`/`Converter` pair per Huawei data type you plan to support, each independently - togglable via a `huawei..enabled` config flag, matching the Oura/Fitbit convention. -6. Add `docker/source-huawei.properties.template`, a `docker-compose.yml` service entry, and a - README config table, following the Fitbit/Oura sections as templates. + catalog version once published — this repo cannot invent schemas locally. +5. One `Route`/`Converter` per vendor data type. For a small number of data types, per-type classes + (Oura's approach) are fine; for a large or schema-reuse-heavy surface (Huawei's ~54 types + sharing a handful of Avro schemas), prefer a single registry (`HuaweiRouteFactory.definitions`) + that both the `ConfigDef` builder and the route-construction code iterate over. +6. Add `docker/source-.properties.template`, a `docker-compose.yml` service entry, and a + README section, following the Fitbit/Oura/Huawei sections as templates. 7. Add the new Docker image to the `IMAGES` matrix in both `.github/workflows/main.yml` and `release.yml`. diff --git a/README.md b/README.md index 585c87e0..09b941e1 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # Kafka Connect REST Source and Fitbit Source This project contains a Kafka Connect source connector for a general REST API, for -specific Fitbit and Oura devices. The documentation of the Kafka Connect REST source still needs to -be done. +specific Fitbit, Oura, and Huawei Health Kit devices. The documentation of the Kafka Connect REST +source still needs to be done. @@ -10,6 +10,7 @@ be done. * [Fitbit source connector](#fitbit-source-connector) * [Installation](#installation) * [Usage](#usage) + * [Huawei Health Kit source connector](#huawei-health-kit-source-connector) * [Sentry monitoring](#sentry-monitoring) * [Contributing](#contributing) @@ -208,9 +209,30 @@ sequenceDiagram connector ->> connector: Update offset times ``` +## Huawei Health Kit source connector + +The `kafka-connect-huawei-source` module polls the +[Huawei Health Kit Data API](https://developer.huawei.com/consumer/en/doc/HMSCore-References/rest-overview-0000001254420693) +for the data types documented in the +[`radar-huawei-connector` schema specification](https://github.com/RADAR-base/RADAR-Schemas/blob/huawei_schemas/specifications/connector/radar-huawei-connector-1.0.0.yml) +(RADAR-Schemas, `huawei_schemas` branch) — activity records, continuous/instantaneous sample +statistics (steps, distance, calories, heart rate, SpO2, blood pressure, breathing rate, ECG, +sleep stages, and more), health records (ambulatory blood pressure, heart rate alerts, +hyperthermia, low SpO2 alerts, menstrual cycle, sleep), and daily summaries. It follows the same +`rest.source.*`, `huawei.api.client`/`huawei.api.secret`, and `huawei.user.repository.*` +configuration conventions as the Fitbit and Oura connectors above, plus one +`huawei..enabled` / `huawei..topic` pair per Huawei data type — see +`org.radarbase.huawei.route.HuaweiRouteFactory` for the full list of `` keys and their +default topic names, and `docker/source-huawei.properties.template` for a minimal example. + +This connector requires a +[published `radar-schemas-commons` build containing the `huawei_schemas` branch](https://github.com/RADAR-base/RADAR-Schemas/tree/huawei_schemas) +(currently `0.9.0-SNAPSHOT`) to be resolvable from one of the repositories declared in +`huawei-library/build.gradle` / `kafka-connect-huawei-source/build.gradle.kts`. + ## Sentry monitoring -To enable Sentry monitoring for the generic REST, Fitbit, or Oura source connector service: +To enable Sentry monitoring for the generic REST, Fitbit, Oura, or Huawei source connector service: 1. Set a `SENTRY_DSN` environment variable that points to the desired Sentry DSN. 2. (Optional) Set the `SENTRY_LOG_LEVEL` environment variable to control the minimum log level of diff --git a/docker-compose.yml b/docker-compose.yml index 53c62257..6243286b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -4,6 +4,7 @@ version: "2.4" volumes: fitbit-logs: {} oura-logs: {} + huawei-logs: {} services: #---------------------------------------------------------------------------# @@ -231,3 +232,49 @@ services: # SENTRY_DSN: 'https://000000000000.ingest.de.sentry.io/000000000000' # SENTRY_ATTACHSTACKTRACE: true # SENTRY_STACKTRACE_APP_PACKAGES: io.confluent.connect,org.radarbase.connect.rest + + #---------------------------------------------------------------------------# + # RADAR Huawei connector # + #---------------------------------------------------------------------------# + radar-huawei-connector: + build: + context: . + dockerfile: ./kafka-connect-huawei-source/Dockerfile + image: radarbase/radar-connect-huawei-source + restart: on-failure + volumes: + - ./docker/source-huawei.properties:/etc/kafka-connect/source-huawei.properties + - ./docker/users:/var/lib/kafka-connect-huawei-source/users + - huawei-logs:/var/lib/kafka-connect-huawei-source/logs + depends_on: + - zookeeper-1 + - zookeeper-2 + - zookeeper-3 + - kafka-1 + - kafka-2 + - kafka-3 + - schema-registry-1 + environment: + CONNECT_BOOTSTRAP_SERVERS: PLAINTEXT://kafka-1:9092,PLAINTEXT://kafka-2:9092,PLAINTEXT://kafka-3:9092 + CONNECT_REST_PORT: 8083 + CONNECT_GROUP_ID: "default" + CONNECT_CONFIG_STORAGE_TOPIC: "default.config" + CONNECT_OFFSET_STORAGE_TOPIC: "default.offsets" + CONNECT_STATUS_STORAGE_TOPIC: "default.status" + CONNECT_KEY_CONVERTER: "io.confluent.connect.avro.AvroConverter" + CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter" + CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry-1:8081" + CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry-1:8081" + CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter" + CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter" + CONNECT_OFFSET_STORAGE_FILE_FILENAME: "/var/lib/kafka-connect-huawei-source/logs/connect.offsets" + CONNECT_REST_ADVERTISED_HOST_NAME: "radar-huawei-connector" + CONNECT_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 + CONNECTOR_PROPERTY_FILE_PREFIX: "source-huawei" + KAFKA_HEAP_OPTS: "-Xms256m -Xmx768m" + KAFKA_BROKERS: 3 + CONNECT_LOG4J_LOGGERS: "org.reflections=ERROR" + # SENTRY_LOG_LEVEL: 'ERROR' + # SENTRY_DSN: 'https://000000000000.ingest.de.sentry.io/000000000000' + # SENTRY_ATTACHSTACKTRACE: true + # SENTRY_STACKTRACE_APP_PACKAGES: io.confluent.connect,org.radarbase.connect.rest diff --git a/docker/source-huawei.properties.template b/docker/source-huawei.properties.template new file mode 100644 index 00000000..1dd5b63e --- /dev/null +++ b/docker/source-huawei.properties.template @@ -0,0 +1,12 @@ +name=radar-huawei-source +connector.class=org.radarbase.connect.rest.huawei.HuaweiSourceConnector +tasks.max=4 +rest.source.base.url=https://health-api.cloud.huawei.com/healthkit/v1 +rest.source.poll.interval.ms=5000 +huawei.api.client=? +huawei.api.secret=? +huawei.user.repository.class=org.radarbase.connect.rest.huawei.user.HuaweiServiceUserRepository +huawei.user.repository.url=http://localhost:8080/ +huawei.user.repository.client.id=radar_huawei_connector +huawei.user.repository.client.secret= +huawei.user.repository.oauth2.token.url= From 089a0912751fe39c4491883d4ed682b83ed88bbf Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 11:19:59 +0000 Subject: [PATCH 06/44] Add tests for huawei-library and the connector config huawei-library tests (verified passing in this sandbox against a locally published radar-schemas-commons 0.9.0-SNAPSHOT): - FieldValuesTest: typed-value array and flattened-object parsing. - HuaweiRouteFactoryTest: builds every one of the ~54 registered route definitions, feeds each a fixture payload shaped for its endpoint kind, and asserts the converter produces exactly one record on the expected topic without throwing - this caught two real bugs during development (a bad endTime field on HuaweiCgmBloodGlucose, and a private extension function shadowing HuaweiDataConverter's). kafka-connect-huawei-source gets a config test mirroring the existing *RestSourceConnectorConfigTest convention, plus checks that enabledTopics() reflects the shared HuaweiRouteFactory.definitions registry and honors per-type huawei..enabled overrides. Could not run this one in-sandbox (see prior commit: packages.confluent.io is blocked here for every kafka-connect-*-source module, pre-existing and unrelated to this change). Also switches huawei-library's JUnit integration from kotlin-test-junit (JUnit4) to kotlin-test-junit5, since oura-library's copy-pasted dependency block conflicts with the JUnit Platform the radar-kotlin Gradle plugin configures the test task with - previously unnoticed only because no *-library module had tests yet. --- gradle/libs.versions.toml | 1 + huawei-library/build.gradle | 5 +- .../huawei/converter/FieldValuesTest.kt | 47 +++++ .../huawei/route/HuaweiRouteFactoryTest.kt | 180 ++++++++++++++++++ kafka-connect-huawei-source/build.gradle.kts | 1 + .../HuaweiRestSourceConnectorConfigTest.kt | 63 ++++++ 6 files changed, 295 insertions(+), 2 deletions(-) create mode 100644 huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt create mode 100644 huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt create mode 100644 kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index bf8a325a..9d14af12 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -59,6 +59,7 @@ mockito-core = { module = "org.mockito:mockito-core", version.ref = "mockito" } wiremock = { module = "com.github.tomakehurst:wiremock", version.ref = "wiremock" } kotlin-test = { module = "org.jetbrains.kotlin:kotlin-test", version.ref = "kotlin" } kotlin-test-junit = { module = "org.jetbrains.kotlin:kotlin-test-junit", version.ref = "kotlin" } +kotlin-test-junit5 = { module = "org.jetbrains.kotlin:kotlin-test-junit5", version.ref = "kotlin" } [plugins] radar-root-project = { id = "org.radarbase.radar-root-project", version.ref = "radarCommons" } diff --git a/huawei-library/build.gradle b/huawei-library/build.gradle index 882ad1e3..d35e82e9 100644 --- a/huawei-library/build.gradle +++ b/huawei-library/build.gradle @@ -51,8 +51,9 @@ dependencies { // Use the Kotlin test library. testImplementation libs.kotlin.test - // Use the Kotlin JUnit integration. - testImplementation libs.kotlin.test.junit + // Use the Kotlin JUnit 5 integration (matches the JUnit Platform the radar-kotlin + // Gradle plugin configures the `test` task with). + testImplementation libs.kotlin.test.junit5 } project.afterEvaluate { diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt new file mode 100644 index 00000000..cdca893f --- /dev/null +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt @@ -0,0 +1,47 @@ +package org.radarbase.huawei.converter + +import com.fasterxml.jackson.databind.ObjectMapper +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertNull + +class FieldValuesTest { + private val mapper = ObjectMapper() + + @Test + fun `parses typed-value array shape`() { + val node = mapper.readTree( + """ + [ + {"fieldName": "steps", "integerValue": 123}, + {"fieldName": "distance", "floatValue": 4.5}, + {"fieldName": "note", "stringValue": "hello"} + ] + """.trimIndent(), + ) + val fields = FieldValues.from(node) + + assertEquals(123, fields.getInt("steps")) + assertEquals(4.5, fields.getDouble("distance")) + assertEquals("hello", fields.getString("note")) + assertNull(fields.getInt("missing")) + } + + @Test + fun `parses flattened object shape`() { + val node = mapper.readTree("""{"avg": 1.5, "max": 3, "min": null}""") + val fields = FieldValues.from(node) + + assertEquals(1.5, fields.getDouble("avg")) + assertEquals(3, fields.getInt("max")) + assertNull(fields.getInt("min")) + } + + @Test + fun `handles missing or null root node`() { + val fields = FieldValues.from(null) + + assertNull(fields.getInt("anything")) + assertNull(fields.getString("anything")) + } +} diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt new file mode 100644 index 00000000..b6437590 --- /dev/null +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -0,0 +1,180 @@ +package org.radarbase.huawei.route + +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.databind.node.ArrayNode +import com.fasterxml.jackson.databind.node.ObjectNode +import org.apache.avro.Schema +import org.radarbase.huawei.user.HuaweiUser +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserRepository +import org.radarcns.connector.huawei.HuaweiHealthRecordDynamicBp +import java.time.Instant +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +/** + * Exercises every [HuaweiRouteFactory] definition end to end: builds the route, feeds a + * generously-populated fixture payload shaped like the endpoint it targets (`sampleSet:polymerize`, + * `healthRecords`, or `activityRecords`), and asserts the converter produces exactly one record on + * the definition's own topic without throwing. This is the main regression test against typos in + * the ~90 hand-written Huawei field-value key strings (and the Avro builder calls around them). + */ +class HuaweiRouteFactoryTest { + private val mapper = ObjectMapper() + private val fakeUser: User = HuaweiUser( + id = "u1", + createdAt = Instant.now(), + projectId = "p", + userId = "u", + humanReadableUserId = null, + sourceId = "s", + externalId = "ext", + isAuthorized = true, + startDate = Instant.parse("2024-01-01T00:00:00Z"), + ) + private val fakeUserRepository = object : UserRepository { + override fun get(key: String): User = fakeUser + override fun stream(): Sequence = sequenceOf(fakeUser) + override fun getAccessToken(user: User): String = "token" + } + + @Test + fun `definitions have unique keys and topics`() { + val keys = HuaweiRouteFactory.definitions.map { it.key } + val topics = HuaweiRouteFactory.definitions.map { it.defaultTopic } + + assertEquals(keys.size, keys.toSet().size, "duplicate route definition keys: $keys") + assertEquals(topics.size, topics.toSet().size, "duplicate route definition topics: $topics") + } + + @Test + fun `every definition converts a fixture payload without error`() { + val failures = mutableListOf() + + HuaweiRouteFactory.definitions.forEach { definition -> + val route = definition.build(fakeUserRepository, definition.defaultTopic) + try { + val payload = fixtureFor(route) + val records = route.converters.single().processRecords(payload, fakeUser).toList() + val successes = records.mapNotNull { it.getOrNull() } + + if (successes.size != 1) { + failures += "${definition.key}: expected 1 record, got ${successes.size} " + + "(errors: ${records.mapNotNull { it.exceptionOrNull() }})" + } else if (successes.first().topic != definition.defaultTopic) { + failures += "${definition.key}: unexpected topic ${successes.first().topic}" + } + } catch (e: Exception) { + failures += "${definition.key}: threw ${e}" + } + } + + assertTrue(failures.isEmpty(), "Failures:\n" + failures.joinToString("\n")) + } + + private fun fixtureFor(route: HuaweiRoute) = when (route) { + is HuaweiActivityRecordRoute -> activityRecordFixture() + is HuaweiHealthRecordRoute -> healthRecordFixture() + is HuaweiSampleSetRoute -> sampleSetFixture() + else -> error("Unknown route type: ${route::class}") + } + + private fun sampleSetFixture(): ObjectNode { + val root = mapper.createObjectNode() + val sampleSet = root.putArray("sampleSet") + val group = sampleSet.addObject() + val samplePoints = group.putArray("samplePoints") + val point = samplePoints.addObject() + point.put("startTime", START_MILLIS) + point.put("endTime", END_MILLIS) + point.set("value", genericValueArray()) + return root + } + + private fun healthRecordFixture(): ObjectNode { + val root = mapper.createObjectNode() + val records = root.putArray("healthRecords") + val record = records.addObject() + record.put("startTime", START_MILLIS) + record.put("endTime", END_MILLIS) + record.set("value", genericValueArray()) + return root + } + + private fun activityRecordFixture(): ObjectNode { + val root = mapper.createObjectNode() + val records = root.putArray("activityRecords") + val record = records.addObject() + record.put("startTime", START_MILLIS) + record.put("endTime", END_MILLIS) + record.put("id", "activity-1") + record.put("name", "Run") + record.put("description", "Morning run") + record.put("timeZone", "Europe/London") + record.put("activityType", "1") + record.put("activeTime", 1000L) + record.put("isKeepGoing", false) + val device = record.putObject("device") + device.put("manufacturer", "Huawei") + device.put("type", 1) + val summary = record.putObject("activitySummary") + summary.put("avgPace", 300.0) + summary.put("bestPace", 250.0) + summary.putObject("paceMap") + summary.putArray("dataSummary") + summary.putArray("sectionSummary") + return root + } + + /** One value entry per literal field-value key used across [HuaweiRouteFactory], plus every + * (snake-cased) field of [HuaweiHealthRecordDynamicBp] - covering both the ad hoc key names + * used for most data types and the mechanically-derived ones used for the 24h ABPM record. */ + private fun genericValueArray(): ArrayNode { + val array = mapper.createArrayNode() + (LITERAL_FIELD_KEYS + dynamicBpFieldKeys()).distinct().forEach { key -> + val entry = array.addObject() + entry.put("fieldName", key) + entry.put("integerValue", 1) + entry.put("floatValue", 1.5) + entry.put("stringValue", "test") + } + return array + } + + private fun dynamicBpFieldKeys(): List = + (HuaweiHealthRecordDynamicBp::class.java.getField("SCHEMA$").get(null) as Schema).fields + .map { it.name() } + .filterNot { it in setOf("time", "timeReceived", "endTime") } + .map(::snake) + + private fun snake(name: String): String = + Regex("([a-z0-9])([A-Z])").replace(name) { "${it.groupValues[1]}_${it.groupValues[2]}" }.lowercase() + + companion object { + private const val START_MILLIS = 1704067200000L // 2024-01-01T00:00:00Z + private const val END_MILLIS = 1704070800000L // 2024-01-01T01:00:00Z + + private val LITERAL_FIELD_KEYS = listOf( + "active_hours", "active_hours_target", "all_sleep_time", "arrhythmia_result", + "arrhythmia_type", "ascent_total", "avg", "avg_breathe_rate", "avg_heart_rate", + "awake_time", "calories", "calories_target", "calories_total", "correlate_mealtime", + "correlate_sleep", "count", "deep_sleep_part", "deep_sleep_time", "descent_total", + "diastolic_pressure_avg", "diastolic_pressure_max", "diastolic_pressure_min", + "distance_delta", "distance_total", "dream_time", "duration", "emotion", "event_name", + "exercise_time", "exercise_time_target", "exercise_type", "extend_data", + "fall_asleep_time", "go_bed_time", "heart_rate_variability_rmssd", + "high_body_temperature_alarm", "last", "level", "light_sleep_time", "max", + "max_breathe_rate", "max_breathrate_baseline", "max_spo2", "meal", "min", + "min_breathe_rate", "min_breathrate_baseline", "min_spo2", "off_bed_time", + "on_off_bed_state", "predicted_calories", "prepare_sleep_time", "record_day", + "record_id", "remarks", "sample_source", "sampling_frequency", "sleep_efficiency", + "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "sphygmus_avg", + "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", "steps_delta", + "steps_target", "sub_status", "systolic_pressure_avg", "systolic_pressure_max", + "systolic_pressure_min", "threshold", "timezone", "total_calories", "type", + "user_symptom", "value", "vo2max", "voltage_data", "wakeup_count", "wakeup_time", + "zone1_duration", "zone2_duration", "zone3_duration", "zone4_duration", "zone5_duration", + ) + } +} diff --git a/kafka-connect-huawei-source/build.gradle.kts b/kafka-connect-huawei-source/build.gradle.kts index 05fcfa3c..3bc0b307 100644 --- a/kafka-connect-huawei-source/build.gradle.kts +++ b/kafka-connect-huawei-source/build.gradle.kts @@ -58,4 +58,5 @@ dependencies { testImplementation(libs.kafka.connect.api) testImplementation(libs.wiremock) testImplementation(libs.mockito.core) + testImplementation(libs.kotlin.test) } diff --git a/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt new file mode 100644 index 00000000..dd7ae19a --- /dev/null +++ b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt @@ -0,0 +1,63 @@ +/* + * Copyright 2018 The Hyve + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.connect.rest.huawei + +import org.junit.jupiter.api.Test +import org.radarbase.huawei.route.HuaweiRouteFactory +import kotlin.test.assertEquals + +class HuaweiRestSourceConnectorConfigTest { + + @Test + fun conf() { + println(HuaweiRestSourceConnectorConfig.conf().toHtmlTable()) + } + + @Test + fun `enabled topics default to every registered data type`() { + val config = HuaweiRestSourceConnectorConfig( + mutableMapOf( + "huawei.api.client" to "client", + "huawei.api.secret" to "secret", + ), + false, + ) + + val enabled = config.enabledTopics() + + assertEquals(HuaweiRouteFactory.definitions.size, enabled.size) + HuaweiRouteFactory.definitions.forEach { definition -> + assertEquals(definition.defaultTopic, enabled[definition.key]) + } + } + + @Test + fun `a data type can be disabled via config`() { + val definition = HuaweiRouteFactory.definitions.first() + val config = HuaweiRestSourceConnectorConfig( + mutableMapOf( + "huawei.api.client" to "client", + "huawei.api.secret" to "secret", + "huawei.${definition.key}.enabled" to "false", + ), + false, + ) + + assertEquals(null, config.enabledTopics()[definition.key]) + } +} From 9267bc83f7bf2e2934fc66cb272459fef1d56920 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 11:22:14 +0000 Subject: [PATCH 07/44] Fix ktlint style violations in huawei-library `./gradlew check` runs ktlint over every module; auto-formatted the mechanical issues (semicolons, wrapping) and manually split a handful of lines ktlint couldn't auto-correct. huawei-library:check is now fully green (ktlint + all 5 tests) in this sandbox. --- .../radarbase/huawei/converter/FieldValues.kt | 11 +- .../converter/HuaweiSampleSetConverter.kt | 3 +- .../huawei/request/HuaweiRequestGenerator.kt | 37 +- .../radarbase/huawei/request/HuaweiResult.kt | 62 +- .../huawei/route/HuaweiActivityRecordRoute.kt | 3 +- .../org/radarbase/huawei/route/HuaweiRoute.kt | 28 +- .../huawei/route/HuaweiRouteFactory.kt | 1129 ++++++++++++----- .../huawei/route/HuaweiSampleSetRoute.kt | 6 +- .../huawei/route/HuaweiRouteFactoryTest.kt | 9 +- 9 files changed, 882 insertions(+), 406 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt index 57386986..f0db5d6a 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt @@ -22,15 +22,20 @@ class FieldValues private constructor(private val values: Map) fun getLong(field: String): Long? = values[field]?.let { if (it.isNull) null else it.asLong() } - fun getDouble(field: String): Double? = values[field]?.let { if (it.isNull) null else it.asDouble() } + fun getDouble(field: String): Double? = values[field]?.let { + if (it.isNull) null else it.asDouble() + } fun getFloat(field: String): Float? = getDouble(field)?.toFloat() - fun getString(field: String): String? = values[field]?.let { if (it.isNull) null else it.asText() } + fun getString(field: String): String? = values[field]?.let { + if (it.isNull) null else it.asText() + } companion object { private const val FIELD_NAME_KEY = "fieldName" - private val VALUE_KEYS = listOf("integerValue", "floatValue", "longValue", "stringValue", "value") + private val VALUE_KEYS = + listOf("integerValue", "floatValue", "longValue", "stringValue", "value") fun from(node: JsonNode?): FieldValues { if (node == null || node.isMissingNode || node.isNull) { diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt index 6e2929f4..e5c6d566 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt @@ -35,7 +35,8 @@ class HuaweiSampleSetConverter( val sampleSets = root.get("sampleSet") ?: root.get("sampleSets") ?: return emptySequence() return sampleSets.asSequence() .flatMap { group -> - (group.get("samplePoints") ?: group.get("samplePoint"))?.asSequence() ?: emptySequence() + val points = group.get("samplePoints") ?: group.get("samplePoint") + points?.asSequence() ?: emptySequence() } .mapCatching { point -> val startTime = point.epochInstant("startTime") diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt index b2d7da11..2a733024 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt @@ -106,7 +106,13 @@ class HuaweiRequestGenerator( logger.debug("Request successful: {}..", request.request) val body = response.body val data = body?.bytes() ?: ByteArray(0) - val records = request.route.converters.flatMap { it.convert(request, response.headers, data) } + val records = request.route.converters.flatMap { + it.convert( + request, + response.headers, + data, + ) + } val offset = records.maxByOrNull { it.offset }?.offset val key = routeKey(request.route, request.user) if (offset != null) { @@ -130,7 +136,10 @@ class HuaweiRequestGenerator( HuaweiRateLimitError("Rate limit reached..", TooManyRequestsException(), "429") } 403 -> { - logger.warn("User {} does not have access to this Huawei Health Kit data type.", request.user) + logger.warn( + "User {} does not have access to this Huawei Health Kit data type.", + request.user, + ) routeNextRequest[key] = Instant.now().plus(USER_BACK_OFF_TIME) HuaweiAccessForbiddenError( "Huawei Health Kit scope not granted or data not available..", @@ -150,22 +159,38 @@ class HuaweiRequestGenerator( 400 -> { logger.warn("Client exception for request {}", request) routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) - HuaweiClientException("Client unsupported or unauthorized..", IOException("Invalid client"), "400") + HuaweiClientException( + "Client unsupported or unauthorized..", + IOException("Invalid client"), + "400", + ) } 422 -> { logger.warn("Request failed (validation error): {}, {}", request, response) routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) - HuaweiValidationError(response.body?.string() ?: "validation error", IOException("Validation error"), "422") + HuaweiValidationError( + response.body?.string() ?: "validation error", + IOException("Validation error"), + "422", + ) } 404 -> { logger.warn("Not found: {}", request) routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) - HuaweiNotFoundError(response.body?.string() ?: "not found", IOException("Data not found"), "404") + HuaweiNotFoundError( + response.body?.string() ?: "not found", + IOException("Data not found"), + "404", + ) } else -> { logger.warn("Request failed: {}, {}", request, response) routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) - HuaweiGenericError(response.body?.string() ?: "unknown error", IOException("Unknown error"), "500") + HuaweiGenericError( + response.body?.string() ?: "unknown error", + IOException("Unknown error"), + "500", + ) } } } diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt index 8a4fa587..fba9e551 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt @@ -13,52 +13,44 @@ sealed class HuaweiErrorBase( val code: String, ) : HuaweiError -class HuaweiRateLimitError(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( - message, - cause, - code, -) +class HuaweiRateLimitError( + message: String, + cause: Exception? = null, + code: String, +) : HuaweiErrorBase(message, cause, code) -class HuaweiClientException(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( - message, - cause, - code, -) +class HuaweiClientException( + message: String, + cause: Exception? = null, + code: String, +) : HuaweiErrorBase(message, cause, code) class HuaweiUnauthorizedAccessError( message: String, cause: Exception? = null, code: String, -) : HuaweiErrorBase( - message, - cause, - code, -) +) : HuaweiErrorBase(message, cause, code) class HuaweiAccessForbiddenError( message: String, cause: Exception? = null, code: String, -) : HuaweiErrorBase( - message, - cause, - code, -) +) : HuaweiErrorBase(message, cause, code) -class HuaweiValidationError(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( - message, - cause, - code, -) +class HuaweiValidationError( + message: String, + cause: Exception? = null, + code: String, +) : HuaweiErrorBase(message, cause, code) -class HuaweiGenericError(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( - message, - cause, - code, -) +class HuaweiGenericError( + message: String, + cause: Exception? = null, + code: String, +) : HuaweiErrorBase(message, cause, code) -class HuaweiNotFoundError(message: String, cause: Exception? = null, code: String) : HuaweiErrorBase( - message, - cause, - code, -) +class HuaweiNotFoundError( + message: String, + cause: Exception? = null, + code: String, +) : HuaweiErrorBase(message, cause, code) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt index 63318b23..07a4f586 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt @@ -18,7 +18,8 @@ class HuaweiActivityRecordRoute( maxIntervalPerRequest: Duration = Duration.ofDays(30L), ) : HuaweiRoute(userRepository, maxIntervalPerRequest) { - override val converters: List = listOf(HuaweiActivityRecordConverter(topic)) + override val converters: List = + listOf(HuaweiActivityRecordConverter(topic)) override fun toString(): String = "huawei_activity_record" diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt index 8fd5bb01..33a47dea 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt @@ -24,7 +24,11 @@ abstract class HuaweiRoute( ) : Route { abstract val converters: List - protected fun createGetRequest(user: User, path: String, queryParams: Map): Request { + protected fun createGetRequest( + user: User, + path: String, + queryParams: Map, + ): Request { val accessToken = userRepository.getAccessToken(user) val urlBuilder = "$HUAWEI_API_BASE_URL/$path".toHttpUrl().newBuilder() queryParams.forEach { (key, value) -> urlBuilder.addQueryParameter(key, value) } @@ -44,15 +48,27 @@ abstract class HuaweiRoute( .build() } - /** Split `[start, end)` into consecutive windows of at most [maxIntervalPerRequest], capped at [max] windows. */ - protected fun chunkedRanges(start: Instant, end: Instant, max: Int): Sequence> = + /** + * Split `[start, end)` into consecutive windows of at most [maxIntervalPerRequest], capped at + * [max] windows. + */ + protected fun chunkedRanges( + start: Instant, + end: Instant, + max: Int, + ): Sequence> = generateSequence(start) { it + maxIntervalPerRequest } .takeWhile { it < end } .take(max) - .map { rangeStart -> rangeStart to (rangeStart + maxIntervalPerRequest).coerceAtMost(end) } + .map { rangeStart -> + rangeStart to (rangeStart + maxIntervalPerRequest).coerceAtMost(end) + } - override fun generateRequests(user: User, start: Instant, end: Instant): Sequence = - generateRequests(user, start, end, Int.MAX_VALUE) + override fun generateRequests( + user: User, + start: Instant, + end: Instant, + ): Sequence = generateRequests(user, start, end, Int.MAX_VALUE) companion object { const val HUAWEI_API_BASE_URL = "https://health-api.cloud.huawei.com/healthkit/v1" diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 2e1df2fa..5d7e2032 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -63,7 +63,9 @@ object HuaweiRouteFactory { /** Best-effort camelCase -> snake_case conversion for deriving a Huawei field key from an Avro field name. */ private fun snake(name: String): String = - SNAKE_CASE_BOUNDARY.replace(name) { "${it.groupValues[1]}_${it.groupValues[2]}" }.lowercase() + SNAKE_CASE_BOUNDARY.replace( + name, + ) { "${it.groupValues[1]}_${it.groupValues[2]}" }.lowercase() private val SNAKE_CASE_BOUNDARY = Regex("([a-z0-9])([A-Z])") @@ -85,333 +87,693 @@ object HuaweiRouteFactory { /** Data types that reuse the generic [HuaweiStatistics] schema: (config key, Huawei data type name, default topic). */ private val genericStatisticsTypes = listOf( - Triple("continuous_body_fat_rate_statistics", "continuous.body.fat.rate.statistics", "connect_huawei_continuous_body_fat_rate_statistics"), - Triple("continuous_body_temperature_rest_statistics", "continuous.body.temperature.rest.statistics", "connect_huawei_continuous_body_temperature_rest_statistics"), - Triple("continuous_body_temperature_statistics", "continuous.body.temperature.statistics", "connect_huawei_continuous_body_temperature_statistics"), - Triple("continuous_calories_bmr_statistics", "continuous.calories.bmr.statistics", "connect_huawei_continuous_calories_bmr_statistics"), - Triple("continuous_exercise_heart_rate_statistics", "continuous.exercise_heart_rate.statistics", "connect_huawei_continuous_exercise_heart_rate_statistics"), - Triple("continuous_heart_rate_statistics", "continuous.heart_rate.statistics", "connect_huawei_continuous_heart_rate_statistics"), - Triple("continuous_power_statistics", "continuous.power.statistics", "connect_huawei_continuous_power_statistics"), - Triple("continuous_skin_temperature_statistics", "continuous.skin.temperature.statistics", "connect_huawei_continuous_skin_temperature_statistics"), - Triple("continuous_speed_statistics", "continuous.speed.statistics", "connect_huawei_continuous_speed_statistics"), - Triple("continuous_steps_rate_statistics", "continuous.steps.rate.statistics", "connect_huawei_continuous_steps_rate_statistics"), - Triple("continuous_stroke_rate_statistics", "continuous.stroke_rate.statistics", "connect_huawei_continuous_stroke_rate_statistics"), - Triple("instantaneous_resting_heart_rate_statistics", "instantaneous.resting_heart_rate.statistics", "connect_huawei_instantaneous_resting_heart_rate_statistics"), - Triple("instantaneous_stress_statistics", "instantaneous.stress.statistics", "connect_huawei_instantaneous_stress_statistics"), + Triple( + "continuous_body_fat_rate_statistics", + "continuous.body.fat.rate.statistics", + "connect_huawei_continuous_body_fat_rate_statistics", + ), + Triple( + "continuous_body_temperature_rest_statistics", + "continuous.body.temperature.rest.statistics", + "connect_huawei_continuous_body_temperature_rest_statistics", + ), + Triple( + "continuous_body_temperature_statistics", + "continuous.body.temperature.statistics", + "connect_huawei_continuous_body_temperature_statistics", + ), + Triple( + "continuous_calories_bmr_statistics", + "continuous.calories.bmr.statistics", + "connect_huawei_continuous_calories_bmr_statistics", + ), + Triple( + "continuous_exercise_heart_rate_statistics", + "continuous.exercise_heart_rate.statistics", + "connect_huawei_continuous_exercise_heart_rate_statistics", + ), + Triple( + "continuous_heart_rate_statistics", + "continuous.heart_rate.statistics", + "connect_huawei_continuous_heart_rate_statistics", + ), + Triple( + "continuous_power_statistics", + "continuous.power.statistics", + "connect_huawei_continuous_power_statistics", + ), + Triple( + "continuous_skin_temperature_statistics", + "continuous.skin.temperature.statistics", + "connect_huawei_continuous_skin_temperature_statistics", + ), + Triple( + "continuous_speed_statistics", + "continuous.speed.statistics", + "connect_huawei_continuous_speed_statistics", + ), + Triple( + "continuous_steps_rate_statistics", + "continuous.steps.rate.statistics", + "connect_huawei_continuous_steps_rate_statistics", + ), + Triple( + "continuous_stroke_rate_statistics", + "continuous.stroke_rate.statistics", + "connect_huawei_continuous_stroke_rate_statistics", + ), + Triple( + "instantaneous_resting_heart_rate_statistics", + "instantaneous.resting_heart_rate.statistics", + "connect_huawei_instantaneous_resting_heart_rate_statistics", + ), + Triple( + "instantaneous_stress_statistics", + "instantaneous.stress.statistics", + "connect_huawei_instantaneous_stress_statistics", + ), Triple("vo2max_statistics", "vo2max.statistics", "connect_huawei_vo2max_statistics"), ) /** Full registry of Huawei Health Kit data types supported by this connector. */ val definitions: List = buildList { add( - HuaweiRouteDefinition("activity_record", "connect_huawei_activity_record") { repo, topic -> + HuaweiRouteDefinition( + "activity_record", + "connect_huawei_activity_record", + ) { repo, topic -> HuaweiActivityRecordRoute(repo, topic) }, ) // cgm_blood_glucose (+ .statistics variant) - add(sampleSetDefinition("cgm_blood_glucose", "cgm_blood_glucose", "connect_huawei_cgm_blood_glucose") { f, start, _, received -> - HuaweiCgmBloodGlucose.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch() - level = f.getDouble("level") - avg = f.getInt("avg"); max = f.getInt("max"); min = f.getInt("min"); last = f.getInt("last") - }.build() - }) - add(sampleSetDefinition("cgm_blood_glucose_statistics", "cgm_blood_glucose.statistics", "connect_huawei_cgm_blood_glucose_statistics") { f, start, _, received -> - HuaweiCgmBloodGlucose.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch() - level = f.getDouble("level") - avg = f.getInt("avg"); max = f.getInt("max"); min = f.getInt("min"); last = f.getInt("last") - }.build() - }) - - add(sampleSetDefinition("daily_activity_summary", "daily_activity_summary", "connect_huawei_daily_activity_summary") { f, start, end, received -> - HuaweiDailyActivitySummary.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - steps = f.getInt("steps") - activeCalories = f.getInt("calories") - exerciseTime = f.getInt("exercise_time") - activeHours = f.getInt("active_hours") - stepsGoal = f.getInt("steps_target") - activeCaloriesGoal = f.getInt("calories_target") - exerciseTimeGoal = f.getInt("exercise_time_target") - activeHoursGoal = f.getInt("active_hours_target") - }.build() - }) - - add(sampleSetDefinition("active_hours", "active_hours", "connect_huawei_active_hours") { f, start, end, received -> - f.toActiveHours(start, end, received) - }) - add(sampleSetDefinition("active_hours_statistics", "active_hours.statistics", "connect_huawei_active_hours_statistics") { f, start, end, received -> - f.toActiveHours(start, end, received) - }) - - add(sampleSetDefinition("continuous_activity_fragment", "continuous.activity.fragment", "connect_huawei_continuous_activity_fragment") { f, start, end, received -> - f.toContinuousActivityStatistics(start, end, received) - }) - add(sampleSetDefinition("continuous_activity_statistics", "continuous.activity.statistics", "connect_huawei_continuous_activity_statistics") { f, start, end, received -> - f.toContinuousActivityStatistics(start, end, received) - }) - - add(sampleSetDefinition("continuous_altitude_statistics", "continuous.altitude.statistics", "connect_huawei_continuous_altitude_statistics") { f, start, end, received -> - HuaweiContinuousAltitudeStatistics.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - avg = f.getDouble("avg"); max = f.getDouble("max"); min = f.getDouble("min") - ascentTotal = f.getDouble("ascent_total") - descentTotal = f.getDouble("descent_total") - }.build() - }) - - add(sampleSetDefinition("continuous_blood_glucose_statistics", "continuous.blood_glucose.statistics", "connect_huawei_continuous_blood_glucose_statistics") { f, start, end, received -> - HuaweiContinuousBloodGlucoseStatistics.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - avg = f.getDouble("avg"); max = f.getDouble("max"); min = f.getDouble("min") - correlationWithMealtime = f.getInt("correlate_mealtime") - meal = f.getInt("meal") - correlationWithSleepState = f.getInt("correlate_sleep") - sampleSource = f.getInt("sample_source") - }.build() - }) - - add(sampleSetDefinition("continuous_breathe_rate_statistics", "continuous.breathe_rate.statistics", "connect_huawei_continuous_breathe_rate_statistics") { f, start, end, received -> - HuaweiContinuousBreatheRateStatistics.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - maxBreatheRate = f.getInt("max_breathe_rate") - minBreatheRate = f.getInt("min_breathe_rate") - avgBreatheRate = f.getInt("avg_breathe_rate") - minBreathrateBaseline = f.getInt("min_breathrate_baseline") - maxBreathrateBaseline = f.getInt("max_breathrate_baseline") - }.build() - }) - - add(sampleSetDefinition("continuous_body_blood_pressure_statistics", "continuous.body.blood_pressure.statistics", "connect_huawei_continuous_body_blood_pressure_statistics") { f, start, end, received -> - HuaweiContinuousBodyBloodPressureStatistics.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - systolicPressureAvg = f.getDouble("systolic_pressure_avg") - systolicPressureMax = f.getDouble("systolic_pressure_max") - systolicPressureMin = f.getDouble("systolic_pressure_min") - diastolicPressureAvg = f.getDouble("diastolic_pressure_avg") - diastolicPressureMax = f.getDouble("diastolic_pressure_max") - diastolicPressureMin = f.getDouble("diastolic_pressure_min") - sphygmusAvg = f.getDouble("sphygmus_avg") - sphygmusMax = f.getDouble("sphygmus_max") - sphygmusMin = f.getDouble("sphygmus_min") - sphygmusLast = f.getDouble("sphygmus_last") - }.build() - }) + add( + sampleSetDefinition( + "cgm_blood_glucose", + "cgm_blood_glucose", + "connect_huawei_cgm_blood_glucose", + ) { f, start, _, received -> + HuaweiCgmBloodGlucose.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + level = f.getDouble("level") + avg = f.getInt("avg") + max = f.getInt("max") + min = f.getInt("min") + last = f.getInt("last") + }.build() + }, + ) + add( + sampleSetDefinition( + "cgm_blood_glucose_statistics", + "cgm_blood_glucose.statistics", + "connect_huawei_cgm_blood_glucose_statistics", + ) { f, start, _, received -> + HuaweiCgmBloodGlucose.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + level = f.getDouble("level") + avg = f.getInt("avg") + max = f.getInt("max") + min = f.getInt("min") + last = f.getInt("last") + }.build() + }, + ) + + add( + sampleSetDefinition( + "daily_activity_summary", + "daily_activity_summary", + "connect_huawei_daily_activity_summary", + ) { f, start, end, received -> + HuaweiDailyActivitySummary.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + steps = f.getInt("steps") + activeCalories = f.getInt("calories") + exerciseTime = f.getInt("exercise_time") + activeHours = f.getInt("active_hours") + stepsGoal = f.getInt("steps_target") + activeCaloriesGoal = f.getInt("calories_target") + exerciseTimeGoal = f.getInt("exercise_time_target") + activeHoursGoal = f.getInt("active_hours_target") + }.build() + }, + ) + + add( + sampleSetDefinition( + "active_hours", + "active_hours", + "connect_huawei_active_hours", + ) { f, start, end, received -> + f.toActiveHours(start, end, received) + }, + ) + add( + sampleSetDefinition( + "active_hours_statistics", + "active_hours.statistics", + "connect_huawei_active_hours_statistics", + ) { f, start, end, received -> + f.toActiveHours(start, end, received) + }, + ) + + add( + sampleSetDefinition( + "continuous_activity_fragment", + "continuous.activity.fragment", + "connect_huawei_continuous_activity_fragment", + ) { f, start, end, received -> + f.toContinuousActivityStatistics(start, end, received) + }, + ) + add( + sampleSetDefinition( + "continuous_activity_statistics", + "continuous.activity.statistics", + "connect_huawei_continuous_activity_statistics", + ) { f, start, end, received -> + f.toContinuousActivityStatistics(start, end, received) + }, + ) + + add( + sampleSetDefinition( + "continuous_altitude_statistics", + "continuous.altitude.statistics", + "connect_huawei_continuous_altitude_statistics", + ) { f, start, end, received -> + HuaweiContinuousAltitudeStatistics.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + avg = f.getDouble("avg") + max = f.getDouble("max") + min = f.getDouble("min") + ascentTotal = f.getDouble("ascent_total") + descentTotal = f.getDouble("descent_total") + }.build() + }, + ) + + add( + sampleSetDefinition( + "continuous_blood_glucose_statistics", + "continuous.blood_glucose.statistics", + "connect_huawei_continuous_blood_glucose_statistics", + ) { f, start, end, received -> + HuaweiContinuousBloodGlucoseStatistics.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + avg = f.getDouble("avg") + max = f.getDouble("max") + min = f.getDouble("min") + correlationWithMealtime = f.getInt("correlate_mealtime") + meal = f.getInt("meal") + correlationWithSleepState = f.getInt("correlate_sleep") + sampleSource = f.getInt("sample_source") + }.build() + }, + ) + + add( + sampleSetDefinition( + "continuous_breathe_rate_statistics", + "continuous.breathe_rate.statistics", + "connect_huawei_continuous_breathe_rate_statistics", + ) { f, start, end, received -> + HuaweiContinuousBreatheRateStatistics.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + maxBreatheRate = f.getInt("max_breathe_rate") + minBreatheRate = f.getInt("min_breathe_rate") + avgBreatheRate = f.getInt("avg_breathe_rate") + minBreathrateBaseline = f.getInt("min_breathrate_baseline") + maxBreathrateBaseline = f.getInt("max_breathrate_baseline") + }.build() + }, + ) + + add( + sampleSetDefinition( + "continuous_body_blood_pressure_statistics", + "continuous.body.blood_pressure.statistics", + "connect_huawei_continuous_body_blood_pressure_statistics", + ) { f, start, end, received -> + HuaweiContinuousBodyBloodPressureStatistics.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + systolicPressureAvg = f.getDouble("systolic_pressure_avg") + systolicPressureMax = f.getDouble("systolic_pressure_max") + systolicPressureMin = f.getDouble("systolic_pressure_min") + diastolicPressureAvg = f.getDouble("diastolic_pressure_avg") + diastolicPressureMax = f.getDouble("diastolic_pressure_max") + diastolicPressureMin = f.getDouble("diastolic_pressure_min") + sphygmusAvg = f.getDouble("sphygmus_avg") + sphygmusMax = f.getDouble("sphygmus_max") + sphygmusMin = f.getDouble("sphygmus_min") + sphygmusLast = f.getDouble("sphygmus_last") + }.build() + }, + ) genericStatisticsTypes.forEach { (key, dataType, topic) -> add( sampleSetDefinition(key, dataType, topic) { f, start, end, received -> - HuaweiStatistics.newBuilder().apply { populateCommon(start, end, received, f) }.build() + HuaweiStatistics.newBuilder().apply { + populateCommon( + start, + end, + received, + f, + ) + }.build() }, ) } - add(sampleSetDefinition("continuous_calories_burnt", "continuous.calories.burnt", "connect_huawei_continuous_calories_burnt") { f, start, end, received -> - HuaweiContinuousCaloriesBurnt.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - calories = f.getDouble("calories") - }.build() - }) - add(sampleSetDefinition("continuous_calories_consumed", "continuous.calories.consumed", "connect_huawei_continuous_calories_consumed") { f, start, end, received -> - HuaweiContinuousCaloriesBurnt.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - calories = f.getDouble("calories") - }.build() - }) - add(sampleSetDefinition("continuous_calories_burnt_total", "continuous.calories.burnt.total", "connect_huawei_continuous_calories_burnt_total") { f, start, end, received -> - HuaweiContinuousCaloriesBurntTotal.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - caloriesTotal = f.getDouble("calories_total") - }.build() - }) - - add(sampleSetDefinition("continuous_distance_delta", "continuous.distance.delta", "connect_huawei_continuous_distance_delta") { f, start, end, received -> - HuaweiContinuousDistanceDelta.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - distanceDelta = f.getDouble("distance_delta") - }.build() - }) - add(sampleSetDefinition("continuous_distance_total", "continuous.distance.total", "connect_huawei_continuous_distance_total") { f, start, end, received -> - HuaweiContinuousDistanceTotal.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - distance = f.getDouble("distance_total") - }.build() - }) - - add(sampleSetDefinition("continuous_ecg_detail", "continuous.ecg_detail", "connect_huawei_continuous_ecg_detail") { f, start, end, received -> - HuaweiContinuousEcgDetail.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - ecgRecordId = f.getString("record_id") - averageHeartRate = f.getInt("avg_heart_rate") - ecgArrhythmiaType = f.getInt("arrhythmia_type") - ecgArrhythmiaResult = f.getInt("arrhythmia_result") - userSymptom = f.getString("user_symptom") - samplingFrequency = f.getInt("sampling_frequency") - voltageData = f.getString("voltage_data") - }.build() - }) - - add(sampleSetDefinition("continuous_exercise_intensity", "continuous.exercise_intensity", "connect_huawei_continuous_exercise_intensity") { f, start, end, received -> - f.toContinuousExerciseIntensity(start, end, received) - }) - add(sampleSetDefinition("continuous_exercise_intensity_statistics", "continuous.exercise_intensity.statistics", "connect_huawei_continuous_exercise_intensity_statistics") { f, start, end, received -> - f.toContinuousExerciseIntensity(start, end, received) - }) - - add(sampleSetDefinition("continuous_exercise_intensity_v2", "continuous.exercise_intensity.v2", "connect_huawei_continuous_exercise_intensity_v2") { f, start, end, received -> - HuaweiContinuousExerciseIntensityV2.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - exerciseType = f.getInt("exercise_type") - }.build() - }) - add(sampleSetDefinition("continuous_exercise_intensity_v2_statistics", "continuous.exercise_intensity.v2.statistics", "connect_huawei_continuous_exercise_intensity_v2_statistics") { f, start, end, received -> - HuaweiContinuousExerciseIntensityV2Statistics.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - zone1Duration = f.getInt("zone1_duration") - zone2Duration = f.getInt("zone2_duration") - zone3Duration = f.getInt("zone3_duration") - zone4Duration = f.getInt("zone4_duration") - zone5Duration = f.getInt("zone5_duration") - }.build() - }) - - add(sampleSetDefinition("continuous_sleep_fragment", "continuous.sleep.fragment", "connect_huawei_continuous_sleep_fragment") { f, start, end, received -> - HuaweiContinuousSleepFragment.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - sleepState = f.getInt("sleep_state") - }.build() - }) - - add(sampleSetDefinition("continuous_spo2_statistics", "continuous.spo2.statistics", "connect_huawei_continuous_spo2_statistics") { f, start, end, received -> - HuaweiContinuousSpo2Statistics.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - saturationAvg = f.getDouble("avg") - saturationMax = f.getDouble("max") - saturationMin = f.getDouble("min") - saturationLast = f.getDouble("last") - }.build() - }) - - add(sampleSetDefinition("continuous_steps_delta", "continuous.steps.delta", "connect_huawei_continuous_steps_delta") { f, start, end, received -> - HuaweiContinuousStepsDelta.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - stepsDelta = f.getInt("steps_delta") - }.build() - }) - add(sampleSetDefinition("continuous_steps_total", "continuous.steps.total", "connect_huawei_continuous_steps_total") { f, start, end, received -> - HuaweiContinuousStepsTotal.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - steps = f.getInt("steps") - duration = f.getInt("duration") - }.build() - }) - - add(sampleSetDefinition("emotion", "emotion", "connect_huawei_emotion") { f, start, _, received -> - HuaweiEmotion.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch() - emotionStatus = f.getInt("emotion") - }.build() - }) - - add(healthRecordDefinition("health_record_dynamic_bp", "health.record.dynamic_bp", "connect_huawei_health_record_dynamic_bp") { f, start, end, received -> - f.toHealthRecordDynamicBp(start, end, received) - }) - add(healthRecordDefinition("health_record_bradycardia", "health.record.bradycardia", "connect_huawei_health_record_bradycardia") { f, start, end, received -> - f.toHealthRecordHeartRateAlert(start, end, received) - }) - add(healthRecordDefinition("health_record_tachycardia", "health.record.tachycardia", "connect_huawei_health_record_tachycardia") { f, start, end, received -> - f.toHealthRecordHeartRateAlert(start, end, received) - }) - add(healthRecordDefinition("health_record_hyperthermia", "health.record.hyperthermia", "connect_huawei_health_record_hyperthermia") { f, start, end, received -> - HuaweiHealthRecordHyperthermia.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - highBodyTemperatureAlarm = f.getFloat("high_body_temperature_alarm") - }.build() - }) - add(healthRecordDefinition("health_record_low_spo2_alert", "health.record.lowSpo2Alert", "connect_huawei_health_record_low_spo2_alert") { f, start, end, received -> - HuaweiHealthRecordLowSpo2Alert.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - threshold = f.getFloat("threshold") - maxSpO2 = f.getFloat("max_spo2") - minSpO2 = f.getFloat("min_spo2") - }.build() - }) - add(healthRecordDefinition("health_record_menstrual_cycle", "health.record.menstrual_cycle", "connect_huawei_health_record_menstrual_cycle") { f, start, end, received -> - HuaweiHealthRecordMenstrualCycle.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - recordday = f.getInt("record_day") - status = f.getInt("status") - substatus = f.getInt("sub_status") - remarks = f.getString("remarks") - timezone = f.getString("timezone") - }.build() - }) - add(healthRecordDefinition("health_record_sleep", "health.record.sleep", "connect_huawei_health_record_sleep") { f, start, end, received -> - HuaweiHealthRecordSleep.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - fallAsleepTime = f.getLong("fall_asleep_time") - wakeupTime = f.getLong("wakeup_time") - lightSleepTime = f.getInt("light_sleep_time") - deepSleepTime = f.getInt("deep_sleep_time") - dreamTime = f.getInt("dream_time") - awakeTime = f.getInt("awake_time") - allSleepTime = f.getInt("all_sleep_time") - wakeupCount = f.getInt("wakeup_count") - deepSleepPart = f.getInt("deep_sleep_part") - sleepScore = f.getInt("sleep_score") - sleepLatency = f.getInt("sleep_latency") - sleepEfficiency = f.getInt("sleep_efficiency") - goBedTime = f.getLong("go_bed_time") - sleepType = f.getInt("sleep_type") - prepareSleepTime = f.getLong("prepare_sleep_time") - offBedTime = f.getLong("off_bed_time") - }.build() - }) - - add(sampleSetDefinition("heart_rate_variability", "heart_rate_variability", "connect_huawei_heart_rate_variability") { f, start, _, received -> - HuaweiHeartRateVariability.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch() - heartRateVariabilityRmssd = f.getInt("heart_rate_variability_rmssd") - }.build() - }) - - add(sampleSetDefinition("resting_calories_statistics", "resting_calories.statistics", "connect_huawei_resting_calories_statistics") { f, start, end, received -> - HuaweiRestingCaloriesStatistics.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - predictedCalories = f.getFloat("predicted_calories") - totalCalories = f.getFloat("total_calories") - }.build() - }) - - add(sampleSetDefinition("sleep_on_off_bed_record", "sleep.on_off_bed_record", "connect_huawei_sleep_on_off_bed_record") { f, start, _, received -> - HuaweiSleepOnOffBedRecord.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch() - onOffBedState = f.getInt("on_off_bed_state") - }.build() - }) - - add(sampleSetDefinition("sleep_respiratory_detail", "sleep_respiratory_detail", "connect_huawei_sleep_respiratory_detail") { f, start, end, received -> - HuaweiSleepRespiratoryDetail.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - type = f.getInt("type") - value = f.getDouble("value") - }.build() - }) - add(sampleSetDefinition("sleep_respiratory_event", "sleep_respiratory_event", "connect_huawei_sleep_respiratory_event") { f, start, end, received -> - HuaweiSleepRespiratoryEvent.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() - eventname = f.getInt("event_name") - }.build() - }) - - add(sampleSetDefinition("vo2max", "vo2max", "connect_huawei_vo2max") { f, start, _, received -> - HuaweiVo2Max.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch() - vo2max = f.getInt("vo2max") - }.build() - }) + add( + sampleSetDefinition( + "continuous_calories_burnt", + "continuous.calories.burnt", + "connect_huawei_continuous_calories_burnt", + ) { f, start, end, received -> + HuaweiContinuousCaloriesBurnt.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + calories = f.getDouble("calories") + }.build() + }, + ) + add( + sampleSetDefinition( + "continuous_calories_consumed", + "continuous.calories.consumed", + "connect_huawei_continuous_calories_consumed", + ) { f, start, end, received -> + HuaweiContinuousCaloriesBurnt.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + calories = f.getDouble("calories") + }.build() + }, + ) + add( + sampleSetDefinition( + "continuous_calories_burnt_total", + "continuous.calories.burnt.total", + "connect_huawei_continuous_calories_burnt_total", + ) { f, start, end, received -> + HuaweiContinuousCaloriesBurntTotal.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + caloriesTotal = f.getDouble("calories_total") + }.build() + }, + ) + + add( + sampleSetDefinition( + "continuous_distance_delta", + "continuous.distance.delta", + "connect_huawei_continuous_distance_delta", + ) { f, start, end, received -> + HuaweiContinuousDistanceDelta.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + distanceDelta = f.getDouble("distance_delta") + }.build() + }, + ) + add( + sampleSetDefinition( + "continuous_distance_total", + "continuous.distance.total", + "connect_huawei_continuous_distance_total", + ) { f, start, end, received -> + HuaweiContinuousDistanceTotal.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + distance = f.getDouble("distance_total") + }.build() + }, + ) + + add( + sampleSetDefinition( + "continuous_ecg_detail", + "continuous.ecg_detail", + "connect_huawei_continuous_ecg_detail", + ) { f, start, end, received -> + HuaweiContinuousEcgDetail.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + ecgRecordId = f.getString("record_id") + averageHeartRate = f.getInt("avg_heart_rate") + ecgArrhythmiaType = f.getInt("arrhythmia_type") + ecgArrhythmiaResult = f.getInt("arrhythmia_result") + userSymptom = f.getString("user_symptom") + samplingFrequency = f.getInt("sampling_frequency") + voltageData = f.getString("voltage_data") + }.build() + }, + ) + + add( + sampleSetDefinition( + "continuous_exercise_intensity", + "continuous.exercise_intensity", + "connect_huawei_continuous_exercise_intensity", + ) { f, start, end, received -> + f.toContinuousExerciseIntensity(start, end, received) + }, + ) + add( + sampleSetDefinition( + "continuous_exercise_intensity_statistics", + "continuous.exercise_intensity.statistics", + "connect_huawei_continuous_exercise_intensity_statistics", + ) { f, start, end, received -> + f.toContinuousExerciseIntensity(start, end, received) + }, + ) + + add( + sampleSetDefinition( + "continuous_exercise_intensity_v2", + "continuous.exercise_intensity.v2", + "connect_huawei_continuous_exercise_intensity_v2", + ) { f, start, end, received -> + HuaweiContinuousExerciseIntensityV2.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + exerciseType = f.getInt("exercise_type") + }.build() + }, + ) + add( + sampleSetDefinition( + "continuous_exercise_intensity_v2_statistics", + "continuous.exercise_intensity.v2.statistics", + "connect_huawei_continuous_exercise_intensity_v2_statistics", + ) { f, start, end, received -> + HuaweiContinuousExerciseIntensityV2Statistics.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + zone1Duration = f.getInt("zone1_duration") + zone2Duration = f.getInt("zone2_duration") + zone3Duration = f.getInt("zone3_duration") + zone4Duration = f.getInt("zone4_duration") + zone5Duration = f.getInt("zone5_duration") + }.build() + }, + ) + + add( + sampleSetDefinition( + "continuous_sleep_fragment", + "continuous.sleep.fragment", + "connect_huawei_continuous_sleep_fragment", + ) { f, start, end, received -> + HuaweiContinuousSleepFragment.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + sleepState = f.getInt("sleep_state") + }.build() + }, + ) + + add( + sampleSetDefinition( + "continuous_spo2_statistics", + "continuous.spo2.statistics", + "connect_huawei_continuous_spo2_statistics", + ) { f, start, end, received -> + HuaweiContinuousSpo2Statistics.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + saturationAvg = f.getDouble("avg") + saturationMax = f.getDouble("max") + saturationMin = f.getDouble("min") + saturationLast = f.getDouble("last") + }.build() + }, + ) + + add( + sampleSetDefinition( + "continuous_steps_delta", + "continuous.steps.delta", + "connect_huawei_continuous_steps_delta", + ) { f, start, end, received -> + HuaweiContinuousStepsDelta.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + stepsDelta = f.getInt("steps_delta") + }.build() + }, + ) + add( + sampleSetDefinition( + "continuous_steps_total", + "continuous.steps.total", + "connect_huawei_continuous_steps_total", + ) { f, start, end, received -> + HuaweiContinuousStepsTotal.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + steps = f.getInt("steps") + duration = f.getInt("duration") + }.build() + }, + ) + + add( + sampleSetDefinition( + "emotion", + "emotion", + "connect_huawei_emotion", + ) { f, start, _, received -> + HuaweiEmotion.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + emotionStatus = f.getInt("emotion") + }.build() + }, + ) + + add( + healthRecordDefinition( + "health_record_dynamic_bp", + "health.record.dynamic_bp", + "connect_huawei_health_record_dynamic_bp", + ) { f, start, end, received -> + f.toHealthRecordDynamicBp(start, end, received) + }, + ) + add( + healthRecordDefinition( + "health_record_bradycardia", + "health.record.bradycardia", + "connect_huawei_health_record_bradycardia", + ) { f, start, end, received -> + f.toHealthRecordHeartRateAlert(start, end, received) + }, + ) + add( + healthRecordDefinition( + "health_record_tachycardia", + "health.record.tachycardia", + "connect_huawei_health_record_tachycardia", + ) { f, start, end, received -> + f.toHealthRecordHeartRateAlert(start, end, received) + }, + ) + add( + healthRecordDefinition( + "health_record_hyperthermia", + "health.record.hyperthermia", + "connect_huawei_health_record_hyperthermia", + ) { f, start, end, received -> + HuaweiHealthRecordHyperthermia.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + highBodyTemperatureAlarm = f.getFloat("high_body_temperature_alarm") + }.build() + }, + ) + add( + healthRecordDefinition( + "health_record_low_spo2_alert", + "health.record.lowSpo2Alert", + "connect_huawei_health_record_low_spo2_alert", + ) { f, start, end, received -> + HuaweiHealthRecordLowSpo2Alert.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + threshold = f.getFloat("threshold") + maxSpO2 = f.getFloat("max_spo2") + minSpO2 = f.getFloat("min_spo2") + }.build() + }, + ) + add( + healthRecordDefinition( + "health_record_menstrual_cycle", + "health.record.menstrual_cycle", + "connect_huawei_health_record_menstrual_cycle", + ) { f, start, end, received -> + HuaweiHealthRecordMenstrualCycle.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + recordday = f.getInt("record_day") + status = f.getInt("status") + substatus = f.getInt("sub_status") + remarks = f.getString("remarks") + timezone = f.getString("timezone") + }.build() + }, + ) + add( + healthRecordDefinition( + "health_record_sleep", + "health.record.sleep", + "connect_huawei_health_record_sleep", + ) { f, start, end, received -> + HuaweiHealthRecordSleep.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + fallAsleepTime = f.getLong("fall_asleep_time") + wakeupTime = f.getLong("wakeup_time") + lightSleepTime = f.getInt("light_sleep_time") + deepSleepTime = f.getInt("deep_sleep_time") + dreamTime = f.getInt("dream_time") + awakeTime = f.getInt("awake_time") + allSleepTime = f.getInt("all_sleep_time") + wakeupCount = f.getInt("wakeup_count") + deepSleepPart = f.getInt("deep_sleep_part") + sleepScore = f.getInt("sleep_score") + sleepLatency = f.getInt("sleep_latency") + sleepEfficiency = f.getInt("sleep_efficiency") + goBedTime = f.getLong("go_bed_time") + sleepType = f.getInt("sleep_type") + prepareSleepTime = f.getLong("prepare_sleep_time") + offBedTime = f.getLong("off_bed_time") + }.build() + }, + ) + + add( + sampleSetDefinition( + "heart_rate_variability", + "heart_rate_variability", + "connect_huawei_heart_rate_variability", + ) { f, start, _, received -> + HuaweiHeartRateVariability.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + heartRateVariabilityRmssd = f.getInt("heart_rate_variability_rmssd") + }.build() + }, + ) + + add( + sampleSetDefinition( + "resting_calories_statistics", + "resting_calories.statistics", + "connect_huawei_resting_calories_statistics", + ) { f, start, end, received -> + HuaweiRestingCaloriesStatistics.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + predictedCalories = f.getFloat("predicted_calories") + totalCalories = f.getFloat("total_calories") + }.build() + }, + ) + + add( + sampleSetDefinition( + "sleep_on_off_bed_record", + "sleep.on_off_bed_record", + "connect_huawei_sleep_on_off_bed_record", + ) { f, start, _, received -> + HuaweiSleepOnOffBedRecord.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + onOffBedState = f.getInt("on_off_bed_state") + }.build() + }, + ) + + add( + sampleSetDefinition( + "sleep_respiratory_detail", + "sleep_respiratory_detail", + "connect_huawei_sleep_respiratory_detail", + ) { f, start, end, received -> + HuaweiSleepRespiratoryDetail.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + type = f.getInt("type") + value = f.getDouble("value") + }.build() + }, + ) + add( + sampleSetDefinition( + "sleep_respiratory_event", + "sleep_respiratory_event", + "connect_huawei_sleep_respiratory_event", + ) { f, start, end, received -> + HuaweiSleepRespiratoryEvent.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + eventname = f.getInt("event_name") + }.build() + }, + ) + + add( + sampleSetDefinition( + "vo2max", + "vo2max", + "connect_huawei_vo2max", + ) { f, start, _, received -> + HuaweiVo2Max.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + vo2max = f.getInt("vo2max") + }.build() + }, + ) } private fun FieldValues.toActiveHours( @@ -419,7 +781,9 @@ object HuaweiRouteFactory { end: Instant?, received: Instant, ): HuaweiActiveHours = HuaweiActiveHours.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() activeHours = getInt("active_hours") moderateIntensityMinutes = getInt("moderate_intensity_minutes") highIntensityMinutes = getInt("high_intensity_minutes") @@ -430,7 +794,9 @@ object HuaweiRouteFactory { end: Instant?, received: Instant, ): HuaweiContinuousActivityStatistics = HuaweiContinuousActivityStatistics.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() typeOfActivity = getInt("activity_type") span = getInt("span") fragments = getInt("fragments") @@ -441,7 +807,9 @@ object HuaweiRouteFactory { end: Instant?, received: Instant, ): HuaweiContinuousExerciseIntensity = HuaweiContinuousExerciseIntensity.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() intensity = getDouble("intensity") span = getInt("span") }.build() @@ -451,7 +819,9 @@ object HuaweiRouteFactory { end: Instant?, received: Instant, ): HuaweiHealthRecordHeartRateAlert = HuaweiHealthRecordHeartRateAlert.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() threshold = getDouble("threshold") avgHeartRate = getDouble("avg_heart_rate") maxHeartRate = getDouble("max_heart_rate") @@ -474,7 +844,9 @@ object HuaweiRouteFactory { fun d(name: String) = f.getDouble(snake(name)) fun l(name: String) = f.getLong(snake(name)) return HuaweiHealthRecordDynamicBp.newBuilder().apply { - time = start.toEpoch(); timeReceived = received.toEpoch(); endTime = end?.toEpoch() + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() planId = f.getString(snake("planId")) planStartTime = l("planStartTime") planEndTime = l("planEndTime") @@ -484,42 +856,99 @@ object HuaweiRouteFactory { sleepStartTime = l("sleepStartTime") sleepEndTime = l("sleepEndTime") - validCntAll = i("validCntAll"); cntAll = i("cntAll") - maxSystolicBpAll = i("maxSystolicBpAll"); maxDiastolicBpAll = i("maxDiastolicBpAll"); maxHeartRateAll = i("maxHeartRateAll") - midSystolicBpAll = i("midSystolicBpAll"); midDiastolicBpAll = i("midDiastolicBpAll"); midHeartRateAll = i("midHeartRateAll") - minSystolicBpAll = i("minSystolicBpAll"); minDiastolicBpAll = i("minDiastolicBpAll"); minHeartRateAll = i("minHeartRateAll") - avgSystolicBpAll = i("avgSystolicBpAll"); avgDiastolicBpAll = i("avgDiastolicBpAll"); avgHeartRateAll = i("avgHeartRateAll") - stdSystolicBpAll = i("stdSystolicBpAll"); stdDiastolicBpAll = i("stdDiastolicBpAll"); stdHeartRateAll = i("stdHeartRateAll") - coefSystolicBpAll = d("coefSystolicBpAll"); coefDiastolicBpAll = d("coefDiastolicBpAll"); coefHeartRateAll = d("coefHeartRateAll") - loadSystolicBpAll = d("loadSystolicBpAll"); loadDiastolicBpAll = d("loadDiastolicBpAll") - dropSystolicBpAll = d("dropSystolicBpAll"); dropDiastolicBpAll = d("dropDiastolicBpAll") - peakSystolicBpAll = i("peakSystolicBpAll"); peakDiastolicBpAll = i("peakDiastolicBpAll") - - validCntWake = i("validCntWake"); cntWake = i("cntWake") - maxSystolicBpWake = i("maxSystolicBpWake"); maxDiastolicBpWake = i("maxDiastolicBpWake"); maxHeartRateWake = i("maxHeartRateWake") - midSystolicBpWake = i("midSystolicBpWake"); midDiastolicBpWake = i("midDiastolicBpWake"); midHeartRateWake = i("midHeartRateWake") - minSystolicBpWake = i("minSystolicBpWake"); minDiastolicBpWake = i("minDiastolicBpWake"); minHeartRateWake = i("minHeartRateWake") - avgSystolicBpWake = i("avgSystolicBpWake"); avgDiastolicBpWake = i("avgDiastolicBpWake"); avgHeartRateWake = i("avgHeartRateWake") - stdSystolicBpWake = i("stdSystolicBpWake"); stdDiastolicBpWake = i("stdDiastolicBpWake"); stdHeartRateWake = i("stdHeartRateWake") - coefSystolicBpWake = d("coefSystolicBpWake"); coefDiastolicBpWake = d("coefDiastolicBpWake"); coefHeartRateWake = d("coefHeartRateWake") - loadSystolicBpWake = d("loadSystolicBpWake"); loadDiastolicBpWake = d("loadDiastolicBpWake") - - validCntSleep = i("validCntSleep"); cntSleep = i("cntSleep") - maxSystolicBpSleep = i("maxSystolicBpSleep"); maxDiastolicBpSleep = i("maxDiastolicBpSleep"); maxHeartRateSleep = i("maxHeartRateSleep") - midSystolicBpSleep = i("midSystolicBpSleep"); midDiastolicBpSleep = i("midDiastolicBpSleep"); midHeartRateSleep = i("midHeartRateSleep") - minSystolicBpSleep = i("minSystolicBpSleep"); minDiastolicBpSleep = i("minDiastolicBpSleep"); minHeartRateSleep = i("minHeartRateSleep") - avgSystolicBpSleep = i("avgSystolicBpSleep"); avgDiastolicBpSleep = i("avgDiastolicBpSleep"); avgHeartRateSleep = i("avgHeartRateSleep") - stdSystolicBpSleep = i("stdSystolicBpSleep"); stdDiastolicBpSleep = i("stdDiastolicBpSleep"); stdHeartRateSleep = i("stdHeartRateSleep") - coefSystolicBpSleep = d("coefSystolicBpSleep"); coefDiastolicBpSleep = d("coefDiastolicBpSleep"); coefHeartRateSleep = d("coefHeartRateSleep") - loadSystolicBpSleep = d("loadSystolicBpSleep"); loadDiastolicBpSleep = d("loadDiastolicBpSleep") - - validCntWakeTwo = i("validCntWakeTwo"); cntWakeTwo = i("cntWakeTwo") - maxSystolicBpWakeTwo = i("maxSystolicBpWakeTwo"); maxDiastolicBpWakeTwo = i("maxDiastolicBpWakeTwo"); maxHeartRateWakeTwo = i("maxHeartRateWakeTwo") - midSystolicBpWakeTwo = i("midSystolicBpWakeTwo"); midDiastolicBpWakeTwo = i("midDiastolicBpWakeTwo"); midHeartRateWakeTwo = i("midHeartRateWakeTwo") - minSystolicBpWakeTwo = i("minSystolicBpWakeTwo"); minDiastolicBpWakeTwo = i("minDiastolicBpWakeTwo"); minHeartRateWakeTwo = i("minHeartRateWakeTwo") - avgSystolicBpWakeTwo = i("avgSystolicBpWakeTwo"); avgDiastolicBpWakeTwo = i("avgDiastolicBpWakeTwo"); avgHeartRateWakeTwo = i("avgHeartRateWakeTwo") - stdSystolicBpWakeTwo = i("stdSystolicBpWakeTwo"); stdDiastolicBpWakeTwo = i("stdDiastolicBpWakeTwo"); stdHeartRateWakeTwo = i("stdHeartRateWakeTwo") - coefSystolicBpWakeTwo = d("coefSystolicBpWakeTwo"); coefDiastolicBpWakeTwo = d("coefDiastolicBpWakeTwo"); coefHeartRateWakeTwo = d("coefHeartRateWakeTwo") + validCntAll = i("validCntAll") + cntAll = i("cntAll") + maxSystolicBpAll = i("maxSystolicBpAll") + maxDiastolicBpAll = i("maxDiastolicBpAll") + maxHeartRateAll = i("maxHeartRateAll") + midSystolicBpAll = i("midSystolicBpAll") + midDiastolicBpAll = i("midDiastolicBpAll") + midHeartRateAll = i("midHeartRateAll") + minSystolicBpAll = i("minSystolicBpAll") + minDiastolicBpAll = i("minDiastolicBpAll") + minHeartRateAll = i("minHeartRateAll") + avgSystolicBpAll = i("avgSystolicBpAll") + avgDiastolicBpAll = i("avgDiastolicBpAll") + avgHeartRateAll = i("avgHeartRateAll") + stdSystolicBpAll = i("stdSystolicBpAll") + stdDiastolicBpAll = i("stdDiastolicBpAll") + stdHeartRateAll = i("stdHeartRateAll") + coefSystolicBpAll = d("coefSystolicBpAll") + coefDiastolicBpAll = d("coefDiastolicBpAll") + coefHeartRateAll = d("coefHeartRateAll") + loadSystolicBpAll = d("loadSystolicBpAll") + loadDiastolicBpAll = d("loadDiastolicBpAll") + dropSystolicBpAll = d("dropSystolicBpAll") + dropDiastolicBpAll = d("dropDiastolicBpAll") + peakSystolicBpAll = i("peakSystolicBpAll") + peakDiastolicBpAll = i("peakDiastolicBpAll") + + validCntWake = i("validCntWake") + cntWake = i("cntWake") + maxSystolicBpWake = i("maxSystolicBpWake") + maxDiastolicBpWake = i("maxDiastolicBpWake") + maxHeartRateWake = i("maxHeartRateWake") + midSystolicBpWake = i("midSystolicBpWake") + midDiastolicBpWake = i("midDiastolicBpWake") + midHeartRateWake = i("midHeartRateWake") + minSystolicBpWake = i("minSystolicBpWake") + minDiastolicBpWake = i("minDiastolicBpWake") + minHeartRateWake = i("minHeartRateWake") + avgSystolicBpWake = i("avgSystolicBpWake") + avgDiastolicBpWake = i("avgDiastolicBpWake") + avgHeartRateWake = i("avgHeartRateWake") + stdSystolicBpWake = i("stdSystolicBpWake") + stdDiastolicBpWake = i("stdDiastolicBpWake") + stdHeartRateWake = i("stdHeartRateWake") + coefSystolicBpWake = d("coefSystolicBpWake") + coefDiastolicBpWake = d("coefDiastolicBpWake") + coefHeartRateWake = d("coefHeartRateWake") + loadSystolicBpWake = d("loadSystolicBpWake") + loadDiastolicBpWake = d("loadDiastolicBpWake") + + validCntSleep = i("validCntSleep") + cntSleep = i("cntSleep") + maxSystolicBpSleep = i("maxSystolicBpSleep") + maxDiastolicBpSleep = i("maxDiastolicBpSleep") + maxHeartRateSleep = i("maxHeartRateSleep") + midSystolicBpSleep = i("midSystolicBpSleep") + midDiastolicBpSleep = i("midDiastolicBpSleep") + midHeartRateSleep = i("midHeartRateSleep") + minSystolicBpSleep = i("minSystolicBpSleep") + minDiastolicBpSleep = i("minDiastolicBpSleep") + minHeartRateSleep = i("minHeartRateSleep") + avgSystolicBpSleep = i("avgSystolicBpSleep") + avgDiastolicBpSleep = i("avgDiastolicBpSleep") + avgHeartRateSleep = i("avgHeartRateSleep") + stdSystolicBpSleep = i("stdSystolicBpSleep") + stdDiastolicBpSleep = i("stdDiastolicBpSleep") + stdHeartRateSleep = i("stdHeartRateSleep") + coefSystolicBpSleep = d("coefSystolicBpSleep") + coefDiastolicBpSleep = d("coefDiastolicBpSleep") + coefHeartRateSleep = d("coefHeartRateSleep") + loadSystolicBpSleep = d("loadSystolicBpSleep") + loadDiastolicBpSleep = d("loadDiastolicBpSleep") + + validCntWakeTwo = i("validCntWakeTwo") + cntWakeTwo = i("cntWakeTwo") + maxSystolicBpWakeTwo = i("maxSystolicBpWakeTwo") + maxDiastolicBpWakeTwo = i("maxDiastolicBpWakeTwo") + maxHeartRateWakeTwo = i("maxHeartRateWakeTwo") + midSystolicBpWakeTwo = i("midSystolicBpWakeTwo") + midDiastolicBpWakeTwo = i("midDiastolicBpWakeTwo") + midHeartRateWakeTwo = i("midHeartRateWakeTwo") + minSystolicBpWakeTwo = i("minSystolicBpWakeTwo") + minDiastolicBpWakeTwo = i("minDiastolicBpWakeTwo") + minHeartRateWakeTwo = i("minHeartRateWakeTwo") + avgSystolicBpWakeTwo = i("avgSystolicBpWakeTwo") + avgDiastolicBpWakeTwo = i("avgDiastolicBpWakeTwo") + avgHeartRateWakeTwo = i("avgHeartRateWakeTwo") + stdSystolicBpWakeTwo = i("stdSystolicBpWakeTwo") + stdDiastolicBpWakeTwo = i("stdDiastolicBpWakeTwo") + stdHeartRateWakeTwo = i("stdHeartRateWakeTwo") + coefSystolicBpWakeTwo = d("coefSystolicBpWakeTwo") + coefDiastolicBpWakeTwo = d("coefDiastolicBpWakeTwo") + coefHeartRateWakeTwo = d("coefHeartRateWakeTwo") extendData = f.getString("extend_data") }.build() diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt index ddea128f..91b207fa 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt @@ -47,7 +47,11 @@ open class HuaweiSampleSetRoute( max: Int, ): Sequence = chunkedRanges(start, end, max).map { (rangeStart, rangeEnd) -> RestRequest( - request = createPostRequest(user, "sampleSet:polymerize", buildRequestBody(rangeStart, rangeEnd)), + request = createPostRequest( + user, + "sampleSet:polymerize", + buildRequestBody(rangeStart, rangeEnd), + ), user = user, route = this, startDate = rangeStart, diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index b6437590..a109ac9b 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -66,7 +66,7 @@ class HuaweiRouteFactoryTest { failures += "${definition.key}: unexpected topic ${successes.first().topic}" } } catch (e: Exception) { - failures += "${definition.key}: threw ${e}" + failures += "${definition.key}: threw $e" } } @@ -149,7 +149,9 @@ class HuaweiRouteFactoryTest { .map(::snake) private fun snake(name: String): String = - Regex("([a-z0-9])([A-Z])").replace(name) { "${it.groupValues[1]}_${it.groupValues[2]}" }.lowercase() + Regex("([a-z0-9])([A-Z])") + .replace(name) { "${it.groupValues[1]}_${it.groupValues[2]}" } + .lowercase() companion object { private const val START_MILLIS = 1704067200000L // 2024-01-01T00:00:00Z @@ -174,7 +176,8 @@ class HuaweiRouteFactoryTest { "steps_target", "sub_status", "systolic_pressure_avg", "systolic_pressure_max", "systolic_pressure_min", "threshold", "timezone", "total_calories", "type", "user_symptom", "value", "vo2max", "voltage_data", "wakeup_count", "wakeup_time", - "zone1_duration", "zone2_duration", "zone3_duration", "zone4_duration", "zone5_duration", + "zone1_duration", "zone2_duration", "zone3_duration", "zone4_duration", + "zone5_duration", ) } } From 9dac4847bbf37bafbc73f6ef78db871a59ccd1dd Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 11:24:00 +0000 Subject: [PATCH 08/44] Fix ktlint style violations in kafka-connect-huawei-source ktlint is static analysis and doesn't need dependency resolution to run, so this module's Kotlin sources could be linted even though they can't be compiled in this sandbox. Wraps long ConfigDef.define() calls and doc-string constants, and simplifies the getUserRepository()/ initialize() flow slightly in the process. --- .../huawei/HuaweiRestSourceConnectorConfig.kt | 158 ++++++++++++++---- .../user/HuaweiServiceUserRepository.kt | 10 +- 2 files changed, 130 insertions(+), 38 deletions(-) diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt index 8178a949..4adbd949 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt @@ -47,7 +47,11 @@ class HuaweiRestSourceConnectorConfig( doLog: Boolean, ) : AbstractConfig(config, parsedConfig, doLog) { - constructor(parsedConfig: MutableMap, doLog: Boolean) : this(conf(), parsedConfig, doLog) + constructor(parsedConfig: MutableMap, doLog: Boolean) : this( + conf(), + parsedConfig, + doLog, + ) private var userRepository: HuaweiUserRepository? = null @@ -58,7 +62,8 @@ class HuaweiRestSourceConnectorConfig( fun getHuaweiClientSecret(): String = getPassword(HUAWEI_API_SECRET_CONFIG).value() fun getUserRepository(reuse: HuaweiUserRepository?): HuaweiUserRepository { - val repo = if (reuse != null && reuse.javaClass == getClass(HUAWEI_USER_REPOSITORY_CONFIG)) { + val configuredClass = getClass(HUAWEI_USER_REPOSITORY_CONFIG) + val repo = if (reuse != null && reuse.javaClass == configuredClass) { reuse } else { createUserRepository() @@ -96,9 +101,13 @@ class HuaweiRestSourceConnectorConfig( ) } - fun getPollIntervalPerUser(): Duration = Duration.ofSeconds(getInt(HUAWEI_USER_POLL_INTERVAL_CONFIG).toLong()) + fun getPollIntervalPerUser(): Duration = Duration.ofSeconds( + getInt(HUAWEI_USER_POLL_INTERVAL_CONFIG).toLong(), + ) - fun getHuaweiUserRepositoryClientId(): String = getString(HUAWEI_USER_REPOSITORY_CLIENT_ID_CONFIG) + fun getHuaweiUserRepositoryClientId(): String = getString( + HUAWEI_USER_REPOSITORY_CLIENT_ID_CONFIG, + ) fun getHuaweiUserRepositoryClientSecret(): String = getPassword(HUAWEI_USER_REPOSITORY_CLIENT_SECRET_CONFIG).value() @@ -147,11 +156,13 @@ class HuaweiRestSourceConnectorConfig( private const val HUAWEI_API_CLIENT_DISPLAY = "Huawei API client ID" const val HUAWEI_API_SECRET_CONFIG = "huawei.api.secret" - private const val HUAWEI_API_SECRET_DOC = "Secret for the Huawei API client set in huawei.api.client." + private const val HUAWEI_API_SECRET_DOC = + "Secret for the Huawei API client set in huawei.api.client." private const val HUAWEI_API_SECRET_DISPLAY = "Huawei API client secret" const val HUAWEI_USER_REPOSITORY_CONFIG = "huawei.user.repository.class" - private const val HUAWEI_USER_REPOSITORY_DOC = "Class for managing users and authentication." + private const val HUAWEI_USER_REPOSITORY_DOC = + "Class for managing users and authentication." private const val HUAWEI_USER_REPOSITORY_DISPLAY = "User repository class" const val HUAWEI_USER_POLL_INTERVAL_CONFIG = "huawei.user.poll.interval" @@ -168,16 +179,22 @@ class HuaweiRestSourceConnectorConfig( private const val HUAWEI_USER_REPOSITORY_URL_DEFAULT = "" const val HUAWEI_USER_REPOSITORY_CLIENT_ID_CONFIG = "huawei.user.repository.client.id" - private const val HUAWEI_USER_REPOSITORY_CLIENT_ID_DOC = "Client ID for connecting to the service repository." - private const val HUAWEI_USER_REPOSITORY_CLIENT_ID_DISPLAY = "Client ID for user repository." + private const val HUAWEI_USER_REPOSITORY_CLIENT_ID_DOC = + "Client ID for connecting to the service repository." + private const val HUAWEI_USER_REPOSITORY_CLIENT_ID_DISPLAY = + "Client ID for user repository." - const val HUAWEI_USER_REPOSITORY_CLIENT_SECRET_CONFIG = "huawei.user.repository.client.secret" + const val HUAWEI_USER_REPOSITORY_CLIENT_SECRET_CONFIG = + "huawei.user.repository.client.secret" private const val HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DOC = "Client secret for connecting to the service repository." - private const val HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DISPLAY = "Client Secret for user repository." + private const val HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DISPLAY = + "Client Secret for user repository." - const val HUAWEI_USER_REPOSITORY_TOKEN_URL_CONFIG = "huawei.user.repository.oauth2.token.url" - private const val HUAWEI_USER_REPOSITORY_TOKEN_URL_DOC = "OAuth 2.0 token url for retrieving client credentials." + const val HUAWEI_USER_REPOSITORY_TOKEN_URL_CONFIG = + "huawei.user.repository.oauth2.token.url" + private const val HUAWEI_USER_REPOSITORY_TOKEN_URL_DOC = + "OAuth 2.0 token url for retrieving client credentials." private const val HUAWEI_USER_REPOSITORY_TOKEN_URL_DISPLAY = "OAuth 2.0 token URL." private fun enabledKey(key: String) = "huawei.$key.enabled" @@ -190,52 +207,125 @@ class HuaweiRestSourceConnectorConfig( val def = ConfigDef() .define( - SOURCE_POLL_INTERVAL_CONFIG, Type.LONG, SOURCE_POLL_INTERVAL_DEFAULT, Importance.LOW, - SOURCE_POLL_INTERVAL_DOC, group, ++order, Width.SHORT, SOURCE_POLL_INTERVAL_DISPLAY, + SOURCE_POLL_INTERVAL_CONFIG, + Type.LONG, + SOURCE_POLL_INTERVAL_DEFAULT, + Importance.LOW, + SOURCE_POLL_INTERVAL_DOC, + group, + ++order, + Width.SHORT, + SOURCE_POLL_INTERVAL_DISPLAY, ) .define( - SOURCE_URL_CONFIG, Type.STRING, SOURCE_URL_DEFAULT, Importance.HIGH, - SOURCE_URL_DOC, group, ++order, Width.SHORT, SOURCE_URL_DISPLAY, + SOURCE_URL_CONFIG, + Type.STRING, + SOURCE_URL_DEFAULT, + Importance.HIGH, + SOURCE_URL_DOC, + group, + ++order, + Width.SHORT, + SOURCE_URL_DISPLAY, ) .define( - HUAWEI_USERS_CONFIG, Type.LIST, emptyList(), Importance.HIGH, - HUAWEI_USERS_DOC, group, ++order, Width.SHORT, HUAWEI_USERS_DISPLAY, + HUAWEI_USERS_CONFIG, + Type.LIST, + emptyList(), + Importance.HIGH, + HUAWEI_USERS_DOC, + group, + ++order, + Width.SHORT, + HUAWEI_USERS_DISPLAY, ) .define( - HUAWEI_API_CLIENT_CONFIG, Type.STRING, ConfigDef.NO_DEFAULT_VALUE, NonEmptyString(), - Importance.HIGH, HUAWEI_API_CLIENT_DOC, group, ++order, Width.SHORT, HUAWEI_API_CLIENT_DISPLAY, + HUAWEI_API_CLIENT_CONFIG, + Type.STRING, + ConfigDef.NO_DEFAULT_VALUE, + NonEmptyString(), + Importance.HIGH, + HUAWEI_API_CLIENT_DOC, + group, + ++order, + Width.SHORT, + HUAWEI_API_CLIENT_DISPLAY, ) .define( - HUAWEI_API_SECRET_CONFIG, Type.PASSWORD, ConfigDef.NO_DEFAULT_VALUE, Importance.HIGH, - HUAWEI_API_SECRET_DOC, group, ++order, Width.SHORT, HUAWEI_API_SECRET_DISPLAY, + HUAWEI_API_SECRET_CONFIG, + Type.PASSWORD, + ConfigDef.NO_DEFAULT_VALUE, + Importance.HIGH, + HUAWEI_API_SECRET_DOC, + group, + ++order, + Width.SHORT, + HUAWEI_API_SECRET_DISPLAY, ) .define( - HUAWEI_USER_POLL_INTERVAL_CONFIG, Type.INT, HUAWEI_USER_POLL_INTERVAL_DEFAULT, Importance.MEDIUM, - HUAWEI_USER_POLL_INTERVAL_DOC, group, ++order, Width.SHORT, HUAWEI_USER_POLL_INTERVAL_DISPLAY, + HUAWEI_USER_POLL_INTERVAL_CONFIG, + Type.INT, + HUAWEI_USER_POLL_INTERVAL_DEFAULT, + Importance.MEDIUM, + HUAWEI_USER_POLL_INTERVAL_DOC, + group, + ++order, + Width.SHORT, + HUAWEI_USER_POLL_INTERVAL_DISPLAY, ) .define( - HUAWEI_USER_REPOSITORY_CONFIG, Type.CLASS, HuaweiServiceUserRepository::class.java, - Importance.MEDIUM, HUAWEI_USER_REPOSITORY_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_CONFIG, + Type.CLASS, + HuaweiServiceUserRepository::class.java, + Importance.MEDIUM, + HUAWEI_USER_REPOSITORY_DOC, + group, + ++order, + Width.SHORT, HUAWEI_USER_REPOSITORY_DISPLAY, ) .define( - HUAWEI_USER_REPOSITORY_URL_CONFIG, Type.STRING, HUAWEI_USER_REPOSITORY_URL_DEFAULT, - Importance.LOW, HUAWEI_USER_REPOSITORY_URL_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_URL_CONFIG, + Type.STRING, + HUAWEI_USER_REPOSITORY_URL_DEFAULT, + Importance.LOW, + HUAWEI_USER_REPOSITORY_URL_DOC, + group, + ++order, + Width.SHORT, HUAWEI_USER_REPOSITORY_URL_DISPLAY, ) .define( - HUAWEI_USER_REPOSITORY_CLIENT_ID_CONFIG, Type.STRING, "", Importance.MEDIUM, - HUAWEI_USER_REPOSITORY_CLIENT_ID_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_CLIENT_ID_CONFIG, + Type.STRING, + "", + Importance.MEDIUM, + HUAWEI_USER_REPOSITORY_CLIENT_ID_DOC, + group, + ++order, + Width.SHORT, HUAWEI_USER_REPOSITORY_CLIENT_ID_DISPLAY, ) .define( - HUAWEI_USER_REPOSITORY_CLIENT_SECRET_CONFIG, Type.PASSWORD, "", Importance.MEDIUM, - HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_CLIENT_SECRET_CONFIG, + Type.PASSWORD, + "", + Importance.MEDIUM, + HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DOC, + group, + ++order, + Width.SHORT, HUAWEI_USER_REPOSITORY_CLIENT_SECRET_DISPLAY, ) .define( - HUAWEI_USER_REPOSITORY_TOKEN_URL_CONFIG, Type.STRING, "", Importance.MEDIUM, - HUAWEI_USER_REPOSITORY_TOKEN_URL_DOC, group, ++order, Width.SHORT, + HUAWEI_USER_REPOSITORY_TOKEN_URL_CONFIG, + Type.STRING, + "", + Importance.MEDIUM, + HUAWEI_USER_REPOSITORY_TOKEN_URL_DOC, + group, + ++order, + Width.SHORT, HUAWEI_USER_REPOSITORY_TOKEN_URL_DISPLAY, ) diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt index 8bf64c2a..5bdfc642 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt @@ -50,14 +50,14 @@ import kotlinx.coroutines.runBlocking import kotlinx.coroutines.withContext import kotlinx.serialization.json.Json import org.radarbase.connect.rest.huawei.HuaweiRestSourceConnectorConfig +import org.radarbase.huawei.user.HuaweiUser +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserNotAuthorizedException import org.radarbase.kotlin.coroutines.CacheConfig import org.radarbase.kotlin.coroutines.CachedSet import org.radarbase.kotlin.coroutines.CachedValue import org.radarbase.ktor.auth.ClientCredentialsConfig import org.radarbase.ktor.auth.clientCredentials -import org.radarbase.huawei.user.HuaweiUser -import org.radarbase.huawei.user.User -import org.radarbase.huawei.user.UserNotAuthorizedException import org.slf4j.LoggerFactory import java.io.IOException import java.util.concurrent.ConcurrentHashMap @@ -90,11 +90,13 @@ class HuaweiServiceUserRepository : HuaweiUserRepository() { override fun initialize(config: HuaweiRestSourceConnectorConfig) { val containedUsers = config.getHuaweiUsers().toHashSet() + val tokenUrl = config.getHuaweiUserRepositoryTokenUrl() + ?.let { URLBuilder(it.toString()).build() } client = createClient( baseUrl = config.getHuaweiUserRepositoryUrl(), - tokenUrl = config.getHuaweiUserRepositoryTokenUrl()?.let { URLBuilder(it.toString()).build() }, + tokenUrl = tokenUrl, clientId = config.getHuaweiUserRepositoryClientId(), clientSecret = config.getHuaweiUserRepositoryClientSecret(), scope = "SUBJECT.READ MEASUREMENT.CREATE", From a6ec537d92ee309a30d15e6c7950b25aa7bac4bc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 13:14:21 +0000 Subject: [PATCH 09/44] Add Onsentia affiliation to Huawei connector author metadata Adds yatharthranjan@onsentia.com alongside the existing KCL address in the Huawei Docker image label and CI workflow author fields. --- .github/workflows/main.yml | 2 +- .github/workflows/release.yml | 4 ++-- kafka-connect-huawei-source/Dockerfile | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index e5397368..dde9bc90 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -25,7 +25,7 @@ env: },{ 'name': 'kafka-connect-huawei-source', 'build_file': 'kafka-connect-huawei-source/Dockerfile', - 'authors': 'Yatharth Ranjan ', + 'authors': 'Yatharth Ranjan , Yatharth Ranjan ', 'description': 'RADAR-base Huawei Health Kit connector application' }] diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 861df14e..6f43aa67 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -16,12 +16,12 @@ env: },{ 'name': 'kafka-connect-oura-source', 'build_file': 'kafka-connect-oura-source/Dockerfile', - 'authors': 'Pauline Conde , Yatharth Ranjan ', + 'authors': 'Pauline Conde , Yatharth Ranjan , Yatharth Ranjan ', 'description': 'RADAR-base Oura connector application' },{ 'name': 'kafka-connect-huawei-source', 'build_file': 'kafka-connect-huawei-source/Dockerfile', - 'authors': 'Yatharth Ranjan ', + 'authors': 'Yatharth Ranjan , Yatharth Ranjan ', 'description': 'RADAR-base Huawei Health Kit connector application' }] diff --git a/kafka-connect-huawei-source/Dockerfile b/kafka-connect-huawei-source/Dockerfile index e01bcb7f..330759cf 100644 --- a/kafka-connect-huawei-source/Dockerfile +++ b/kafka-connect-huawei-source/Dockerfile @@ -36,7 +36,7 @@ FROM confluentinc/cp-kafka-connect-base:7.8.7 USER appuser -LABEL org.opencontainers.image.authors="yatharth.ranjan@kcl.ac.uk" +LABEL org.opencontainers.image.authors="yatharth.ranjan@kcl.ac.uk, yatharthranjan@onsentia.com" LABEL description="Kafka Huawei Health Kit REST API Source connector" From 2af0bb943980cb0a0471a2d3dbbf2636517fc52d Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 13:45:40 +0000 Subject: [PATCH 10/44] Fix CI type errors in HuaweiServiceUserRepository - Wrap the okhttp3.HttpUrl from getHuaweiUserRepositoryUrl() in URLBuilder(...).build() before passing it as createClient's Ktor Url parameter, matching how the token URL was already handled (and how OuraServiceUserRepository does it). - Throwable.message is nullable; fall back to a default string before passing it to UserNotAuthorizedException's non-null constructor. These only surfaced in real CI since packages.confluent.io is blocked in this sandbox, so kafka-connect-huawei-source couldn't be compiled here - verified via ktlint (which needs no dependency resolution) that the fix introduces no new style issues. --- .../connect/rest/huawei/user/HuaweiServiceUserRepository.kt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt index 5bdfc642..ae86aea6 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt @@ -95,7 +95,7 @@ class HuaweiServiceUserRepository : HuaweiUserRepository() { client = createClient( - baseUrl = config.getHuaweiUserRepositoryUrl(), + baseUrl = URLBuilder(config.getHuaweiUserRepositoryUrl().toString()).build(), tokenUrl = tokenUrl, clientId = config.getHuaweiUserRepositoryClientId(), clientSecret = config.getHuaweiUserRepositoryClientSecret(), @@ -237,7 +237,7 @@ class HuaweiServiceUserRepository : HuaweiUserRepository() { } catch (ex: HttpResponseException) { if (ex.statusCode == 407) { credentialCaches -= user.id - throw UserNotAuthorizedException(ex.message) + throw UserNotAuthorizedException(ex.message ?: "User is not authorized") } throw ex } From 59cdbc56a51dac4b0781015b9184c01dad35ea52 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 14 Jul 2026 14:05:36 +0000 Subject: [PATCH 11/44] Fix ktlint line-length violation in build.gradle.kts ktlintKotlinScriptCheck also lints .gradle.kts files; wraps the two GitHub Packages credential lines that exceeded 100 chars. --- kafka-connect-huawei-source/build.gradle.kts | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kafka-connect-huawei-source/build.gradle.kts b/kafka-connect-huawei-source/build.gradle.kts index 3bc0b307..d2e0a528 100644 --- a/kafka-connect-huawei-source/build.gradle.kts +++ b/kafka-connect-huawei-source/build.gradle.kts @@ -18,8 +18,10 @@ repositories { maven { url = uri("https://maven.pkg.github.com/RADAR-base/RADAR-Schemas") credentials { - username = project.findProperty("public.gpr.user") as String? ?: System.getenv("GPR_USER") - password = project.findProperty("public.gpr.token") as String? ?: System.getenv("GPR_TOKEN") + username = project.findProperty("public.gpr.user") as String? + ?: System.getenv("GPR_USER") + password = project.findProperty("public.gpr.token") as String? + ?: System.getenv("GPR_TOKEN") } } } From d7dbf15937aeacbe9433c9d713a68a2ac908d2d5 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 23 Jul 2026 15:19:49 +0000 Subject: [PATCH 12/44] Add Onsentia copyright header to Huawei connector source files Applies a standard Apache-2.0 copyright header (Copyright 2026 Onsentia) to every .kt/.java file in huawei-library and kafka-connect-huawei-source - the two modules added for the Huawei Health Kit integration. Files that had copied The Hyve's 2018 header from the Oura pattern get it replaced; files with no header get one prepended. Pre-existing Fitbit/Oura files are left untouched since their copyright belongs to their original authors. --- .../radarbase/huawei/converter/FieldValues.kt | 17 +++++++++++++++++ .../converter/HuaweiActivityRecordConverter.kt | 17 +++++++++++++++++ .../huawei/converter/HuaweiDataConverter.kt | 17 +++++++++++++++++ .../converter/HuaweiHealthRecordConverter.kt | 17 +++++++++++++++++ .../converter/HuaweiSampleSetConverter.kt | 17 +++++++++++++++++ .../huawei/converter/RecordConverter.kt | 17 +++++++++++++++++ .../huawei/converter/SequenceExtensions.kt | 17 +++++++++++++++++ .../org/radarbase/huawei/converter/TopicData.kt | 17 +++++++++++++++++ .../org/radarbase/huawei/offset/Offset.kt | 17 +++++++++++++++++ .../org/radarbase/huawei/offset/Offsets.kt | 17 +++++++++++++++++ .../huawei/request/HuaweiOffsetManager.kt | 17 +++++++++++++++++ .../huawei/request/HuaweiRequestGenerator.kt | 17 +++++++++++++++++ .../radarbase/huawei/request/HuaweiResult.kt | 17 +++++++++++++++++ .../huawei/request/RequestGenerator.kt | 17 +++++++++++++++++ .../org/radarbase/huawei/request/RestRequest.kt | 17 +++++++++++++++++ .../huawei/request/TooManyRequestsException.kt | 17 +++++++++++++++++ .../huawei/route/HuaweiActivityRecordRoute.kt | 17 +++++++++++++++++ .../huawei/route/HuaweiHealthRecordRoute.kt | 17 +++++++++++++++++ .../org/radarbase/huawei/route/HuaweiRoute.kt | 17 +++++++++++++++++ .../huawei/route/HuaweiRouteDefinition.kt | 17 +++++++++++++++++ .../huawei/route/HuaweiRouteFactory.kt | 17 +++++++++++++++++ .../huawei/route/HuaweiSampleSetRoute.kt | 17 +++++++++++++++++ .../kotlin/org/radarbase/huawei/route/Route.kt | 17 +++++++++++++++++ .../org/radarbase/huawei/user/HuaweiUser.kt | 17 +++++++++++++++++ .../kotlin/org/radarbase/huawei/user/User.kt | 17 +++++++++++++++++ .../huawei/user/UserNotAuthorizedException.kt | 17 +++++++++++++++++ .../org/radarbase/huawei/user/UserRepository.kt | 17 +++++++++++++++++ .../huawei/converter/FieldValuesTest.kt | 17 +++++++++++++++++ .../huawei/route/HuaweiRouteFactoryTest.kt | 17 +++++++++++++++++ .../huawei/AbstractRestSourceConnector.java | 2 +- .../huawei/HuaweiRestSourceConnectorConfig.kt | 2 +- .../rest/huawei/HuaweiSourceConnector.java | 2 +- .../connect/rest/huawei/HuaweiSourceTask.java | 2 +- .../rest/huawei/offset/KafkaOffsetManager.java | 17 +++++++++++++++++ .../rest/huawei/user/HttpResponseException.java | 2 +- .../huawei/user/HuaweiServiceUserRepository.kt | 2 +- .../rest/huawei/user/HuaweiUserRepository.kt | 2 +- .../connect/rest/huawei/user/HuaweiUsers.java | 2 +- .../rest/huawei/user/OAuth2UserCredentials.java | 2 +- .../connect/rest/huawei/util/VersionUtil.java | 2 +- .../HuaweiRestSourceConnectorConfigTest.kt | 2 +- 41 files changed, 521 insertions(+), 11 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt index f0db5d6a..2fb3d0ea 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.converter import com.fasterxml.jackson.databind.JsonNode diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt index ebe7d40f..43e4bd9e 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.converter import com.fasterxml.jackson.databind.JsonNode diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt index 6b095587..473bd938 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.converter import com.fasterxml.jackson.databind.JsonNode diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt index 013307bc..505836f1 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.converter import com.fasterxml.jackson.databind.JsonNode diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt index e5c6d566..e44ab0c5 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.converter import com.fasterxml.jackson.databind.JsonNode diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt index f9e9e16b..71e40465 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.converter import okhttp3.Headers diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt index fe1dc73f..eb25544a 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.converter import org.slf4j.LoggerFactory diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt index a537af98..af32d046 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.converter import org.apache.avro.specific.SpecificRecord diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt index 9da597c0..6ef10272 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.offset import org.radarbase.huawei.route.Route diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt index 88c67afb..ff34db97 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.offset data class Offsets( diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt index 03b23c34..8c40ea5f 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.request import org.radarbase.huawei.offset.Offset diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt index 2a733024..07d7d4b9 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.request import com.fasterxml.jackson.core.JsonFactory diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt index fba9e551..8e4e5648 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.request sealed class HuaweiResult { diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt index 39bb60e9..08ad8f15 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.request import okhttp3.Response diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt index 502ecc26..e46f3e01 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.request import okhttp3.Request diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt index 3dc5aa9c..f8b583bc 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.request class TooManyRequestsException : RuntimeException() diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt index 07a4f586..58dd8ca2 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.route import org.radarbase.huawei.converter.HuaweiActivityRecordConverter diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt index ae2fa2b5..e1823aa7 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.route import org.apache.avro.specific.SpecificRecord diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt index 33a47dea..9653cd4e 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.route import okhttp3.HttpUrl.Companion.toHttpUrl diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt index 5a311e47..d117a36e 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.route import org.radarbase.huawei.user.UserRepository diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 5d7e2032..e4a17802 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.route import org.apache.avro.specific.SpecificRecord diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt index 91b207fa..922766ff 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.route import com.fasterxml.jackson.databind.ObjectMapper diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt index 39e2cf71..0a3b0c8f 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.route import org.radarbase.huawei.request.RestRequest diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt index 7d1521e3..06ff3933 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.user import com.fasterxml.jackson.annotation.JsonIgnoreProperties diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt index b84dfe76..9e15d7b9 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.user import org.radarcns.kafka.ObservationKey diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt index 1bd513b0..1a3ed168 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.user class UserNotAuthorizedException(message: String) : Exception(message) { diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt index 6f26b74b..da59958c 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.user import java.io.IOException diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt index cdca893f..717676b3 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.converter import com.fasterxml.jackson.databind.ObjectMapper diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index a109ac9b..4942376b 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.huawei.route import com.fasterxml.jackson.databind.ObjectMapper diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java index ad09e22a..57dc6c05 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java @@ -1,7 +1,7 @@ package org.radarbase.connect.rest.huawei; /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt index 4adbd949..150512bf 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java index 63c9debf..cf1ec845 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java index e3e6520e..bc672d4d 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java index dcc044ef..c8b32539 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java @@ -1,3 +1,20 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + package org.radarbase.connect.rest.huawei.offset; import java.time.Duration; diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java index c4f8c30c..4b94e045 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt index ae86aea6..2e9ac67c 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt index 010a5b5a..aa24c836 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java index e01aeff8..b23f5ef6 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java index 88bad3b8..3ea2c8a1 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java index 8c23ac79..e3bf8e09 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt index dd7ae19a..04d4e1b3 100644 --- a/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt +++ b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt @@ -1,5 +1,5 @@ /* - * Copyright 2018 The Hyve + * Copyright 2026 Onsentia * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. From ac15a680e028a8c8a33756ad7a1d7679cf52f9d3 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 23 Jul 2026 15:24:27 +0000 Subject: [PATCH 13/44] Fix copyright header in Huawei connector Dockerfile Was still copied from Oura's Dockerfile (Copyright 2018 The Hyve); updates to match the Onsentia header applied to the rest of this module's files. --- kafka-connect-huawei-source/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kafka-connect-huawei-source/Dockerfile b/kafka-connect-huawei-source/Dockerfile index 330759cf..5da90434 100644 --- a/kafka-connect-huawei-source/Dockerfile +++ b/kafka-connect-huawei-source/Dockerfile @@ -1,4 +1,4 @@ -# Copyright 2018 The Hyve +# Copyright 2026 Onsentia # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. From 401291b1d84caaf38348673d5ff887f7004ed270 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 23 Jul 2026 15:27:54 +0000 Subject: [PATCH 14/44] Add @author yatharthranjan to Huawei connector source docs Adds an @author yatharthranjan KDoc/Javadoc tag to the primary class/interface/object of every .kt/.java file in huawei-library and kafka-connect-huawei-source: appended to existing class-level doc comments where present, added as a new minimal doc comment otherwise. --- .../main/kotlin/org/radarbase/huawei/converter/FieldValues.kt | 2 ++ .../huawei/converter/HuaweiActivityRecordConverter.kt | 2 ++ .../org/radarbase/huawei/converter/HuaweiDataConverter.kt | 2 ++ .../radarbase/huawei/converter/HuaweiHealthRecordConverter.kt | 2 ++ .../org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt | 2 ++ .../kotlin/org/radarbase/huawei/converter/RecordConverter.kt | 3 +++ .../org/radarbase/huawei/converter/SequenceExtensions.kt | 3 +++ .../main/kotlin/org/radarbase/huawei/converter/TopicData.kt | 3 +++ .../src/main/kotlin/org/radarbase/huawei/offset/Offset.kt | 3 +++ .../src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt | 3 +++ .../kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt | 3 +++ .../org/radarbase/huawei/request/HuaweiRequestGenerator.kt | 3 +++ .../main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt | 3 +++ .../kotlin/org/radarbase/huawei/request/RequestGenerator.kt | 3 +++ .../main/kotlin/org/radarbase/huawei/request/RestRequest.kt | 3 +++ .../org/radarbase/huawei/request/TooManyRequestsException.kt | 3 +++ .../org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt | 2 ++ .../org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt | 2 ++ .../src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt | 2 ++ .../kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt | 2 ++ .../kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt | 2 ++ .../kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt | 2 ++ .../src/main/kotlin/org/radarbase/huawei/route/Route.kt | 3 +++ .../src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt | 3 +++ .../src/main/kotlin/org/radarbase/huawei/user/User.kt | 3 +++ .../org/radarbase/huawei/user/UserNotAuthorizedException.kt | 3 +++ .../main/kotlin/org/radarbase/huawei/user/UserRepository.kt | 3 +++ .../kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt | 3 +++ .../org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt | 2 ++ .../connect/rest/huawei/AbstractRestSourceConnector.java | 3 +++ .../connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt | 2 ++ .../radarbase/connect/rest/huawei/HuaweiSourceConnector.java | 3 +++ .../org/radarbase/connect/rest/huawei/HuaweiSourceTask.java | 3 +++ .../connect/rest/huawei/offset/KafkaOffsetManager.java | 3 +++ .../connect/rest/huawei/user/HttpResponseException.java | 3 +++ .../connect/rest/huawei/user/HuaweiServiceUserRepository.kt | 2 ++ .../radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt | 3 +++ .../org/radarbase/connect/rest/huawei/user/HuaweiUsers.java | 3 +++ .../connect/rest/huawei/user/OAuth2UserCredentials.java | 3 +++ .../org/radarbase/connect/rest/huawei/util/VersionUtil.java | 3 +++ .../connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt | 3 +++ 41 files changed, 109 insertions(+) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt index 2fb3d0ea..2bb49b49 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt @@ -32,6 +32,8 @@ import com.fasterxml.jackson.databind.JsonNode * * Field name constants follow Huawei's public `Field` identifiers (e.g. `steps_delta`, `calories`, * `avg`, `max`, `min`), as documented for the on-device and REST Health Kit APIs. + * + * @author yatharthranjan */ class FieldValues private constructor(private val values: Map) { diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt index 43e4bd9e..26248540 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt @@ -30,6 +30,8 @@ import java.time.Instant * type, and a nested activity summary with pace/data/section statistics). Nested JSON structures * that map to free-form Avro `string` fields (pace map, data summary, section summary) are kept as * their raw JSON text, since their internal shape varies by activity type. + * + * @author yatharthranjan */ class HuaweiActivityRecordConverter( private val topic: String = "connect_huawei_activity_record", diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt index 473bd938..51d79be4 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt @@ -26,6 +26,8 @@ import java.time.Instant /** * Converts a Huawei Health Kit HTTP JSON response body to zero or more [TopicData] records. + * + * @author yatharthranjan */ interface HuaweiDataConverter : RecordConverter { /** Process the JSON records generated by given request. */ diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt index 505836f1..b68c660a 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt @@ -33,6 +33,8 @@ private fun JsonNode.epochInstant(field: String): Instant? { * Generic converter for `GET /healthkit/v1/healthRecords` responses: iterates every record * returned for the requested `subDataTypeName` and builds one Avro record per entry via * [buildRecord]. + * + * @author yatharthranjan */ class HuaweiHealthRecordConverter( private val topic: String, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt index e44ab0c5..192ad7d5 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt @@ -36,6 +36,8 @@ private fun JsonNode.epochInstant(field: String): Instant? { * This single converter is reused for the large majority of Huawei Health Kit data types, since * they all share the same `sampleSet[].samplePoints[]` response envelope and differ only in which * Avro record type their field values are mapped onto. + * + * @author yatharthranjan */ class HuaweiSampleSetConverter( private val topic: String, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt index 71e40465..ca82e213 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/RecordConverter.kt @@ -22,6 +22,9 @@ import org.radarbase.huawei.request.RestRequest import org.slf4j.LoggerFactory import java.io.IOException +/** + * @author yatharthranjan + */ interface RecordConverter { @Throws(IOException::class) fun convert( diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt index eb25544a..62746512 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/SequenceExtensions.kt @@ -19,6 +19,9 @@ package org.radarbase.huawei.converter import org.slf4j.LoggerFactory +/** + * @author yatharthranjan + */ val logger = LoggerFactory.getLogger("org.radarbase.huawei.converter.SequenceExtensions") internal fun Sequence.mapCatching(fn: (T) -> S): Sequence> = map { t -> diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt index af32d046..aadd6a09 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/TopicData.kt @@ -20,6 +20,9 @@ package org.radarbase.huawei.converter import org.apache.avro.specific.SpecificRecord /** Single value for a topic. */ +/** + * @author yatharthranjan + */ data class TopicData( val topic: String, val key: SpecificRecord, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt index 6ef10272..791b0065 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offset.kt @@ -21,6 +21,9 @@ import org.radarbase.huawei.route.Route import org.radarbase.huawei.user.User import java.time.Instant +/** + * @author yatharthranjan + */ data class Offset( val user: User, val route: Route, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt index ff34db97..1c31c8b7 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/offset/Offsets.kt @@ -17,6 +17,9 @@ package org.radarbase.huawei.offset +/** + * @author yatharthranjan + */ data class Offsets( val offsets: List, ) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt index 8c40ea5f..0709396f 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiOffsetManager.kt @@ -22,6 +22,9 @@ import org.radarbase.huawei.route.Route import org.radarbase.huawei.user.User import java.time.Instant +/** + * @author yatharthranjan + */ interface HuaweiOffsetManager { fun getOffset(route: Route, user: User): Offset? diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt index 07d7d4b9..6891ffb0 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt @@ -30,6 +30,9 @@ import java.io.IOException import java.time.Duration import java.time.Instant +/** + * @author yatharthranjan + */ class HuaweiRequestGenerator( private val userRepository: UserRepository, private val huaweiOffsetManager: HuaweiOffsetManager, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt index 8e4e5648..0f9dda6d 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiResult.kt @@ -17,6 +17,9 @@ package org.radarbase.huawei.request +/** + * @author yatharthranjan + */ sealed class HuaweiResult { data class Success(val value: T) : HuaweiResult() data class Error(val error: HuaweiError) : HuaweiResult() diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt index 08ad8f15..bf48a457 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RequestGenerator.kt @@ -22,6 +22,9 @@ import org.radarbase.huawei.converter.TopicData import org.radarbase.huawei.route.Route import org.radarbase.huawei.user.User +/** + * @author yatharthranjan + */ interface RequestGenerator { fun requests(user: User, max: Int): Sequence diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt index e46f3e01..ef199dbf 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/RestRequest.kt @@ -22,6 +22,9 @@ import org.radarbase.huawei.route.HuaweiRoute import org.radarbase.huawei.user.User import java.time.Instant +/** + * @author yatharthranjan + */ data class RestRequest( val request: Request, val user: User, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt index f8b583bc..db54d069 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/TooManyRequestsException.kt @@ -17,4 +17,7 @@ package org.radarbase.huawei.request +/** + * @author yatharthranjan + */ class TooManyRequestsException : RuntimeException() diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt index 58dd8ca2..ebd75def 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt @@ -28,6 +28,8 @@ import java.time.Instant /** * Route backed by `GET /healthkit/v1/activityRecords`, covering the Huawei Health Kit Activity * Records API (workout / physical-activity sessions). + * + * @author yatharthranjan */ class HuaweiActivityRecordRoute( userRepository: UserRepository, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt index e1823aa7..4d7a8c44 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt @@ -31,6 +31,8 @@ import java.time.Instant * Route backed by `GET /healthkit/v1/healthRecords`, used for the `health.record.*` data types * (ambulatory blood pressure sessions, heart rate alerts, hyperthermia, low SpO2 alerts, * menstrual cycle phases, and comprehensive sleep records). + * + * @author yatharthranjan */ open class HuaweiHealthRecordRoute( userRepository: UserRepository, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt index 9653cd4e..02bf19b8 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt @@ -34,6 +34,8 @@ import java.time.Instant * Handles OAuth2-authorized request construction (both `GET` with query parameters and `POST` * with a JSON body, since the Health Kit Data API mixes both styles across its endpoints) and * generic time-range chunking, shared by all concrete route types. + * + * @author yatharthranjan */ abstract class HuaweiRoute( private val userRepository: UserRepository, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt index d117a36e..80a3679d 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteDefinition.kt @@ -27,6 +27,8 @@ import org.radarbase.huawei.user.UserRepository * Using one shared registry (see [HuaweiRouteFactory]) for both the Kafka Connect config * definition and the set of routes actually polled avoids hand-duplicating each of the ~54 Huawei * data types across a `ConfigDef` and a route-construction switch. + * + * @author yatharthranjan */ data class HuaweiRouteDefinition( val key: String, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index e4a17802..ad34ad9d 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -71,6 +71,8 @@ import java.time.Instant * documented constants, the snake_case form of the Avro field's own name is used as a best-effort * default (see [snake]) — verify against a live API response and adjust the key strings in this * file if Huawei's actual response uses different names. + * + * @author yatharthranjan */ object HuaweiRouteFactory { diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt index 922766ff..664e80a7 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt @@ -37,6 +37,8 @@ import java.time.Instant * When [groupByTimeUnit] is set, the request aggregates sample points into buckets of that size — * this is how Huawei's `.statistics` data types are queried. When it is `null`, the endpoint * returns raw, un-aggregated sample points for [dataTypeName] over the requested time range. + * + * @author yatharthranjan */ open class HuaweiSampleSetRoute( userRepository: UserRepository, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt index 0a3b0c8f..ba2ae506 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/Route.kt @@ -22,6 +22,9 @@ import org.radarbase.huawei.user.User import java.time.Duration import java.time.Instant +/** + * @author yatharthranjan + */ interface Route { fun generateRequests(user: User, start: Instant, end: Instant): Sequence diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt index 06ff3933..692120c8 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/HuaweiUser.kt @@ -22,6 +22,9 @@ import com.fasterxml.jackson.annotation.JsonProperty import org.radarcns.kafka.ObservationKey import java.time.Instant +/** + * @author yatharthranjan + */ @JsonIgnoreProperties(ignoreUnknown = true) data class HuaweiUser( @JsonProperty("id") override val id: String, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt index 9e15d7b9..e2092cda 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/User.kt @@ -20,6 +20,9 @@ package org.radarbase.huawei.user import org.radarcns.kafka.ObservationKey import java.time.Instant +/** + * @author yatharthranjan + */ interface User { val id: String val projectId: String diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt index 1a3ed168..ace22016 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserNotAuthorizedException.kt @@ -17,6 +17,9 @@ package org.radarbase.huawei.user +/** + * @author yatharthranjan + */ class UserNotAuthorizedException(message: String) : Exception(message) { constructor(user: User) : this("User ${user.id} is not authorized") } diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt index da59958c..34d50e78 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt @@ -20,6 +20,9 @@ package org.radarbase.huawei.user import java.io.IOException /** User repository for Huawei Health Kit users. */ +/** + * @author yatharthranjan + */ interface UserRepository { /** * Get specified user. diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt index 717676b3..428f5065 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt @@ -22,6 +22,9 @@ import kotlin.test.Test import kotlin.test.assertEquals import kotlin.test.assertNull +/** + * @author yatharthranjan + */ class FieldValuesTest { private val mapper = ObjectMapper() diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 4942376b..55f3c473 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -36,6 +36,8 @@ import kotlin.test.assertTrue * `healthRecords`, or `activityRecords`), and asserts the converter produces exactly one record on * the definition's own topic without throwing. This is the main regression test against typos in * the ~90 hand-written Huawei field-value key strings (and the Avro builder calls around them). + * + * @author yatharthranjan */ class HuaweiRouteFactoryTest { private val mapper = ObjectMapper() diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java index 57dc6c05..479ac97a 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/AbstractRestSourceConnector.java @@ -25,6 +25,9 @@ import org.apache.kafka.connect.source.SourceConnector; import org.radarbase.connect.rest.huawei.util.VersionUtil; +/** + * @author yatharthranjan + */ @SuppressWarnings("unused") public abstract class AbstractRestSourceConnector extends SourceConnector { protected HuaweiRestSourceConnectorConfig config; diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt index 150512bf..b8e1a2e8 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt @@ -40,6 +40,8 @@ import java.time.Duration * boolean and a `huawei..topic` string config, generated from that single shared registry * instead of ~110 hand-duplicated `ConfigDef` entries (one connector, one config, one canonical * list of Huawei data types). + * + * @author yatharthranjan */ class HuaweiRestSourceConnectorConfig( config: ConfigDef, diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java index cf1ec845..75e8397c 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceConnector.java @@ -39,6 +39,9 @@ import static org.radarbase.connect.rest.huawei.HuaweiRestSourceConnectorConfig.HUAWEI_USERS_CONFIG; +/** + * @author yatharthranjan + */ public class HuaweiSourceConnector extends AbstractRestSourceConnector { private static final Logger logger = LoggerFactory.getLogger(HuaweiSourceConnector.class); diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java index bc672d4d..6fd00a65 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java @@ -53,6 +53,9 @@ import okhttp3.OkHttpClient; import okhttp3.Response; +/** + * @author yatharthranjan + */ public class HuaweiSourceTask extends SourceTask { private static final Logger logger = LoggerFactory.getLogger(HuaweiSourceTask.class); diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java index c8b32539..8f2aec3c 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java @@ -31,6 +31,9 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +/** + * @author yatharthranjan + */ public class KafkaOffsetManager implements HuaweiOffsetManager { private static final Logger logger = LoggerFactory.getLogger(KafkaOffsetManager.class); private static final String TIMESTAMP_OFFSET_KEY = "timestamp"; diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java index 4b94e045..4a8573a9 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HttpResponseException.java @@ -19,6 +19,9 @@ import java.io.IOException; +/** + * @author yatharthranjan + */ public class HttpResponseException extends IOException { private final int statusCode; diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt index 2e9ac67c..d0c5d088 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt @@ -72,6 +72,8 @@ import kotlin.time.Duration.Companion.seconds * [org.radarbase.connect.rest.oura.user.OuraServiceUserRepository]. Retrieves the list of Huawei * users configured for a study (`GET users?source-type=Huawei`) and their Huawei Health Kit OAuth2 * access/refresh tokens (`users//token`). + * + * @author yatharthranjan */ @Suppress("unused") class HuaweiServiceUserRepository : HuaweiUserRepository() { diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt index aa24c836..44efe62e 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUserRepository.kt @@ -22,6 +22,9 @@ import org.radarbase.huawei.user.UserNotAuthorizedException import org.radarbase.huawei.user.UserRepository import java.io.IOException +/** + * @author yatharthranjan + */ @Suppress("unused") abstract class HuaweiUserRepository : UserRepository { abstract fun initialize(config: HuaweiRestSourceConnectorConfig) diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java index b23f5ef6..db225b9a 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiUsers.java @@ -25,6 +25,9 @@ import java.util.List; import org.radarbase.huawei.user.HuaweiUser; +/** + * @author yatharthranjan + */ @JsonIgnoreProperties(ignoreUnknown = true) public class HuaweiUsers { private final List users; diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java index 3ea2c8a1..42f2edf8 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java @@ -24,6 +24,9 @@ import java.time.Duration; import java.time.Instant; +/** + * @author yatharthranjan + */ @JsonIgnoreProperties(ignoreUnknown = true) public class OAuth2UserCredentials { private static final Duration DEFAULT_EXPIRY = Duration.ofHours(1); diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java index e3bf8e09..1c74b19f 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/util/VersionUtil.java @@ -17,6 +17,9 @@ package org.radarbase.connect.rest.huawei.util; +/** + * @author yatharthranjan + */ public final class VersionUtil { private VersionUtil() { // utility class diff --git a/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt index 04d4e1b3..585d8775 100644 --- a/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt +++ b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt @@ -21,6 +21,9 @@ import org.junit.jupiter.api.Test import org.radarbase.huawei.route.HuaweiRouteFactory import kotlin.test.assertEquals +/** + * @author yatharthranjan + */ class HuaweiRestSourceConnectorConfigTest { @Test From 4e23dbf5da49970b6c53d2fe25a953c75e361d33 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Tue, 28 Jul 2026 13:25:43 +0000 Subject: [PATCH 15/44] Add file-based HuaweiYamlUserRepository for local testing Lets the Huawei connector run against per-user YAML credential files under huawei.user.dir, mirroring Fitbit's YamlUserRepository, so it can be tested locally without a rest-source-authorizer webservice. Adds a docker/huawei-user.yml.template and README instructions for the flow. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- README.md | 36 +++ docker/huawei-user.yml.template | 24 ++ docker/source-huawei.properties.template | 6 + .../huawei/HuaweiRestSourceConnectorConfig.kt | 29 +++ .../rest/huawei/user/HuaweiLocalUser.kt | 104 ++++++++ .../huawei/user/HuaweiYamlUserRepository.kt | 234 ++++++++++++++++++ 6 files changed, 433 insertions(+) create mode 100644 docker/huawei-user.yml.template create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt create mode 100644 kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepository.kt diff --git a/README.md b/README.md index 09b941e1..7fe83445 100644 --- a/README.md +++ b/README.md @@ -230,6 +230,42 @@ This connector requires a (currently `0.9.0-SNAPSHOT`) to be resolvable from one of the repositories declared in `huawei-library/build.gradle` / `kafka-connect-huawei-source/build.gradle.kts`. +### Testing locally + +The easiest way to try out the Huawei connector without standing up a +`rest-source-authorizer` webservice is the file-based +`org.radarbase.connect.rest.huawei.user.HuaweiYamlUserRepository`, which reads one YAML file per +user from a local directory, mirroring the Fitbit `YamlUserRepository` above. + +1. [Register a Huawei Health Kit OAuth 2.0 app](https://developer.huawei.com/consumer/en/doc/HMSCore-Guides/config-agc-0000001050170137) + and obtain an access token and refresh token for one test user by hand, using Huawei's + [OAuth 2.0 authorization code flow](https://developer.huawei.com/consumer/en/doc/HMSCore-Guides/authorization-code-0000001053629189). +2. Copy `docker/huawei-user.yml.template` to a file in `docker/users/` (e.g. `docker/users/test.yml`) + and fill in the `externalUserId`, `oauth2.accessToken`, and `oauth2.refreshToken` fields. +3. Copy `docker/source-huawei.properties.template` to `docker/source-huawei.properties`, set + `huawei.api.client` / `huawei.api.secret` to your Huawei app's client ID and secret, and change + `huawei.user.repository.class` to `org.radarbase.connect.rest.huawei.user.HuaweiYamlUserRepository` + (the `huawei.user.repository.url`/`.client.id`/`.client.secret`/`.oauth2.token.url` properties + are only used by the webservice-based repository and can be left as-is or removed). +4. Run the full stack with `docker-compose up -d --build` and inspect the connector's progress with + `docker-compose logs -f radar-huawei-connector`. +5. To inspect the data coming out of a specific route, run, for example: + + ```shell + docker-compose exec schema-registry-1 kafka-avro-console-consumer \ + --bootstrap-server kafka-1:9092,kafka-2:9092,kafka-3:9092 \ + --from-beginning \ + --topic connect_huawei_activity_record + ``` + + (replace the topic with any `huawei..topic` default from + `org.radarbase.huawei.route.HuaweiRouteFactory`). + +For a full RADAR-base deployment, use the webservice-based +`org.radarbase.connect.rest.huawei.user.HuaweiServiceUserRepository` (the default) against a +`rest-source-authorizer` instance instead, following the same pattern as the Fitbit connector's +ManagementPortal setup above. + ## Sentry monitoring To enable Sentry monitoring for the generic REST, Fitbit, Oura, or Huawei source connector service: diff --git a/docker/huawei-user.yml.template b/docker/huawei-user.yml.template new file mode 100644 index 00000000..99e5187e --- /dev/null +++ b/docker/huawei-user.yml.template @@ -0,0 +1,24 @@ +--- +# Unique user key +id: test +# Project ID to be used in org.radarcns.kafka.ObservationKey record keys +projectId: radar-test +# User ID to be used in org.radarcns.kafka.ObservationKey record keys +userId: test +# Source ID to be used in org.radarcns.kafka.ObservationKey record keys +sourceId: huawei-watch +# Date from when to collect data. +startDate: 2018-08-06T00:00:00Z +# Date until when to collect data. +endDate: 2099-01-01T00:00:00Z +# Huawei user ID as returned by the Huawei OAuth 2.0 authentication procedure +externalUserId: ? +oauth2: + # Huawei Health Kit OAuth 2.0 access token as returned by the Huawei authentication procedure + accessToken: ? + # Huawei Health Kit OAuth 2.0 refresh token as returned by the Huawei authentication procedure + refreshToken: ? + # Optional expiry time of the access token. If absent, it will be estimated to one hour + # when the source connector starts. When an authentication error occurs, a new access token will + # be fetched regardless of the value in this field. + #expiresAt: 2018-08-06T00:00:00Z diff --git a/docker/source-huawei.properties.template b/docker/source-huawei.properties.template index 1dd5b63e..69607aa6 100644 --- a/docker/source-huawei.properties.template +++ b/docker/source-huawei.properties.template @@ -5,6 +5,12 @@ rest.source.base.url=https://health-api.cloud.huawei.com/healthkit/v1 rest.source.poll.interval.ms=5000 huawei.api.client=? huawei.api.secret=? +# For local testing without a rest-source-authorizer webservice, use the file-based +# repository instead, backed by per-user YAML files under huawei.user.dir - see +# docker/huawei-user.yml.template and the "Testing locally" section in README.md. +#huawei.user.repository.class=org.radarbase.connect.rest.huawei.user.HuaweiYamlUserRepository +#huawei.user.dir=/var/lib/kafka-connect-huawei-source/users + huawei.user.repository.class=org.radarbase.connect.rest.huawei.user.HuaweiServiceUserRepository huawei.user.repository.url=http://localhost:8080/ huawei.user.repository.client.id=radar_huawei_connector diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt index b8e1a2e8..d797cce2 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfig.kt @@ -31,6 +31,8 @@ import org.radarbase.connect.rest.huawei.user.HuaweiUserRepository import org.radarbase.huawei.route.HuaweiRouteFactory import java.net.MalformedURLException import java.net.URL +import java.nio.file.Path +import java.nio.file.Paths import java.time.Duration /** @@ -90,6 +92,14 @@ class HuaweiRestSourceConnectorConfig( throw ConnectException("Invalid class. $e") } + /** + * Directory containing per-user YAML credential files, for the file-based + * [org.radarbase.connect.rest.huawei.user.HuaweiYamlUserRepository]. Only used if that + * repository is configured via [HUAWEI_USER_REPOSITORY_CONFIG]. + */ + fun getHuaweiUserCredentialsPath(): Path = + Paths.get(getString(HUAWEI_USER_CREDENTIALS_DIR_CONFIG)) + fun getHuaweiUserRepositoryUrl(): HttpUrl { var urlString = getString(HUAWEI_USER_REPOSITORY_URL_CONFIG).trim() if (urlString.isNotEmpty() && urlString.last() != '/') { @@ -173,6 +183,14 @@ class HuaweiRestSourceConnectorConfig( private const val HUAWEI_USER_POLL_INTERVAL_DEFAULT = 150 private const val HUAWEI_USER_POLL_INTERVAL_DISPLAY = "Per-user per-route polling interval." + const val HUAWEI_USER_CREDENTIALS_DIR_CONFIG = "huawei.user.dir" + private const val HUAWEI_USER_CREDENTIALS_DIR_DOC = + "Directory containing Huawei user information and credentials. Only used if a " + + "file-based user repository is configured." + private const val HUAWEI_USER_CREDENTIALS_DIR_DISPLAY = "User directory" + private const val HUAWEI_USER_CREDENTIALS_DIR_DEFAULT = + "/var/lib/kafka-connect-huawei-source/users" + const val HUAWEI_USER_REPOSITORY_URL_CONFIG = "huawei.user.repository.url" private const val HUAWEI_USER_REPOSITORY_URL_DOC = "URL for webservice containing user credentials. Only used if a webservice-based " + @@ -286,6 +304,17 @@ class HuaweiRestSourceConnectorConfig( Width.SHORT, HUAWEI_USER_REPOSITORY_DISPLAY, ) + .define( + HUAWEI_USER_CREDENTIALS_DIR_CONFIG, + Type.STRING, + HUAWEI_USER_CREDENTIALS_DIR_DEFAULT, + Importance.LOW, + HUAWEI_USER_CREDENTIALS_DIR_DOC, + group, + ++order, + Width.SHORT, + HUAWEI_USER_CREDENTIALS_DIR_DISPLAY, + ) .define( HUAWEI_USER_REPOSITORY_URL_CONFIG, Type.STRING, diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt new file mode 100644 index 00000000..895d22fb --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt @@ -0,0 +1,104 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.radarbase.connect.rest.huawei.user + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties +import com.fasterxml.jackson.annotation.JsonInclude +import com.fasterxml.jackson.annotation.JsonProperty +import org.radarbase.huawei.user.User +import org.radarcns.kafka.ObservationKey +import java.time.Instant + +/** + * A single user's Huawei Health Kit credentials, read from (and written back to) a local YAML + * file by [HuaweiYamlUserRepository]. Mirrors Fitbit's `LocalUser`. See + * `docker/huawei-user.yml.template` for the expected file format. + * + * @author yatharthranjan + */ +@JsonInclude(JsonInclude.Include.NON_EMPTY) +@JsonIgnoreProperties(ignoreUnknown = true) +class HuaweiLocalUser : User { + @JsonProperty("id") + override var id: String = "" + + @JsonProperty("projectId") + override var projectId: String = "" + + @JsonProperty("userId") + override var userId: String = "" + + @JsonProperty("sourceId") + override var sourceId: String = "" + + @JsonProperty("externalUserId") + override var externalId: String? = null + + @JsonProperty("startDate") + override var startDate: Instant = Instant.parse("2017-01-01T00:00:00Z") + + @JsonProperty("endDate") + override var endDate: Instant? = Instant.parse("9999-12-31T23:59:59.999Z") + + @JsonProperty("createdAt") + override var createdAt: Instant = Instant.now() + + @JsonProperty("humanReadableUserId") + override var humanReadableUserId: String? = null + + @JsonProperty("serviceUserId") + override var serviceUserId: String? = null + + @JsonProperty("version") + override var version: String? = null + + @JsonProperty("oauth2") + var oauth2Credentials: OAuth2UserCredentials = OAuth2UserCredentials() + + @JsonProperty("isAuthorized") + var isAuthorizedOverride: Boolean? = null + + override val isAuthorized: Boolean + get() = isAuthorizedOverride + ?: (!oauth2Credentials.isAccessTokenExpired || oauth2Credentials.hasRefreshToken()) + + override val observationKey: ObservationKey + get() = ObservationKey(projectId, userId, sourceId) + + override val versionedId: String + get() = "$id${version?.let { "#$it" } ?: ""}" + + fun copy(): HuaweiLocalUser { + val copy = HuaweiLocalUser() + copy.id = id + copy.projectId = projectId + copy.userId = userId + copy.sourceId = sourceId + copy.externalId = externalId + copy.startDate = startDate + copy.endDate = endDate + copy.createdAt = createdAt + copy.humanReadableUserId = humanReadableUserId + copy.serviceUserId = serviceUserId + copy.version = version + copy.oauth2Credentials = oauth2Credentials + copy.isAuthorizedOverride = isAuthorizedOverride + return copy + } + + override fun toString(): String = "HuaweiLocalUser(id='$id', versionedId='$versionedId')" +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepository.kt new file mode 100644 index 00000000..97e09b6f --- /dev/null +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepository.kt @@ -0,0 +1,234 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.radarbase.connect.rest.huawei.user + +import com.fasterxml.jackson.databind.JsonNode +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.databind.SerializationFeature +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory +import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule +import com.fasterxml.jackson.module.kotlin.registerKotlinModule +import okhttp3.FormBody +import okhttp3.Headers +import okhttp3.OkHttpClient +import okhttp3.Request +import org.radarbase.connect.rest.huawei.HuaweiRestSourceConnectorConfig +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserNotAuthorizedException +import org.slf4j.LoggerFactory +import java.io.IOException +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.StandardCopyOption +import java.time.Duration +import java.time.Instant +import java.util.Base64 +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.atomic.AtomicReference +import java.util.concurrent.locks.ReentrantLock +import java.util.stream.Collectors + +/** + * User repository that reads (and writes refreshed tokens back to) YAML files in a local + * directory, one file per user - mirrors Fitbit's `YamlUserRepository`. This is the easiest way + * to run this connector locally without standing up a rest-source-authorizer webservice: register + * a Huawei Health Kit OAuth2 app, obtain one user's access/refresh token by hand (e.g. via + * Huawei's OAuth 2.0 authorization code flow), and drop them into a file under the directory + * configured by `huawei.user.dir` - see `docker/huawei-user.yml.template`. + * + * @author yatharthranjan + */ +@Suppress("unused") +class HuaweiYamlUserRepository : HuaweiUserRepository() { + private val client = OkHttpClient() + private val users = ConcurrentHashMap() + private val nextFetch = AtomicReference(Instant.EPOCH) + private lateinit var credentialsDir: Path + private lateinit var clientCredentials: Headers + + override fun initialize(config: HuaweiRestSourceConnectorConfig) { + credentialsDir = config.getHuaweiUserCredentialsPath() + Files.createDirectories(credentialsDir) + val credentialString = "${config.getHuaweiClient()}:${config.getHuaweiClientSecret()}" + val credentialsBase64 = Base64.getEncoder().encodeToString(credentialString.toByteArray()) + clientCredentials = Headers.headersOf("Authorization", "Basic $credentialsBase64") + } + + override operator fun get(key: String): User? { + updateUsers() + return users[key]?.locked { it.copy() } + } + + override fun stream(): Sequence { + if (nextFetch.get() == Instant.EPOCH) { + applyPendingUpdates() + } + return users.values.asSequence() + .filter { it.locked { u -> u.oauth2Credentials.hasRefreshToken() } } + .map { it.locked { u -> u.copy() } } + } + + @Throws(IOException::class, UserNotAuthorizedException::class) + override fun getAccessToken(user: User): String { + updateUsers() + val actual = users[user.id] + ?: throw NoSuchElementException("User $user is not present in this user repository.") + val current = actual.locked { u -> + if (!u.oauth2Credentials.isAccessTokenExpired) u.oauth2Credentials.accessToken else null + } + return current ?: refreshAccessToken(user) + } + + @Throws(IOException::class, UserNotAuthorizedException::class) + override fun refreshAccessToken(user: User): String { + val actual = users[user.id] + ?: throw NoSuchElementException("User $user is not present in this user repository.") + val refreshToken = actual.locked { it.oauth2Credentials.refreshToken } + val node = requestAccessToken(refreshToken) + + val expiresIn = node["expires_in"]?.asLong() + val accessToken = node["access_token"]?.asText() + ?: throw UserNotAuthorizedException("Did not get an access token") + val newRefreshToken = node["refresh_token"]?.asText() ?: refreshToken + + actual.update { u -> + u.oauth2Credentials = OAuth2UserCredentials(newRefreshToken, accessToken, expiresIn) + store(actual.path, u) + } + return accessToken + } + + override fun hasPendingUpdates(): Boolean = Instant.now().isAfter(nextFetch.get()) + + @Throws(IOException::class) + override fun applyPendingUpdates() { + forceUpdateUsers() + nextFetch.set(Instant.now().plus(FETCH_THRESHOLD)) + } + + private fun updateUsers() { + val next = nextFetch.get() + val now = Instant.now() + if (!now.isAfter(next) || !nextFetch.compareAndSet(next, now.plus(FETCH_THRESHOLD))) { + return + } + forceUpdateUsers() + } + + private fun forceUpdateUsers() { + try { + Files.walk(credentialsDir).use { walker -> + val newUsers = walker + .filter { + Files.isRegularFile(it) && + it.fileName.toString().lowercase().endsWith(".yml") + } + .map { path -> + LockedUser( + YAML_READER.readValue(path.toFile(), HuaweiLocalUser::class.java), + path, + ) + } + .collect(Collectors.toMap({ it.locked { u -> u.id } }, { it })) + users.keys.retainAll(newUsers.keys) + newUsers.forEach { (id, u) -> users.putIfAbsent(id, u) } + } + } catch (ex: IOException) { + logger.error("Failed to read user directory: {}", ex.toString()) + } + } + + private fun requestAccessToken(refreshToken: String?): JsonNode { + if (refreshToken.isNullOrEmpty()) { + throw UserNotAuthorizedException("Refresh token is not set") + } + val request = Request.Builder() + .url(HUAWEI_TOKEN_URL) + .headers(clientCredentials) + .post( + FormBody.Builder() + .add("grant_type", "refresh_token") + .add("refresh_token", refreshToken) + .build(), + ) + .build() + + client.newCall(request).execute().use { response -> + val body = response.body?.string() + return when { + response.isSuccessful && body != null -> JSON_READER.readTree(body) + response.code == 400 || response.code == 401 -> + throw UserNotAuthorizedException("Refresh token is no longer valid.") + else -> throw IOException( + "Failed to request refresh token, HTTP status ${response.code}" + + (body?.let { " and content $it" } ?: ""), + ) + } + } + } + + private fun store(path: Path, user: HuaweiLocalUser) { + try { + val temp = Files.createTempFile(user.id, ".tmp") + try { + Files.newOutputStream(temp).use { out -> YAML_WRITER.writeValue(out, user) } + Files.move(temp, path, StandardCopyOption.REPLACE_EXISTING) + } finally { + Files.deleteIfExists(temp) + } + } catch (ex: IOException) { + logger.error("Failed to store user file: {}", ex.toString()) + } + } + + /** Guards a mutable [HuaweiLocalUser] against concurrent read/refresh/store. */ + private class LockedUser(val user: HuaweiLocalUser, val path: Path) { + private val lock = ReentrantLock() + + fun locked(block: (HuaweiLocalUser) -> V): V { + lock.lock() + try { + return block(user) + } finally { + lock.unlock() + } + } + + fun update(block: (HuaweiLocalUser) -> Unit) { + lock.lock() + try { + block(user) + } finally { + lock.unlock() + } + } + } + + companion object { + private val logger = LoggerFactory.getLogger(HuaweiYamlUserRepository::class.java) + private const val HUAWEI_TOKEN_URL = "https://oauth-login.cloud.huawei.com/oauth2/v3/token" + private val FETCH_THRESHOLD = Duration.ofHours(1L) + private val YAML_MAPPER = ObjectMapper(YAMLFactory()).apply { + registerKotlinModule() + registerModule(JavaTimeModule()) + configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false) + } + private val YAML_READER = YAML_MAPPER.reader() + private val YAML_WRITER = YAML_MAPPER.writerFor(HuaweiLocalUser::class.java) + private val JSON_READER = ObjectMapper().registerModule(JavaTimeModule()).reader() + } +} From 9577b6794301e739ce546e356e6fb305ac562bbb Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Tue, 4 Aug 2026 11:51:49 +0000 Subject: [PATCH 16/44] Fix Jackson getter conflict on HuaweiLocalUser.isAuthorized isAuthorizedOverride was mapped to the same JSON key ("isAuthorized") as the isAuthorized computed property, causing Jackson to fail with "Conflicting getter definitions" when reading user YAML files. Rename the manual-override field's JSON key to "authorized". Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt index 895d22fb..a8a665ec 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt @@ -69,7 +69,7 @@ class HuaweiLocalUser : User { @JsonProperty("oauth2") var oauth2Credentials: OAuth2UserCredentials = OAuth2UserCredentials() - @JsonProperty("isAuthorized") + @JsonProperty("authorized") var isAuthorizedOverride: Boolean? = null override val isAuthorized: Boolean From 5f69d0d1c46336145302bb1dadb8fcf55ad66350 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Tue, 4 Aug 2026 12:20:14 +0000 Subject: [PATCH 17/44] Log actual Huawei API error body on 400/401/403 responses The 400/401/403 branches previously logged only a hardcoded generic message, discarding the real error body Huawei's API returned. This made it impossible to diagnose the actual cause of a failed request (e.g. wrong client/app, invalid scope) from the logs. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/request/HuaweiRequestGenerator.kt | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt index 6891ffb0..f039d2ca 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt @@ -156,31 +156,39 @@ class HuaweiRequestGenerator( HuaweiRateLimitError("Rate limit reached..", TooManyRequestsException(), "429") } 403 -> { + val body = response.body?.string() ?: "no response body" logger.warn( - "User {} does not have access to this Huawei Health Kit data type.", + "User {} does not have access to this Huawei Health Kit data type: {}", request.user, + body, ) routeNextRequest[key] = Instant.now().plus(USER_BACK_OFF_TIME) HuaweiAccessForbiddenError( - "Huawei Health Kit scope not granted or data not available..", + "Huawei Health Kit scope not granted or data not available: $body", IOException("Forbidden"), "403", ) } 401 -> { - logger.warn("User {} access token is expired, malformed, or revoked.", request.user) + val body = response.body?.string() ?: "no response body" + logger.warn( + "User {} access token is expired, malformed, or revoked: {}", + request.user, + body, + ) routeNextRequest[key] = Instant.now().plus(USER_BACK_OFF_TIME) HuaweiUnauthorizedAccessError( - "Access token expired or revoked..", + "Access token expired or revoked: $body", IOException("Unauthorized"), "401", ) } 400 -> { - logger.warn("Client exception for request {}", request) + val body = response.body?.string() ?: "no response body" + logger.warn("Client exception for request {}: {}", request, body) routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) HuaweiClientException( - "Client unsupported or unauthorized..", + "Client unsupported or unauthorized: $body", IOException("Invalid client"), "400", ) From 1ace5c11dc3a9fa725b6c6921bd79cd51dae90b1 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Tue, 4 Aug 2026 12:30:17 +0000 Subject: [PATCH 18/44] Fix wrong query param name on the healthRecords GET route HuaweiHealthRecordRoute sent the health record type identifier under a "subDataTypeName" query parameter, but Huawei's healthRecords API expects it under "dataTypeName" - the API was silently rejecting every health_record_* request with "DataTypeName is null" since it never received the parameter it actually looks for. Renamed the parameter (and the route/factory field) to dataTypeName throughout. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../radarbase/huawei/converter/HuaweiHealthRecordConverter.kt | 2 +- .../org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt | 4 ++-- .../kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt index b68c660a..671301db 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt @@ -31,7 +31,7 @@ private fun JsonNode.epochInstant(field: String): Instant? { /** * Generic converter for `GET /healthkit/v1/healthRecords` responses: iterates every record - * returned for the requested `subDataTypeName` and builds one Avro record per entry via + * returned for the requested `dataTypeName` and builds one Avro record per entry via * [buildRecord]. * * @author yatharthranjan diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt index 4d7a8c44..ef9e6740 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt @@ -36,7 +36,7 @@ import java.time.Instant */ open class HuaweiHealthRecordRoute( userRepository: UserRepository, - private val subDataTypeName: String, + private val dataTypeName: String, private val topic: String, maxIntervalPerRequest: Duration = Duration.ofDays(30L), buildRecord: ( @@ -63,7 +63,7 @@ open class HuaweiHealthRecordRoute( user, "healthRecords", mapOf( - "subDataTypeName" to subDataTypeName, + "dataTypeName" to dataTypeName, "startTime" to rangeStart.toEpochMilli().toString(), "endTime" to rangeEnd.toEpochMilli().toString(), ), diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index ad34ad9d..c93b5606 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -995,7 +995,7 @@ object HuaweiRouteFactory { private fun healthRecordDefinition( key: String, - subDataTypeName: String, + dataTypeName: String, defaultTopic: String, buildRecord: ( fields: FieldValues, @@ -1006,7 +1006,7 @@ object HuaweiRouteFactory { ): HuaweiRouteDefinition = HuaweiRouteDefinition(key, defaultTopic) { repo, topic -> HuaweiHealthRecordRoute( userRepository = repo, - subDataTypeName = VENDOR_PREFIX + subDataTypeName, + dataTypeName = VENDOR_PREFIX + dataTypeName, topic = topic, buildRecord = buildRecord, ) From 33512fc795f4f0d825fbb571f43b2134eeb3e83d Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Tue, 4 Aug 2026 14:34:12 +0000 Subject: [PATCH 19/44] Fix healthRecords route against the official REST API reference Per Huawei's official "Querying Health Records of a Data Type" spec: - The endpoint is on API version v2, not v1. - The data type query parameter is named "dataType", not "dataTypeName" (and not "subDataTypeName" as it was before that). - startTime/endTime, both in the request and in each returned record, are in nanoseconds since the epoch, not milliseconds. This was causing every health_record_* request to fail with "DataTypeName is null", and would have produced wildly wrong timestamps for any record that did come back. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../converter/HuaweiHealthRecordConverter.kt | 9 +++++---- .../huawei/route/HuaweiHealthRecordRoute.kt | 17 +++++++++++++---- .../org/radarbase/huawei/route/HuaweiRoute.kt | 4 +++- .../huawei/route/HuaweiRouteFactoryTest.kt | 6 ++++-- 4 files changed, 25 insertions(+), 11 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt index 671301db..aa522ff1 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt @@ -22,16 +22,17 @@ import org.apache.avro.specific.SpecificRecord import org.radarbase.huawei.user.User import java.time.Instant +/** Huawei's healthRecords v2 endpoint reports startTime/endTime in nanoseconds since the epoch. */ private fun JsonNode.epochInstant(field: String): Instant? { val value = this.get(field) ?: return null if (value.isNull) return null - val millis = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() - return millis?.let { Instant.ofEpochMilli(it) } + val nanos = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() + return nanos?.let { Instant.ofEpochSecond(it / 1_000_000_000L, it % 1_000_000_000L) } } /** - * Generic converter for `GET /healthkit/v1/healthRecords` responses: iterates every record - * returned for the requested `dataTypeName` and builds one Avro record per entry via + * Generic converter for `GET /healthkit/v2/healthRecords` responses: iterates every record + * returned for the requested `dataType` and builds one Avro record per entry via * [buildRecord]. * * @author yatharthranjan diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt index ef9e6740..9a376e36 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt @@ -28,10 +28,16 @@ import java.time.Duration import java.time.Instant /** - * Route backed by `GET /healthkit/v1/healthRecords`, used for the `health.record.*` data types + * Route backed by `GET /healthkit/v2/healthRecords`, used for the `health.record.*` data types * (ambulatory blood pressure sessions, heart rate alerts, hyperthermia, low SpO2 alerts, * menstrual cycle phases, and comprehensive sleep records). * + * Per the official Health Kit REST API reference, this endpoint is on API version `v2` (unlike + * `sampleSet:polymerize`/`activityRecords`, which are on `v1`), takes the data type under the + * `dataType` query parameter (not `dataTypeName`), and its `startTime`/`endTime` parameters (and + * the `startTime`/`endTime` fields of each returned record) are in **nanoseconds** since the + * epoch, not milliseconds. + * * @author yatharthranjan */ open class HuaweiHealthRecordRoute( @@ -63,10 +69,11 @@ open class HuaweiHealthRecordRoute( user, "healthRecords", mapOf( - "dataTypeName" to dataTypeName, - "startTime" to rangeStart.toEpochMilli().toString(), - "endTime" to rangeEnd.toEpochMilli().toString(), + "dataType" to dataTypeName, + "startTime" to rangeStart.toEpochNanos().toString(), + "endTime" to rangeEnd.toEpochNanos().toString(), ), + baseUrl = HUAWEI_API_BASE_URL_V2, ), user = user, route = this, @@ -74,4 +81,6 @@ open class HuaweiHealthRecordRoute( endDate = rangeEnd, ) } + + private fun Instant.toEpochNanos(): Long = epochSecond * 1_000_000_000L + nano.toLong() } diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt index 02bf19b8..81f94a32 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt @@ -47,9 +47,10 @@ abstract class HuaweiRoute( user: User, path: String, queryParams: Map, + baseUrl: String = HUAWEI_API_BASE_URL, ): Request { val accessToken = userRepository.getAccessToken(user) - val urlBuilder = "$HUAWEI_API_BASE_URL/$path".toHttpUrl().newBuilder() + val urlBuilder = "$baseUrl/$path".toHttpUrl().newBuilder() queryParams.forEach { (key, value) -> urlBuilder.addQueryParameter(key, value) } return Request.Builder() .url(urlBuilder.build()) @@ -91,6 +92,7 @@ abstract class HuaweiRoute( companion object { const val HUAWEI_API_BASE_URL = "https://health-api.cloud.huawei.com/healthkit/v1" + const val HUAWEI_API_BASE_URL_V2 = "https://health-api.cloud.huawei.com/healthkit/v2" private val JSON_MEDIA_TYPE = "application/json; charset=utf-8".toMediaType() private val DEFAULT_INTERVAL_PER_REQUEST = Duration.ofDays(30L) } diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 55f3c473..dd7aa8e3 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -115,8 +115,8 @@ class HuaweiRouteFactoryTest { val root = mapper.createObjectNode() val records = root.putArray("healthRecords") val record = records.addObject() - record.put("startTime", START_MILLIS) - record.put("endTime", END_MILLIS) + record.put("startTime", START_NANOS) + record.put("endTime", END_NANOS) record.set("value", genericValueArray()) return root } @@ -175,6 +175,8 @@ class HuaweiRouteFactoryTest { companion object { private const val START_MILLIS = 1704067200000L // 2024-01-01T00:00:00Z private const val END_MILLIS = 1704070800000L // 2024-01-01T01:00:00Z + private const val START_NANOS = START_MILLIS * 1_000_000L + private const val END_NANOS = END_MILLIS * 1_000_000L private val LITERAL_FIELD_KEYS = listOf( "active_hours", "active_hours_target", "all_sleep_time", "arrhythmia_result", From 1a1b6dfa090f08024f66e38c9daaeb6cc781bc06 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Tue, 4 Aug 2026 14:35:09 +0000 Subject: [PATCH 20/44] Report the real HTTP status code for unclassified error responses The generic error branch (405/409/500/502/503/590, and anything else not explicitly handled) hardcoded its HuaweiGenericError's code to "500" regardless of the actual response status, which was misleading for diagnostics. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../org/radarbase/huawei/request/HuaweiRequestGenerator.kt | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt index f039d2ca..5d7ce2aa 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt @@ -212,12 +212,13 @@ class HuaweiRequestGenerator( ) } else -> { - logger.warn("Request failed: {}, {}", request, response) + val body = response.body?.string() ?: "unknown error" + logger.warn("Request failed: {}: {}", request, body) routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) HuaweiGenericError( - response.body?.string() ?: "unknown error", + body, IOException("Unknown error"), - "500", + response.code.toString(), ) } } From 358584156aafff6d9b4a314b41f9ecccacd9b818 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Tue, 4 Aug 2026 14:43:25 +0000 Subject: [PATCH 21/44] Strip ".statistics" suffix from the dataTypeName sent to polymerize Per the official Health Kit REST API reference (Postman "HMS Core" collection, "Querying Sampling Data Statistics of Multiple Days"), the groupByTime-aggregated variant of a data type is obtained by polymerizing the underlying *raw* dataTypeName with groupByTime, not by sending a literal "*.statistics"-suffixed dataTypeName - that suffix is only RADAR-Schemas'/this connector's own label for "the daily-aggregated route", not a real Huawei data type identifier. Sending it verbatim is exactly why Huawei's API had no dataCollector for e.g. "com.huawei.continuous.heart_rate.statistics", "com.huawei.vo2max.statistics", "com.huawei.resting_calories.statistics", etc. - across every single statistics route built via sampleSetDefinition(). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index c93b5606..d98c7fa9 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -986,7 +986,11 @@ object HuaweiRouteFactory { ): HuaweiRouteDefinition = HuaweiRouteDefinition(key, defaultTopic) { repo, topic -> HuaweiSampleSetRoute( userRepository = repo, - dataTypeName = VENDOR_PREFIX + dataTypeSuffix, + // Huawei's polymerize API has no dataCollector for a literal "*.statistics" data + // type - ".statistics" is only this connector's/RADAR-Schemas' label for "the + // groupByTime-aggregated variant of the underlying raw data type", so it must be + // stripped from the dataTypeName actually sent on the wire. + dataTypeName = VENDOR_PREFIX + dataTypeSuffix.removeSuffix(".statistics"), topic = topic, groupByTimeUnit = if (dataTypeSuffix.endsWith(".statistics")) "day" else null, buildRecord = buildRecord, From d8c63885f14364c81f8328ad3cbc3c68ea118d26 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Tue, 4 Aug 2026 16:01:31 +0000 Subject: [PATCH 22/44] Query .statistics data types via sampleSet:dailyPolymerize, not polymerize Huawei's live API confirmed the root cause behind most "no default dataCollector found"/"Invalid dataTypeName" errors on statistics routes: sampleSet:polymerize does not support a groupByTime-aggregated query for every data type (confirmed live: "com.huawei.resting_calories does not support the query mode, please use dailyPolymerize API"). Per the official REST API reference for "Querying Sampling Data Statistics of Multiple Days", the day-aggregated variant of a data type must instead go through the dedicated POST /healthkit/v2/sampleSet:dailyPolymerize endpoint, which takes a startDay/endDay (yyyyMMdd) + timeZone request body and returns a differently-shaped, doubly-nested group[].sampleSet[].samplePoints[] response (with group-level times in milliseconds but sample-point times in nanoseconds). Adds HuaweiDailyPolymerizeRoute/HuaweiDailyPolymerizeConverter and routes every "*.statistics" definition through it instead of HuaweiSampleSetRoute, which now only handles raw (non-statistics) data types and has had its now-dead groupByTime support removed. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../HuaweiDailyPolymerizeConverter.kt | 70 ++++++++++++++ .../route/HuaweiDailyPolymerizeRoute.kt | 96 +++++++++++++++++++ .../org/radarbase/huawei/route/HuaweiRoute.kt | 9 +- .../huawei/route/HuaweiRouteFactory.kt | 33 ++++--- .../huawei/route/HuaweiSampleSetRoute.kt | 19 ++-- .../huawei/route/HuaweiRouteFactoryTest.kt | 17 ++++ 6 files changed, 218 insertions(+), 26 deletions(-) create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDailyPolymerizeConverter.kt create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiDailyPolymerizeRoute.kt diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDailyPolymerizeConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDailyPolymerizeConverter.kt new file mode 100644 index 00000000..3e4d5d72 --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDailyPolymerizeConverter.kt @@ -0,0 +1,70 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.huawei.converter + +import com.fasterxml.jackson.databind.JsonNode +import org.apache.avro.specific.SpecificRecord +import org.radarbase.huawei.user.User +import java.time.Instant + +/** Sample points inside `sampleSet:dailyPolymerize`'s response report their times in nanoseconds. */ +private fun JsonNode.epochNanoInstant(field: String): Instant? { + val value = this.get(field) ?: return null + if (value.isNull) return null + val nanos = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() + return nanos?.let { Instant.ofEpochSecond(it / 1_000_000_000L, it % 1_000_000_000L) } +} + +/** + * Converter for `POST /healthkit/v2/sampleSet:dailyPolymerize` responses: unlike + * `sampleSet:polymerize`, each day's result is wrapped in a `group[]` entry containing its own + * `sampleSet[].samplePoints[]`, so this walks two levels of nesting instead of one before reaching + * the same `{"fieldName": ..., "value": ...}` point shape used elsewhere. + * + * @author yatharthranjan + */ +class HuaweiDailyPolymerizeConverter( + private val topic: String, + private val buildRecord: ( + fields: FieldValues, + startTime: Instant, + endTime: Instant?, + timeReceived: Instant, + ) -> SpecificRecord, +) : HuaweiDataConverter { + + override fun processRecords(root: JsonNode, user: User): Sequence> { + val timeReceived = Instant.now() + val groups = root.get("group") ?: return emptySequence() + return groups.asSequence() + .flatMap { group -> group.get("sampleSet")?.asSequence() ?: emptySequence() } + .flatMap { sampleSet -> sampleSet.get("samplePoints")?.asSequence() ?: emptySequence() } + .mapCatching { point -> + val startTime = point.epochNanoInstant("startTime") + ?: error("Huawei daily polymerize sample point is missing startTime") + val endTime = point.epochNanoInstant("endTime") + val fieldValues = FieldValues.from(point.get("value")) + TopicData( + topic = topic, + key = user.observationKey, + offset = startTime.epochSecond, + value = buildRecord(fieldValues, startTime, endTime, timeReceived), + ) + } + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiDailyPolymerizeRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiDailyPolymerizeRoute.kt new file mode 100644 index 00000000..dafbdd8c --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiDailyPolymerizeRoute.kt @@ -0,0 +1,96 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.huawei.route + +import com.fasterxml.jackson.databind.ObjectMapper +import org.apache.avro.specific.SpecificRecord +import org.radarbase.huawei.converter.FieldValues +import org.radarbase.huawei.converter.HuaweiDailyPolymerizeConverter +import org.radarbase.huawei.converter.HuaweiDataConverter +import org.radarbase.huawei.request.RestRequest +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserRepository +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.time.format.DateTimeFormatter + +/** + * Route backed by `POST /healthkit/v2/sampleSet:dailyPolymerize`, used for every Huawei + * `.statistics` data type. + * + * Huawei's `sampleSet:polymerize` endpoint (see [HuaweiSampleSetRoute]) does not accept a + * `groupByTime`-aggregated query for every data type - some (confirmed live: `resting_calories`) + * reject it with `"does not support the query mode, please use dailyPolymerize API"`. This route + * calls that dedicated day-granularity statistics endpoint instead, which takes a day-string range + * (`startDay`/`endDay`, format `yyyyMMdd`, at most 31 days apart) rather than epoch timestamps. + * + * @author yatharthranjan + */ +open class HuaweiDailyPolymerizeRoute( + userRepository: UserRepository, + private val dataTypeName: String, + private val topic: String, + maxIntervalPerRequest: Duration = Duration.ofDays(30L), + buildRecord: ( + fields: FieldValues, + startTime: Instant, + endTime: Instant?, + timeReceived: Instant, + ) -> SpecificRecord, +) : HuaweiRoute(userRepository, maxIntervalPerRequest) { + + override val converters: List = + listOf(HuaweiDailyPolymerizeConverter(topic, buildRecord)) + + override fun toString(): String = "huawei_" + topic.removePrefix("connect_huawei_") + + override fun generateRequests( + user: User, + start: Instant, + end: Instant, + max: Int, + ): Sequence = chunkedRanges(start, end, max).map { (rangeStart, rangeEnd) -> + RestRequest( + request = createPostRequest( + user, + "sampleSet:dailyPolymerize", + buildRequestBody(rangeStart, rangeEnd), + baseUrl = HUAWEI_API_BASE_URL_V2, + ), + user = user, + route = this, + startDate = rangeStart, + endDate = rangeEnd, + ) + } + + private fun buildRequestBody(start: Instant, end: Instant): String { + val root = MAPPER.createObjectNode() + root.putArray("dataTypes").add(dataTypeName) + root.put("startDay", DAY_FORMATTER.format(start)) + root.put("endDay", DAY_FORMATTER.format(end)) + root.put("timeZone", "+0000") + return MAPPER.writeValueAsString(root) + } + + companion object { + private val MAPPER = ObjectMapper() + private val DAY_FORMATTER = DateTimeFormatter.ofPattern("yyyyMMdd").withZone(ZoneOffset.UTC) + } +} diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt index 81f94a32..9c7cd11e 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRoute.kt @@ -59,10 +59,15 @@ abstract class HuaweiRoute( .build() } - protected fun createPostRequest(user: User, path: String, jsonBody: String): Request { + protected fun createPostRequest( + user: User, + path: String, + jsonBody: String, + baseUrl: String = HUAWEI_API_BASE_URL, + ): Request { val accessToken = userRepository.getAccessToken(user) return Request.Builder() - .url("$HUAWEI_API_BASE_URL/$path".toHttpUrl()) + .url("$baseUrl/$path".toHttpUrl()) .header("Authorization", "Bearer $accessToken") .post(jsonBody.toRequestBody(JSON_MEDIA_TYPE)) .build() diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index d98c7fa9..f9f0b85e 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -984,17 +984,28 @@ object HuaweiRouteFactory { timeReceived: Instant, ) -> SpecificRecord, ): HuaweiRouteDefinition = HuaweiRouteDefinition(key, defaultTopic) { repo, topic -> - HuaweiSampleSetRoute( - userRepository = repo, - // Huawei's polymerize API has no dataCollector for a literal "*.statistics" data - // type - ".statistics" is only this connector's/RADAR-Schemas' label for "the - // groupByTime-aggregated variant of the underlying raw data type", so it must be - // stripped from the dataTypeName actually sent on the wire. - dataTypeName = VENDOR_PREFIX + dataTypeSuffix.removeSuffix(".statistics"), - topic = topic, - groupByTimeUnit = if (dataTypeSuffix.endsWith(".statistics")) "day" else null, - buildRecord = buildRecord, - ) + // Huawei's polymerize API rejects a groupByTime-aggregated query for at least some data + // types (confirmed live: "com.huawei.resting_calories does not support the query mode, + // please use dailyPolymerize API") - the day-aggregated ("*.statistics") variant of every + // data type must go through the dedicated sampleSet:dailyPolymerize endpoint instead, using + // the underlying raw data type name (the ".statistics" suffix is only this + // connector's/RADAR-Schemas' label and is never sent on the wire). + val rawDataTypeName = VENDOR_PREFIX + dataTypeSuffix.removeSuffix(".statistics") + if (dataTypeSuffix.endsWith(".statistics")) { + HuaweiDailyPolymerizeRoute( + userRepository = repo, + dataTypeName = rawDataTypeName, + topic = topic, + buildRecord = buildRecord, + ) + } else { + HuaweiSampleSetRoute( + userRepository = repo, + dataTypeName = rawDataTypeName, + topic = topic, + buildRecord = buildRecord, + ) + } } private fun healthRecordDefinition( diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt index 664e80a7..3e15485e 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt @@ -30,13 +30,13 @@ import java.time.Instant /** * Route backed by `POST /healthkit/v1/sampleSet:polymerize`, which covers the large majority of - * Huawei Health Kit data types (all `continuous.*`, `instantaneous.*`, `cgm_blood_glucose`, - * `active_hours`, `daily_activity_summary`, `emotion`, `heart_rate_variability`, `vo2max`, - * `resting_calories.statistics`, `sleep.on_off_bed_record`, and `sleep_respiratory_*` types). + * raw (non-`.statistics`) Huawei Health Kit data types (all `continuous.*`, `instantaneous.*`, + * `cgm_blood_glucose`, `active_hours`, `daily_activity_summary`, `emotion`, + * `heart_rate_variability`, `vo2max`, `sleep.on_off_bed_record`, and `sleep_respiratory_*` types). + * Returns raw, un-aggregated sample points for [dataTypeName] over the requested time range. * - * When [groupByTimeUnit] is set, the request aggregates sample points into buckets of that size — - * this is how Huawei's `.statistics` data types are queried. When it is `null`, the endpoint - * returns raw, un-aggregated sample points for [dataTypeName] over the requested time range. + * The day-aggregated `.statistics` variant of a data type is not queried through this route - + * see [HuaweiDailyPolymerizeRoute]. * * @author yatharthranjan */ @@ -44,7 +44,6 @@ open class HuaweiSampleSetRoute( userRepository: UserRepository, private val dataTypeName: String, private val topic: String, - private val groupByTimeUnit: String? = null, maxIntervalPerRequest: Duration = Duration.ofDays(30L), buildRecord: ( fields: FieldValues, @@ -83,12 +82,6 @@ open class HuaweiSampleSetRoute( root.putArray("polymerizeWith").addObject().put("dataTypeName", dataTypeName) root.put("startTime", start.toEpochMilli()) root.put("endTime", end.toEpochMilli()) - if (groupByTimeUnit != null) { - val groupPeriod = root.putObject("groupByTime").putObject("groupPeriod") - groupPeriod.put("unit", groupByTimeUnit) - groupPeriod.put("value", 1) - groupPeriod.put("timeZone", "+0000") - } return MAPPER.writeValueAsString(root) } diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index dd7aa8e3..48e05947 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -95,6 +95,7 @@ class HuaweiRouteFactoryTest { private fun fixtureFor(route: HuaweiRoute) = when (route) { is HuaweiActivityRecordRoute -> activityRecordFixture() is HuaweiHealthRecordRoute -> healthRecordFixture() + is HuaweiDailyPolymerizeRoute -> dailyPolymerizeFixture() is HuaweiSampleSetRoute -> sampleSetFixture() else -> error("Unknown route type: ${route::class}") } @@ -111,6 +112,22 @@ class HuaweiRouteFactoryTest { return root } + private fun dailyPolymerizeFixture(): ObjectNode { + val root = mapper.createObjectNode() + val groups = root.putArray("group") + val group = groups.addObject() + group.put("startTime", START_MILLIS) + group.put("endTime", END_MILLIS) + val sampleSet = group.putArray("sampleSet") + val collector = sampleSet.addObject() + val samplePoints = collector.putArray("samplePoints") + val point = samplePoints.addObject() + point.put("startTime", START_NANOS) + point.put("endTime", END_NANOS) + point.set("value", genericValueArray()) + return root + } + private fun healthRecordFixture(): ObjectNode { val root = mapper.createObjectNode() val records = root.putArray("healthRecords") From da4ddfd45bae969a9167d44aaf973d08dbd14f0d Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 6 Aug 2026 10:48:37 +0000 Subject: [PATCH 23/44] Fix activityRecords route against the official REST API reference Per Huawei's "Querying Created Exercise Records" spec: - The endpoint is on API version v2, not v1. - The response's array of records is under the key "activityRecord" (singular), not "activityRecords" - we were reading the wrong key, so every activity_record request was silently producing zero records regardless of what the API actually returned, with no error logged. - Each record's description field is "desc", not "description". Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/converter/HuaweiActivityRecordConverter.kt | 9 ++++++--- .../radarbase/huawei/route/HuaweiActivityRecordRoute.kt | 3 ++- .../org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt | 4 ++-- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt index 26248540..e70c6b63 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt @@ -23,7 +23,7 @@ import org.radarcns.connector.huawei.HuaweiActivityRecord import java.time.Instant /** - * Converts `GET /healthkit/v1/activityRecords` responses into [HuaweiActivityRecord]s. + * Converts `GET /healthkit/v2/activityRecords` responses into [HuaweiActivityRecord]s. * * Field names below follow the Huawei Health Kit `ActivityRecord`/`Device`/`ActivitySummary` * model (activity record id, name, description, time zone, activity type, device manufacturer and @@ -39,7 +39,10 @@ class HuaweiActivityRecordConverter( override fun processRecords(root: JsonNode, user: User): Sequence> { val timeReceived = Instant.now() - val records = root.get("activityRecords") ?: root.get("records") ?: return emptySequence() + val records = root.get("activityRecord") + ?: root.get("activityRecords") + ?: root.get("records") + ?: return emptySequence() return records.asSequence() .mapCatching { record -> val startTime = record.epochInstant("startTime") @@ -65,7 +68,7 @@ class HuaweiActivityRecordConverter( endTime = epochInstant("endTime")?.toEpoch() activityRecordId = textOrNull("id") ?: textOrNull("activityRecordId") name = textOrNull("name") - description = textOrNull("description") + description = textOrNull("desc") ?: textOrNull("description") timeZone = textOrNull("timeZone") activityTypeId = textOrNull("activityType") ?: textOrNull("activityTypeId") activeTimeMillis = longOrNull("activeTime") ?: longOrNull("activeTimeMillis") diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt index ebd75def..c8c8b347 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiActivityRecordRoute.kt @@ -26,7 +26,7 @@ import java.time.Duration import java.time.Instant /** - * Route backed by `GET /healthkit/v1/activityRecords`, covering the Huawei Health Kit Activity + * Route backed by `GET /healthkit/v2/activityRecords`, covering the Huawei Health Kit Activity * Records API (workout / physical-activity sessions). * * @author yatharthranjan @@ -56,6 +56,7 @@ class HuaweiActivityRecordRoute( "startTime" to rangeStart.toEpochMilli().toString(), "endTime" to rangeEnd.toEpochMilli().toString(), ), + baseUrl = HUAWEI_API_BASE_URL_V2, ), user = user, route = this, diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 48e05947..d7483909 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -140,13 +140,13 @@ class HuaweiRouteFactoryTest { private fun activityRecordFixture(): ObjectNode { val root = mapper.createObjectNode() - val records = root.putArray("activityRecords") + val records = root.putArray("activityRecord") val record = records.addObject() record.put("startTime", START_MILLIS) record.put("endTime", END_MILLIS) record.put("id", "activity-1") record.put("name", "Run") - record.put("description", "Morning run") + record.put("desc", "Morning run") record.put("timeZone", "Europe/London") record.put("activityType", "1") record.put("activeTime", 1000L) From 9c90c4ed96c2b9161b957bab02cad437f5af7f81 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 6 Aug 2026 11:01:23 +0000 Subject: [PATCH 24/44] Disable daily_activity_summary by default pending a redesign "com.huawei.daily_activity_summary" is not a real sampleSet dataTypeName (confirmed live: "no default dataCollector found"). The goal fields this route maps to actually belong to a separate endpoint (GET /healthkit/v2/sampleConfigs?type=9002&id=<...>, "Querying Activity Goals" - one call per goal type), while the achieved-value fields would need to come from the existing continuous/statistics routes. That needs a route that issues multiple requests and merges them, which is a real design decision rather than an endpoint/field-name fix like the other routes touched this session, so disable it by default (huawei.daily_activity_summary.enabled=false) rather than leave it erroring or guess at a composite implementation. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/route/HuaweiRouteFactory.kt | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index f9f0b85e..661b2d9e 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -222,10 +222,20 @@ object HuaweiRouteFactory { ) add( + // Disabled by default: "com.huawei.daily_activity_summary" is not a real Huawei + // sampleSet dataTypeName (confirmed live: "no default dataCollector found"). The + // goal fields this route maps (stepsGoal/activeCaloriesGoal/exerciseTimeGoal/ + // activeHoursGoal) actually belong to a completely different endpoint + // (GET /healthkit/v2/sampleConfigs?type=9002&id=<900200006..900200009>, "Querying + // Activity Goals"), and the achieved-value fields would need to come from the + // existing continuous/statistics routes instead. Needs a real redesign (a route that + // issues multiple requests and merges them) before this can work - not a simple + // endpoint/field-name fix like the other routes here. sampleSetDefinition( "daily_activity_summary", "daily_activity_summary", "connect_huawei_daily_activity_summary", + enabledByDefault = false, ) { f, start, end, received -> HuaweiDailyActivitySummary.newBuilder().apply { time = start.toEpoch() @@ -977,13 +987,18 @@ object HuaweiRouteFactory { key: String, dataTypeSuffix: String, defaultTopic: String, + enabledByDefault: Boolean = true, buildRecord: ( fields: FieldValues, startTime: Instant, endTime: Instant?, timeReceived: Instant, ) -> SpecificRecord, - ): HuaweiRouteDefinition = HuaweiRouteDefinition(key, defaultTopic) { repo, topic -> + ): HuaweiRouteDefinition = HuaweiRouteDefinition( + key, + defaultTopic, + enabledByDefault, + ) { repo, topic -> // Huawei's polymerize API rejects a groupByTime-aggregated query for at least some data // types (confirmed live: "com.huawei.resting_calories does not support the query mode, // please use dailyPolymerize API") - the day-aggregated ("*.statistics") variant of every From 89ea098b9ae55ee0f437763f615da76100e94532 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 6 Aug 2026 11:15:57 +0000 Subject: [PATCH 25/44] Query *.total routes by their raw/delta data type via dailyPolymerize "*.total" is never a valid Huawei request dataTypeName (confirmed live: "no default dataCollector found for: com.huawei.continuous.steps.total"), matching the official dailyPolymerize doc's own example: it requests "com.huawei.continuous.steps.delta" and gets back a response labelled "com.huawei.continuous.steps.total". Adds queryDataTypeSuffix/ useDailyPolymerize overrides to sampleSetDefinition() and points continuous_steps_total, continuous_distance_total, and continuous_calories_burnt_total at their sibling raw/delta data type through the dailyPolymerize endpoint instead of requesting the ".total" name directly via plain polymerize. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/route/HuaweiRouteFactory.kt | 27 +++++++++++++------ 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 661b2d9e..c9c98aed 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -422,6 +422,8 @@ object HuaweiRouteFactory { "continuous_calories_burnt_total", "continuous.calories.burnt.total", "connect_huawei_continuous_calories_burnt_total", + queryDataTypeSuffix = "continuous.calories.burnt", + useDailyPolymerize = true, ) { f, start, end, received -> HuaweiContinuousCaloriesBurntTotal.newBuilder().apply { time = start.toEpoch() @@ -451,6 +453,8 @@ object HuaweiRouteFactory { "continuous_distance_total", "continuous.distance.total", "connect_huawei_continuous_distance_total", + queryDataTypeSuffix = "continuous.distance.delta", + useDailyPolymerize = true, ) { f, start, end, received -> HuaweiContinuousDistanceTotal.newBuilder().apply { time = start.toEpoch() @@ -586,6 +590,8 @@ object HuaweiRouteFactory { "continuous_steps_total", "continuous.steps.total", "connect_huawei_continuous_steps_total", + queryDataTypeSuffix = "continuous.steps.delta", + useDailyPolymerize = true, ) { f, start, end, received -> HuaweiContinuousStepsTotal.newBuilder().apply { time = start.toEpoch() @@ -988,6 +994,17 @@ object HuaweiRouteFactory { dataTypeSuffix: String, defaultTopic: String, enabledByDefault: Boolean = true, + // Huawei's polymerize API rejects a groupByTime-aggregated query for at least some data + // types (confirmed live: "com.huawei.resting_calories does not support the query mode, + // please use dailyPolymerize API") - the day-aggregated ("*.statistics") variant of every + // data type must go through the dedicated sampleSet:dailyPolymerize endpoint instead. + // Likewise, "*.total" data types are never valid *request* dataTypeNames (confirmed live: + // "no default dataCollector found for: com.huawei.continuous.steps.total") - Huawei's own + // dailyPolymerize example queries the "*.delta" data type and gets a "*.total"-labelled + // response back, so a "*.total" route must override [queryDataTypeSuffix] to name its + // sibling raw/delta data type instead. + queryDataTypeSuffix: String = dataTypeSuffix.removeSuffix(".statistics"), + useDailyPolymerize: Boolean = dataTypeSuffix.endsWith(".statistics"), buildRecord: ( fields: FieldValues, startTime: Instant, @@ -999,14 +1016,8 @@ object HuaweiRouteFactory { defaultTopic, enabledByDefault, ) { repo, topic -> - // Huawei's polymerize API rejects a groupByTime-aggregated query for at least some data - // types (confirmed live: "com.huawei.resting_calories does not support the query mode, - // please use dailyPolymerize API") - the day-aggregated ("*.statistics") variant of every - // data type must go through the dedicated sampleSet:dailyPolymerize endpoint instead, using - // the underlying raw data type name (the ".statistics" suffix is only this - // connector's/RADAR-Schemas' label and is never sent on the wire). - val rawDataTypeName = VENDOR_PREFIX + dataTypeSuffix.removeSuffix(".statistics") - if (dataTypeSuffix.endsWith(".statistics")) { + val rawDataTypeName = VENDOR_PREFIX + queryDataTypeSuffix + if (useDailyPolymerize) { HuaweiDailyPolymerizeRoute( userRepository = repo, dataTypeName = rawDataTypeName, From 309b8074946fc6f1d2ef437321f3d8fd38bb1b8c Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 6 Aug 2026 11:23:27 +0000 Subject: [PATCH 26/44] Fix continuous_spo2_statistics data type name and field keys Per the official "SpO2" data type reference: the statistics data type is documented under the "continuous." namespace (com.huawei.continuous.spo2.statistics), but its underlying raw detailed data type is under a different namespace entirely (com.huawei.instantaneous.spo2, not com.huawei.continuous.spo2, which doesn't exist - matching the live "Invalid dataTypeName." error). Also corrects the field-value keys read from the response (saturation_avg/max/min/last, not the generic avg/max/min/last this route had guessed). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../radarbase/huawei/route/HuaweiRouteFactory.kt | 13 +++++++++---- .../huawei/route/HuaweiRouteFactoryTest.kt | 1 + 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index c9c98aed..07098e65 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -558,15 +558,20 @@ object HuaweiRouteFactory { "continuous_spo2_statistics", "continuous.spo2.statistics", "connect_huawei_continuous_spo2_statistics", + // Statistics variant is documented under "continuous.", but its underlying raw + // detailed data type is "com.huawei.instantaneous.spo2" - a different namespace, + // per the official "SpO2" data type reference. + queryDataTypeSuffix = "instantaneous.spo2", + useDailyPolymerize = true, ) { f, start, end, received -> HuaweiContinuousSpo2Statistics.newBuilder().apply { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - saturationAvg = f.getDouble("avg") - saturationMax = f.getDouble("max") - saturationMin = f.getDouble("min") - saturationLast = f.getDouble("last") + saturationAvg = f.getDouble("saturation_avg") + saturationMax = f.getDouble("saturation_max") + saturationMin = f.getDouble("saturation_min") + saturationLast = f.getDouble("saturation_last") }.build() }, ) diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index d7483909..eb25e897 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -209,6 +209,7 @@ class HuaweiRouteFactoryTest { "min_breathe_rate", "min_breathrate_baseline", "min_spo2", "off_bed_time", "on_off_bed_state", "predicted_calories", "prepare_sleep_time", "record_day", "record_id", "remarks", "sample_source", "sampling_frequency", "sleep_efficiency", + "saturation_avg", "saturation_last", "saturation_max", "saturation_min", "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "sphygmus_avg", "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", "steps_delta", "steps_target", "sub_status", "systolic_pressure_avg", "systolic_pressure_max", From 1bf0b9e0e6d1e40ac6ace5a2ae8d1b6ebad1ac1e Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 6 Aug 2026 11:39:32 +0000 Subject: [PATCH 27/44] Fix daily_activity_summary, distance_total, altitude_statistics, active_hours Per the official "Daily Activity" data type reference pages: - daily_activity_summary is itself a real "Atomic Sampling Statistical Data Type" queried by day via dailyPolymerize - it's not a derived label needing a different raw type, and not the separate sampleConfigs-based "Workout Goals" endpoint as previously assumed. Re-enabled by default and switched to dailyPolymerize, with field keys corrected to the documented camelCase names (steps, activeCalories, exerciseTime, activeHours, stepsGoal, activeCaloriesGoal, exerciseTimeGoal, activeHoursGoal), matching the Avro schema's own field names. - continuous_distance_total's single field is "distance", not "distance_total". - continuous_altitude_statistics's underlying raw data type is "com.huawei.instantaneous.altitude" (a different namespace than its "continuous."-labelled statistics name), the same namespace mismatch already fixed for SpO2. - active_hours (raw) and active_hours_statistics were incorrectly sharing one builder function reading the same field keys, but they have genuinely different response shapes: the raw type's only field is "isActive", while the statistics type's only field is "activeHours" (with no moderate/high intensity minute breakdown on either, unlike what the shared builder assumed). Split into two builder functions with the correct field keys for each. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/route/HuaweiRouteFactory.kt | 66 ++++++++++++------- .../huawei/route/HuaweiRouteFactoryTest.kt | 35 +++++----- 2 files changed, 60 insertions(+), 41 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 07098e65..abff6274 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -222,33 +222,30 @@ object HuaweiRouteFactory { ) add( - // Disabled by default: "com.huawei.daily_activity_summary" is not a real Huawei - // sampleSet dataTypeName (confirmed live: "no default dataCollector found"). The - // goal fields this route maps (stepsGoal/activeCaloriesGoal/exerciseTimeGoal/ - // activeHoursGoal) actually belong to a completely different endpoint - // (GET /healthkit/v2/sampleConfigs?type=9002&id=<900200006..900200009>, "Querying - // Activity Goals"), and the achieved-value fields would need to come from the - // existing continuous/statistics routes instead. Needs a real redesign (a route that - // issues multiple requests and merges them) before this can work - not a simple - // endpoint/field-name fix like the other routes here. + // "com.huawei.daily_activity_summary" is itself a documented "Atomic Sampling + // Statistical Data Type" (per the official "Daily Activity Data" reference) queried by + // day via dailyPolymerize - it is not derived from a separate raw type, and it is NOT + // the separate sampleConfigs-based "Workout Goals" endpoint. Its field names are + // camelCase (matching the Avro schema field names directly), unlike most other Huawei + // data types' snake_case field names. sampleSetDefinition( "daily_activity_summary", "daily_activity_summary", "connect_huawei_daily_activity_summary", - enabledByDefault = false, + useDailyPolymerize = true, ) { f, start, end, received -> HuaweiDailyActivitySummary.newBuilder().apply { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() steps = f.getInt("steps") - activeCalories = f.getInt("calories") - exerciseTime = f.getInt("exercise_time") - activeHours = f.getInt("active_hours") - stepsGoal = f.getInt("steps_target") - activeCaloriesGoal = f.getInt("calories_target") - exerciseTimeGoal = f.getInt("exercise_time_target") - activeHoursGoal = f.getInt("active_hours_target") + activeCalories = f.getInt("activeCalories") + exerciseTime = f.getInt("exerciseTime") + activeHours = f.getInt("activeHours") + stepsGoal = f.getInt("stepsGoal") + activeCaloriesGoal = f.getInt("activeCaloriesGoal") + exerciseTimeGoal = f.getInt("exerciseTimeGoal") + activeHoursGoal = f.getInt("activeHoursGoal") }.build() }, ) @@ -259,7 +256,7 @@ object HuaweiRouteFactory { "active_hours", "connect_huawei_active_hours", ) { f, start, end, received -> - f.toActiveHours(start, end, received) + f.toRawActiveHours(start, end, received) }, ) add( @@ -268,7 +265,7 @@ object HuaweiRouteFactory { "active_hours.statistics", "connect_huawei_active_hours_statistics", ) { f, start, end, received -> - f.toActiveHours(start, end, received) + f.toActiveHoursStatistics(start, end, received) }, ) @@ -296,6 +293,11 @@ object HuaweiRouteFactory { "continuous_altitude_statistics", "continuous.altitude.statistics", "connect_huawei_continuous_altitude_statistics", + // Statistics variant is documented under "continuous.", but its underlying raw + // detailed data type is "com.huawei.instantaneous.altitude" - a different + // namespace, per the official "Altitude" data type reference. + queryDataTypeSuffix = "instantaneous.altitude", + useDailyPolymerize = true, ) { f, start, end, received -> HuaweiContinuousAltitudeStatistics.newBuilder().apply { time = start.toEpoch() @@ -460,7 +462,7 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - distance = f.getDouble("distance_total") + distance = f.getDouble("distance") }.build() }, ) @@ -816,7 +818,25 @@ object HuaweiRouteFactory { ) } - private fun FieldValues.toActiveHours( + /** + * The raw `com.huawei.active_hours` data type's only documented field is `isActive` (whether + * that hour had at least moderate-intensity activity) - it has no moderate/high intensity + * minute breakdown, unlike what [toActiveHoursStatistics] reads. + */ + private fun FieldValues.toRawActiveHours( + start: Instant, + end: Instant?, + received: Instant, + ): HuaweiActiveHours = HuaweiActiveHours.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + activeHours = getInt("isActive") + }.build() + + /** The `com.huawei.active_hours.statistics` data type's only documented field is `activeHours` + * (the number of active hours in the statistical period). */ + private fun FieldValues.toActiveHoursStatistics( start: Instant, end: Instant?, received: Instant, @@ -824,9 +844,7 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - activeHours = getInt("active_hours") - moderateIntensityMinutes = getInt("moderate_intensity_minutes") - highIntensityMinutes = getInt("high_intensity_minutes") + activeHours = getInt("activeHours") }.build() private fun FieldValues.toContinuousActivityStatistics( diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index eb25e897..53355c7b 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -196,27 +196,28 @@ class HuaweiRouteFactoryTest { private const val END_NANOS = END_MILLIS * 1_000_000L private val LITERAL_FIELD_KEYS = listOf( - "active_hours", "active_hours_target", "all_sleep_time", "arrhythmia_result", + "active_hours", "active_hours_target", "activeCalories", "activeHours", + "activeCaloriesGoal", "activeHoursGoal", "all_sleep_time", "arrhythmia_result", "arrhythmia_type", "ascent_total", "avg", "avg_breathe_rate", "avg_heart_rate", "awake_time", "calories", "calories_target", "calories_total", "correlate_mealtime", "correlate_sleep", "count", "deep_sleep_part", "deep_sleep_time", "descent_total", "diastolic_pressure_avg", "diastolic_pressure_max", "diastolic_pressure_min", - "distance_delta", "distance_total", "dream_time", "duration", "emotion", "event_name", - "exercise_time", "exercise_time_target", "exercise_type", "extend_data", - "fall_asleep_time", "go_bed_time", "heart_rate_variability_rmssd", - "high_body_temperature_alarm", "last", "level", "light_sleep_time", "max", - "max_breathe_rate", "max_breathrate_baseline", "max_spo2", "meal", "min", - "min_breathe_rate", "min_breathrate_baseline", "min_spo2", "off_bed_time", - "on_off_bed_state", "predicted_calories", "prepare_sleep_time", "record_day", - "record_id", "remarks", "sample_source", "sampling_frequency", "sleep_efficiency", - "saturation_avg", "saturation_last", "saturation_max", "saturation_min", - "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "sphygmus_avg", - "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", "steps_delta", - "steps_target", "sub_status", "systolic_pressure_avg", "systolic_pressure_max", - "systolic_pressure_min", "threshold", "timezone", "total_calories", "type", - "user_symptom", "value", "vo2max", "voltage_data", "wakeup_count", "wakeup_time", - "zone1_duration", "zone2_duration", "zone3_duration", "zone4_duration", - "zone5_duration", + "distance", "distance_delta", "distance_total", "dream_time", "duration", "emotion", + "event_name", "exercise_time", "exercise_time_target", "exerciseTime", + "exerciseTimeGoal", "exercise_type", "extend_data", "fall_asleep_time", "go_bed_time", + "heart_rate_variability_rmssd", "high_body_temperature_alarm", "isActive", "last", + "level", "light_sleep_time", "max", "max_breathe_rate", "max_breathrate_baseline", + "max_spo2", "meal", "min", "min_breathe_rate", "min_breathrate_baseline", "min_spo2", + "off_bed_time", "on_off_bed_state", "predicted_calories", "prepare_sleep_time", + "record_day", "record_id", "remarks", "sample_source", "sampling_frequency", + "sleep_efficiency", "saturation_avg", "saturation_last", "saturation_max", + "saturation_min", "sleep_latency", "sleep_score", "sleep_state", "sleep_type", + "sphygmus_avg", "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", + "steps_delta", "steps_target", "stepsGoal", "sub_status", "systolic_pressure_avg", + "systolic_pressure_max", "systolic_pressure_min", "threshold", "timezone", + "total_calories", "type", "user_symptom", "value", "vo2max", "voltage_data", + "wakeup_count", "wakeup_time", "zone1_duration", "zone2_duration", "zone3_duration", + "zone4_duration", "zone5_duration", ) } } From d92f5b42beb4369055d362ac26cac497f19c52f4 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 18:16:35 +0000 Subject: [PATCH 28/44] Update continuous_exercise_intensity_v2_statistics for the new intensityMap schema RADAR-Schemas PR #430 replaced HuaweiContinuousExerciseIntensityV2Statistics's five fixed heart-rate-zone fields (zone1Duration..zone5Duration) with a single intensityMap field (Map of Huawei exercise type code to duration in minutes), matching what Huawei's API actually returns for this data type - not per-HR-zone durations. Adds FieldValues.getIntMap() to read a Map-typed field from the sampleSet response, and wires the route to populate intensityMap from the "intensity" field instead of the old per-zone keys. The wire shape for a map-typed field ("mapValue", by the same Value convention as the scalar types) is a best-effort guess pending live verification, same caveat as documented on FieldValues. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../org/radarbase/huawei/converter/FieldValues.kt | 15 ++++++++++++++- .../radarbase/huawei/route/HuaweiRouteFactory.kt | 9 ++++----- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt index 2bb49b49..7d147581 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt @@ -51,10 +51,23 @@ class FieldValues private constructor(private val values: Map) if (it.isNull) null else it.asText() } + /** + * Reads a `Map`-typed field (e.g. Huawei's exercise-type-to-duration map), + * as an object whose keys are stringified (Avro maps require string keys). Best-effort: the + * exact wire shape of a Huawei map-typed field is not confirmed against a live API response + * (Huawei's typed-value array uses `integerValue`/`floatValue`/`stringValue`/`longValue` for + * scalars, so `mapValue` is assumed by the same `Value` convention) - verify and adjust + * if this doesn't match what the API actually returns. + */ + fun getIntMap(field: String): Map? = values[field] + ?.takeIf { it.isObject } + ?.properties() + ?.associate { (key, value) -> key to value.asInt() } + companion object { private const val FIELD_NAME_KEY = "fieldName" private val VALUE_KEYS = - listOf("integerValue", "floatValue", "longValue", "stringValue", "value") + listOf("integerValue", "floatValue", "longValue", "stringValue", "mapValue", "value") fun from(node: JsonNode?): FieldValues { if (node == null || node.isMissingNode || node.isNull) { diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index abff6274..20b84fd5 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -531,11 +531,10 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - zone1Duration = f.getInt("zone1_duration") - zone2Duration = f.getInt("zone2_duration") - zone3Duration = f.getInt("zone3_duration") - zone4Duration = f.getInt("zone4_duration") - zone5Duration = f.getInt("zone5_duration") + // Huawei reports this as a single Map field ("intensity"), + // not per-heart-rate-zone durations - see FieldValues.getIntMap for the wire-shape + // caveat. + intensityMap = f.getIntMap("intensity") }.build() }, ) From 41251a2ad8066b1651277a26bc9020fdc024428a Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 20:22:17 +0000 Subject: [PATCH 29/44] Fix raw data type namespace for body/skin temperature and breathe rate statistics Per the official "Health Sampling" data type reference pages, three more ".statistics" routes have the same "continuous."-labelled statistics name vs. "instantaneous."-namespaced raw detailed type mismatch already confirmed and fixed for SpO2 and Altitude: - continuous_body_temperature_statistics's raw type is com.huawei.instantaneous.body.temperature, not com.huawei.continuous.body.temperature. - continuous_skin_temperature_statistics's raw type is com.huawei.instantaneous.skin.temperature. - continuous_breathe_rate_statistics's raw type is com.huawei.instantaneous.breathe_rate. The first two were previously built through the shared genericStatisticsTypes loop, which has no way to override the query data type, so they're pulled out into their own definitions (same HuaweiStatistics schema and populateCommon builder) alongside the explicit queryDataTypeSuffix override. Also confirmed (no code change needed): resting_calories_statistics already resolves correctly - com.huawei.resting_calories is a real, separately-documented raw data type, and continuous.resting_heart_rate.statistics's raw type (com.huawei.instantaneous.resting_heart_rate) already matches what our default suffix-stripping produces. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/route/HuaweiRouteFactory.kt | 52 +++++++++++++++---- 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 20b84fd5..b2f406c5 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -116,11 +116,6 @@ object HuaweiRouteFactory { "continuous.body.temperature.rest.statistics", "connect_huawei_continuous_body_temperature_rest_statistics", ), - Triple( - "continuous_body_temperature_statistics", - "continuous.body.temperature.statistics", - "connect_huawei_continuous_body_temperature_statistics", - ), Triple( "continuous_calories_bmr_statistics", "continuous.calories.bmr.statistics", @@ -141,11 +136,6 @@ object HuaweiRouteFactory { "continuous.power.statistics", "connect_huawei_continuous_power_statistics", ), - Triple( - "continuous_skin_temperature_statistics", - "continuous.skin.temperature.statistics", - "connect_huawei_continuous_skin_temperature_statistics", - ), Triple( "continuous_speed_statistics", "continuous.speed.statistics", @@ -338,6 +328,11 @@ object HuaweiRouteFactory { "continuous_breathe_rate_statistics", "continuous.breathe_rate.statistics", "connect_huawei_continuous_breathe_rate_statistics", + // Statistics variant is documented under "continuous.", but its underlying raw + // detailed data type is "com.huawei.instantaneous.breathe_rate" - the same + // namespace mismatch already confirmed for SpO2/Altitude. + queryDataTypeSuffix = "instantaneous.breathe_rate", + useDailyPolymerize = true, ) { f, start, end, received -> HuaweiContinuousBreatheRateStatistics.newBuilder().apply { time = start.toEpoch() @@ -391,6 +386,43 @@ object HuaweiRouteFactory { ) } + // Statistics variants whose "continuous."-labelled name doesn't match their underlying + // raw detailed data type's namespace (it's "instantaneous." instead) - the same mismatch + // already confirmed live for SpO2 and Altitude - so they need an explicit + // queryDataTypeSuffix override rather than the generic loop above. + listOf( + Triple( + "continuous_body_temperature_statistics", + "continuous.body.temperature.statistics", + "connect_huawei_continuous_body_temperature_statistics", + ) to "instantaneous.body.temperature", + Triple( + "continuous_skin_temperature_statistics", + "continuous.skin.temperature.statistics", + "connect_huawei_continuous_skin_temperature_statistics", + ) to "instantaneous.skin.temperature", + ).forEach { (definition, rawDataType) -> + val (key, dataType, topic) = definition + add( + sampleSetDefinition( + key, + dataType, + topic, + queryDataTypeSuffix = rawDataType, + useDailyPolymerize = true, + ) { f, start, end, received -> + HuaweiStatistics.newBuilder().apply { + populateCommon( + start, + end, + received, + f, + ) + }.build() + }, + ) + } + add( sampleSetDefinition( "continuous_calories_burnt", From da3ba83f59a2386c4226b72b823590386a34db50 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 20:24:31 +0000 Subject: [PATCH 30/44] Stop reading fields that don't exist on each cgm_blood_glucose variant Per the official "Blood Glucose" data type reference: the raw com.huawei.cgm_blood_glucose type only has a "level" field, while its .statistics variant only has avg/max/min/last (no "level"). Both builders were reading all five keys regardless, so each always left some fields silently null - harmless, but misleading. Each builder now only reads the fields its own data type actually returns. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt | 5 ----- 1 file changed, 5 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index b2f406c5..7ba5b19d 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -186,10 +186,6 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() level = f.getDouble("level") - avg = f.getInt("avg") - max = f.getInt("max") - min = f.getInt("min") - last = f.getInt("last") }.build() }, ) @@ -202,7 +198,6 @@ object HuaweiRouteFactory { HuaweiCgmBloodGlucose.newBuilder().apply { time = start.toEpoch() timeReceived = received.toEpoch() - level = f.getDouble("level") avg = f.getInt("avg") max = f.getInt("max") min = f.getInt("min") From cd37306afbbd209bc6afb29d32eaa2a3b38d4aa9 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 20:32:11 +0000 Subject: [PATCH 31/44] Fix health.record.* field keys to camelCase (not snake_case) Per the official "Low SpO2 Level Data", "ABPM Reports", and "High Body Temperature Data" references, every health.record.* data type uses camelCase field keys matching the Avro field names directly - the same convention already confirmed for daily_activity_summary and active_hours.statistics, and distinct from the snake_case convention used by continuous.*/instantaneous.* sampleSet types. Confirmed and fixed directly from these docs: - health_record_low_spo2_alert: maxSpO2/minSpO2 (was max_spo2/min_spo2) - health_record_hyperthermia: highBodyTemperatureAlarm (was high_body_temperature_alarm) - health_record_dynamic_bp (the ~90-field ABPM record): the entire field mapper was routing every lookup through a camelCase->snake_case "snake()" conversion, which is now removed - every one of its ~90 fields was being queried under the wrong key. Also fixes extendData (was extend_data). Given the now 5-for-5 confirmed pattern across every health.record.* type actually checked (also daily_activity_summary, active_hours.statistics from earlier), also applies the same camelCase correction to the remaining health.record.* routes not covered by these specific docs (bradycardia/tachycardia's shared heart-rate-alert builder, menstrual_cycle, sleep) - high confidence by the established pattern, though not individually doc-verified like the three above. Removes the now-dead snake() helper (in both the main file and the test's dynamicBpFieldKeys()) since nothing converts case anymore. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/route/HuaweiRouteFactory.kt | 68 ++++++++----------- .../huawei/route/HuaweiRouteFactoryTest.kt | 41 +++++------ 2 files changed, 48 insertions(+), 61 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 7ba5b19d..73f7f66a 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -80,14 +80,6 @@ object HuaweiRouteFactory { private fun Instant.toEpoch(): Double = toEpochMilli() / 1000.0 - /** Best-effort camelCase -> snake_case conversion for deriving a Huawei field key from an Avro field name. */ - private fun snake(name: String): String = - SNAKE_CASE_BOUNDARY.replace( - name, - ) { "${it.groupValues[1]}_${it.groupValues[2]}" }.lowercase() - - private val SNAKE_CASE_BOUNDARY = Regex("([a-z0-9])([A-Z])") - private fun HuaweiStatistics.Builder.populateCommon( startTime: Instant, endTime: Instant?, @@ -687,7 +679,7 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - highBodyTemperatureAlarm = f.getFloat("high_body_temperature_alarm") + highBodyTemperatureAlarm = f.getFloat("highBodyTemperatureAlarm") }.build() }, ) @@ -702,8 +694,8 @@ object HuaweiRouteFactory { timeReceived = received.toEpoch() endTime = end?.toEpoch() threshold = f.getFloat("threshold") - maxSpO2 = f.getFloat("max_spo2") - minSpO2 = f.getFloat("min_spo2") + maxSpO2 = f.getFloat("maxSpO2") + minSpO2 = f.getFloat("minSpO2") }.build() }, ) @@ -717,11 +709,11 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - recordday = f.getInt("record_day") + recordday = f.getInt("recordDay") status = f.getInt("status") - substatus = f.getInt("sub_status") + substatus = f.getInt("subStatus") remarks = f.getString("remarks") - timezone = f.getString("timezone") + timezone = f.getString("timeZone") }.build() }, ) @@ -735,22 +727,22 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - fallAsleepTime = f.getLong("fall_asleep_time") - wakeupTime = f.getLong("wakeup_time") - lightSleepTime = f.getInt("light_sleep_time") - deepSleepTime = f.getInt("deep_sleep_time") - dreamTime = f.getInt("dream_time") - awakeTime = f.getInt("awake_time") - allSleepTime = f.getInt("all_sleep_time") - wakeupCount = f.getInt("wakeup_count") - deepSleepPart = f.getInt("deep_sleep_part") - sleepScore = f.getInt("sleep_score") - sleepLatency = f.getInt("sleep_latency") - sleepEfficiency = f.getInt("sleep_efficiency") - goBedTime = f.getLong("go_bed_time") - sleepType = f.getInt("sleep_type") - prepareSleepTime = f.getLong("prepare_sleep_time") - offBedTime = f.getLong("off_bed_time") + fallAsleepTime = f.getLong("fallAsleepTime") + wakeupTime = f.getLong("wakeupTime") + lightSleepTime = f.getInt("lightSleepTime") + deepSleepTime = f.getInt("deepSleepTime") + dreamTime = f.getInt("dreamTime") + awakeTime = f.getInt("awakeTime") + allSleepTime = f.getInt("allSleepTime") + wakeupCount = f.getInt("wakeupCount") + deepSleepPart = f.getInt("deepSleepPart") + sleepScore = f.getInt("sleepScore") + sleepLatency = f.getInt("sleepLatency") + sleepEfficiency = f.getInt("sleepEfficiency") + goBedTime = f.getLong("goBedTime") + sleepType = f.getInt("sleepType") + prepareSleepTime = f.getLong("prepareSleepTime") + offBedTime = f.getLong("offBedTime") }.build() }, ) @@ -907,9 +899,9 @@ object HuaweiRouteFactory { timeReceived = received.toEpoch() endTime = end?.toEpoch() threshold = getDouble("threshold") - avgHeartRate = getDouble("avg_heart_rate") - maxHeartRate = getDouble("max_heart_rate") - minHeartRate = getDouble("min_heart_rate") + avgHeartRate = getDouble("avgHeartRate") + maxHeartRate = getDouble("maxHeartRate") + minHeartRate = getDouble("minHeartRate") }.build() /** @@ -924,14 +916,14 @@ object HuaweiRouteFactory { received: Instant, ): HuaweiHealthRecordDynamicBp { val f = this - fun i(name: String) = f.getInt(snake(name)) - fun d(name: String) = f.getDouble(snake(name)) - fun l(name: String) = f.getLong(snake(name)) + fun i(name: String) = f.getInt(name) + fun d(name: String) = f.getDouble(name) + fun l(name: String) = f.getLong(name) return HuaweiHealthRecordDynamicBp.newBuilder().apply { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - planId = f.getString(snake("planId")) + planId = f.getString("planId") planStartTime = l("planStartTime") planEndTime = l("planEndTime") planActualTime = l("planActualTime") @@ -1034,7 +1026,7 @@ object HuaweiRouteFactory { coefDiastolicBpWakeTwo = d("coefDiastolicBpWakeTwo") coefHeartRateWakeTwo = d("coefHeartRateWakeTwo") - extendData = f.getString("extend_data") + extendData = f.getString("extendData") }.build() } diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 53355c7b..29c7cfb7 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -182,12 +182,6 @@ class HuaweiRouteFactoryTest { (HuaweiHealthRecordDynamicBp::class.java.getField("SCHEMA$").get(null) as Schema).fields .map { it.name() } .filterNot { it in setOf("time", "timeReceived", "endTime") } - .map(::snake) - - private fun snake(name: String): String = - Regex("([a-z0-9])([A-Z])") - .replace(name) { "${it.groupValues[1]}_${it.groupValues[2]}" } - .lowercase() companion object { private const val START_MILLIS = 1704067200000L // 2024-01-01T00:00:00Z @@ -197,26 +191,27 @@ class HuaweiRouteFactoryTest { private val LITERAL_FIELD_KEYS = listOf( "active_hours", "active_hours_target", "activeCalories", "activeHours", - "activeCaloriesGoal", "activeHoursGoal", "all_sleep_time", "arrhythmia_result", + "activeCaloriesGoal", "activeHoursGoal", "allSleepTime", "arrhythmia_result", "arrhythmia_type", "ascent_total", "avg", "avg_breathe_rate", "avg_heart_rate", - "awake_time", "calories", "calories_target", "calories_total", "correlate_mealtime", - "correlate_sleep", "count", "deep_sleep_part", "deep_sleep_time", "descent_total", - "diastolic_pressure_avg", "diastolic_pressure_max", "diastolic_pressure_min", - "distance", "distance_delta", "distance_total", "dream_time", "duration", "emotion", - "event_name", "exercise_time", "exercise_time_target", "exerciseTime", - "exerciseTimeGoal", "exercise_type", "extend_data", "fall_asleep_time", "go_bed_time", - "heart_rate_variability_rmssd", "high_body_temperature_alarm", "isActive", "last", - "level", "light_sleep_time", "max", "max_breathe_rate", "max_breathrate_baseline", - "max_spo2", "meal", "min", "min_breathe_rate", "min_breathrate_baseline", "min_spo2", - "off_bed_time", "on_off_bed_state", "predicted_calories", "prepare_sleep_time", - "record_day", "record_id", "remarks", "sample_source", "sampling_frequency", - "sleep_efficiency", "saturation_avg", "saturation_last", "saturation_max", - "saturation_min", "sleep_latency", "sleep_score", "sleep_state", "sleep_type", + "avgHeartRate", "awakeTime", "calories", "calories_target", "calories_total", + "correlate_mealtime", "correlate_sleep", "count", "deepSleepPart", "deepSleepTime", + "descent_total", "diastolic_pressure_avg", "diastolic_pressure_max", + "diastolic_pressure_min", "distance", "distance_delta", "distance_total", + "dreamTime", "duration", "emotion", "event_name", "exercise_time", + "exercise_time_target", "exerciseTime", "exerciseTimeGoal", "exercise_type", + "extendData", "fallAsleepTime", "goBedTime", "heart_rate_variability_rmssd", + "highBodyTemperatureAlarm", "isActive", "last", "level", "lightSleepTime", "max", + "max_breathe_rate", "max_breathrate_baseline", "maxHeartRate", "maxSpO2", "meal", + "min", "min_breathe_rate", "min_breathrate_baseline", "minHeartRate", "minSpO2", + "offBedTime", "on_off_bed_state", "predicted_calories", "prepareSleepTime", + "recordDay", "record_id", "remarks", "sample_source", "sampling_frequency", + "sleepEfficiency", "saturation_avg", "saturation_last", "saturation_max", + "saturation_min", "sleepLatency", "sleepScore", "sleep_state", "sleepType", "sphygmus_avg", "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", - "steps_delta", "steps_target", "stepsGoal", "sub_status", "systolic_pressure_avg", - "systolic_pressure_max", "systolic_pressure_min", "threshold", "timezone", + "steps_delta", "steps_target", "stepsGoal", "subStatus", "systolic_pressure_avg", + "systolic_pressure_max", "systolic_pressure_min", "threshold", "timeZone", "total_calories", "type", "user_symptom", "value", "vo2max", "voltage_data", - "wakeup_count", "wakeup_time", "zone1_duration", "zone2_duration", "zone3_duration", + "wakeupCount", "wakeupTime", "zone1_duration", "zone2_duration", "zone3_duration", "zone4_duration", "zone5_duration", ) } From 643326ca51d6948ca646bcb5a90a1d7929780c45 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 20:39:14 +0000 Subject: [PATCH 32/44] Revert unconfirmed camelCase guess for health_record_sleep and others The official "Sleep Records" reference confirms com.huawei.health.record.sleep actually uses snake_case field keys (fall_asleep_time, wakeup_time, sleep_type, etc.), contradicting the "all health.record.* types use camelCase" pattern inferred from lowSpo2Alert/dynamic_bp/hyperthermia in the previous commit. That inference doesn't hold universally, so also reverts the other two changes that were extended from it without direct doc evidence (health_record_bradycardia/tachycardia's shared heart-rate-alert builder, health_record_menstrual_cycle) back to their original snake_case keys, pending actual documentation for those. The three fixes with direct doc confirmation (lowSpo2Alert/dynamic_bp/hyperthermia, and daily_activity_summary/ active_hours.statistics from earlier) are unaffected. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/route/HuaweiRouteFactory.kt | 44 +++++++++---------- .../huawei/route/HuaweiRouteFactoryTest.kt | 38 ++++++++-------- 2 files changed, 41 insertions(+), 41 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 73f7f66a..7ec7b783 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -709,11 +709,11 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - recordday = f.getInt("recordDay") + recordday = f.getInt("record_day") status = f.getInt("status") - substatus = f.getInt("subStatus") + substatus = f.getInt("sub_status") remarks = f.getString("remarks") - timezone = f.getString("timeZone") + timezone = f.getString("timezone") }.build() }, ) @@ -727,22 +727,22 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - fallAsleepTime = f.getLong("fallAsleepTime") - wakeupTime = f.getLong("wakeupTime") - lightSleepTime = f.getInt("lightSleepTime") - deepSleepTime = f.getInt("deepSleepTime") - dreamTime = f.getInt("dreamTime") - awakeTime = f.getInt("awakeTime") - allSleepTime = f.getInt("allSleepTime") - wakeupCount = f.getInt("wakeupCount") - deepSleepPart = f.getInt("deepSleepPart") - sleepScore = f.getInt("sleepScore") - sleepLatency = f.getInt("sleepLatency") - sleepEfficiency = f.getInt("sleepEfficiency") - goBedTime = f.getLong("goBedTime") - sleepType = f.getInt("sleepType") - prepareSleepTime = f.getLong("prepareSleepTime") - offBedTime = f.getLong("offBedTime") + fallAsleepTime = f.getLong("fall_asleep_time") + wakeupTime = f.getLong("wakeup_time") + lightSleepTime = f.getInt("light_sleep_time") + deepSleepTime = f.getInt("deep_sleep_time") + dreamTime = f.getInt("dream_time") + awakeTime = f.getInt("awake_time") + allSleepTime = f.getInt("all_sleep_time") + wakeupCount = f.getInt("wakeup_count") + deepSleepPart = f.getInt("deep_sleep_part") + sleepScore = f.getInt("sleep_score") + sleepLatency = f.getInt("sleep_latency") + sleepEfficiency = f.getInt("sleep_efficiency") + goBedTime = f.getLong("go_bed_time") + sleepType = f.getInt("sleep_type") + prepareSleepTime = f.getLong("prepare_sleep_time") + offBedTime = f.getLong("off_bed_time") }.build() }, ) @@ -899,9 +899,9 @@ object HuaweiRouteFactory { timeReceived = received.toEpoch() endTime = end?.toEpoch() threshold = getDouble("threshold") - avgHeartRate = getDouble("avgHeartRate") - maxHeartRate = getDouble("maxHeartRate") - minHeartRate = getDouble("minHeartRate") + avgHeartRate = getDouble("avg_heart_rate") + maxHeartRate = getDouble("max_heart_rate") + minHeartRate = getDouble("min_heart_rate") }.build() /** diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 29c7cfb7..e23d3091 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -191,27 +191,27 @@ class HuaweiRouteFactoryTest { private val LITERAL_FIELD_KEYS = listOf( "active_hours", "active_hours_target", "activeCalories", "activeHours", - "activeCaloriesGoal", "activeHoursGoal", "allSleepTime", "arrhythmia_result", + "activeCaloriesGoal", "activeHoursGoal", "all_sleep_time", "arrhythmia_result", "arrhythmia_type", "ascent_total", "avg", "avg_breathe_rate", "avg_heart_rate", - "avgHeartRate", "awakeTime", "calories", "calories_target", "calories_total", - "correlate_mealtime", "correlate_sleep", "count", "deepSleepPart", "deepSleepTime", - "descent_total", "diastolic_pressure_avg", "diastolic_pressure_max", - "diastolic_pressure_min", "distance", "distance_delta", "distance_total", - "dreamTime", "duration", "emotion", "event_name", "exercise_time", - "exercise_time_target", "exerciseTime", "exerciseTimeGoal", "exercise_type", - "extendData", "fallAsleepTime", "goBedTime", "heart_rate_variability_rmssd", - "highBodyTemperatureAlarm", "isActive", "last", "level", "lightSleepTime", "max", - "max_breathe_rate", "max_breathrate_baseline", "maxHeartRate", "maxSpO2", "meal", - "min", "min_breathe_rate", "min_breathrate_baseline", "minHeartRate", "minSpO2", - "offBedTime", "on_off_bed_state", "predicted_calories", "prepareSleepTime", - "recordDay", "record_id", "remarks", "sample_source", "sampling_frequency", - "sleepEfficiency", "saturation_avg", "saturation_last", "saturation_max", - "saturation_min", "sleepLatency", "sleepScore", "sleep_state", "sleepType", - "sphygmus_avg", "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", - "steps_delta", "steps_target", "stepsGoal", "subStatus", "systolic_pressure_avg", - "systolic_pressure_max", "systolic_pressure_min", "threshold", "timeZone", + "awake_time", "calories", "calories_target", "calories_total", "correlate_mealtime", + "correlate_sleep", "count", "deep_sleep_part", "deep_sleep_time", "descent_total", + "diastolic_pressure_avg", "diastolic_pressure_max", "diastolic_pressure_min", + "distance", "distance_delta", "distance_total", "dream_time", "duration", "emotion", + "event_name", "exercise_time", "exercise_time_target", "exerciseTime", + "exerciseTimeGoal", "exercise_type", "extendData", "fall_asleep_time", "go_bed_time", + "heart_rate_variability_rmssd", "highBodyTemperatureAlarm", "isActive", "last", + "level", "light_sleep_time", "max", "max_breathe_rate", "max_breathrate_baseline", + "max_heart_rate", "maxSpO2", "meal", "min", "min_breathe_rate", + "min_breathrate_baseline", "min_heart_rate", "minSpO2", "off_bed_time", + "on_off_bed_state", "predicted_calories", "prepare_sleep_time", "record_day", + "record_id", "remarks", "sample_source", "sampling_frequency", "sleep_efficiency", + "saturation_avg", "saturation_last", "saturation_max", "saturation_min", + "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "sphygmus_avg", + "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", "steps_delta", + "steps_target", "stepsGoal", "sub_status", "systolic_pressure_avg", + "systolic_pressure_max", "systolic_pressure_min", "threshold", "timezone", "total_calories", "type", "user_symptom", "value", "vo2max", "voltage_data", - "wakeupCount", "wakeupTime", "zone1_duration", "zone2_duration", "zone3_duration", + "wakeup_count", "wakeup_time", "zone1_duration", "zone2_duration", "zone3_duration", "zone4_duration", "zone5_duration", ) } From 4a807843c80c3476417058347180e73bc9a8987d Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 20:42:17 +0000 Subject: [PATCH 33/44] Confirm health_record_menstrual_cycle uses camelCase field keys Per the official "Menstrual Cycle Data" reference: recordDay, status, subStatus, timeZone, remarks - confirms and re-applies the camelCase fix reverted in the previous commit for lack of evidence at the time. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt | 6 +++--- .../org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 7ec7b783..638b0078 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -709,11 +709,11 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - recordday = f.getInt("record_day") + recordday = f.getInt("recordDay") status = f.getInt("status") - substatus = f.getInt("sub_status") + substatus = f.getInt("subStatus") remarks = f.getString("remarks") - timezone = f.getString("timezone") + timezone = f.getString("timeZone") }.build() }, ) diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index e23d3091..2292db08 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -203,13 +203,13 @@ class HuaweiRouteFactoryTest { "level", "light_sleep_time", "max", "max_breathe_rate", "max_breathrate_baseline", "max_heart_rate", "maxSpO2", "meal", "min", "min_breathe_rate", "min_breathrate_baseline", "min_heart_rate", "minSpO2", "off_bed_time", - "on_off_bed_state", "predicted_calories", "prepare_sleep_time", "record_day", + "on_off_bed_state", "predicted_calories", "prepare_sleep_time", "recordDay", "record_id", "remarks", "sample_source", "sampling_frequency", "sleep_efficiency", "saturation_avg", "saturation_last", "saturation_max", "saturation_min", "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "sphygmus_avg", "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", "steps_delta", - "steps_target", "stepsGoal", "sub_status", "systolic_pressure_avg", - "systolic_pressure_max", "systolic_pressure_min", "threshold", "timezone", + "steps_target", "stepsGoal", "subStatus", "systolic_pressure_avg", + "systolic_pressure_max", "systolic_pressure_min", "threshold", "timeZone", "total_calories", "type", "user_symptom", "value", "vo2max", "voltage_data", "wakeup_count", "wakeup_time", "zone1_duration", "zone2_duration", "zone3_duration", "zone4_duration", "zone5_duration", From 0964fff184f8122422ca91ee09329e633672bc58 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 20:51:55 +0000 Subject: [PATCH 34/44] Align Huawei field keys and raw data types with official references - Use documented field keys: emotionStatus, heartRateVariabilityRMSSD, onOffBedState, eventName, voltage_datas, camelCase breathe-rate and resting-calories statistics, measure_count for stress statistics. - Query heart rate, blood pressure and blood glucose statistics via their instantaneous.* raw types; read body fat statistics from instantaneous.body_weight's *_body_fat_rate fields. - Disable continuous_calories_consumed by default (rejected live, not a documented data type). - FieldValues: accept fallback keys, serialize list/object values in getString, tolerate unknown typed-value keys and more map shapes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../radarbase/huawei/converter/FieldValues.kt | 62 ++++++++--- .../huawei/route/HuaweiRouteFactory.kt | 104 ++++++++++++------ .../huawei/converter/FieldValuesTest.kt | 37 +++++++ .../huawei/route/HuaweiRouteFactoryTest.kt | 42 ++++--- 4 files changed, 174 insertions(+), 71 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt index 7d147581..0513835f 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt @@ -37,18 +37,23 @@ import com.fasterxml.jackson.databind.JsonNode */ class FieldValues private constructor(private val values: Map) { - fun getInt(field: String): Int? = values[field]?.let { if (it.isNull) null else it.asInt() } + /* + * Every accessor accepts one or more candidate keys and returns the first one present, so a + * field whose casing Huawei's docs don't pin down unambiguously can list both spellings. + */ - fun getLong(field: String): Long? = values[field]?.let { if (it.isNull) null else it.asLong() } + fun getInt(vararg fields: String): Int? = scalar(fields)?.asInt() - fun getDouble(field: String): Double? = values[field]?.let { - if (it.isNull) null else it.asDouble() - } + fun getLong(vararg fields: String): Long? = scalar(fields)?.asLong() + + fun getDouble(vararg fields: String): Double? = scalar(fields)?.asDouble() - fun getFloat(field: String): Float? = getDouble(field)?.toFloat() + fun getFloat(vararg fields: String): Float? = getDouble(*fields)?.toFloat() - fun getString(field: String): String? = values[field]?.let { - if (it.isNull) null else it.asText() + /** Textual fields are returned as-is; array/object-valued fields (e.g. the ECG voltage sample + * list) are returned as their JSON serialization rather than Jackson's empty `asText()`. */ + fun getString(vararg fields: String): String? = lookup(fields)?.let { + if (it.isContainerNode) it.toString() else it.asText() } /** @@ -56,19 +61,44 @@ class FieldValues private constructor(private val values: Map) * as an object whose keys are stringified (Avro maps require string keys). Best-effort: the * exact wire shape of a Huawei map-typed field is not confirmed against a live API response * (Huawei's typed-value array uses `integerValue`/`floatValue`/`stringValue`/`longValue` for - * scalars, so `mapValue` is assumed by the same `Value` convention) - verify and adjust - * if this doesn't match what the API actually returns. + * scalars, so `mapValue` is assumed by the same `Value` convention). Accepts a plain + * `{"key": 1}` object, an object of typed values `{"key": {"integerValue": 1}}`, or an array of + * `{"key": ..., "value": ...}` entries. */ - fun getIntMap(field: String): Map? = values[field] - ?.takeIf { it.isObject } - ?.properties() - ?.associate { (key, value) -> key to value.asInt() } + fun getIntMap(vararg fields: String): Map? { + val node = lookup(fields) ?: return null + return when { + node.isObject -> node.properties().mapNotNull { (key, value) -> + unwrap(value)?.let { key to it.asInt() } + }.toMap() + node.isArray -> node.mapNotNull { entry -> + val key = entry.get("key")?.takeUnless { it.isNull }?.asText() + ?: return@mapNotNull null + unwrap(entry.get("value"))?.let { key to it.asInt() } + }.toMap() + else -> null + } + } + + private fun lookup(fields: Array): JsonNode? = + fields.firstNotNullOfOrNull { field -> values[field]?.takeUnless { it.isNull } } + + private fun scalar(fields: Array): JsonNode? = + lookup(fields)?.takeIf { it.isValueNode } companion object { private const val FIELD_NAME_KEY = "fieldName" private val VALUE_KEYS = listOf("integerValue", "floatValue", "longValue", "stringValue", "mapValue", "value") + /** Unwraps a typed-value wrapper (`{"integerValue": 1}`) to its value; returns other + * nodes unchanged. */ + private fun unwrap(node: JsonNode?): JsonNode? { + if (node == null || node.isNull || node.isMissingNode) return null + if (!node.isObject) return node + return VALUE_KEYS.firstNotNullOfOrNull { key -> node.get(key) } ?: node + } + fun from(node: JsonNode?): FieldValues { if (node == null || node.isMissingNode || node.isNull) { return FieldValues(emptyMap()) @@ -77,7 +107,11 @@ class FieldValues private constructor(private val values: Map) val map = LinkedHashMap() node.forEach { entry -> val name = entry.get(FIELD_NAME_KEY)?.asText() ?: return@forEach + // Prefer the known typed-value keys; fall back to whatever other single + // property the entry carries, in case Huawei uses a type name not listed here. val value = VALUE_KEYS.firstNotNullOfOrNull { key -> entry.get(key) } + ?: entry.properties().firstOrNull { (key, _) -> key != FIELD_NAME_KEY } + ?.value if (value != null) { map[name] = value } diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 638b0078..658f5656 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -65,12 +65,10 @@ import java.time.Instant * specification's `doc` strings (prefixed with the vendor namespace `com.huawei.`), which in turn * describe the Huawei Health Kit REST Data API's own data type identifiers. * - * Field-value key names used in the record builders are Huawei Health Kit `Field` identifiers - * (snake_case, matching the on-device HiHealth SDK's public `Field.FIELD_*` constant family, e.g. - * `steps_delta`, `calories`, `avg`/`max`/`min`/`last`). Where a field is not among Huawei's widely - * documented constants, the snake_case form of the Avro field's own name is used as a best-effort - * default (see [snake]) — verify against a live API response and adjust the key strings in this - * file if Huawei's actual response uses different names. + * Field-value key names used in the record builders are the `fieldName`s documented in Huawei's + * per-data-type references. Their casing is not consistent across data types (e.g. `steps_delta` + * and `sleep_state` but `emotionStatus` and `onOffBedState`), so each key is taken from its own data + * type's reference; keys not documented there are marked best-effort. * * @author yatharthranjan */ @@ -93,16 +91,12 @@ object HuaweiRouteFactory { max = fields.getDouble("max") min = fields.getDouble("min") last = fields.getDouble("last") - count = fields.getInt("count") + // Stress statistics documents its count as "measure_count". + count = fields.getInt("count", "measure_count") } /** Data types that reuse the generic [HuaweiStatistics] schema: (config key, Huawei data type name, default topic). */ private val genericStatisticsTypes = listOf( - Triple( - "continuous_body_fat_rate_statistics", - "continuous.body.fat.rate.statistics", - "connect_huawei_continuous_body_fat_rate_statistics", - ), Triple( "continuous_body_temperature_rest_statistics", "continuous.body.temperature.rest.statistics", @@ -118,11 +112,6 @@ object HuaweiRouteFactory { "continuous.exercise_heart_rate.statistics", "connect_huawei_continuous_exercise_heart_rate_statistics", ), - Triple( - "continuous_heart_rate_statistics", - "continuous.heart_rate.statistics", - "connect_huawei_continuous_heart_rate_statistics", - ), Triple( "continuous_power_statistics", "continuous.power.statistics", @@ -294,6 +283,9 @@ object HuaweiRouteFactory { "continuous_blood_glucose_statistics", "continuous.blood_glucose.statistics", "connect_huawei_continuous_blood_glucose_statistics", + // Raw detailed type is "com.huawei.instantaneous.blood_glucose" per the official + // "Blood Glucose" reference. + queryDataTypeSuffix = "instantaneous.blood_glucose", ) { f, start, end, received -> HuaweiContinuousBloodGlucoseStatistics.newBuilder().apply { time = start.toEpoch() @@ -325,11 +317,12 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - maxBreatheRate = f.getInt("max_breathe_rate") - minBreatheRate = f.getInt("min_breathe_rate") - avgBreatheRate = f.getInt("avg_breathe_rate") - minBreathrateBaseline = f.getInt("min_breathrate_baseline") - maxBreathrateBaseline = f.getInt("max_breathrate_baseline") + // camelCase per the official "Respiratory Rate" data type reference. + maxBreatheRate = f.getInt("maxBreatheRate") + minBreatheRate = f.getInt("minBreatheRate") + avgBreatheRate = f.getInt("avgBreatheRate") + minBreathrateBaseline = f.getInt("minBreathrateBaseline") + maxBreathrateBaseline = f.getInt("maxBreathrateBaseline") }.build() }, ) @@ -339,6 +332,9 @@ object HuaweiRouteFactory { "continuous_body_blood_pressure_statistics", "continuous.body.blood_pressure.statistics", "connect_huawei_continuous_body_blood_pressure_statistics", + // Raw detailed type is "com.huawei.instantaneous.blood_pressure" per the official + // "Blood Pressure" reference. + queryDataTypeSuffix = "instantaneous.blood_pressure", ) { f, start, end, received -> HuaweiContinuousBodyBloodPressureStatistics.newBuilder().apply { time = start.toEpoch() @@ -374,10 +370,15 @@ object HuaweiRouteFactory { } // Statistics variants whose "continuous."-labelled name doesn't match their underlying - // raw detailed data type's namespace (it's "instantaneous." instead) - the same mismatch - // already confirmed live for SpO2 and Altitude - so they need an explicit - // queryDataTypeSuffix override rather than the generic loop above. + // raw detailed data type's namespace (it's "instantaneous." instead, per the official + // Health Sampling references) - so they need an explicit queryDataTypeSuffix override + // rather than the generic loop above. listOf( + Triple( + "continuous_heart_rate_statistics", + "continuous.heart_rate.statistics", + "connect_huawei_continuous_heart_rate_statistics", + ) to "instantaneous.heart_rate", Triple( "continuous_body_temperature_statistics", "continuous.body.temperature.statistics", @@ -410,6 +411,29 @@ object HuaweiRouteFactory { ) } + add( + sampleSetDefinition( + "continuous_body_fat_rate_statistics", + "continuous.body.fat.rate.statistics", + "connect_huawei_continuous_body_fat_rate_statistics", + // Huawei has no separate body fat data type: per the official "Weight" reference, + // body fat percentage is part of com.huawei.instantaneous.body_weight, whose daily + // statistics carry it as avg/max/min_body_fat_rate (avg/max/min/last there are the + // weight itself). + queryDataTypeSuffix = "instantaneous.body_weight", + useDailyPolymerize = true, + ) { f, start, end, received -> + HuaweiStatistics.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + endTime = end?.toEpoch() + avg = f.getDouble("avg_body_fat_rate") + max = f.getDouble("max_body_fat_rate") + min = f.getDouble("min_body_fat_rate") + }.build() + }, + ) + add( sampleSetDefinition( "continuous_calories_burnt", @@ -429,6 +453,10 @@ object HuaweiRouteFactory { "continuous_calories_consumed", "continuous.calories.consumed", "connect_huawei_continuous_calories_consumed", + // Rejected live with "no default dataCollector found for: + // com.huawei.continuous.calories.consumed" and absent from Huawei's data type + // references, so disabled unless explicitly enabled. + enabledByDefault = false, ) { f, start, end, received -> HuaweiContinuousCaloriesBurnt.newBuilder().apply { time = start.toEpoch() @@ -496,13 +524,15 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() + // The official "ECG" reference only documents "ecg_type" and "voltage_datas" + // (a list, serialized here as JSON); the remaining keys are best-effort. ecgRecordId = f.getString("record_id") averageHeartRate = f.getInt("avg_heart_rate") ecgArrhythmiaType = f.getInt("arrhythmia_type") ecgArrhythmiaResult = f.getInt("arrhythmia_result") userSymptom = f.getString("user_symptom") samplingFrequency = f.getInt("sampling_frequency") - voltageData = f.getString("voltage_data") + voltageData = f.getString("voltage_datas", "voltage_data") }.build() }, ) @@ -637,7 +667,7 @@ object HuaweiRouteFactory { HuaweiEmotion.newBuilder().apply { time = start.toEpoch() timeReceived = received.toEpoch() - emotionStatus = f.getInt("emotion") + emotionStatus = f.getInt("emotionStatus") }.build() }, ) @@ -756,7 +786,12 @@ object HuaweiRouteFactory { HuaweiHeartRateVariability.newBuilder().apply { time = start.toEpoch() timeReceived = received.toEpoch() - heartRateVariabilityRmssd = f.getInt("heart_rate_variability_rmssd") + // The official reference's field column is truncated to "...tRateVariabilityRMSSD"; + // the doc's value range is (0, 200] ms, so fractional values are truncated. + heartRateVariabilityRmssd = f.getInt( + "heartRateVariabilityRMSSD", + "heartRateVariabilityRmssd", + ) }.build() }, ) @@ -771,8 +806,8 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - predictedCalories = f.getFloat("predicted_calories") - totalCalories = f.getFloat("total_calories") + predictedCalories = f.getFloat("predictedCalories") + totalCalories = f.getFloat("totalCalories") }.build() }, ) @@ -786,7 +821,7 @@ object HuaweiRouteFactory { HuaweiSleepOnOffBedRecord.newBuilder().apply { time = start.toEpoch() timeReceived = received.toEpoch() - onOffBedState = f.getInt("on_off_bed_state") + onOffBedState = f.getInt("onOffBedState") }.build() }, ) @@ -816,7 +851,7 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - eventname = f.getInt("event_name") + eventname = f.getInt("eventName") }.build() }, ) @@ -906,9 +941,8 @@ object HuaweiRouteFactory { /** * The 24h ambulatory blood pressure monitoring record has ~80 numeric fields, all following - * the same `` naming (e.g. `avgSystolicBpAll`, `maxHeartRateWake`). - * [snake] derives each Huawei field key mechanically from the Avro field name to avoid - * hand-transcribing ~80 near-identical key strings. + * the same `` naming (e.g. `avgSystolicBpAll`, `maxHeartRateWake`), which + * Huawei uses verbatim as its camelCase field keys. */ private fun FieldValues.toHealthRecordDynamicBp( start: Instant, diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt index 428f5065..9bb24183 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt @@ -57,6 +57,43 @@ class FieldValuesTest { assertNull(fields.getInt("min")) } + @Test + fun `falls back to later candidate keys and serializes list values`() { + val node = mapper.readTree( + """ + [ + {"fieldName": "heartRateVariabilityRmssd", "integerValue": 42}, + {"fieldName": "voltage_datas", "value": [1.5, -2.0]}, + {"fieldName": "custom", "doubleValue": 7.5} + ] + """.trimIndent(), + ) + val fields = FieldValues.from(node) + + assertEquals(42, fields.getInt("heartRateVariabilityRMSSD", "heartRateVariabilityRmssd")) + assertEquals("[1.5,-2.0]", fields.getString("voltage_datas")) + assertNull(fields.getInt("voltage_datas")) + assertEquals(7.5, fields.getDouble("custom")) + } + + @Test + fun `parses map-typed values`() { + val node = mapper.readTree( + """ + [ + {"fieldName": "plain", "mapValue": {"1": 10, "2": 20}}, + {"fieldName": "typed", "mapValue": {"1": {"integerValue": 5}}}, + {"fieldName": "entries", "mapValue": [{"key": "3", "value": {"integerValue": 7}}]} + ] + """.trimIndent(), + ) + val fields = FieldValues.from(node) + + assertEquals(mapOf("1" to 10, "2" to 20), fields.getIntMap("plain")) + assertEquals(mapOf("1" to 5), fields.getIntMap("typed")) + assertEquals(mapOf("3" to 7), fields.getIntMap("entries")) + } + @Test fun `handles missing or null root node`() { val fields = FieldValues.from(null) diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 2292db08..ee21805f 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -190,29 +190,27 @@ class HuaweiRouteFactoryTest { private const val END_NANOS = END_MILLIS * 1_000_000L private val LITERAL_FIELD_KEYS = listOf( - "active_hours", "active_hours_target", "activeCalories", "activeHours", - "activeCaloriesGoal", "activeHoursGoal", "all_sleep_time", "arrhythmia_result", - "arrhythmia_type", "ascent_total", "avg", "avg_breathe_rate", "avg_heart_rate", - "awake_time", "calories", "calories_target", "calories_total", "correlate_mealtime", - "correlate_sleep", "count", "deep_sleep_part", "deep_sleep_time", "descent_total", + "activeCalories", "activeCaloriesGoal", "activeHours", "activeHoursGoal", + "activity_type", "all_sleep_time", "arrhythmia_result", "arrhythmia_type", + "ascent_total", "avg", "avg_body_fat_rate", "avg_heart_rate", "avgBreatheRate", + "awake_time", "calories", "calories_total", "correlate_mealtime", "correlate_sleep", + "count", "deep_sleep_part", "deep_sleep_time", "descent_total", "diastolic_pressure_avg", "diastolic_pressure_max", "diastolic_pressure_min", - "distance", "distance_delta", "distance_total", "dream_time", "duration", "emotion", - "event_name", "exercise_time", "exercise_time_target", "exerciseTime", - "exerciseTimeGoal", "exercise_type", "extendData", "fall_asleep_time", "go_bed_time", - "heart_rate_variability_rmssd", "highBodyTemperatureAlarm", "isActive", "last", - "level", "light_sleep_time", "max", "max_breathe_rate", "max_breathrate_baseline", - "max_heart_rate", "maxSpO2", "meal", "min", "min_breathe_rate", - "min_breathrate_baseline", "min_heart_rate", "minSpO2", "off_bed_time", - "on_off_bed_state", "predicted_calories", "prepare_sleep_time", "recordDay", - "record_id", "remarks", "sample_source", "sampling_frequency", "sleep_efficiency", - "saturation_avg", "saturation_last", "saturation_max", "saturation_min", - "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "sphygmus_avg", - "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", "steps_delta", - "steps_target", "stepsGoal", "subStatus", "systolic_pressure_avg", - "systolic_pressure_max", "systolic_pressure_min", "threshold", "timeZone", - "total_calories", "type", "user_symptom", "value", "vo2max", "voltage_data", - "wakeup_count", "wakeup_time", "zone1_duration", "zone2_duration", "zone3_duration", - "zone4_duration", "zone5_duration", + "distance", "distance_delta", "dream_time", "duration", "emotionStatus", "eventName", + "exercise_type", "exerciseTime", "exerciseTimeGoal", "extendData", "fall_asleep_time", + "fragments", "go_bed_time", "heartRateVariabilityRMSSD", "heartRateVariabilityRmssd", + "highBodyTemperatureAlarm", "intensity", "isActive", "last", "level", + "light_sleep_time", "max", "max_body_fat_rate", "max_heart_rate", "maxBreatheRate", + "maxBreathrateBaseline", "maxSpO2", "meal", "measure_count", "min", "min_body_fat_rate", + "min_heart_rate", "minBreatheRate", "minBreathrateBaseline", "minSpO2", "off_bed_time", + "onOffBedState", "predictedCalories", "prepare_sleep_time", "record_id", "recordDay", + "remarks", "sample_source", "sampling_frequency", "saturation_avg", "saturation_last", + "saturation_max", "saturation_min", "sleep_efficiency", "sleep_latency", "sleep_score", + "sleep_state", "sleep_type", "span", "sphygmus_avg", "sphygmus_last", "sphygmus_max", + "sphygmus_min", "status", "steps", "steps_delta", "stepsGoal", "subStatus", + "systolic_pressure_avg", "systolic_pressure_max", "systolic_pressure_min", "threshold", + "timeZone", "totalCalories", "type", "user_symptom", "value", "vo2max", "voltage_data", + "voltage_datas", "wakeup_count", "wakeup_time", ) } } From 84a4e9f4a00df4480a58cdbc8f3fe0ce3411b8e4 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 20:53:07 +0000 Subject: [PATCH 35/44] Parse Huawei sample sets with or without group wrapper, infer time units sampleSet:polymerize responses wrap sample sets in group[] (like dailyPolymerize) and report sample point times in nanoseconds; the converter only read a top-level sampleSet[] in milliseconds, so it would have emitted nothing (or wrong times). Share one tolerant parser across both endpoints and infer s/ms/us/ns from each timestamp's magnitude in all converters. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../HuaweiActivityRecordConverter.kt | 7 -- .../HuaweiDailyPolymerizeConverter.kt | 44 ++------- .../huawei/converter/HuaweiDataConverter.kt | 2 +- .../converter/HuaweiHealthRecordConverter.kt | 8 -- .../converter/HuaweiSampleSetConverter.kt | 47 +++++---- .../huawei/converter/HuaweiTimestamps.kt | 58 +++++++++++ .../converter/HuaweiSampleSetConverterTest.kt | 97 +++++++++++++++++++ 7 files changed, 192 insertions(+), 71 deletions(-) create mode 100644 huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiTimestamps.kt create mode 100644 huawei-library/src/test/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverterTest.kt diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt index e70c6b63..cbc228df 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiActivityRecordConverter.kt @@ -83,13 +83,6 @@ class HuaweiActivityRecordConverter( }.build() } - private fun JsonNode.epochInstant(field: String): Instant? { - val value = this.get(field) ?: return null - if (value.isNull) return null - val millis = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() - return millis?.let { Instant.ofEpochMilli(it) } - } - private fun JsonNode.textOrNull(field: String): String? = this.get(field)?.takeUnless { it.isNull }?.asText() diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDailyPolymerizeConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDailyPolymerizeConverter.kt index 3e4d5d72..7a52460d 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDailyPolymerizeConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDailyPolymerizeConverter.kt @@ -17,54 +17,22 @@ package org.radarbase.huawei.converter -import com.fasterxml.jackson.databind.JsonNode import org.apache.avro.specific.SpecificRecord -import org.radarbase.huawei.user.User import java.time.Instant -/** Sample points inside `sampleSet:dailyPolymerize`'s response report their times in nanoseconds. */ -private fun JsonNode.epochNanoInstant(field: String): Instant? { - val value = this.get(field) ?: return null - if (value.isNull) return null - val nanos = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() - return nanos?.let { Instant.ofEpochSecond(it / 1_000_000_000L, it % 1_000_000_000L) } -} - /** - * Converter for `POST /healthkit/v2/sampleSet:dailyPolymerize` responses: unlike - * `sampleSet:polymerize`, each day's result is wrapped in a `group[]` entry containing its own - * `sampleSet[].samplePoints[]`, so this walks two levels of nesting instead of one before reaching - * the same `{"fieldName": ..., "value": ...}` point shape used elsewhere. + * Converter for `POST /healthkit/v2/sampleSet:dailyPolymerize` responses. Each day's result is + * wrapped in a `group[]` entry containing its own `sampleSet[].samplePoints[]`, which + * [HuaweiSampleSetConverter] already handles, so this only exists to name the endpoint. * * @author yatharthranjan */ class HuaweiDailyPolymerizeConverter( - private val topic: String, - private val buildRecord: ( + topic: String, + buildRecord: ( fields: FieldValues, startTime: Instant, endTime: Instant?, timeReceived: Instant, ) -> SpecificRecord, -) : HuaweiDataConverter { - - override fun processRecords(root: JsonNode, user: User): Sequence> { - val timeReceived = Instant.now() - val groups = root.get("group") ?: return emptySequence() - return groups.asSequence() - .flatMap { group -> group.get("sampleSet")?.asSequence() ?: emptySequence() } - .flatMap { sampleSet -> sampleSet.get("samplePoints")?.asSequence() ?: emptySequence() } - .mapCatching { point -> - val startTime = point.epochNanoInstant("startTime") - ?: error("Huawei daily polymerize sample point is missing startTime") - val endTime = point.epochNanoInstant("endTime") - val fieldValues = FieldValues.from(point.get("value")) - TopicData( - topic = topic, - key = user.observationKey, - offset = startTime.epochSecond, - value = buildRecord(fieldValues, startTime, endTime, timeReceived), - ) - } - } -} +) : HuaweiDataConverter by HuaweiSampleSetConverter(topic, buildRecord) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt index 51d79be4..52f1f75b 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt @@ -48,7 +48,7 @@ interface HuaweiDataConverter : RecordConverter { r.fold( { it }, { - logger.error("Data conversion failed.. " + it.message) + logger.error("Data conversion failed for {}: {}", request, it.toString()) null }, ) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt index aa522ff1..d1eb7480 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt @@ -22,14 +22,6 @@ import org.apache.avro.specific.SpecificRecord import org.radarbase.huawei.user.User import java.time.Instant -/** Huawei's healthRecords v2 endpoint reports startTime/endTime in nanoseconds since the epoch. */ -private fun JsonNode.epochInstant(field: String): Instant? { - val value = this.get(field) ?: return null - if (value.isNull) return null - val nanos = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() - return nanos?.let { Instant.ofEpochSecond(it / 1_000_000_000L, it % 1_000_000_000L) } -} - /** * Generic converter for `GET /healthkit/v2/healthRecords` responses: iterates every record * returned for the requested `dataType` and builds one Avro record per entry via diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt index 192ad7d5..837732d7 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverter.kt @@ -22,20 +22,19 @@ import org.apache.avro.specific.SpecificRecord import org.radarbase.huawei.user.User import java.time.Instant -private fun JsonNode.epochInstant(field: String): Instant? { - val value = this.get(field) ?: return null - if (value.isNull) return null - val millis = if (value.isTextual) value.asText().toLongOrNull() else value.asLong() - return millis?.let { Instant.ofEpochMilli(it) } -} - /** - * Generic converter for `sampleSet:polymerize` responses: iterates every sample point of every - * data-type group in the response and builds one Avro record per point via [buildRecord]. + * Generic converter for `sampleSet:polymerize` and `sampleSet:dailyPolymerize` responses: + * iterates every sample point of every sample set in the response and builds one Avro record per + * point via [buildRecord]. + * + * Huawei wraps sample sets in a `group[]` array (one entry per aggregation bucket or day), each + * holding `sampleSet[].samplePoints[]`. A bare top-level `sampleSet[]` is also accepted. Sample + * point times are nanoseconds in Huawei's examples, while group times are milliseconds; units are + * inferred per value (see [epochInstant]). * * This single converter is reused for the large majority of Huawei Health Kit data types, since - * they all share the same `sampleSet[].samplePoints[]` response envelope and differ only in which - * Avro record type their field values are mapped onto. + * they all share the same response envelope and differ only in which Avro record type their field + * values are mapped onto. * * @author yatharthranjan */ @@ -51,12 +50,7 @@ class HuaweiSampleSetConverter( override fun processRecords(root: JsonNode, user: User): Sequence> { val timeReceived = Instant.now() - val sampleSets = root.get("sampleSet") ?: root.get("sampleSets") ?: return emptySequence() - return sampleSets.asSequence() - .flatMap { group -> - val points = group.get("samplePoints") ?: group.get("samplePoint") - points?.asSequence() ?: emptySequence() - } + return root.samplePoints() .mapCatching { point -> val startTime = point.epochInstant("startTime") ?: error("Huawei sample point is missing startTime") @@ -70,4 +64,23 @@ class HuaweiSampleSetConverter( ) } } + + companion object { + private fun JsonNode.child(vararg names: String): JsonNode? = + names.firstNotNullOfOrNull { name -> get(name)?.takeIf { it.isArray } } + + /** All sample points in a (daily) polymerize response, with or without `group[]`. */ + internal fun JsonNode.samplePoints(): Sequence { + val sampleSets = child("group", "groups") + ?.asSequence() + ?.flatMap { group -> + group.child("sampleSet", "sampleSets")?.asSequence().orEmpty() + } + ?: child("sampleSet", "sampleSets")?.asSequence() + ?: emptySequence() + return sampleSets.flatMap { sampleSet -> + sampleSet.child("samplePoints", "samplePoint")?.asSequence().orEmpty() + } + } + } } diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiTimestamps.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiTimestamps.kt new file mode 100644 index 00000000..7a00c70d --- /dev/null +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiTimestamps.kt @@ -0,0 +1,58 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.radarbase.huawei.converter + +import com.fasterxml.jackson.databind.JsonNode +import java.time.Instant + +/** + * Reads an epoch timestamp field from a Huawei response node, whatever its unit. + * + * Huawei's endpoints are inconsistent about timestamp units - `activityRecords` uses milliseconds, + * `healthRecords` and sample points use nanoseconds, and `dailyPolymerize` groups use milliseconds + * around nanosecond sample points - and the docs don't always say which. Any plausible timestamp + * (after 2001) differs by at least three orders of magnitude between units, so the unit is + * inferred from the value's magnitude instead of hard-coded per endpoint. Both numeric and + * stringified numbers are accepted. + * + * @author yatharthranjan + */ +internal fun JsonNode.epochInstant(field: String): Instant? { + val node = this.get(field) ?: return null + if (node.isNull) return null + val value = if (node.isTextual) node.asText().trim().toLongOrNull() else node.asLong() + return value?.let { epochInstantOf(it) } +} + +internal fun epochInstantOf(value: Long): Instant = when { + value >= NANOS_THRESHOLD -> Instant.ofEpochSecond( + Math.floorDiv(value, 1_000_000_000L), + Math.floorMod(value, 1_000_000_000L), + ) + value >= MICROS_THRESHOLD -> Instant.ofEpochSecond( + Math.floorDiv(value, 1_000_000L), + Math.floorMod(value, 1_000_000L) * 1_000L, + ) + value >= MILLIS_THRESHOLD -> Instant.ofEpochMilli(value) + else -> Instant.ofEpochSecond(value) +} + +// 1e9 s is 2001-09-09, so any post-2001 time is >= 1e12 in ms, >= 1e15 in us and >= 1e18 in ns. +// Lower thresholds by 10x to keep boundaries far from any realistic value in the adjacent unit. +private const val MILLIS_THRESHOLD = 100_000_000_000L +private const val MICROS_THRESHOLD = 100_000_000_000_000L +private const val NANOS_THRESHOLD = 100_000_000_000_000_000L diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverterTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverterTest.kt new file mode 100644 index 00000000..1ee7855e --- /dev/null +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/HuaweiSampleSetConverterTest.kt @@ -0,0 +1,97 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.huawei.converter + +import com.fasterxml.jackson.databind.ObjectMapper +import org.radarbase.huawei.user.HuaweiUser +import org.radarcns.connector.huawei.HuaweiContinuousStepsDelta +import java.time.Instant +import kotlin.test.Test +import kotlin.test.assertEquals + +/** + * @author yatharthranjan + */ +class HuaweiSampleSetConverterTest { + private val mapper = ObjectMapper() + private val user = HuaweiUser( + id = "u1", + createdAt = Instant.now(), + projectId = "p", + userId = "u", + humanReadableUserId = null, + sourceId = "s", + externalId = "ext", + isAuthorized = true, + startDate = Instant.parse("2024-01-01T00:00:00Z"), + ) + private val converter = HuaweiSampleSetConverter("topic") { f, start, end, received -> + HuaweiContinuousStepsDelta.newBuilder().apply { + time = start.toEpochMilli() / 1000.0 + timeReceived = received.toEpochMilli() / 1000.0 + endTime = end?.let { it.toEpochMilli() / 1000.0 } + stepsDelta = f.getInt("steps_delta") + }.build() + } + + @Test + fun `reads group-wrapped sample sets with nanosecond times`() { + val root = mapper.readTree( + """ + {"group": [{ + "startTime": 1704067200000, "endTime": 1704153600000, + "sampleSet": [{"samplePoints": [ + {"startTime": 1704067200000000000, "endTime": "1704067260000000000", + "value": [{"fieldName": "steps_delta", "integerValue": 12}]} + ]}] + }]} + """.trimIndent(), + ) + val record = converter.processRecords(root, user).single().getOrThrow() + val value = record.value as HuaweiContinuousStepsDelta + + assertEquals(1704067200L, record.offset) + assertEquals(1704067200.0, value.time) + assertEquals(1704067260.0, value.endTime) + assertEquals(12, value.stepsDelta) + } + + @Test + fun `reads top-level sample sets with millisecond times`() { + val root = mapper.readTree( + """ + {"sampleSet": [{"samplePoints": [ + {"startTime": 1704067200000, "endTime": 1704067260000, + "value": [{"fieldName": "steps_delta", "integerValue": 3}]} + ]}]} + """.trimIndent(), + ) + val record = converter.processRecords(root, user).single().getOrThrow() + + assertEquals(1704067200L, record.offset) + } + + @Test + fun `infers timestamp units from magnitude`() { + val expected = Instant.parse("2024-01-01T00:00:00.123Z") + assertEquals(expected, epochInstantOf(1704067200123L)) + assertEquals(expected, epochInstantOf(1704067200123000L)) + assertEquals(expected, epochInstantOf(1704067200123000000L)) + assertEquals(Instant.parse("2024-01-01T00:00:00Z"), epochInstantOf(1704067200L)) + } +} From e07e2b6effead75748016ce6b68754ba5b7bf269 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 21:00:53 +0000 Subject: [PATCH 36/44] Harden Huawei request scheduling, offsets and error handling - Keep querying the last 7 days until data arrives, since devices sync to the Huawei cloud late; advance offsets only past records received, drop records already emitted, and follow hasMoreData by continuing after the latest record. - Request daily statistics for whole, completed UTC days only, with an inclusive endDay, so consecutive requests neither overlap nor emit partial days. - Stop the remaining chunks of a route after a failure or rate limit, so offsets can't skip a failed window; honour the global 429 back-off. - Contain token/user-listing errors raised while building requests instead of letting them kill the source task; back off the user. - On HTTP 401, invalidate the cached access token and retry after 10 min. - poll(): single pass, no wait while catching up, wakes on stop(). - Store Kafka offsets just past the last record and tolerate a missing offset reader. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/converter/HuaweiDataConverter.kt | 8 +- .../huawei/request/HuaweiRequestGenerator.kt | 214 ++++++++++++----- .../route/HuaweiDailyPolymerizeRoute.kt | 54 +++-- .../huawei/route/HuaweiSampleSetRoute.kt | 2 +- .../radarbase/huawei/user/UserRepository.kt | 6 + .../request/HuaweiRequestGeneratorTest.kt | 225 ++++++++++++++++++ .../connect/rest/huawei/HuaweiSourceTask.java | 28 ++- .../huawei/offset/KafkaOffsetManager.java | 14 +- .../user/HuaweiServiceUserRepository.kt | 4 + .../huawei/user/OAuth2UserCredentials.java | 5 + 10 files changed, 461 insertions(+), 99 deletions(-) create mode 100644 huawei-library/src/test/kotlin/org/radarbase/huawei/request/HuaweiRequestGeneratorTest.kt diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt index 52f1f75b..532e2ecd 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiDataConverter.kt @@ -40,10 +40,11 @@ interface HuaweiDataConverter : RecordConverter { request: RestRequest, headers: Headers, data: ByteArray, - ): List { - val node = JSON_READER.readTree(data) + ): List = convert(request, JSON_READER.readTree(data)) - return this.processRecords(node, request.user) + /** Convert an already-parsed response body, logging and skipping records that fail. */ + fun convert(request: RestRequest, root: JsonNode): List = + this.processRecords(root, request.user) .mapNotNull { r -> r.fold( { it }, @@ -54,7 +55,6 @@ interface HuaweiDataConverter : RecordConverter { ) } .toList() - } fun Instant.toEpoch(): Double = this.toEpochMilli() / 1000.0 } diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt index 5d7ce2aa..c8016394 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/request/HuaweiRequestGenerator.kt @@ -18,17 +18,21 @@ package org.radarbase.huawei.request import com.fasterxml.jackson.core.JsonFactory +import com.fasterxml.jackson.databind.JsonNode import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.databind.node.MissingNode import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule import okhttp3.Response import org.radarbase.huawei.converter.TopicData import org.radarbase.huawei.route.Route import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserNotAuthorizedException import org.radarbase.huawei.user.UserRepository import org.slf4j.LoggerFactory import java.io.IOException import java.time.Duration import java.time.Instant +import java.util.concurrent.ConcurrentHashMap /** * @author yatharthranjan @@ -38,47 +42,28 @@ class HuaweiRequestGenerator( private val huaweiOffsetManager: HuaweiOffsetManager, val routes: List, ) : RequestGenerator { - private val routeNextRequest: MutableMap = mutableMapOf() + private val routeNextRequest: MutableMap = ConcurrentHashMap() + + /** Routes whose last request failed, until when. Stops the rest of a lazily generated + * request sequence for that route and user, so a later chunk can't advance the offset past + * a chunk that failed. */ + private val routeFailedUntil: MutableMap = ConcurrentHashMap() var nextRequestTime: Instant = Instant.MIN override fun requests(user: User, max: Int): Sequence = routes.asSequence() - .flatMap { route -> - if (routeReady(user, route)) { - generateRequests(route, user) - } else { - logger.info( - "Skip {} for {}: route in backoff until {}", - route, - user.versionedId, - routeNextRequest[routeKey(route, user)], - ) - emptySequence() - } - } + .flatMap { route -> requests(route, user, max) } override fun requests(route: Route, max: Int): Sequence = userRepository.stream() - .flatMap { user -> - if (routeReady(user, route)) { - generateRequests(route, user) - } else { - logger.info( - "Skip {} for {}: route in backoff until {}", - route, - user.versionedId, - routeNextRequest[routeKey(route, user)], - ) - emptySequence() - } - } + .flatMap { user -> requests(route, user, max) } override fun requests(route: Route, user: User, max: Int): Sequence = if (routeReady(user, route)) { generateRequests(route, user) } else { - logger.info( + logger.debug( "Skip {} for {}: route in backoff until {}", route, user.versionedId, @@ -88,17 +73,10 @@ class HuaweiRequestGenerator( } fun generateRequests(route: Route, user: User): Sequence { - val offset = huaweiOffsetManager.getOffset(route, user) - val startDate = user.startDate - val startOffset: Instant = if (offset == null) { - logger.info("No offsets found for $user, using the start date.") - startDate - } else { - offset.offset.coerceAtLeast(startDate) - } + val startOffset = currentOffset(route, user) val endDate = user.endDate?.coerceAtMost(Instant.now()) ?: Instant.now() if (!startOffset.isBefore(endDate)) { - logger.info( + logger.debug( "Skip {} for {}: interval empty (startOffset={} >= endDate={})", route, user.versionedId, @@ -107,12 +85,64 @@ class HuaweiRequestGenerator( ) return emptySequence() } + val key = routeKey(route, user) return route.generateRequests(user, startOffset, endDate, USER_MAX_REQUESTS) + .guarded(route, user) + .takeWhile { !isBlocked(key) } + } + + /** + * Requests are built lazily, and building one fetches the user's access token, which throws + * if the user is no longer authorized or the token endpoint is unreachable. Contain that to + * this route and user (backing it off) instead of letting it escape the source task's poll. + */ + private fun Sequence.guarded(route: Route, user: User): Sequence { + val source = this + return sequence { + val iterator = source.iterator() + while (true) { + val next = try { + if (!iterator.hasNext()) break + iterator.next() + } catch (ex: UserNotAuthorizedException) { + logger.warn("User {} is not authorized: {}", user.versionedId, ex.message) + backOff(route, user, USER_BACK_OFF_TIME) + break + } catch (ex: Exception) { + logger.warn( + "Failed to create {} request for {}: {}", + route, + user.versionedId, + ex.toString(), + ) + backOff(route, user, BACK_OFF_TIME) + break + } + yield(next) + } + } + } + + private fun currentOffset(route: Route, user: User): Instant { + val offset = huaweiOffsetManager.getOffset(route, user) + return if (offset == null) { + logger.info("No offsets found for {} on {}, using the start date.", user, route) + user.startDate + } else { + offset.offset.coerceAtLeast(user.startDate) + } } fun handleResponse(req: RestRequest, response: Response): HuaweiResult> { return if (response.isSuccessful) { - HuaweiResult.Success(requestSuccessful(req, response)) + try { + HuaweiResult.Success(requestSuccessful(req, response)) + } catch (ex: IOException) { + // Unreadable or malformed body: don't advance the offset, and retry later. + logger.warn("Failed to read response of {}: {}", req, ex.toString()) + backOff(req.route, req.user, BACK_OFF_TIME) + HuaweiResult.Success(emptyList()) + } } else { try { HuaweiResult.Error(requestFailed(req, response)) @@ -122,37 +152,73 @@ class HuaweiRequestGenerator( } } + /** + * Converts the response and advances the route's offset. + * + * Huawei devices sync to the Huawei cloud with a delay of minutes to days, so data for a + * period can appear after that period was already queried. The offset is therefore never + * advanced past [LATE_SYNC_WINDOW] ago on the basis of an empty response: only past the + * latest record actually received. Records starting before the current offset were already + * emitted (the offset sits just past the latest one) and are dropped, since Huawei also + * returns samples that merely overlap the queried window. + */ override fun requestSuccessful(request: RestRequest, response: Response): List { logger.debug("Request successful: {}..", request.request) - val body = response.body - val data = body?.bytes() ?: ByteArray(0) - val records = request.route.converters.flatMap { - it.convert( - request, - response.headers, - data, - ) + val now = Instant.now() + val data = response.body?.bytes() ?: ByteArray(0) + val root: JsonNode = if (data.isEmpty()) { + MissingNode.getInstance() + } else { + JSON_READER.readTree(data) ?: MissingNode.getInstance() } - val offset = records.maxByOrNull { it.offset }?.offset - val key = routeKey(request.route, request.user) - if (offset != null) { - val maxOffsetTime = Instant.ofEpochSecond(offset) - val nextOffset = maxOffsetTime.plus(OFFSET_BUFFER).coerceAtLeast(request.endDate) + val currentOffset = currentOffset(request.route, request.user) + val records = request.route.converters + .flatMap { it.convert(request, root) } + .filter { it.offset >= currentOffset.epochSecond } + + val settledEnd = request.endDate.coerceAtMost(now.minus(LATE_SYNC_WINDOW)) + val maxOffset = records.maxOfOrNull { it.offset } + val nextOffset = if (maxOffset != null) { + val afterLatest = Instant.ofEpochSecond(maxOffset + 1) + if (root.path("hasMoreData").asBoolean(false)) { + // Only part of this window was returned; continue right after the latest record. + logger.info( + "More {} data available for {} than returned; continuing after {}", + request.route, + request.user.versionedId, + afterLatest, + ) + afterLatest + } else { + afterLatest.coerceAtLeast(settledEnd) + } + } else { + settledEnd.coerceAtLeast(currentOffset) + } + if (nextOffset.isAfter(currentOffset)) { huaweiOffsetManager.updateOffsets(request.route, request.user, nextOffset) + } + + val key = routeKey(request.route, request.user) + routeFailedUntil -= key + routeNextRequest[key] = if (records.isEmpty() && request.endDate > settledEnd) { + // Caught up to data that may still be syncing: check again later. + now.plus(CAUGHT_UP_BACK_OFF_TIME) } else { - huaweiOffsetManager.updateOffsets(request.route, request.user, request.endDate) + now.plus(SUCCESS_BACK_OFF_TIME) } - routeNextRequest[key] = Instant.now().plus(SUCCESS_BACK_OFF_TIME) return records } override fun requestFailed(request: RestRequest, response: Response): HuaweiError { - val key = routeKey(request.route, request.user) return when (response.code) { 429 -> { - logger.info("Too many requests, rate limit reached. Backing off...") + logger.info( + "Too many requests, rate limit reached. Backing off... {}", + response.body?.string(), + ) nextRequestTime = Instant.now() + BACK_OFF_TIME - routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + backOff(request.route, request.user, BACK_OFF_TIME) HuaweiRateLimitError("Rate limit reached..", TooManyRequestsException(), "429") } 403 -> { @@ -162,7 +228,7 @@ class HuaweiRequestGenerator( request.user, body, ) - routeNextRequest[key] = Instant.now().plus(USER_BACK_OFF_TIME) + backOff(request.route, request.user, USER_BACK_OFF_TIME) HuaweiAccessForbiddenError( "Huawei Health Kit scope not granted or data not available: $body", IOException("Forbidden"), @@ -176,7 +242,10 @@ class HuaweiRequestGenerator( request.user, body, ) - routeNextRequest[key] = Instant.now().plus(USER_BACK_OFF_TIME) + // Refetch the token on the next attempt; a user whose refresh token is also no + // longer valid is then backed off when building that request. + userRepository.invalidateAccessToken(request.user) + backOff(request.route, request.user, BACK_OFF_TIME) HuaweiUnauthorizedAccessError( "Access token expired or revoked: $body", IOException("Unauthorized"), @@ -186,7 +255,7 @@ class HuaweiRequestGenerator( 400 -> { val body = response.body?.string() ?: "no response body" logger.warn("Client exception for request {}: {}", request, body) - routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + backOff(request.route, request.user, BACK_OFF_TIME) HuaweiClientException( "Client unsupported or unauthorized: $body", IOException("Invalid client"), @@ -195,7 +264,7 @@ class HuaweiRequestGenerator( } 422 -> { logger.warn("Request failed (validation error): {}, {}", request, response) - routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + backOff(request.route, request.user, BACK_OFF_TIME) HuaweiValidationError( response.body?.string() ?: "validation error", IOException("Validation error"), @@ -204,7 +273,7 @@ class HuaweiRequestGenerator( } 404 -> { logger.warn("Not found: {}", request) - routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + backOff(request.route, request.user, BACK_OFF_TIME) HuaweiNotFoundError( response.body?.string() ?: "not found", IOException("Data not found"), @@ -214,7 +283,7 @@ class HuaweiRequestGenerator( else -> { val body = response.body?.string() ?: "unknown error" logger.warn("Request failed: {}: {}", request, body) - routeNextRequest[key] = Instant.now().plus(BACK_OFF_TIME) + backOff(request.route, request.user, BACK_OFF_TIME) HuaweiGenericError( body, IOException("Unknown error"), @@ -224,9 +293,23 @@ class HuaweiRequestGenerator( } } + private fun backOff(route: Route, user: User, duration: Duration) { + val key = routeKey(route, user) + val until = Instant.now().plus(duration) + routeNextRequest[key] = until + routeFailedUntil[key] = until + } + private fun routeReady(user: User, route: Route): Boolean { + val now = Instant.now() val key = routeKey(route, user) - return routeNextRequest[key]?.let { Instant.now() > it } ?: true + return now > nextRequestTime && routeNextRequest[key]?.let { now > it } ?: true + } + + /** Whether requests for this route and user must stop, after a failure or rate limit. */ + private fun isBlocked(key: String): Boolean { + val now = Instant.now() + return now <= nextRequestTime || routeFailedUntil[key]?.let { now <= it } ?: false } private fun routeKey(route: Route, user: User): String = user.versionedId + "#" + route @@ -236,7 +319,10 @@ class HuaweiRequestGenerator( private val BACK_OFF_TIME = Duration.ofMinutes(10L) private val USER_BACK_OFF_TIME = Duration.ofHours(12L) private val SUCCESS_BACK_OFF_TIME = Duration.ofSeconds(10L) - private val OFFSET_BUFFER = Duration.ofHours(1) + private val CAUGHT_UP_BACK_OFF_TIME = Duration.ofMinutes(30L) + + /** How long after the fact Huawei data may still be synced to the cloud. */ + private val LATE_SYNC_WINDOW = Duration.ofDays(7L) private const val USER_MAX_REQUESTS = 1000 val JSON_FACTORY = JsonFactory() val JSON_READER = ObjectMapper(JSON_FACTORY).registerModule(JavaTimeModule()).reader() diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiDailyPolymerizeRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiDailyPolymerizeRoute.kt index dafbdd8c..211c952f 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiDailyPolymerizeRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiDailyPolymerizeRoute.kt @@ -29,6 +29,7 @@ import java.time.Duration import java.time.Instant import java.time.ZoneOffset import java.time.format.DateTimeFormatter +import java.time.temporal.ChronoUnit /** * Route backed by `POST /healthkit/v2/sampleSet:dailyPolymerize`, used for every Huawei @@ -46,6 +47,7 @@ open class HuaweiDailyPolymerizeRoute( userRepository: UserRepository, private val dataTypeName: String, private val topic: String, + // dailyPolymerize accepts at most 31 days (inclusive) per request. maxIntervalPerRequest: Duration = Duration.ofDays(30L), buildRecord: ( fields: FieldValues, @@ -60,37 +62,57 @@ open class HuaweiDailyPolymerizeRoute( override fun toString(): String = "huawei_" + topic.removePrefix("connect_huawei_") + /** + * Requests whole UTC days only: from the first midnight at or after [start] up to the last + * midnight at least [COMPLETION_DELAY] ago (capped at [end]), so a day's statistics are only + * fetched once that day is over and has had time to sync, and consecutive requests never + * overlap. Each request's `endDay` is inclusive, so it names the day before the exclusive + * range end. + */ override fun generateRequests( user: User, start: Instant, end: Instant, max: Int, - ): Sequence = chunkedRanges(start, end, max).map { (rangeStart, rangeEnd) -> - RestRequest( - request = createPostRequest( - user, - "sampleSet:dailyPolymerize", - buildRequestBody(rangeStart, rangeEnd), - baseUrl = HUAWEI_API_BASE_URL_V2, - ), - user = user, - route = this, - startDate = rangeStart, - endDate = rangeEnd, - ) + ): Sequence { + val firstDay = start.ceilToDay() + val lastDayEnd = end.coerceAtMost(Instant.now().minus(COMPLETION_DELAY)).floorToDay() + if (!firstDay.isBefore(lastDayEnd)) return emptySequence() + return chunkedRanges(firstDay, lastDayEnd, max).map { (rangeStart, rangeEnd) -> + RestRequest( + request = createPostRequest( + user, + "sampleSet:dailyPolymerize", + buildRequestBody(rangeStart, rangeEnd.minus(Duration.ofDays(1))), + baseUrl = HUAWEI_API_BASE_URL_V2, + ), + user = user, + route = this, + startDate = rangeStart, + endDate = rangeEnd, + ) + } } - private fun buildRequestBody(start: Instant, end: Instant): String { + private fun buildRequestBody(firstDay: Instant, lastDay: Instant): String { val root = MAPPER.createObjectNode() root.putArray("dataTypes").add(dataTypeName) - root.put("startDay", DAY_FORMATTER.format(start)) - root.put("endDay", DAY_FORMATTER.format(end)) + root.put("startDay", DAY_FORMATTER.format(firstDay)) + root.put("endDay", DAY_FORMATTER.format(lastDay)) root.put("timeZone", "+0000") return MAPPER.writeValueAsString(root) } companion object { + /** How long after the end of a UTC day its statistics are first requested. */ + private val COMPLETION_DELAY = Duration.ofHours(12L) private val MAPPER = ObjectMapper() + + private fun Instant.floorToDay(): Instant = truncatedTo(ChronoUnit.DAYS) + + private fun Instant.ceilToDay(): Instant = floorToDay().let { + if (it == this) it else it.plus(Duration.ofDays(1)) + } private val DAY_FORMATTER = DateTimeFormatter.ofPattern("yyyyMMdd").withZone(ZoneOffset.UTC) } } diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt index 3e15485e..7ca1db1c 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiSampleSetRoute.kt @@ -31,7 +31,7 @@ import java.time.Instant /** * Route backed by `POST /healthkit/v1/sampleSet:polymerize`, which covers the large majority of * raw (non-`.statistics`) Huawei Health Kit data types (all `continuous.*`, `instantaneous.*`, - * `cgm_blood_glucose`, `active_hours`, `daily_activity_summary`, `emotion`, + * `cgm_blood_glucose`, `active_hours`, `emotion`, * `heart_rate_variability`, `vo2max`, `sleep.on_off_bed_record`, and `sleep_respiratory_*` types). * Returns raw, un-aggregated sample points for [dataTypeName] over the requested time range. * diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt index 34d50e78..835b44cc 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/user/UserRepository.kt @@ -50,4 +50,10 @@ interface UserRepository { */ @Throws(IOException::class, UserNotAuthorizedException::class) fun getAccessToken(user: User): String + + /** + * Discard any cached access token of given user, e.g. after the Huawei API rejected it with + * HTTP 401 before its advertised expiry, so the next [getAccessToken] call fetches a new one. + */ + fun invalidateAccessToken(user: User) {} } diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/request/HuaweiRequestGeneratorTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/request/HuaweiRequestGeneratorTest.kt new file mode 100644 index 00000000..c32e68e3 --- /dev/null +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/request/HuaweiRequestGeneratorTest.kt @@ -0,0 +1,225 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.radarbase.huawei.request + +import okhttp3.Protocol +import okhttp3.Request +import okhttp3.Response +import okhttp3.ResponseBody.Companion.toResponseBody +import org.radarbase.huawei.offset.Offset +import org.radarbase.huawei.route.HuaweiDailyPolymerizeRoute +import org.radarbase.huawei.route.HuaweiRoute +import org.radarbase.huawei.route.HuaweiSampleSetRoute +import org.radarbase.huawei.route.Route +import org.radarbase.huawei.user.HuaweiUser +import org.radarbase.huawei.user.User +import org.radarbase.huawei.user.UserNotAuthorizedException +import org.radarbase.huawei.user.UserRepository +import org.radarcns.connector.huawei.HuaweiContinuousStepsDelta +import java.time.Duration +import java.time.Instant +import java.time.temporal.ChronoUnit +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +/** + * @author yatharthranjan + */ +class HuaweiRequestGeneratorTest { + private val user: User = HuaweiUser( + id = "u1", + createdAt = Instant.now(), + projectId = "p", + userId = "u", + humanReadableUserId = null, + sourceId = "s", + externalId = "ext", + isAuthorized = true, + startDate = Instant.parse("2024-01-01T00:00:00Z"), + ) + + private var tokenError: Exception? = null + private var invalidated = 0 + private val repository = object : UserRepository { + override fun get(key: String): User = user + override fun stream(): Sequence = sequenceOf(user) + override fun getAccessToken(user: User): String = tokenError?.let { throw it } ?: "token" + override fun invalidateAccessToken(user: User) { + invalidated++ + } + } + + private val offsets = mutableMapOf() + private val offsetManager = object : HuaweiOffsetManager { + override fun getOffset(route: Route, user: User): Offset? = + offsets[route.toString()]?.let { Offset(user, route, it) } + + override fun updateOffsets(route: Route, user: User, offset: Instant) { + offsets[route.toString()] = offset + } + } + + private val route = HuaweiSampleSetRoute( + repository, + "com.huawei.continuous.steps.delta", + "topic", + ) { f, start, _, received -> + HuaweiContinuousStepsDelta.newBuilder().apply { + time = start.epochSecond.toDouble() + timeReceived = received.epochSecond.toDouble() + stepsDelta = f.getInt("steps_delta") + }.build() + } + + private val generator = HuaweiRequestGenerator(repository, offsetManager, listOf(route)) + + private fun request(route: HuaweiRoute, start: Instant, end: Instant) = RestRequest( + Request.Builder().url("https://example.com").build(), + user, + route, + start, + end, + ) + + private fun response(req: RestRequest, code: Int, body: String) = Response.Builder() + .request(req.request) + .protocol(Protocol.HTTP_1_1) + .code(code) + .message("msg") + .body(body.toResponseBody()) + .build() + + private fun pointsBody(vararg starts: Instant, hasMoreData: Boolean = false): String { + val points = starts.joinToString(",") { start -> + """{"startTime": ${start.toEpochMilli() * 1_000_000}, + "value": [{"fieldName": "steps_delta", "integerValue": 1}]}""" + } + return """{"hasMoreData": $hasMoreData, + "group": [{"sampleSet": [{"samplePoints": [$points]}]}]}""" + } + + @Test + fun `empty historic window advances to its end`() { + val start = Instant.parse("2024-01-01T00:00:00Z") + val end = Instant.parse("2024-01-31T00:00:00Z") + val req = request(route, start, end) + + val records = generator.requestSuccessful(req, response(req, 200, pointsBody())) + + assertTrue(records.isEmpty()) + assertEquals(end, offsets[route.toString()]) + } + + @Test + fun `empty recent window does not advance past the late sync window`() { + val now = Instant.now() + val start = now.minus(Duration.ofDays(20)) + offsets[route.toString()] = start + val req = request(route, start, now) + + generator.requestSuccessful(req, response(req, 200, pointsBody())) + + val offset = offsets.getValue(route.toString()) + val limit = Instant.now().minus(Duration.ofDays(7)) + assertTrue(offset <= limit, "offset $offset advanced too far") + assertTrue(offset > now.minus(Duration.ofDays(8))) + } + + @Test + fun `recent records advance to just after the latest record and drop already-seen ones`() { + val now = Instant.now().truncatedTo(ChronoUnit.SECONDS) + val offset = now.minus(Duration.ofHours(2)) + offsets[route.toString()] = offset + val req = request(route, offset, now) + val seen = offset.minusSeconds(30) + val latest = now.minus(Duration.ofHours(1)) + + val records = generator.requestSuccessful( + req, + response(req, 200, pointsBody(seen, offset, latest)), + ) + + assertEquals(listOf(offset.epochSecond, latest.epochSecond), records.map { it.offset }) + assertEquals(latest.plusSeconds(1), offsets[route.toString()]) + } + + @Test + fun `partial historic response continues after the latest record`() { + val start = Instant.parse("2024-01-01T00:00:00Z") + val latest = Instant.parse("2024-01-05T00:00:00Z") + val req = request(route, start, Instant.parse("2024-01-31T00:00:00Z")) + + generator.requestSuccessful(req, response(req, 200, pointsBody(latest, hasMoreData = true))) + + assertEquals(latest.plusSeconds(1), offsets[route.toString()]) + } + + @Test + fun `failed chunk stops the remaining chunks of that route`() { + val requests = generator.requests(route, user, 100).iterator() + val first = requests.next() + generator.handleResponse(first, response(first, 500, "boom")) + + assertTrue(!requests.hasNext()) + assertEquals(null, offsets[route.toString()]) + } + + @Test + fun `unauthorized user is backed off instead of throwing`() { + tokenError = UserNotAuthorizedException("revoked") + + assertTrue(generator.requests(route, user, 100).toList().isEmpty()) + + tokenError = null + assertTrue(generator.requests(route, user, 100).toList().isEmpty(), "not backed off") + } + + @Test + fun `401 invalidates the cached access token`() { + val req = request(route, user.startDate, user.startDate.plus(Duration.ofDays(1))) + + generator.handleResponse(req, response(req, 401, "{}")) + + assertEquals(1, invalidated) + } + + @Test + fun `daily requests cover whole completed UTC days only`() { + val daily = HuaweiDailyPolymerizeRoute( + repository, + "com.huawei.continuous.steps.delta", + "daily", + ) { _, start, _, received -> + HuaweiContinuousStepsDelta.newBuilder().apply { + time = start.epochSecond.toDouble() + timeReceived = received.epochSecond.toDouble() + }.build() + } + val start = Instant.parse("2024-01-01T10:00:00Z") + val end = Instant.parse("2024-01-04T10:00:00Z") + + val requests = daily.generateRequests(user, start, end, 10).toList() + + assertEquals(1, requests.size) + assertEquals(Instant.parse("2024-01-02T00:00:00Z"), requests[0].startDate) + assertEquals(Instant.parse("2024-01-04T00:00:00Z"), requests[0].endDate) + val body = okio.Buffer().also { requests[0].request.body!!.writeTo(it) }.readUtf8() + assertTrue(body.contains("\"startDay\":\"20240102\""), body) + assertTrue(body.contains("\"endDay\":\"20240103\""), body) + } +} diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java index 6fd00a65..62cfffa1 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/HuaweiSourceTask.java @@ -24,6 +24,8 @@ import java.util.Iterator; import java.util.List; import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; import java.util.stream.Collectors; import java.util.stream.Stream; @@ -68,6 +70,8 @@ public class HuaweiSourceTask extends SourceTask { private static final String TIMESTAMP_OFFSET_KEY = "timestamp"; private static final long TIMEOUT = 60000L; private int routeStartIndex = 0; + private final CountDownLatch stopLatch = new CountDownLatch(1); + private boolean lastPollHadRecords = false; public void initialize(HuaweiRestSourceConnectorConfig config, OffsetStorageReader offsetStorageReader) { this.baseClient = new OkHttpClient(); @@ -136,7 +140,8 @@ public Stream handleRequest(RestRequest req) throws IOException { SchemaAndValue avro = avroData.toConnectData(r.getValue().getSchema(), r.getValue()); SchemaAndValue key = avroData.toConnectData(r.getKey().getSchema(), r.getKey()); Map partition = getPartition(req.getRoute().toString(), req.getUser()); - Map offset = Collections.singletonMap(TIMESTAMP_OFFSET_KEY, r.getOffset()); + // Stored offsets mark where to resume: just after this record's start time (seconds). + Map offset = Collections.singletonMap(TIMESTAMP_OFFSET_KEY, r.getOffset() + 1); return new SourceRecord(partition, offset, r.getTopic(), key.schema(), key.value(), avro.schema(), avro.value()); @@ -167,15 +172,19 @@ public void start(Map map) { @Override public List poll() throws InterruptedException { + // Only wait between polls when there is nothing left to catch up on, and wake up as soon + // as the task is stopped. + if (!lastPollHadRecords && stopLatch.await(TIMEOUT, TimeUnit.MILLISECONDS)) { + return null; + } + long requestsGenerated = 0; List sourceRecords = Collections.emptyList(); - do { - Thread.sleep(TIMEOUT); - + try { Iterator requestIterator = this.requests().iterator(); - while (sourceRecords.isEmpty() && requestIterator.hasNext()) { + while (sourceRecords.isEmpty() && stopLatch.getCount() > 0 && requestIterator.hasNext()) { RestRequest request = requestIterator.next(); logger.info("Requesting for user {}, url: {}", request.getUser().getUserId(), request.getRequest().url()); @@ -184,12 +193,16 @@ public List poll() throws InterruptedException { try { sourceRecords = this.handleRequest(request) .collect(Collectors.toList()); - } catch (IOException ex) { + } catch (IOException | RuntimeException ex) { logger.warn("Failed to make request: {}", ex.toString()); } } - } while (sourceRecords.isEmpty()); + } catch (Exception ex) { + // Never let a failure to list users or build requests kill the task; retry next poll. + logger.error("Failed to generate Huawei requests: {}", ex.toString(), ex); + } + lastPollHadRecords = !sourceRecords.isEmpty(); logger.info("Processed {} records from {} URLs", sourceRecords.size(), requestsGenerated); return sourceRecords; @@ -198,6 +211,7 @@ public List poll() throws InterruptedException { @Override public void stop() { logger.debug("Stopping source task"); + stopLatch.countDown(); } @Override diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java index 8f2aec3c..7c84198d 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/offset/KafkaOffsetManager.java @@ -21,7 +21,7 @@ import java.time.Instant; import java.util.List; import java.util.Map; -import java.util.stream.Collectors; +import java.util.concurrent.ConcurrentHashMap; import static java.time.temporal.ChronoUnit.NANOS; import org.apache.kafka.connect.storage.OffsetStorageReader; import org.radarbase.huawei.offset.Offset; @@ -40,7 +40,7 @@ public class KafkaOffsetManager implements HuaweiOffsetManager { private static final Duration ONE_NANO = NANOS.getDuration(); private final OffsetStorageReader offsetStorageReader; - private Map offsets; + private final Map offsets = new ConcurrentHashMap<>(); public KafkaOffsetManager(OffsetStorageReader offsetStorageReader) { this.offsetStorageReader = offsetStorageReader; @@ -48,11 +48,11 @@ public KafkaOffsetManager(OffsetStorageReader offsetStorageReader) { public void initialize(List> partitions) { if (this.offsetStorageReader != null) { - this.offsets = this.offsetStorageReader.offsets(partitions).entrySet().stream() - .filter(e -> e.getValue() != null && e.getValue().containsKey(TIMESTAMP_OFFSET_KEY)) - .collect(Collectors.toMap( - e -> e.getKey().get("user") + "-" + e.getKey().get("route"), - e -> Instant.ofEpochSecond(((Number) e.getValue().get(TIMESTAMP_OFFSET_KEY)).longValue()))); + this.offsetStorageReader.offsets(partitions).entrySet().stream() + .filter(e -> e.getValue() != null && e.getValue().get(TIMESTAMP_OFFSET_KEY) instanceof Number) + .forEach(e -> offsets.put( + e.getKey().get("user") + "-" + e.getKey().get("route"), + Instant.ofEpochSecond(((Number) e.getValue().get(TIMESTAMP_OFFSET_KEY)).longValue()))); } else { logger.warn("Offset storage reader is null, will resume from an empty state."); } diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt index d0c5d088..002ea1d4 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiServiceUserRepository.kt @@ -204,6 +204,10 @@ class HuaweiServiceUserRepository : HuaweiUserRepository() { } } + override fun invalidateAccessToken(user: User) { + credentialCaches -= user.id + } + @Throws(IOException::class, UserNotAuthorizedException::class) override fun refreshAccessToken(user: User): String { if (!user.isAuthorized) { diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java index 42f2edf8..7c0ecc16 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/OAuth2UserCredentials.java @@ -75,6 +75,11 @@ protected static Instant getExpiresAt(Duration expiresIn) { .minus(EXPIRY_TIME_MARGIN); } + /** Mark the access token as expired, so it is refreshed before its next use. */ + public void invalidateAccessToken() { + expiresAt = Instant.EPOCH; + } + @JsonIgnore public boolean isAccessTokenExpired() { return expiresAt == null || Instant.now().isAfter(expiresAt); From ea00a4d841c8581cbb8c4447ed58e9f8bd3c56e2 Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Wed, 23 Sep 2026 21:00:53 +0000 Subject: [PATCH 37/44] Fix YAML user repository for multi-task and long-running use - Only stream users assigned to the task (huawei.users), so users aren't polled by every task. - Give HuaweiLocalUser identity-based equals/hashCode so token refreshes don't trigger an hourly task reconfiguration. - Reload user files edited on disk; skip unreadable or duplicate files instead of dropping all users; write temp files next to the target. - Send client credentials as form parameters to Huawei's token endpoint and include the error body when a refresh token is rejected. - Support access-token invalidation; update config test for routes that are disabled by default. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../rest/huawei/user/HuaweiLocalUser.kt | 24 ++++ .../huawei/user/HuaweiYamlUserRepository.kt | 97 +++++++++++---- .../HuaweiRestSourceConnectorConfigTest.kt | 6 +- .../user/HuaweiYamlUserRepositoryTest.kt | 116 ++++++++++++++++++ 4 files changed, 216 insertions(+), 27 deletions(-) create mode 100644 kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepositoryTest.kt diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt index a8a665ec..b3f15b06 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiLocalUser.kt @@ -22,6 +22,7 @@ import com.fasterxml.jackson.annotation.JsonProperty import org.radarbase.huawei.user.User import org.radarcns.kafka.ObservationKey import java.time.Instant +import java.util.Objects /** * A single user's Huawei Health Kit credentials, read from (and written back to) a local YAML @@ -100,5 +101,28 @@ class HuaweiLocalUser : User { return copy } + /** + * Equality covers the user's identity and polling configuration, but not its OAuth2 tokens or + * [createdAt] (which defaults to the read time when absent from the file): the connector + * compares user sets to decide whether tasks need reconfiguring, and a token refresh must + * not trigger that. + */ + override fun equals(other: Any?): Boolean { + if (this === other) return true + if (other !is HuaweiLocalUser) return false + return id == other.id && + projectId == other.projectId && + userId == other.userId && + sourceId == other.sourceId && + externalId == other.externalId && + startDate == other.startDate && + endDate == other.endDate && + serviceUserId == other.serviceUserId && + version == other.version && + isAuthorizedOverride == other.isAuthorizedOverride + } + + override fun hashCode(): Int = Objects.hash(id, projectId, userId, sourceId, version) + override fun toString(): String = "HuaweiLocalUser(id='$id', versionedId='$versionedId')" } diff --git a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepository.kt b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepository.kt index 97e09b6f..adde2b86 100644 --- a/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepository.kt +++ b/kafka-connect-huawei-source/src/main/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepository.kt @@ -23,7 +23,6 @@ import com.fasterxml.jackson.dataformat.yaml.YAMLFactory import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule import com.fasterxml.jackson.module.kotlin.registerKotlinModule import okhttp3.FormBody -import okhttp3.Headers import okhttp3.OkHttpClient import okhttp3.Request import org.radarbase.connect.rest.huawei.HuaweiRestSourceConnectorConfig @@ -36,7 +35,6 @@ import java.nio.file.Path import java.nio.file.StandardCopyOption import java.time.Duration import java.time.Instant -import java.util.Base64 import java.util.concurrent.ConcurrentHashMap import java.util.concurrent.atomic.AtomicReference import java.util.concurrent.locks.ReentrantLock @@ -58,14 +56,17 @@ class HuaweiYamlUserRepository : HuaweiUserRepository() { private val users = ConcurrentHashMap() private val nextFetch = AtomicReference(Instant.EPOCH) private lateinit var credentialsDir: Path - private lateinit var clientCredentials: Headers + private lateinit var clientId: String + private lateinit var clientSecret: String + private var containedUsers: Set = emptySet() override fun initialize(config: HuaweiRestSourceConnectorConfig) { credentialsDir = config.getHuaweiUserCredentialsPath() + // Each task is assigned a subset of users (by versioned ID); empty means all users. + containedUsers = config.getHuaweiUsers().toHashSet() Files.createDirectories(credentialsDir) - val credentialString = "${config.getHuaweiClient()}:${config.getHuaweiClientSecret()}" - val credentialsBase64 = Base64.getEncoder().encodeToString(credentialString.toByteArray()) - clientCredentials = Headers.headersOf("Authorization", "Basic $credentialsBase64") + clientId = config.getHuaweiClient() + clientSecret = config.getHuaweiClientSecret() } override operator fun get(key: String): User? { @@ -78,7 +79,12 @@ class HuaweiYamlUserRepository : HuaweiUserRepository() { applyPendingUpdates() } return users.values.asSequence() - .filter { it.locked { u -> u.oauth2Credentials.hasRefreshToken() } } + .filter { + it.locked { u -> + u.oauth2Credentials.hasRefreshToken() && + (containedUsers.isEmpty() || u.versionedId in containedUsers) + } + } .map { it.locked { u -> u.copy() } } } @@ -93,6 +99,10 @@ class HuaweiYamlUserRepository : HuaweiUserRepository() { return current ?: refreshAccessToken(user) } + override fun invalidateAccessToken(user: User) { + users[user.id]?.update { it.oauth2Credentials.invalidateAccessToken() } + } + @Throws(IOException::class, UserNotAuthorizedException::class) override fun refreshAccessToken(user: User): String { val actual = users[user.id] @@ -107,7 +117,7 @@ class HuaweiYamlUserRepository : HuaweiUserRepository() { actual.update { u -> u.oauth2Credentials = OAuth2UserCredentials(newRefreshToken, accessToken, expiresIn) - store(actual.path, u) + store(actual, u) } return accessToken } @@ -130,25 +140,51 @@ class HuaweiYamlUserRepository : HuaweiUserRepository() { } private fun forceUpdateUsers() { - try { + val paths = try { Files.walk(credentialsDir).use { walker -> - val newUsers = walker + walker .filter { Files.isRegularFile(it) && it.fileName.toString().lowercase().endsWith(".yml") } - .map { path -> - LockedUser( - YAML_READER.readValue(path.toFile(), HuaweiLocalUser::class.java), - path, - ) - } - .collect(Collectors.toMap({ it.locked { u -> u.id } }, { it })) - users.keys.retainAll(newUsers.keys) - newUsers.forEach { (id, u) -> users.putIfAbsent(id, u) } + .sorted() + .collect(Collectors.toList()) } } catch (ex: IOException) { logger.error("Failed to read user directory: {}", ex.toString()) + return + } + val newUsers = LinkedHashMap() + paths.forEach { path -> + // A single malformed or unreadable file must not hide every other user. + val lockedUser = try { + LockedUser( + YAML_READER.readValue(path.toFile(), HuaweiLocalUser::class.java), + path, + Files.getLastModifiedTime(path).toInstant(), + ) + } catch (ex: IOException) { + logger.error("Failed to read user file {}: {}", path, ex.toString()) + return@forEach + } + val id = lockedUser.user.id + val existing = newUsers.putIfAbsent(id, lockedUser) + if (existing != null) { + logger.warn( + "Ignoring {}: user ID {} already defined in {}", + path, + id, + existing.path, + ) + } + } + users.keys.retainAll(newUsers.keys) + // Keep in-memory state (e.g. freshly refreshed tokens) unless the file was edited since it + // was last read or written here, e.g. to add a new refresh token. + newUsers.forEach { (id, u) -> + users.merge(id, u) { old, new -> + if (new.modifiedAt > old.modifiedAt || new.path != old.path) new else old + } } } @@ -158,10 +194,13 @@ class HuaweiYamlUserRepository : HuaweiUserRepository() { } val request = Request.Builder() .url(HUAWEI_TOKEN_URL) - .headers(clientCredentials) .post( + // Huawei's OAuth 2.0 token endpoint takes the client credentials as form + // parameters, not as HTTP Basic authentication. FormBody.Builder() .add("grant_type", "refresh_token") + .add("client_id", clientId) + .add("client_secret", clientSecret) .add("refresh_token", refreshToken) .build(), ) @@ -172,7 +211,9 @@ class HuaweiYamlUserRepository : HuaweiUserRepository() { return when { response.isSuccessful && body != null -> JSON_READER.readTree(body) response.code == 400 || response.code == 401 -> - throw UserNotAuthorizedException("Refresh token is no longer valid.") + throw UserNotAuthorizedException( + "Refresh token was rejected (HTTP ${response.code}): $body", + ) else -> throw IOException( "Failed to request refresh token, HTTP status ${response.code}" + (body?.let { " and content $it" } ?: ""), @@ -181,12 +222,16 @@ class HuaweiYamlUserRepository : HuaweiUserRepository() { } } - private fun store(path: Path, user: HuaweiLocalUser) { + private fun store(lockedUser: LockedUser, user: HuaweiLocalUser) { + val path = lockedUser.path try { - val temp = Files.createTempFile(user.id, ".tmp") + // Temp file next to the target (not ending in .yml, so it is never read as a user), + // so the move is a same-directory rename. + val temp = Files.createTempFile(path.parent, ".${user.id}", ".tmp") try { Files.newOutputStream(temp).use { out -> YAML_WRITER.writeValue(out, user) } Files.move(temp, path, StandardCopyOption.REPLACE_EXISTING) + lockedUser.modifiedAt = Files.getLastModifiedTime(path).toInstant() } finally { Files.deleteIfExists(temp) } @@ -196,7 +241,11 @@ class HuaweiYamlUserRepository : HuaweiUserRepository() { } /** Guards a mutable [HuaweiLocalUser] against concurrent read/refresh/store. */ - private class LockedUser(val user: HuaweiLocalUser, val path: Path) { + private class LockedUser( + val user: HuaweiLocalUser, + val path: Path, + @Volatile var modifiedAt: Instant, + ) { private val lock = ReentrantLock() fun locked(block: (HuaweiLocalUser) -> V): V { diff --git a/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt index 585d8775..6ce94da9 100644 --- a/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt +++ b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/HuaweiRestSourceConnectorConfigTest.kt @@ -32,7 +32,7 @@ class HuaweiRestSourceConnectorConfigTest { } @Test - fun `enabled topics default to every registered data type`() { + fun `enabled topics default to every data type enabled by default`() { val config = HuaweiRestSourceConnectorConfig( mutableMapOf( "huawei.api.client" to "client", @@ -43,9 +43,9 @@ class HuaweiRestSourceConnectorConfigTest { val enabled = config.enabledTopics() - assertEquals(HuaweiRouteFactory.definitions.size, enabled.size) HuaweiRouteFactory.definitions.forEach { definition -> - assertEquals(definition.defaultTopic, enabled[definition.key]) + val expected = definition.defaultTopic.takeIf { definition.enabledByDefault } + assertEquals(expected, enabled[definition.key], definition.key) } } diff --git a/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepositoryTest.kt b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepositoryTest.kt new file mode 100644 index 00000000..94538f1c --- /dev/null +++ b/kafka-connect-huawei-source/src/test/java/org/radarbase/connect/rest/huawei/user/HuaweiYamlUserRepositoryTest.kt @@ -0,0 +1,116 @@ +/* + * Copyright 2026 Onsentia + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.radarbase.connect.rest.huawei.user + +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.io.TempDir +import org.radarbase.connect.rest.huawei.HuaweiRestSourceConnectorConfig +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.attribute.FileTime +import java.time.Instant +import kotlin.test.assertEquals + +/** + * @author yatharthranjan + */ +class HuaweiYamlUserRepositoryTest { + @TempDir + lateinit var dir: Path + + private fun writeUser(id: String, refreshToken: String, accessToken: String = "access") { + Files.writeString( + dir.resolve("$id.yml"), + """ + id: $id + projectId: p + userId: $id + sourceId: s + startDate: 2024-01-01T00:00:00Z + oauth2: + accessToken: $accessToken + refreshToken: $refreshToken + expiresAt: 2999-01-01T00:00:00Z + """.trimIndent(), + ) + } + + private fun repository(users: String = ""): HuaweiYamlUserRepository { + val config = HuaweiRestSourceConnectorConfig( + mutableMapOf( + "huawei.api.client" to "client", + "huawei.api.secret" to "secret", + "huawei.user.dir" to dir.toString(), + "huawei.users" to users, + ), + false, + ) + return HuaweiYamlUserRepository().apply { initialize(config) } + } + + @Test + fun `streams only users assigned to the task and skips unreadable files`() { + writeUser("a", "refresh-a") + writeUser("b", "refresh-b") + writeUser("c", "") + Files.writeString(dir.resolve("broken.yml"), "id: [unterminated") + + assertEquals(setOf("a", "b"), repository().stream().map { it.id }.toSet()) + assertEquals(setOf("b"), repository("b").stream().map { it.id }.toSet()) + } + + @Test + fun `users are equal regardless of tokens and read time`() { + writeUser("a", "refresh-a") + val first = repository().stream().single() + writeUser("a", "refresh-a2", accessToken = "other") + val second = repository().stream().single() + + assertEquals(first, second) + assertEquals(first.hashCode(), second.hashCode()) + } + + @Test + fun `invalidated access token is no longer returned as valid`() { + writeUser("a", "refresh-a") + val repository = repository() + val user = repository.stream().single() + assertEquals("access", repository.getAccessToken(user)) + + repository.invalidateAccessToken(user) + + val credentials = (repository["a"] as HuaweiLocalUser).oauth2Credentials + assertEquals(true, credentials.isAccessTokenExpired) + } + + @Test + fun `edited user files are reloaded`() { + writeUser("a", "refresh-a", accessToken = "old") + val repository = repository() + assertEquals("old", repository.getAccessToken(repository.stream().single())) + + writeUser("a", "refresh-a", accessToken = "new") + Files.setLastModifiedTime( + dir.resolve("a.yml"), + FileTime.from(Instant.now().plusSeconds(60)), + ) + repository.applyPendingUpdates() + + assertEquals("new", repository.getAccessToken(repository.stream().single())) + } +} From e9e8d84e26c92b44b629eaef961abc589917882a Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 24 Sep 2026 09:46:36 +0000 Subject: [PATCH 38/44] Use documented heartRateVariabilityRMSSD key for HRV Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../org/radarbase/huawei/route/HuaweiRouteFactory.kt | 9 +++------ .../org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt | 2 +- 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 658f5656..d24462d1 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -786,12 +786,9 @@ object HuaweiRouteFactory { HuaweiHeartRateVariability.newBuilder().apply { time = start.toEpoch() timeReceived = received.toEpoch() - // The official reference's field column is truncated to "...tRateVariabilityRMSSD"; - // the doc's value range is (0, 200] ms, so fractional values are truncated. - heartRateVariabilityRmssd = f.getInt( - "heartRateVariabilityRMSSD", - "heartRateVariabilityRmssd", - ) + // int, milliseconds, (0, 200], per the official "Heart Rate Variability" + // reference. + heartRateVariabilityRmssd = f.getInt("heartRateVariabilityRMSSD") }.build() }, ) diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index ee21805f..e8e9e3f3 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -198,7 +198,7 @@ class HuaweiRouteFactoryTest { "diastolic_pressure_avg", "diastolic_pressure_max", "diastolic_pressure_min", "distance", "distance_delta", "dream_time", "duration", "emotionStatus", "eventName", "exercise_type", "exerciseTime", "exerciseTimeGoal", "extendData", "fall_asleep_time", - "fragments", "go_bed_time", "heartRateVariabilityRMSSD", "heartRateVariabilityRmssd", + "fragments", "go_bed_time", "heartRateVariabilityRMSSD", "highBodyTemperatureAlarm", "intensity", "isActive", "last", "level", "light_sleep_time", "max", "max_body_fat_rate", "max_heart_rate", "maxBreatheRate", "maxBreathrateBaseline", "maxSpO2", "meal", "measure_count", "min", "min_body_fat_rate", From 98c090255f3f25393881a2e9d352d1f9b71d0fef Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 24 Sep 2026 09:48:11 +0000 Subject: [PATCH 39/44] Read only documented ECG detail fields Per the official ECG Measurement Details reference, continuous.ecg_detail has only ecg_type and voltage_datas; stop reading guessed keys. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../radarbase/huawei/route/HuaweiRouteFactory.kt | 15 ++++++--------- .../huawei/route/HuaweiRouteFactoryTest.kt | 8 ++++---- 2 files changed, 10 insertions(+), 13 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index d24462d1..506bd3aa 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -524,15 +524,12 @@ object HuaweiRouteFactory { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - // The official "ECG" reference only documents "ecg_type" and "voltage_datas" - // (a list, serialized here as JSON); the remaining keys are best-effort. - ecgRecordId = f.getString("record_id") - averageHeartRate = f.getInt("avg_heart_rate") - ecgArrhythmiaType = f.getInt("arrhythmia_type") - ecgArrhythmiaResult = f.getInt("arrhythmia_result") - userSymptom = f.getString("user_symptom") - samplingFrequency = f.getInt("sampling_frequency") - voltageData = f.getString("voltage_datas", "voltage_data") + // The official "ECG Measurement Details" reference documents only two fields: + // "ecg_type" (int, mandatory; 1/6/12/18-lead), which this schema has no field + // for yet, and "voltage_datas" (String; a JSON list is serialized as-is). The + // schema's record ID, heart rate, arrhythmia, symptom and sampling frequency + // fields are not part of this data type and are left null. + voltageData = f.getString("voltage_datas") }.build() }, ) diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index e8e9e3f3..2b50e7ac 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -191,7 +191,7 @@ class HuaweiRouteFactoryTest { private val LITERAL_FIELD_KEYS = listOf( "activeCalories", "activeCaloriesGoal", "activeHours", "activeHoursGoal", - "activity_type", "all_sleep_time", "arrhythmia_result", "arrhythmia_type", + "activity_type", "all_sleep_time", "ascent_total", "avg", "avg_body_fat_rate", "avg_heart_rate", "avgBreatheRate", "awake_time", "calories", "calories_total", "correlate_mealtime", "correlate_sleep", "count", "deep_sleep_part", "deep_sleep_time", "descent_total", @@ -203,13 +203,13 @@ class HuaweiRouteFactoryTest { "light_sleep_time", "max", "max_body_fat_rate", "max_heart_rate", "maxBreatheRate", "maxBreathrateBaseline", "maxSpO2", "meal", "measure_count", "min", "min_body_fat_rate", "min_heart_rate", "minBreatheRate", "minBreathrateBaseline", "minSpO2", "off_bed_time", - "onOffBedState", "predictedCalories", "prepare_sleep_time", "record_id", "recordDay", - "remarks", "sample_source", "sampling_frequency", "saturation_avg", "saturation_last", + "onOffBedState", "predictedCalories", "prepare_sleep_time", "recordDay", + "remarks", "sample_source", "saturation_avg", "saturation_last", "saturation_max", "saturation_min", "sleep_efficiency", "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "span", "sphygmus_avg", "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", "steps_delta", "stepsGoal", "subStatus", "systolic_pressure_avg", "systolic_pressure_max", "systolic_pressure_min", "threshold", - "timeZone", "totalCalories", "type", "user_symptom", "value", "vo2max", "voltage_data", + "timeZone", "totalCalories", "type", "value", "vo2max", "voltage_datas", "wakeup_count", "wakeup_time", ) } From e0ab90893bb8d1bef717ecb4a36367cbea2e3ceb Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 24 Sep 2026 09:51:31 +0000 Subject: [PATCH 40/44] Query ECG via continuous.ecg_record health records ECG measurement details are only open through health record queries, so the continuous_ecg_detail route (same key, topic and schema) now queries healthRecords for com.huawei.continuous.ecg_record with subDataType com.huawei.continuous.ecg_detail. Record fields map to the documented ecg_record keys, ecgRecordId comes from the record id, and voltageData from the associated detail points' voltage_datas. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../radarbase/huawei/converter/FieldValues.kt | 17 ++++ .../converter/HuaweiHealthRecordConverter.kt | 18 ++++- .../huawei/route/HuaweiHealthRecordRoute.kt | 21 +++-- .../huawei/route/HuaweiRouteFactory.kt | 47 +++++++++-- .../huawei/route/HuaweiRouteFactoryTest.kt | 79 ++++++++++++++----- 5 files changed, 147 insertions(+), 35 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt index 0513835f..971ef4d8 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt @@ -37,6 +37,23 @@ import com.fasterxml.jackson.databind.JsonNode */ class FieldValues private constructor(private val values: Map) { + /** Huawei's own ID of the record these fields belong to, where the endpoint returns one + * (e.g. `healthRecords`). */ + var recordId: String? = null + private set + + /** Field values of the associated detail sample points of a health record (its + * `subDataDetails`), in response order, when they were requested and returned. */ + var subData: List = emptyList() + private set + + /** Copy of these field values with the given record-level context attached. */ + fun withRecord(recordId: String?, subData: List): FieldValues = + FieldValues(values).also { + it.recordId = recordId + it.subData = subData + } + /* * Every accessor accepts one or more candidate keys and returns the first one present, so a * field whose casing Huawei's docs don't pin down unambiguously can list both spellings. diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt index d1eb7480..dfa22f90 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/HuaweiHealthRecordConverter.kt @@ -25,7 +25,8 @@ import java.time.Instant /** * Generic converter for `GET /healthkit/v2/healthRecords` responses: iterates every record * returned for the requested `dataType` and builds one Avro record per entry via - * [buildRecord]. + * [buildRecord]. The record's `id` and any associated detail sample points are exposed through + * [FieldValues.recordId] and [FieldValues.subData]. * * @author yatharthranjan */ @@ -39,6 +40,16 @@ class HuaweiHealthRecordConverter( ) -> SpecificRecord, ) : HuaweiDataConverter { + /** Detail sample points returned with a record when `subDataType` was requested. The + * HealthRecord model nests them as sample sets; bare sample points are accepted too. */ + private fun JsonNode.subDataPoints(): Sequence { + val details = get("subDataDetails")?.takeIf { it.isArray } ?: return emptySequence() + return details.asSequence().flatMap { detail -> + detail.get("samplePoints")?.takeIf { it.isArray }?.asSequence() + ?: sequenceOf(detail) + } + } + override fun processRecords(root: JsonNode, user: User): Sequence> { val timeReceived = Instant.now() val records = root.get("healthRecords") ?: root.get("records") ?: return emptySequence() @@ -49,6 +60,11 @@ class HuaweiHealthRecordConverter( val endTime = record.epochInstant("endTime") val fieldValues = FieldValues.from( record.get("value") ?: record.get("fieldValues") ?: record.get("field"), + ).withRecord( + recordId = record.get("id")?.takeIf { it.isTextual }?.asText(), + subData = record.subDataPoints() + .map { FieldValues.from(it.get("value") ?: it.get("fieldValues")) } + .toList(), ) TopicData( topic = topic, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt index 9a376e36..3d21a59c 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiHealthRecordRoute.kt @@ -30,7 +30,8 @@ import java.time.Instant /** * Route backed by `GET /healthkit/v2/healthRecords`, used for the `health.record.*` data types * (ambulatory blood pressure sessions, heart rate alerts, hyperthermia, low SpO2 alerts, - * menstrual cycle phases, and comprehensive sleep records). + * menstrual cycle phases, and comprehensive sleep records) and other record-style types such as + * ECG measurement records, optionally with their associated detail data (`subDataType`). * * Per the official Health Kit REST API reference, this endpoint is on API version `v2` (unlike * `sampleSet:polymerize`/`activityRecords`, which are on `v1`), takes the data type under the @@ -44,6 +45,8 @@ open class HuaweiHealthRecordRoute( userRepository: UserRepository, private val dataTypeName: String, private val topic: String, + /** Associated detail data types to return with each record (`subDataType`). */ + private val subDataTypes: List = emptyList(), maxIntervalPerRequest: Duration = Duration.ofDays(30L), buildRecord: ( fields: FieldValues, @@ -68,11 +71,17 @@ open class HuaweiHealthRecordRoute( request = createGetRequest( user, "healthRecords", - mapOf( - "dataType" to dataTypeName, - "startTime" to rangeStart.toEpochNanos().toString(), - "endTime" to rangeEnd.toEpochNanos().toString(), - ), + buildMap { + put("dataType", dataTypeName) + put("startTime", rangeStart.toEpochNanos().toString()) + put("endTime", rangeEnd.toEpochNanos().toString()) + if (subDataTypes.isNotEmpty()) { + put( + "subDataType", + subDataTypes.joinToString(","), + ) + } + }, baseUrl = HUAWEI_API_BASE_URL_V2, ), user = user, diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 506bd3aa..27e024ce 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -17,6 +17,7 @@ package org.radarbase.huawei.route +import com.fasterxml.jackson.databind.ObjectMapper import org.apache.avro.specific.SpecificRecord import org.radarbase.huawei.converter.FieldValues import org.radarcns.connector.huawei.HuaweiActiveHours @@ -75,6 +76,7 @@ import java.time.Instant object HuaweiRouteFactory { private const val VENDOR_PREFIX = "com.huawei." + private val MAPPER = ObjectMapper() private fun Instant.toEpoch(): Double = toEpochMilli() / 1000.0 @@ -515,21 +517,34 @@ object HuaweiRouteFactory { ) add( - sampleSetDefinition( + // Keeps its historical key/topic (matching the HuaweiContinuousEcgDetail schema), but + // is backed by the "ECG Measurement Records" health record type + // com.huawei.continuous.ecg_record: ECG measurement details + // (com.huawei.continuous.ecg_detail) are only open through health record queries, as + // the detail data associated with each record. + healthRecordDefinition( "continuous_ecg_detail", - "continuous.ecg_detail", + "continuous.ecg_record", "connect_huawei_continuous_ecg_detail", + subDataTypes = listOf(VENDOR_PREFIX + "continuous.ecg_detail"), ) { f, start, end, received -> HuaweiContinuousEcgDetail.newBuilder().apply { time = start.toEpoch() timeReceived = received.toEpoch() endTime = end?.toEpoch() - // The official "ECG Measurement Details" reference documents only two fields: - // "ecg_type" (int, mandatory; 1/6/12/18-lead), which this schema has no field - // for yet, and "voltage_datas" (String; a JSON list is serialized as-is). The - // schema's record ID, heart rate, arrhythmia, symptom and sampling frequency - // fields are not part of this data type and are left null. - voltageData = f.getString("voltage_datas") + ecgRecordId = f.recordId + // Documented as a float (bpm); the schema field is an int. + averageHeartRate = f.getDouble("avg_heart_rate")?.let { Math.round(it).toInt() } + // Bit flags (1: sinus rhythm, 2: atrial premature beats, ..., 128: poor + // signals), documented as a long but only using the low 8 bits. + ecgArrhythmiaType = f.getLong("ecg_arrhythmia_type")?.toInt() + // ecgArrhythmiaResult has no counterpart among the documented ecg_record + // fields and is left null. + // Bit flags of user-selected symptoms, documented as a long; the schema field is + // a string, so the decimal value is kept as-is. + userSymptom = f.getLong("user_symptom")?.toString() + samplingFrequency = f.getInt("sampling_frequency") + voltageData = f.subData.voltageData() }.build() }, ) @@ -1103,10 +1118,25 @@ object HuaweiRouteFactory { } } + /** + * ECG voltage data of all measurement detail points associated with an ECG record: a single + * point's `voltage_datas` string as-is, or a JSON array of each point's value when there are + * several. + */ + private fun List.voltageData(): String? { + val segments = mapNotNull { it.getString("voltage_datas") } + return when (segments.size) { + 0 -> null + 1 -> segments.single() + else -> MAPPER.writeValueAsString(segments) + } + } + private fun healthRecordDefinition( key: String, dataTypeName: String, defaultTopic: String, + subDataTypes: List = emptyList(), buildRecord: ( fields: FieldValues, startTime: Instant, @@ -1118,6 +1148,7 @@ object HuaweiRouteFactory { userRepository = repo, dataTypeName = VENDOR_PREFIX + dataTypeName, topic = topic, + subDataTypes = subDataTypes, buildRecord = buildRecord, ) } diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 2b50e7ac..62c809c1 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -24,6 +24,7 @@ import org.apache.avro.Schema import org.radarbase.huawei.user.HuaweiUser import org.radarbase.huawei.user.User import org.radarbase.huawei.user.UserRepository +import org.radarcns.connector.huawei.HuaweiContinuousEcgDetail import org.radarcns.connector.huawei.HuaweiHealthRecordDynamicBp import java.time.Instant import kotlin.test.Test @@ -92,6 +93,45 @@ class HuaweiRouteFactoryTest { assertTrue(failures.isEmpty(), "Failures:\n" + failures.joinToString("\n")) } + @Test + fun `ECG route reads record id, record fields and associated voltage data`() { + val definition = HuaweiRouteFactory.definitions.single { it.key == "continuous_ecg_detail" } + val route = definition.build(fakeUserRepository, definition.defaultTopic) + val payload = mapper.readTree( + """ + {"healthRecords": [{ + "startTime": $START_NANOS, "endTime": $END_NANOS, + "dataTypeName": "com.huawei.continuous.ecg_record", + "id": "ecg-1", + "value": [ + {"fieldName": "ecg_type", "integerValue": 1}, + {"fieldName": "avg_heart_rate", "floatValue": 75.6}, + {"fieldName": "ecg_arrhythmia_type", "longValue": 8}, + {"fieldName": "user_symptom", "longValue": 1022}, + {"fieldName": "sampling_frequency", "integerValue": 500} + ], + "subDataDetails": [{ + "dataTypeName": "com.huawei.continuous.ecg_detail", + "samplePoints": [ + {"startTime": $START_NANOS, "value": [ + {"fieldName": "voltage_datas", "stringValue": "0.1,0.2"}]} + ] + }] + }]} + """.trimIndent(), + ) + + val record = route.converters.single().processRecords(payload, fakeUser) + .single().getOrThrow().value as HuaweiContinuousEcgDetail + + assertEquals("ecg-1", record.ecgRecordId) + assertEquals(76, record.averageHeartRate) + assertEquals(8, record.ecgArrhythmiaType) + assertEquals("1022", record.userSymptom) + assertEquals(500, record.samplingFrequency) + assertEquals("0.1,0.2", record.voltageData) + } + private fun fixtureFor(route: HuaweiRoute) = when (route) { is HuaweiActivityRecordRoute -> activityRecordFixture() is HuaweiHealthRecordRoute -> healthRecordFixture() @@ -191,26 +231,25 @@ class HuaweiRouteFactoryTest { private val LITERAL_FIELD_KEYS = listOf( "activeCalories", "activeCaloriesGoal", "activeHours", "activeHoursGoal", - "activity_type", "all_sleep_time", - "ascent_total", "avg", "avg_body_fat_rate", "avg_heart_rate", "avgBreatheRate", - "awake_time", "calories", "calories_total", "correlate_mealtime", "correlate_sleep", - "count", "deep_sleep_part", "deep_sleep_time", "descent_total", - "diastolic_pressure_avg", "diastolic_pressure_max", "diastolic_pressure_min", - "distance", "distance_delta", "dream_time", "duration", "emotionStatus", "eventName", - "exercise_type", "exerciseTime", "exerciseTimeGoal", "extendData", "fall_asleep_time", - "fragments", "go_bed_time", "heartRateVariabilityRMSSD", - "highBodyTemperatureAlarm", "intensity", "isActive", "last", "level", - "light_sleep_time", "max", "max_body_fat_rate", "max_heart_rate", "maxBreatheRate", - "maxBreathrateBaseline", "maxSpO2", "meal", "measure_count", "min", "min_body_fat_rate", - "min_heart_rate", "minBreatheRate", "minBreathrateBaseline", "minSpO2", "off_bed_time", - "onOffBedState", "predictedCalories", "prepare_sleep_time", "recordDay", - "remarks", "sample_source", "saturation_avg", "saturation_last", - "saturation_max", "saturation_min", "sleep_efficiency", "sleep_latency", "sleep_score", - "sleep_state", "sleep_type", "span", "sphygmus_avg", "sphygmus_last", "sphygmus_max", - "sphygmus_min", "status", "steps", "steps_delta", "stepsGoal", "subStatus", - "systolic_pressure_avg", "systolic_pressure_max", "systolic_pressure_min", "threshold", - "timeZone", "totalCalories", "type", "value", "vo2max", - "voltage_datas", "wakeup_count", "wakeup_time", + "activity_type", "all_sleep_time", "ascent_total", "avg", "avg_body_fat_rate", + "avg_heart_rate", "avgBreatheRate", "awake_time", "calories", "calories_total", + "correlate_mealtime", "correlate_sleep", "count", "deep_sleep_part", "deep_sleep_time", + "descent_total", "diastolic_pressure_avg", "diastolic_pressure_max", + "diastolic_pressure_min", "distance", "distance_delta", "dream_time", "duration", + "ecg_arrhythmia_type", "emotionStatus", "eventName", "exercise_type", "exerciseTime", + "exerciseTimeGoal", "extendData", "fall_asleep_time", "fragments", "go_bed_time", + "heartRateVariabilityRMSSD", "highBodyTemperatureAlarm", "intensity", "isActive", + "last", "level", "light_sleep_time", "max", "max_body_fat_rate", "max_heart_rate", + "maxBreatheRate", "maxBreathrateBaseline", "maxSpO2", "meal", "measure_count", "min", + "min_body_fat_rate", "min_heart_rate", "minBreatheRate", "minBreathrateBaseline", + "minSpO2", "off_bed_time", "onOffBedState", "predictedCalories", "prepare_sleep_time", + "recordDay", "remarks", "sample_source", "sampling_frequency", "saturation_avg", + "saturation_last", "saturation_max", "saturation_min", "sleep_efficiency", + "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "span", "sphygmus_avg", + "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", "steps_delta", + "stepsGoal", "subStatus", "systolic_pressure_avg", "systolic_pressure_max", + "systolic_pressure_min", "threshold", "timeZone", "totalCalories", "type", + "user_symptom", "value", "vo2max", "voltage_datas", "wakeup_count", "wakeup_time", ) } } From a874c6559e31c285fd1d5cd4cde07b7b112bf98f Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 24 Sep 2026 09:55:05 +0000 Subject: [PATCH 41/44] Disable undocumented body temperature rest statistics by default The official Body Temperature reference defines only body and skin temperature (detailed and statistics); there is no resting variant. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../radarbase/huawei/route/HuaweiRouteFactory.kt | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 27e024ce..c8d714fe 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -147,6 +147,13 @@ object HuaweiRouteFactory { Triple("vo2max_statistics", "vo2max.statistics", "connect_huawei_vo2max_statistics"), ) + /** + * Keys of [genericStatisticsTypes] absent from Huawei's data type references (the official + * "Body Temperature" reference only defines body and skin temperature, with no resting + * variant), so disabled unless explicitly enabled. + */ + private val undocumentedStatisticsTypes = setOf("continuous_body_temperature_rest_statistics") + /** Full registry of Huawei Health Kit data types supported by this connector. */ val definitions: List = buildList { add( @@ -358,7 +365,12 @@ object HuaweiRouteFactory { genericStatisticsTypes.forEach { (key, dataType, topic) -> add( - sampleSetDefinition(key, dataType, topic) { f, start, end, received -> + sampleSetDefinition( + key, + dataType, + topic, + enabledByDefault = key !in undocumentedStatisticsTypes, + ) { f, start, end, received -> HuaweiStatistics.newBuilder().apply { populateCommon( start, From d8aaa57d43018811403797fa3dd2d6439335e57c Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 24 Sep 2026 10:05:31 +0000 Subject: [PATCH 42/44] Map new ECG schema fields; don't coerce non-numeric text to 0 - Fill ecgType, float averageHeartRate, long userSymptom, ecgAlgorithmVersion, ecgDataSources, ecgDataLength and packageName from the documented ecg_record fields (RADAR-Schemas #430 update). - FieldValues numeric getters now return null for non-numeric text and non-scalar values instead of Jackson's default 0. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../radarbase/huawei/converter/FieldValues.kt | 18 +++++++++++++----- .../huawei/route/HuaweiRouteFactory.kt | 17 ++++++++++------- .../huawei/converter/FieldValuesTest.kt | 18 ++++++++++++++++++ .../huawei/route/HuaweiRouteFactoryTest.kt | 15 ++++++++++++--- 4 files changed, 53 insertions(+), 15 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt index 971ef4d8..10c97002 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/converter/FieldValues.kt @@ -59,11 +59,17 @@ class FieldValues private constructor(private val values: Map) * field whose casing Huawei's docs don't pin down unambiguously can list both spellings. */ - fun getInt(vararg fields: String): Int? = scalar(fields)?.asInt() + fun getInt(vararg fields: String): Int? = number(fields)?.let { + if (it.isNumber) it.asInt() else it.asText().trim().toIntOrNull() + } - fun getLong(vararg fields: String): Long? = scalar(fields)?.asLong() + fun getLong(vararg fields: String): Long? = number(fields)?.let { + if (it.isNumber) it.asLong() else it.asText().trim().toLongOrNull() + } - fun getDouble(vararg fields: String): Double? = scalar(fields)?.asDouble() + fun getDouble(vararg fields: String): Double? = number(fields)?.let { + if (it.isNumber) it.asDouble() else it.asText().trim().toDoubleOrNull() + } fun getFloat(vararg fields: String): Float? = getDouble(*fields)?.toFloat() @@ -100,8 +106,10 @@ class FieldValues private constructor(private val values: Map) private fun lookup(fields: Array): JsonNode? = fields.firstNotNullOfOrNull { field -> values[field]?.takeUnless { it.isNull } } - private fun scalar(fields: Array): JsonNode? = - lookup(fields)?.takeIf { it.isValueNode } + /** Numeric or textual node; Jackson's `asInt()` etc. would turn anything else (and + * non-numeric text) into 0 rather than null. */ + private fun number(fields: Array): JsonNode? = + lookup(fields)?.takeIf { it.isNumber || it.isTextual } companion object { private const val FIELD_NAME_KEY = "fieldName" diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index c8d714fe..be464f0d 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -545,17 +545,20 @@ object HuaweiRouteFactory { timeReceived = received.toEpoch() endTime = end?.toEpoch() ecgRecordId = f.recordId - // Documented as a float (bpm); the schema field is an int. - averageHeartRate = f.getDouble("avg_heart_rate")?.let { Math.round(it).toInt() } + ecgType = f.getInt("ecg_type") + averageHeartRate = f.getFloat("avg_heart_rate") // Bit flags (1: sinus rhythm, 2: atrial premature beats, ..., 128: poor // signals), documented as a long but only using the low 8 bits. ecgArrhythmiaType = f.getLong("ecg_arrhythmia_type")?.toInt() - // ecgArrhythmiaResult has no counterpart among the documented ecg_record - // fields and is left null. - // Bit flags of user-selected symptoms, documented as a long; the schema field is - // a string, so the decimal value is kept as-is. - userSymptom = f.getLong("user_symptom")?.toString() + // Bit flags of user-selected symptoms (bit 0: no discomfort ... bit 9: other). + userSymptom = f.getLong("user_symptom") samplingFrequency = f.getInt("sampling_frequency") + ecgAlgorithmVersion = f.getString("ecg_algorithm_version") + // Documented as a String (device vendor name) but an int in the schema, so + // only numeric values are kept. + ecgDataSources = f.getInt("ecg_data_sources") + ecgDataLength = f.getInt("ecg_data_length") + packageName = f.getString("package_name") voltageData = f.subData.voltageData() }.build() }, diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt index 9bb24183..c52d3e19 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/converter/FieldValuesTest.kt @@ -76,6 +76,24 @@ class FieldValuesTest { assertEquals(7.5, fields.getDouble("custom")) } + @Test + fun `non-numeric text is null rather than zero for numeric getters`() { + val node = mapper.readTree( + """ + [ + {"fieldName": "vendor", "stringValue": "HUAWEI"}, + {"fieldName": "count", "stringValue": " 42 "} + ] + """.trimIndent(), + ) + val fields = FieldValues.from(node) + + assertNull(fields.getInt("vendor")) + assertNull(fields.getDouble("vendor")) + assertEquals(42, fields.getInt("count")) + assertEquals(42L, fields.getLong("count")) + } + @Test fun `parses map-typed values`() { val node = mapper.readTree( diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 62c809c1..285f94fa 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -108,7 +108,11 @@ class HuaweiRouteFactoryTest { {"fieldName": "avg_heart_rate", "floatValue": 75.6}, {"fieldName": "ecg_arrhythmia_type", "longValue": 8}, {"fieldName": "user_symptom", "longValue": 1022}, - {"fieldName": "sampling_frequency", "integerValue": 500} + {"fieldName": "sampling_frequency", "integerValue": 500}, + {"fieldName": "ecg_algorithm_version", "stringValue": "1.0"}, + {"fieldName": "ecg_data_sources", "stringValue": "HUAWEI"}, + {"fieldName": "ecg_data_length", "integerValue": 7500}, + {"fieldName": "package_name", "stringValue": "com.huawei.health"} ], "subDataDetails": [{ "dataTypeName": "com.huawei.continuous.ecg_detail", @@ -125,10 +129,15 @@ class HuaweiRouteFactoryTest { .single().getOrThrow().value as HuaweiContinuousEcgDetail assertEquals("ecg-1", record.ecgRecordId) - assertEquals(76, record.averageHeartRate) + assertEquals(1, record.ecgType) + assertEquals(75.6f, record.averageHeartRate) assertEquals(8, record.ecgArrhythmiaType) - assertEquals("1022", record.userSymptom) + assertEquals(1022L, record.userSymptom) assertEquals(500, record.samplingFrequency) + assertEquals("1.0", record.ecgAlgorithmVersion) + assertEquals(null, record.ecgDataSources) + assertEquals(7500, record.ecgDataLength) + assertEquals("com.huawei.health", record.packageName) assertEquals("0.1,0.2", record.voltageData) } From 12b2ba7872da13ae271599bab378c9c4dec46efe Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 24 Sep 2026 11:05:42 +0000 Subject: [PATCH 43/44] Add instantaneous body and skin temperature routes Both query their raw detailed data types via sampleSet:polymerize and share the new HuaweiTemperature schema (RADAR-Schemas #430), reading the documented float "temperature" field. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../huawei/route/HuaweiRouteFactory.kt | 18 +++++++++++++++++ .../huawei/route/HuaweiRouteFactoryTest.kt | 20 ++++++++++--------- 2 files changed, 29 insertions(+), 9 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index be464f0d..0e3b8459 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -53,6 +53,7 @@ import org.radarcns.connector.huawei.HuaweiSleepOnOffBedRecord import org.radarcns.connector.huawei.HuaweiSleepRespiratoryDetail import org.radarcns.connector.huawei.HuaweiSleepRespiratoryEvent import org.radarcns.connector.huawei.HuaweiStatistics +import org.radarcns.connector.huawei.HuaweiTemperature import org.radarcns.connector.huawei.HuaweiVo2Max import java.time.Instant @@ -448,6 +449,23 @@ object HuaweiRouteFactory { }, ) + // Individual temperature readings (both share the HuaweiTemperature schema), per the + // official "Body Temperature" reference: a single float "temperature" field (°C). + listOf( + "instantaneous_body_temperature" to "instantaneous.body.temperature", + "instantaneous_skin_temperature" to "instantaneous.skin.temperature", + ).forEach { (key, dataType) -> + add( + sampleSetDefinition(key, dataType, "connect_huawei_$key") { f, start, _, received -> + HuaweiTemperature.newBuilder().apply { + time = start.toEpoch() + timeReceived = received.toEpoch() + temperature = f.getFloat("temperature") + }.build() + }, + ) + } + add( sampleSetDefinition( "continuous_calories_burnt", diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 285f94fa..94dd8a4a 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -245,20 +245,22 @@ class HuaweiRouteFactoryTest { "correlate_mealtime", "correlate_sleep", "count", "deep_sleep_part", "deep_sleep_time", "descent_total", "diastolic_pressure_avg", "diastolic_pressure_max", "diastolic_pressure_min", "distance", "distance_delta", "dream_time", "duration", - "ecg_arrhythmia_type", "emotionStatus", "eventName", "exercise_type", "exerciseTime", + "ecg_algorithm_version", "ecg_arrhythmia_type", "ecg_data_length", "ecg_data_sources", + "ecg_type", "emotionStatus", "eventName", "exercise_type", "exerciseTime", "exerciseTimeGoal", "extendData", "fall_asleep_time", "fragments", "go_bed_time", "heartRateVariabilityRMSSD", "highBodyTemperatureAlarm", "intensity", "isActive", "last", "level", "light_sleep_time", "max", "max_body_fat_rate", "max_heart_rate", "maxBreatheRate", "maxBreathrateBaseline", "maxSpO2", "meal", "measure_count", "min", "min_body_fat_rate", "min_heart_rate", "minBreatheRate", "minBreathrateBaseline", - "minSpO2", "off_bed_time", "onOffBedState", "predictedCalories", "prepare_sleep_time", - "recordDay", "remarks", "sample_source", "sampling_frequency", "saturation_avg", - "saturation_last", "saturation_max", "saturation_min", "sleep_efficiency", - "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "span", "sphygmus_avg", - "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", "steps_delta", - "stepsGoal", "subStatus", "systolic_pressure_avg", "systolic_pressure_max", - "systolic_pressure_min", "threshold", "timeZone", "totalCalories", "type", - "user_symptom", "value", "vo2max", "voltage_datas", "wakeup_count", "wakeup_time", + "minSpO2", "off_bed_time", "onOffBedState", "package_name", "predictedCalories", + "prepare_sleep_time", "recordDay", "remarks", "sample_source", "sampling_frequency", + "saturation_avg", "saturation_last", "saturation_max", "saturation_min", + "sleep_efficiency", "sleep_latency", "sleep_score", "sleep_state", "sleep_type", "span", + "sphygmus_avg", "sphygmus_last", "sphygmus_max", "sphygmus_min", "status", "steps", + "steps_delta", "stepsGoal", "subStatus", "systolic_pressure_avg", + "systolic_pressure_max", "systolic_pressure_min", "temperature", "threshold", + "timeZone", "totalCalories", "type", "user_symptom", "value", "vo2max", "voltage_datas", + "wakeup_count", "wakeup_time", ) } } From 387081170c565ac20912082e9da2ab08926a6d2f Mon Sep 17 00:00:00 2001 From: yatharthranjan Date: Thu, 24 Sep 2026 13:08:02 +0000 Subject: [PATCH 44/44] Read ecgDataSources as the vendor name string Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun --- .../kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt | 5 ++--- .../org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt index 0e3b8459..880984fd 100644 --- a/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt +++ b/huawei-library/src/main/kotlin/org/radarbase/huawei/route/HuaweiRouteFactory.kt @@ -572,9 +572,8 @@ object HuaweiRouteFactory { userSymptom = f.getLong("user_symptom") samplingFrequency = f.getInt("sampling_frequency") ecgAlgorithmVersion = f.getString("ecg_algorithm_version") - // Documented as a String (device vendor name) but an int in the schema, so - // only numeric values are kept. - ecgDataSources = f.getInt("ecg_data_sources") + // Name of the device vendor that provided the ECG data. + ecgDataSources = f.getString("ecg_data_sources") ecgDataLength = f.getInt("ecg_data_length") packageName = f.getString("package_name") voltageData = f.subData.voltageData() diff --git a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt index 94dd8a4a..cd8e09ae 100644 --- a/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt +++ b/huawei-library/src/test/kotlin/org/radarbase/huawei/route/HuaweiRouteFactoryTest.kt @@ -135,7 +135,7 @@ class HuaweiRouteFactoryTest { assertEquals(1022L, record.userSymptom) assertEquals(500, record.samplingFrequency) assertEquals("1.0", record.ecgAlgorithmVersion) - assertEquals(null, record.ecgDataSources) + assertEquals("HUAWEI", record.ecgDataSources) assertEquals(7500, record.ecgDataLength) assertEquals("com.huawei.health", record.packageName) assertEquals("0.1,0.2", record.voltageData)