KOKKOS: port surf collide/react, computes, fix temp/global/rescale, and QK gas reactions - #17
Open
stanmoore1 wants to merge 31 commits into
Open
KOKKOS: port surf collide/react, computes, fix temp/global/rescale, and QK gas reactions#17stanmoore1 wants to merge 31 commits into
stanmoore1 wants to merge 31 commits into
Conversation
Add SurfCollideAdiabaticKokkos, the GPU-capable port of the adiabatic surface collision model (isotropic scattering conserving particle speed). Follows the established surf_collide_diffuse_kokkos pattern: device-callable collide_kokkos<REACT,ATOMIC_REDUCTION>, Kokkos RNG pool with SPARTA_KOKKOS_EXACT support, DualView counters, and the surf-react KKCopy dispatch. Wire the new sc_type id 5 into update_kokkos (model selection, the 3D/2D/ boundary collision dispatch ladders, post_collide, and backup/restore), and register the files in the KOKKOS Install.sh. Verified: builds with the Kokkos serial backend and runs examples/surf_collide/in.circle.adiabatic with -sf kk, producing surface- collision statistics consistent with the CPU path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add SurfCollideImpulsiveKokkos, the GPU-capable port of the impulsive surface collision model (Rettner-style rejection-sampled reflection with optional soft-sphere energy exchange, step/double-cosine angular variants, and internal-energy redistribution). Follows the surf_collide_diffuse_kokkos pattern: device-callable collide_kokkos<REACT,ATOMIC_REDUCTION>, the impulsive() reflection kernel as KOKKOS_INLINE_FUNCTION (v_f_avg demoted to a local so the method is const/device-safe), Kokkos RNG pool with SPARTA_KOKKOS_EXACT support, DualView counters, dynamic() for VARSURF/CUSTOM Tsurf, and surf-react dispatch. Give the base SurfCollideImpulsive a Kokkos empty constructor, make its members protected, and guard its destructor with copy so KKCopy shallow copies don't double-free the RNG. Wire sc_type id 6 through update_kokkos (selection, the 3D/2D/boundary dispatch ladders, post_collide, backup/restore) and register the files in the KOKKOS Install.sh. Verified: exact-match gate (Serial + SPARTA_KOKKOS_EXACT, 1 thread) is bit-for-bit identical CPU vs -sf kk on examples/surf_collide/in.circle.impulsive and in.beam.impulsive; OpenMP 4-thread run is clean and statistically consistent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add SurfCollideTDKokkos, the GPU-capable port of the td (thermal desorption) surface collision model, including the barrier/initenergy/bond options. Follows the surf_collide_diffuse_kokkos pattern: device-callable collide_kokkos<REACT,ATOMIC_REDUCTION>, the td() reflection kernel plus device erot()/evib() helpers as KOKKOS_INLINE_FUNCTION, Kokkos RNG pool with SPARTA_KOKKOS_EXACT support, DualView counters, dynamic() for VARSURF/CUSTOM Tsurf, and surf-react dispatch. Give the base SurfCollideTD a Kokkos empty constructor, make its members protected, and guard its destructor with copy. Wire sc_type id 7 through update_kokkos (selection, the 3D/2D/boundary dispatch ladders, post_collide, backup/restore) and register the files in the KOKKOS Install.sh. Verified: exact-match gate (Serial + SPARTA_KOKKOS_EXACT, 1 thread) is bit-for-bit identical CPU vs -sf kk on examples/surf_collide/in.circle.td and in.beam.td; OpenMP 4-thread run is clean and statistically consistent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add SurfCollideCLLKokkos, the GPU-capable port of the cll (Cercignani-Lampis- Lord) surface collision model, including the partial (eccentricity) and translate/rotate options and the rotational/vibrational accommodation. Follows the surf_collide_diffuse_kokkos pattern: device-callable collide_kokkos<REACT,ATOMIC_REDUCTION>, the cll() reflection kernel as KOKKOS_INLINE_FUNCTION, Kokkos RNG pool with SPARTA_KOKKOS_EXACT support, DualView counters, dynamic() for VARSURF/CUSTOM Tsurf, and surf-react dispatch. The translate path's random->gaussian() maps to rand_gen.normal(), which is RanKnuth::gaussian() under SPARTA_KOKKOS_EXACT (via RandWrap::normal). Give the base SurfCollideCLL a Kokkos empty constructor, make its members protected, and guard its destructor with copy. Wire sc_type id 8 through update_kokkos (selection, the 3D/2D/boundary dispatch ladders, post_collide, backup/restore) and register the files in the KOKKOS Install.sh. Verified: exact-match gate (Serial + SPARTA_KOKKOS_EXACT, 1 thread) is bit-for-bit identical CPU vs -sf kk on examples/surf_collide/in.circle.cll and in.beam.cll; OpenMP 4-thread run is clean and statistically consistent. Note: those inputs use default cll, so the translate/gaussian branch is not exercised by the gate (it matches by construction in exact mode). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Prerequisite for the surf_react adsorb Kokkos port (GS chemistry re-scatters a
product particle via cmodels[...]->wrapper()). Add a device-callable
KOKKOS_INLINE_FUNCTION wrapper_kokkos(p,norm,flags,coeffs) to
SurfCollideDiffuseKokkos and SurfCollideSpecularKokkos mirroring the host
wrapper(): diffuse applies coeffs[0]=tsurf, coeffs[1]=acc then reflects;
specular reflects. To keep the object const in the functor copy, the device
diffuse() now takes acc as a parameter (collide_kokkos passes the member, so
that path is unchanged) and wrapper_kokkos passes the reaction's coeffs.
These wrappers are not yet called (SurfReactAdsorbKokkos lands in later slices);
this commit only adds the infrastructure. Verified no regression: diffuse and
specular remain bit-for-bit identical CPU vs -sf kk under Serial+EXACT on
examples/surf_collide/in.circle.{diffuse,specular}.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Stan Moore <stanmoore1@gmail.com>
Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add FixTempGlobalRescaleKokkos (style temp/global/rescale/kk), a device port of fix temp/global/rescale. end_of_step() runs two device passes over all particles: a parallel_reduce to accumulate t = sum mass*(v.v), and a parallel_for to rescale velocities by vscale = sqrt(t_target/t_current). The global reduction / MPI_Allreduce / scale-factor math is unchanged from the host version, and particle data is synced to Device and marked modified after rescaling. Verified on a 2d circle flow (in.circle.diffuse + fix temp/global/rescale): CPU vs -sf kk are bit-for-bit identical under Serial+EXACT (the Serial reduce accumulates in index order, matching the host serial sum), the fix demonstrably changes the run vs the no-fix baseline, and a 4-thread OpenMP run is clean and statistically consistent (np within ~0.4% of serial). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Make the standard ablation pipeline (compute isurf/grid -> fix ave/grid -> fix ablate) run under -sf kk. Previously update_kokkos errored with "Kokkos doesn't yet support compute isurf/grid" and fix ave/grid errored on grid/surf (PERGRIDSURF) inputs, so only random-decrement ablation worked on device. - ComputeISurfGridKokkos: new Kokkos compute that tallies per-surf flux/force on-device in surf_tally_kk() (mirrors ComputeSurfKokkos, with the isurf/grid keyword subset). tallyinfo() syncs+compresses the tally to the host arrays the host fix ave/grid PERGRIDSURF path consumes; post-processing to per-grid (grid->collate_array_implicit) stays on the host. Base ComputeISurfGrid gets the Kokkos copy ctor, virtual init_normflux/grow_tally, and the copy/copymode destructor guard. - update_kokkos: partition the active surf-tally computes into compute surf (slist_active_copy) and compute isurf/grid (slist_active_isurf_copy) typed copy arrays; the move kernel's surface-collision loop now invokes surf_tally_kk() on both. nsurf_tally still counts the total (so iorig is saved correctly); nslist_surf + nslist_isurf == nsurf_tally. - fix_ave_grid_kokkos: PERGRIDSURF runs on the host (the device per-surf tally is brought to the host by the compute's tallyinfo(), then the host base class collates to per-grid). Skip Kokkos allocation in the ctor and delegate init/setup/end_of_step/grow_percell to FixAveGrid; the grow_percell delegation fixes a shutdown invalid-free when fix balance changed the grid (host arrays were being reallocated with Kokkos memory). Verified bit-for-bit CPU vs -sf kk under Serial+EXACT on examples/ablation/ in.ablation.2d (full compute isurf/grid -> fix ave/grid -> fix ablate -> fix balance chain): Np, Nscoll, Nscheck and f_ablate (surface state 716805 -> 714047 over 500 steps) all identical. Also verified no regression in the compute surf tally path (adjust_temp/in.circle.constant, compute surf etot) and the no-tally path (surf_collide/in.circle.diffuse). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add SurfReactAdsorbKokkos (style adsorb/kk), a device port of the gas-surface
(GS) chemistry in surf_react adsorb for box-face (face) mode.
- SurfReactAdsorbKokkos: flattens the GS reaction tables (type/style/k_react,
kisliuk + energy coeffs, per-reactant/product state/part/stoich/ad-index,
product species) and the per-face state (total_state, area, weight,
species_state, atomic species_delta) into device views. react_kokkos()
mirrors SurfReactAdsorb::react() for face/GS: per-reaction probability
(including kisliuk and surf-coverage S_theta), scatter probability, and the
reaction executions that need no post-reaction collision model
(AA/EXCHANGE/RECOMBINATION/LH3/CD, plus DISSOCIATION particle creation).
tally_update() pulls device counts + per-face deltas to the host and reuses
the host MPI state-sync (update_state_face). Unsupported features (ps
chemistry, surf-element mode, and the DA/LH1/ER/CI types or any cmodel
post-reaction scatter) error clearly at init rather than silently mis-running.
- surf_collide cll: dispatch sr_type==2 (adsorb) in collide_kokkos, with the
setup/backup/restore wiring; guard the react block with isr >= 0 so box-face
collisions on non-reacting faces are safe.
- update_kokkos: select a REACT=1 move variant when reactions exist without
explicit surfs (move<DIM,0,1,0>), so box-face/boundary reactions are invoked
on device (previously REACT was only enabled when surf->exist).
- surf_react_adsorb base: add the Kokkos copy ctor, make init/tally_update
virtual, members protected, and guard the destructor for copies.
Verified bit-for-bit CPU vs -sf kk under Serial+EXACT on a face-mode GS run
(beam onto a zlo box face, cll collide + adsorb, sample-GS_1.surf O(g)->O(s)):
identical stats and 28194 surface reactions. GS_2 (cmodel reaction types)
errors cleanly under -sf kk. No regression: surf_collide/in.circle.{diffuse,
cll} and adjust_temp/in.circle.constant (compute surf) remain bit-for-bit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Stan Moore <stanmoore1@gmail.com>
Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Extend SurfReactAdsorbKokkos::react_kokkos() to the remaining GS reaction types (DA/LH1/ER/CI) and post-reaction collision-model (cmodel) scatter for the NOMODEL and SPECULAR cases (specular mirrors SurfCollideSpecular::wrapper, a velocity reflect with no RNG, so it is exactly bit-for-bit). - react_kokkos: DA (gas-product handling + optional second-particle creation), LH1/ER (set product species + cmodel scatter), CI (set product species + optional second-particle creation + cmodel scatter), mirroring the host SurfReactAdsorb::react() control flow and RNG draw order (particle-id draws use the surf-react RNG, matching the host). Add apply_cmodel() (SPECULAR reflect via MathExtraKokkos::reflect3) and create_particle() helpers. - init guard relaxed: DA/LH1/ER/CI are now supported; only RNG-based cmodels (diffuse/cll/td/adiabatic/impulsive) still error clearly at init. Verified bit-for-bit CPU vs -sf kk (Serial+EXACT) on the beam/face GS_2 set with cmodels switched to specular: identical stats and per-reaction tallies (AA 21055, LH1 2, LH3 6475, ER 662, CI 10; total 28204). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add device post-reaction collision-model (cmodel) scatter so the real gas-surface examples (GS_2: cll + td cmodels, DA/LH1/ER/CI types) run under -sf kk and match the host bit-for-bit. RNG approach (per guidance): each cmodel scatter draws from the Kokkos RNG wrapper (RandPoolWrap) initialized from that cmodel's own RanKnuth (cmodels[idx]->random, reached via a new SurfCollide::kokkos_random() accessor). In EXACT serial RandPoolWrap::init points thread 0 at the same RanKnuth the host uses, so the replicated scatter draws in the host order and matches exactly. The Kokkos scatter device functions for diffuse/cll/td (incl. rotational/vibrational energy accommodation, mirroring the surf-collide Kokkos kernels) are replicated inline in SurfReactAdsorbKokkos to avoid the cll<->adsorb include cycle; specular is an inline reflect (no RNG). - SurfReactAdsorbKokkos: scatter_cmodel() dispatch + diffuse_scatter/cll_scatter/ td_scatter/erot_kk/evib_kk; flatten per-reaction cmodel coeffs/flags (ip+jp) to device; one RandPoolWrap per cmodel type built in init_cmodels_kokkos(); capture boltz + collide rot/vib styles + per-cmodel RNG in pre_react(). - surf_collide: add kokkos_random() accessor (base returns NULL; cll/td/diffuse/ adiabatic/impulsive return their RanKnuth). - init guard now only rejects adiabatic/impulsive cmodels. Verified bit-for-bit CPU vs -sf kk (Serial+EXACT) on examples/surf_react_adsorb/ in.beam.face.gs (GS_2): identical stats and per-reaction tallies (total 28160; AA 20986, LH1 2, LH3 6529, ER 638, CI 5). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Generalize SurfReactAdsorbKokkos from box-face-only to also support surf mode (reactions on explicit surface elements), covering in.beam.surf.gs and in.circle.gs (2d/3d). - react_kokkos: state index is the box face (FACE) or the local surf index (SURF); in SURF mode mark the reacting surf for the periodic collate. - per-state device arrays sized nstate_ = nface (FACE) or nlocal+nghost surfs (SURF); add a per-surf mark dual view. - pre_react: in SURF mode refresh the host state pointers from the surf custom arrays (nstick_total/nstick_species/area/weight) before syncing to device. - tally_update: bring device per-surf deltas + mark to the host and reuse the host update_state_surf() (rendezvous collate_array + spread_custom), then re-sync the zeroed deltas/mark to device. - surf_custom: guard Surf::remove_custom() against a freed ewhich. In a Kokkos run SurfKokkos's destructor frees the Kokkos-managed custom data (and nulls ewhich) before the base Surf destructor deletes surf-react instances, whose destructors call remove_custom; without the guard that dereferenced NULL. Verified bit-for-bit CPU vs -sf kk (Serial+EXACT), identical stats and per-reaction tallies: in.beam.surf.gs (3d, 56634 reactions) and in.circle.gs (2d, 26588). FACE mode (in.beam.face.gs) remains bit-for-bit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Enable periodic on-surface (PS) chemistry under -sf kk, completing adsorb
Kokkos support across all shipped examples.
PS_chemistry() runs on the host (it desorbs/inserts particles from the surface
state every nsync steps). In a Kokkos run particles live on the device during
the timestep, so SurfReactAdsorbKokkos::tally_update() now brings the particle
list to the host (sync Host) before the base tally_update()/PS_chemistry()
appends new particles via Particle::add_particle(), and marks the particle data
host-modified afterward so the device picks them up on the next sync.
- remove the gs/ps init guard; support gs, ps, and gs/ps modes.
- init_reactions_gs_kokkos(): tolerate gsflag==0 (PS-only) where reactions_gs is
not allocated (d_reactions_n set to zero so react_kokkos is a no-op).
Verified bit-for-bit CPU vs -sf kk (Serial+EXACT) on all 9 surf_react_adsorb
examples: in.{beam.face,beam.surf,circle}.{gs,gs_ps,ps} -- identical stats and
per-reaction tallies (e.g. beam.surf.gs_ps 104924, circle.gs_ps 50713).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Stan Moore <stanmoore1@gmail.com>
Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Device per-surf reaction tally for implicit surfaces, mirroring the compute isurf/grid Kokkos port. surf_tally_kk() counts reactions per surf (rpflag/reaction2col column mapping), tallied on-device from the move kernel's surface-collision loop; tallyinfo() syncs+compresses to the host, and the host base collates to per-grid. - ComputeReactISurfGridKokkos: surf_tally_kk + lifecycle (init/clear/pre_surf_ tally/post_surf_tally/tallyinfo/grow_tally), reaction2col flattened to device. post_process_isurf_grid() overridden to sync the device tally to the host first, since consumers like fix ablate read the compute directly (no fix ave/grid in between to call tallyinfo()). - base compute_react_isurf_grid: Kokkos copy ctor, virtual grow_tally/ post_process_isurf_grid, copy/copymode destructor guard. - update_kokkos: add a third surf-tally partition (react/isurf/grid) alongside compute surf and compute isurf/grid. Verified bit-for-bit CPU vs -sf kk (Serial+EXACT) on ablation/in.ablation.3d. reactions (reaction-count-driven ablation): identical np/nscoll/nsreact and f_ablate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Device per-surf reaction tally for explicit surfaces, mirroring compute react/isurf/grid but with per-surf output (post_process_surf + collate_array over owned surfs). surf_tally_kk() counts reactions per surf element (rpflag/reaction2col), tallied on-device from the move kernel surface-collision loop; tallyinfo() syncs+compresses to the host; post_process_surf() is overridden to sync first since consumers (dump surf, compute reduce, ...) read the compute directly. - ComputeReactSurfKokkos + base Kokkos copy ctor / virtual grow_tally / copy-guard destructor. - update_kokkos: 4th surf-tally partition (react/surf). Also fixes the nslist_react_surf counter not being reset in tally_set. Verified bit-for-bit CPU vs -sf kk (Serial+EXACT): explicit circle with diffuse collide + global surf react; compute react/surf summed via compute reduce in stats matches (per-step destroy/create reaction counts identical, sum == nsreact). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Device per-owned-surf geometry extraction (id, vertices, centroid, area, normal), mirroring compute property/grid. A RangePolicy over the owned in-group surfs (cglobal) packs each requested field on-device via an index switch, then results sync to the host vector_surf/array_surf for consumers (dump surf, compute reduce, ...). Verified bit-for-bit CPU vs -sf kk (Serial+EXACT): 2d circle, dump surf of id/area/xc/yc/normx/normy -- identical dump files. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add device gas-phase quantum-kinetic chemistry, dispatched from the collide vss/kk kernel. - ReactQKKokkos (qk/kk): device attempt_kk mirroring ReactQK::attempt -- DISSOCIATION + EXCHANGE via discrete vibrational-level thresholds and (for exchange) the rejection-sampling loop, drawing one random_prob per attempt. - ReactTCEQKKokkos (tce/qk/kk): device attempt_kk mirroring ReactTCEQK::attempt -- per reaction, ARRHENIUS style uses the simple TCE probability, QUANTUM style uses the QK model; each evaluated reaction draws its own random number before its inner energy screen, matching the host RNG order exactly. - Both flatten the VSS omega for all species pairs to a device array at init (collide->extract) and reuse the ReactBirdKokkos rand_pool / d_rlist / d_reactions / d_tally_reactions; recombination + compute_chem_rates rejected at init as on the host. - collide_vss_kokkos: dispatch the per-collision react attempt by style (TCE / QK / TCE-QK) via dynamic_cast at setup + a react_style branch in perform_collision_kokkos; previously the kernel hardcoded ReactTCEKokkos. Verified bit-for-bit CPU vs -sf kk (Serial+EXACT) on an N2/N dissociation chem box: react qk (identical np/nreact/temp trajectory) and react tce/qk with a mixed A/Q reaction file (QK path firing). Both run clean and reproducible at OMP_NUM_THREADS=4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add Kokkos accelerator documentation for the styles ported in this
branch. In Section_commands.txt, append the "(k)" marker to:
fix temp/global/rescale
compute isurf/grid, property/surf, react/surf, react/isurf/grid
surf_collide adiabatic, cll, impulsive, td
surf_react adsorb
In each style's doc page, add the "/kk" style name (syntax heading
and/or style/args list) and the standard KOKKOS accelerator boilerplate
paragraph:
surf_collide.txt - cll/kk, adiabatic/kk, impulsive/kk, td/kk
(also transparent/kk in the style line)
surf_react_adsorb.txt - adsorb/kk + boilerplate
react.txt - qk/kk, tce/qk/kk
compute_isurf_grid.txt, compute_property_surf.txt,
compute_react_surf.txt, compute_react_isurf_grid.txt,
fix_temp_global_rescale.txt - /kk heading + boilerplate
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Stan Moore <stanmoore1@gmail.com>
Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
This binary read_isurf data file was unintentionally removed during the KOKKOS porting work and is still referenced by examples/ablation/in.ablation.3d.reactions. Restore it from master. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Stan Moore <stanmoore1@gmail.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Sync the generated .html doc pages with the .txt changes from this
branch (the /kk style names, accelerator boilerplate, and "(k)" command
list markers), so the published html matches the source:
- surf_collide.html: cll/kk, adiabatic/kk, impulsive/kk, td/kk (and
transparent/kk) in the style line and per-style args
- surf_react_adsorb.html: adsorb/kk heading + style + boilerplate
- react.html: qk/kk and tce/qk/kk args
- compute_isurf_grid.html, compute_property_surf.html,
compute_react_surf.html, compute_react_isurf_grid.html,
fix_temp_global_rescale.html: /kk heading + boilerplate
- Section_commands.html: "(k)" markers for the ported styles
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Stan Moore <stanmoore1@gmail.com>
Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add a Kokkos NTC collision path for mixtures with ngroup > 1, mirroring the non-Kokkos Collide::collisions_group algorithm. Previously the vss/kk style errored out whenever the collision mixture defined more than one group. Scope is limited to the case where group membership is static within a timestep, so collisions can run in a parallel per-cell kernel and the result is bit-for-bit identical to the non-Kokkos version: no gas-phase reactions, no ambipolar approximation, and no near-neighbor (nearcp) selection. Those combinations still raise a clear "not (yet) supported" error. Implementation: - new TagCollideCollisionsGroup kernel and collisions_group() launcher - per-cell group partitioning built on device into d_glist with per-group offsets, in the same per-group order as the CPU version - attempt counts pre-computed per group pair into d_nattempt_pair, with RN drawn for every pair to preserve the CPU collision RNG ordering - attempt_collision_kokkos() overload for a pair of groups - device copy of mixture species-to-group map (d_species2group) Verified bit-for-bit identical to the non-Kokkos path (Serial build with SPARTA_KOKKOS_EXACT, 1 thread) over 500 steps of a 2-group air case, and statistically consistent with 4 OpenMP threads. Adds examples/collide/ in.collide.group as a 2-group regression case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM Co-authored-by: stanmoore1 <stanmoore1@gmail.com>
…s/kk Add Kokkos-enabled compute gas/collision/grid and compute gas/reaction/grid (the per-grid gas tally computes) and wire them into the vss/kk collision kernels. Previously any active gas/gas tally compute caused the Kokkos path to error out. New computes (compute_gas_collision_grid_kokkos, compute_gas_reaction_grid_kokkos): - subclass the CPU computes, allocate the per-grid output as a DualView (vector_grid, or array_grid for reaction every/select modes) and override clear()/reallocate() to manage it on device - provide an on-device gas_tally_kk() that mirrors the CPU gas_tally() filtering (grid group mask + mixture species2group) and accumulates the tally directly into the per-cell slot - pre_gas_tally()/post_gas_tally() bind device views before the kernel and sync the result back to the host array afterward Because Collide parallelizes over grid cells (one icell per work item), the per-cell tally has no cross-thread write contention, so it is written directly with no atomics or ScatterView duplication. collide_vss_kokkos: - setup_gas_tally()/finish_gas_tally() partition update->glist_active into typed KKCopy lists, validate each is a supported per-grid Kokkos compute (the per-event gas/collision/tally and gas/reaction/tally are rejected with a clear error), and call pre/post hooks - the collisions_one, collisions_group, and ambipolar kernels invoke gas_tally_kk() on each active compute when GASTALLY is set Base ComputeGasCollisionGrid/ComputeGasReactionGrid get a SPARTA-only ctor for the Kokkos copy pattern. UpdateKokkos::tally_set no longer errors on gas tallies (validation moved to the collide setup). Verified bit-for-bit identical to the non-Kokkos path (Serial build with SPARTA_KOKKOS_EXACT, 1 thread) over 500 steps of a reacting air case, both for stats reductions and a per-cell grid dump covering vector output and the reaction every/select array modes; also verified for non-reacting group collisions. With 4 OpenMP threads the per-cell tally identities (sum gas/collision/grid == ncoll-nreact, sum gas/reaction/grid == nreact) hold exactly. Adds examples/chem/in.chem.gastally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM Co-authored-by: stanmoore1 <stanmoore1@gmail.com>
Add a Kokkos-enabled fix ave/surf that accelerates the dominant use case: time-averaging the per-surf output of surf-tally computes (compute surf, compute react/surf). Previously fix ave/surf ran only on the host in a Kokkos run, syncing each compute's per-surf tally device->host every sample. Design (mirrors fix_ave_grid_kokkos's selective-acceleration philosophy): - The frequent per-sample work runs on device: each value reads a column of its compute's per-local-surf device tally and adds it into a device accumulator d_acc, with no per-sample device->host sync. Collide/move already produce the compute tally on device (post_surf_tally); the new query_tally_surf_kokkos() accessor exposes it without a host round-trip. The accumulation parallelizes over local surfs (one row per work item), so it needs no atomics. - The infrequent per-output work stays on host and mirrors the CPU base class exactly: surf->collate_* merges per-local-surf tallies to owned surfs (MPI), then normalize by sample count and apply the group mask. - The non-tally path (fix/variable/custom inputs, count_tally == 0) is delegated entirely to the host base class. Also fixes a latent bug: FixAveSurf::~FixAveSurf lacked the copymode guard that FixAveGrid has, so when Kokkos copies the fix functor for a kernel and destroys the copy, the shared which/argindex/value2index/ids arrays were freed out from under the real fix. Added the guard and made the base members protected so the Kokkos subclass can reach them. Verified bit-for-bit identical to the non-Kokkos path (Serial build with SPARTA_KOKKOS_EXACT, 1 thread) for both the vector path (ave one) and the multi-column array path (ave running), via surf dumps of the averaged output over 500 steps of examples/adjust_temp/in.circle.constant. The 4-thread OpenMP build runs cleanly. Marks fix ave/surf (k) in the command list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM Co-authored-by: stanmoore1 <stanmoore1@gmail.com>
…de vss/kk Add a Kokkos NTC collision path for ambipolar collisions with more than one mixture group (ngroup > 1), mirroring the non-Kokkos Collide::collisions_group_ambipolar algorithm. Previously the vss/kk style errored out for multigroup ambipolar collisions. As with the plain multigroup port, scope is limited to the non-reacting case (react == NULL), so group membership and the per-cell electron list are static within a timestep and the result is bit-for-bit identical to the non-Kokkos version. Reacting multigroup ambipolar still raises a clear "not (yet) supported" error. Implementation (new TagCollideCollisionsGroupAmbipolar kernel and collisions_group_ambipolar launcher): - per cell, builds group-contiguous lists of real particles in d_glist and a separate electron list d_elist (one electron per ambipolar ion, copied from the ion with its velambi velocity), with the electron species placed in its own group (egroup, required and checked at init) - pre-computes attempt counts per group pair into d_nattempt_pair, skipping electron/electron pairs and drawing RN for every other pair to preserve the non-Kokkos collision RNG ordering - keeps the electron group on the J side of each pair (matching the non-Kokkos gpair igroup/jgroup flip) so collision velocity assignment is identical - after collisions, copies the (scattered) electron velocities back into velambi, exactly as the non-Kokkos version does Verified bit-for-bit identical to the non-Kokkos path (Serial build with SPARTA_KOKKOS_EXACT, 1 thread) over 300 steps of a two-group thermal plasma (heavy species + ambipolar electrons); a non-ambipolar control run confirms the electron-group path is actually exercised. The 4-thread OpenMP build runs cleanly. Adds examples/ambi/in.ambi.group. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM Co-authored-by: stanmoore1 <stanmoore1@gmail.com>
Address issues found in code review: - (major) Make compute gas/collision/grid/kk and gas/reaction/grid/kk inherit KokkosBase and expose their per-grid output through KokkosBase::d_vector_grid / d_array_grid (with a no-op compute_per_grid_kokkos). They set kokkos_flag=1 but previously did not inherit KokkosBase, so feeding one into fix ave/grid/kk passed its kokkos_flag guard and then null-dereferenced the dynamic_cast<KokkosBase*> result. Now fix ave/grid/kk can average these per-grid gas tallies; verified bit-for-bit identical to the non-Kokkos path. - (minor) collide vss/kk: reallocate d_nattempt_pair when ngroups grows, not just when nglocal grows (it is sized ngroups x ngroups); previously a later run with more collision groups could index it out of bounds. - initialize the egroup member (-1) and the SPARTA-only-ctor members of the gas computes; correct the gas-tally instance-limit error text; add a static_assert coupling the VAL_4 KKCopy initializers to KOKKOS_MAX_GLIST. All existing bit-exact checks (group collisions, multigroup ambipolar, per-grid gas tallies, fix ave/surf) still pass on the Serial+EXACT build, and the OpenMP build compiles and runs cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM Co-authored-by: stanmoore1 <stanmoore1@gmail.com>
stanmoore1
force-pushed
the
claude/zen-babbage-qmpx1m
branch
from
June 25, 2026 16:20
3647a58 to
3bd339c
Compare
…qmpx1m # Conflicts: # doc/Section_commands.html
Remove the fix ave/surf/kk port and revert the associated compute surf
device-tally accessors back to master. The device-direct read path in
fix ave/surf/kk bypassed compute surf's host tallyinfo()/post_process_surf()
pipeline while still marking the compute invoked, so a downstream surf-style
variable (e.g. the torque computation in examples/torque) read stale host
tally state and produced zero output.
Unlike the per-grid case, compute surf's output is a compressed/remapped
tally fronted by a stateful host collate stage, which the device fast path
cannot safely skip without porting post_process_surf to device (out of scope).
Reverts:
- src/KOKKOS/fix_ave_surf_kokkos.{h,cpp} (deleted)
- src/fix_ave_surf.{h,cpp} restored to master
- src/KOKKOS/compute_surf_kokkos.h, compute_react_surf_kokkos.h
device-tally accessors removed
- doc ave/surf (k) entry reverted
The remaining Kokkos ports in this branch (non-reacting group collisions,
per-grid gas tallies, non-reacting multigroup ambipolar) are unaffected and
remain bit-exact under SPARTA_KOKKOS_EXACT.
Co-authored-by: stanmoore1 <stanmoore1@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Now that this branch ports the surf_collide adiabatic/cll/td/impulsive
models, surf_react adsorb, and the gas-phase collide features, the KOKKOS
exclusion list (used when SPARTA_KOKKOS_EXACT runs the regression suite with
"-k on -sf kk") no longer needs to skip those inputs. Remove from the
SPARTA_KOKKOS_EXACT skip list the 18 inputs that now run bit-for-bit
identical to the non-KOKKOS gold logs:
in.ablation.2d
in.beam.{adiabatic,cll,impulsive,td}
in.circle.{adiabatic,cll,impulsive,td}
in.beam.face.{gs,gs_ps,ps}
in.beam.surf.{gs,gs_ps,ps}
in.circle.{gs,gs_ps,ps}
Kept excluded (still not KOKKOS-supported):
in.ablation.3d - implicit-surf collision hits zero cell volume under KOKKOS
in.bfield - external field fix not KOKKOS-enabled
in.bfield.grid - external field fix not KOKKOS-enabled
Also add committed CPU gold logs for the new feature examples
(in.collide.group, in.chem.gastally, in.ambi.group) so the KOKKOS job
compares each "-sf kk" run against the non-KOKKOS reference rather than
auto-generating a (trivially passing) KOKKOS-vs-KOKKOS log.
All enabled inputs verified bit-for-bit (Serial backend, SPARTA_KOKKOS_EXACT,
1 thread, np=1) against the non-KOKKOS reference.
Co-authored-by: stanmoore1 <stanmoore1@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
The committed gold logs for in.collide.group, in.chem.gastally, and
in.ambi.group were blessed on a dev machine whose libm/FP behavior differs
from the CI runner (ubuntu-22.04). Over a multi-hundred-step stochastic
collision run, last-bit differences in exp/sqrt/pow accumulate until an
integer collision count flips, so a bit-exact (1e-7) comparison against an
off-platform gold log fails in upstream CI. This is cross-OS numerical drift,
not a correctness bug: on any single machine the KOKKOS ("-k on -sf kk") and
non-KOKKOS runs of these inputs are bit-for-bit identical under
SPARTA_KOKKOS_EXACT.
Drop the gold logs so the regression harness auto-generates the reference on
the CI platform per run (self-consistent, same OS), matching the many other
examples that ship without a committed log. The KOKKOS-vs-non-KOKKOS
bit-exactness of these inputs remains verified locally.
Co-authored-by: stanmoore1 <stanmoore1@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add committed gold logs for the group-collision example, generated on the CI runner (ubuntu-22.04, SPARTA 24 Sep 2025) at 1 and 4 MPI ranks. These replace the dev-machine logs removed earlier: a gold log is only bit-exact on the OS it was blessed on, so the reference must come from the CI platform. With these in place the regression harness performs a real bit-exact comparison (KOKKOS "-k on -sf kk" and non-KOKKOS runs vs the committed reference) rather than auto-generating a self-consistent log. Co-authored-by: stanmoore1 <stanmoore1@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Add committed gold logs for the multigroup-ambipolar and per-grid gas-tally examples, generated on the CI runner (ubuntu-22.04, SPARTA 24 Sep 2025) at 1 and 4 MPI ranks. Same rationale as in.collide.group: a gold log is only bit-exact on the OS it was blessed on, so the reference must come from the CI platform. With these in place all three new KOKKOS examples are held to a real bit-exact comparison (KOKKOS "-k on -sf kk" and non-KOKKOS runs vs the committed reference). Co-authored-by: stanmoore1 <stanmoore1@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Running in.ablation.3d under KOKKOS (SPARTA_KOKKOS_EXACT, "-k on -sf kk") errored at the first post-ablation step with "Collision cell volume is zero". Two distinct KOKKOS-only bugs in the mid-run implicit-surface ablation path were responsible; the first masked the second. 1. Stale device per-cell surf graphs after ablation. fix ablate regenerates the implicit surfaces on the host every Nevery steps, but the device per-cell surf lists (d_csurfs/d_csplits/d_csubs) are only built by wrap_kokkos_graphs() in UpdateKokkos::setup(), i.e. once per run. After ablation the surf move used stale per-cell surf lists, so particles penetrated newly-solid cells and were left in fully-inside (zero-volume) cells, tripping the collision volume check. Fix: add Grid::changed, set by notify_changed(), and have the KOKKOS run loop resync the device grid/surf graphs (mirroring setup()) after any end-of-step fix that changes grid/surf topology. The resync happens between moves, where grid_kk_copy is not live, so it is KKCopy-safe. 2. Per-surf isurf/grid tally arrays not regrown on surf-count change. ComputeISurfGridKokkos sizes its isurf-indexed per-surf tally arrays in init_normflux(), called from reallocate(), which early-returns when grid->nlocal is unchanged. Ablation changes the surf count without changing the cell count, so when the surf count grew the device tally kernel and host tallyinfo() wrote/read out of bounds, corrupting the heap (manifesting as a later double-free/segfault). Fix: override ComputeISurfGridKokkos::reallocate() to re-run init_normflux() (which regrows the arrays and recomputes normflux) when the surf count changes. The sibling react/isurf/grid compute already self-heals in clear(). With both fixes in.ablation.3d runs to completion and is bit-for-bit identical to the non-KOKKOS reference over 100 steps (thermo + f_ablate), matching the committed gold logs. Remove it from the SPARTA_KOKKOS_EXACT skip list. in.ablation.2d and the full KOKKOS regression suite are unaffected. Co-authored-by: stanmoore1 <stanmoore1@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports a substantial set of previously host-only physics to the KOKKOS package, so GPU/threaded runs no longer fall back to (or hard-error on) these models. Every ported style was validated bit-for-bit against the CPU code in exact mode (Serial backend +
SPARTA_KOKKOS_EXACT, 1 thread, the hostRanKnuthRNG path) on its example inputs, and additionally run clean under OpenMP with 4 threads for thread-safety/statistical sanity.Styles ported (new
/kkvariants)Surface collision models (dispatched in the
update_kokkosmove kernel):surf_collide adiabatic/kksurf_collide impulsive/kksurf_collide td/kksurf_collide cll/kkSurface reaction model:
surf_react adsorb/kk— full support: GS (gas-surface) chemistry in both FACE and SURF modes, plus PS (on-surface) chemistry. GS reactions re-scatter via a devicewrapper_kokkosadded to the diffuse/specular/cll/td collide models.Computes (per-surf / per-grid tallies):
compute property/surf/kkcompute react/surf/kkcompute react/isurf/grid/kkcompute isurf/grid/kkFix:
fix temp/global/rescale/kkGas-phase reactions (dispatched in
collide_vss_kokkos):react qk/kk— Bird's quantum-kinetic modelreact tce/qk/kk— hybrid TCE + QK modelAblation: enables the flux-driven implicit-surface ablation path end-to-end under Kokkos.
Implementation notes
surf_collide_diffuse_kokkospattern; sharederot()/evib()/gaussian device helpers. Exact-mode RNG viaRandPoolWrap/RandWrap(thread 0 = hostRanKnuth) for bit-for-bit agreement.protected:access, andif (copy) return;destructor guards (to avoidKKCopydouble-free); aSurf::remove_customguard was added.collide_vss_kokkosselects TCE/QK/TCEQK by an integerreact_styleresolved viadynamic_castat setup.Documentation
Added the
/kkstyle names + standard KOKKOS accelerator boilerplate to each affected doc page, and the(k)marker to the command list inSection_commands.txt.Other
examples/ablation/binary.21x21x21, which had been accidentally removed during the porting work and is still referenced byin.ablation.3d.reactions.🤖 Generated with Claude Code
https://claude.ai/code/session_01Q6nQVuxEiTUqSzDoDppCUM
Generated by Claude Code