docs: dv-connect troubleshooting for auth re-prompt & first-query hang (#63)#69
Open
aadharshkannan wants to merge 1 commit into
Open
Conversation
…sue microsoft#63) Addresses two failure modes reported in microsoft#63 that the skill did not cover: repeated sign-in every session despite valid pac auth, and a simple query that hangs ~2 min then returns nothing. Adds a Troubleshooting section to dv-connect/SKILL.md that: - Clarifies the three separate auth surfaces (PAC CLI, MCP proxy, Python SDK) each have their own token cache, so a valid pac profile does not prove the MCP/Python paths are authenticated -> do not 'fix' by re-running pac auth. - Symptom A (reauth every session): diagnose at the prompting surface (claude mcp list, re-register, --validate GA /api/mcp, verify consent + allowlist); stabilize the proxy's cached auth. - Symptom B (first query hangs ~2 min): the non-interactive device-code trap -- auth.py's interactive DeviceCodeCredential blocks a non-interactive turn until timeout. First-line fix: warm the token cache once (python scripts/auth.py) so the AuthenticationRecord persists and later runs refresh silently; prefer MCP; use a service principal for CI. - Symptom C (sign-in loops after completing login): corrupt token cache; clear the relevant cache (not the PAC profile), re-auth, re-verify. Also softens the over-confident 'this only happens once' note to point at the new section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
d6361e6 to
ff87568
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dv-connect: troubleshooting for "auth keeps re-prompting" and "first query hangs" (issue #63)
Closes the documentation gap behind #63. This PR adds a focused Troubleshooting section to
dv-connect/SKILL.mdfor two symptoms the skill previously did not cover, and softens an over-confident claim that contradicted the reported behavior.The problem (from #63)
A user running the Dataverse plugin in GitHub Copilot reports:
pac auth listinstantly shows a valid profile for the same environment.The previous
dv-connect/SKILL.mdactively reinforced the wrong mental model: at MCP setup it stated the browser sign-in "only happens once; the token is cached for future sessions," and it had no troubleshooting entry for either symptom.Root cause
There are three separate authentication surfaces, each with its own token cache: the PAC CLI (
pac auth), the MCP proxy (@microsoft/dataverse mcp), and the Python SDK path (scripts/auth.py, azure-identity). A validpac auth listonly proves the first. The two reported symptoms come from the other two surfaces:scripts/auth.pybuilds an interactiveDeviceCodeCredentialwhenever there is no saved authentication record and no service principal. In a non-interactive Copilot session, nobody can complete the device-code prompt, so the call blocks until the execution timeout (~2 min) and produces no result.pac authdoes nothing because the PAC profile is already valid.The fix
A new Troubleshooting: auth keeps re-prompting, or the first query hangs section that:
pac auth create/selectwhen the PAC profile is valid.claude mcp listshould show ✓ Connected; re-register the MCP server; run--validateagainst the GA/api/mcpendpoint; verify tenant admin consent + environment allowlist; stabilize the proxy's cached auth.python scripts/auth.pyinteractively so theAuthenticationRecordpersists and later runs refresh silently. Prefer the MCP server for queries; use a service principal (CLIENT_ID/CLIENT_SECRET) for unattended/CI.It also softens the "only happens once" MCP note to point at this section.
How SkillOpt derived and validated this
This change was found and validated with SkillOpt, our skill-evaluation harness. SkillOpt materializes the full Dataverse plugin, swaps in a candidate
SKILL.mdfor the target skill, runs the GitHub Copilot CLI agent against held-out probe prompts, and has an LLM judge score the response against a set of semantic claims. The eval set (7 probes) ships in PR #66 (evals/skillopt/dv_connect_auth.jsonl) and is reproducible.Goldilocks baseline on the current production
dv-connect/SKILL.md:The single baseline failure was the non-interactive device-code item. The judge's verdict on the production skill names the exact gap:
Without the documented guidance, the strong target model jumps straight to a heavyweight service-principal rewrite and omits the lightweight first-line fix (warm the cache once).
A/B stability — the recovered item run 6× against the pristine original vs. 6× against this PR's skill (same harness, same judge):
dv-connect/SKILL.mdThe distributions do not overlap: every original run scored soft ≤ 0.87 and only half passed
hard; every PR run scored a perfect 1.00. The production skill leaves the model at coin-flip reliability on this remediation; documenting it makes the correct guidance deterministic.Scope / hedging
This is a documentation-only change to one
SKILL.md. It frames the global-binary MCP registration as a remediation option for persistent reauth, not as a replacement for the defaultnpxflow, and ties each remedy to an observable symptom rather than asserting a single absolute cause. The companion eval set is in PR #66.