[16.0][FIX] currency_rate_update_xe: use XE.com JSON API (HTML scraping returns 403)#216
Conversation
XE.com started returning HTTP 403 to requests without a User-Agent
header. The provider was parsing the 403 error page, finding no rates,
and silently marking the scheduled run as successful (last_successful_run
gets advanced because _obtain_rates returns {date.today(): {}} which is
truthy in res.currency.rate.provider._update). No res.currency.rate
records were created, with no warning posted to the provider chatter.
Send a Mozilla User-Agent so XE responds with the rates page, and call
raise_for_status() so future HTTP errors are surfaced as a chatter
message via the existing exception handling in _update, instead of
appearing as a successful no-op.
OCA-git-bot
added
mod:currency_rate_update_xe
_get_historical_rate built ?date=YYYY-MM-DD URLs for every day in the range, but XE.com returns HTTP 404 for that query when the date is the current day (or in the future). The scheduled cron path always hits today's URL when date_to == today, which is the common case. Before the previous commit added raise_for_status() this 404 was silently parsed as an empty HTML and produced no rate for today without an exception. With raise_for_status() it now interrupts the whole iteration and skips even the past-day rates that were already fetched successfully. Fall back to the latest endpoint (no date parameter) when the iterated day is today or in the future. This is what _get_latest_rate already uses, so we just reuse the same URL shape inside the loop. It also unblocks the module's own test_cron, which exercises this exact flow (date_from = yesterday, date_to = today).
|
Pushed a follow-up commit that fixes the underlying cause of the test failure: Before this PR the 404 was silently parsed as empty HTML, so today's rate was not produced but past-day rates from the same iteration were. Now with The fix falls back to the latest endpoint (no Note: the |
|
Side note on the red |
With a fresh provider (no last_successful_run) and the default next_run = today, _scheduled_update computes date_from = today - 1 day and date_to = today, so _update produces one rate per day in the range. Before this PR, today's URL silently returned no data (XE 404 parsed as empty HTML), so the assertion `len(rates) == 1` only held by accident. Set last_successful_run = yesterday in the test so the cron computes date_from = today and routes through _get_latest_rate, producing exactly one rate. This is also the realistic shape of a daily cron that is up to date.
…raping XE.com now answers automated requests to its public HTML currency tables with a CloudFront 403, regardless of the User-Agent sent, so the previous scraping approach no longer works from server IP ranges. Switch the provider to the JSON endpoint that powers xe.com's own converter (midmarket-converter). It returns the latest USD-based mid-market rates and is reachable from datacenter hosts. Rates are converted to be relative to the company base currency. Since the endpoint only exposes the latest rates, the provider now always returns today's rates regardless of the requested range, which is enough for the scheduled daily update.
DarioLodeiros
changed the title
2 participants
Problem
XE.com now answers automated requests to its public HTML currency tables
(
www.xe.com/currencytables) with a CloudFront 403 "Request blocked",regardless of the
User-Agentsent. This breaks_obtain_rates, whichscrapes that page: the scheduled update fails every run and posts a
"Currency Rate Provider Failure" message, leaving rates frozen.
The earlier mitigation in this PR (sending a
User-Agentand raising onHTTP errors) is no longer enough — the block is applied at the edge for
datacenter IP ranges, so no header tweak gets through.
Fix
Switch the provider to the JSON endpoint that powers xe.com's own
converter (
/api/protected/midmarket-converter/), which is reachablefrom datacenter hosts. It returns the latest USD-based mid-market rates;
these are converted to be relative to the company base currency.
Since the endpoint only exposes the latest rates, the provider now
always returns today's rates regardless of the requested range, which is
sufficient for the scheduled daily update.
Notes
_get_supported_currenciesis unchanged (the API uses the same ISO codes).(DOP-based company, USD rate): scraping returns 403, the JSON API
returns 200 with correct pegs (e.g. AED 3.6725, BHD 0.376).