Source pages misrepresenting the actual process

[stefanb]

http://results.openaddresses.io/sources/si/countrywide
Is not getting any new rows with​ each new batch, just the date in first row changes, but everything else stays the same.
In the last batch run date changed from 2017-05-15 to 2017-05-20, but linked log (https://s3.amazonaws.com/data.openaddresses.io/runs/188019/output.txt ) is still showing timestamps from 2017-05-11 (one of the previous batches). I would expect existing rows to remain the same, but new added on top of the table with new links and new addresses count (now 554000+, as new source zip is prepared daily).

Diff:

[migurski]

We have a 10 day timeout on batch runs — older versions are re-used if a new batch process is requested within that time. Is it possible that’s what you are seeing?

[stefanb]

What I am seeing makes no sense. The previous processing (2017-05-11) was finished way before last batch was started, and (according to the log) was finished in approximately 6 minutes. No chance of reaching 10 day timeout limit. If nothing was being processed for some legitimate reason then the date should not be changed in that row, where all other links indicate to older data and log. I suspect the same thing happened in the previous (2015-05-15) batch run, but have no diff to support my suspicion.

[migurski]

Here’s a simplified view of what’s in the database for Slovenia:

id	success?	copy of	set_id	version	datetime
187276	f			ba13428	2017-05-08 17:22:16+00
187278	f			39ed867	2017-05-08 18:11:15+00
187280	f	187278		9e6a5c9	2017-05-08 19:20:37+00
187282	t			d8d0991	2017-05-08 19:39:11+00
187284	t	187282		e753eca	2017-05-08 19:51:19+00
187286	t	187282		ea80fa7	2017-05-08 21:18:10+00
187296	t	187282		e753eca	2017-05-08 21:48:45+00
187301	t			12261e6	2017-05-08 23:13:20+00
187302	t			12261e6	2017-05-08 23:13:23+00
187351	t	187282	187303	822dcb5	2017-05-08 23:13:34+00
187480	t	187302		12261e6	2017-05-08 23:18:33+00
187740	t	187302		12261e6	2017-05-08 23:30:19+00
187889	t	187302		0fe3dd6	2017-05-08 23:36:56+00
187991	t	187302		594fff8	2017-05-09 09:42:23+00
188005	t	187302		7a6319b	2017-05-10 06:57:21+00
188008	t			98513dc	2017-05-10 08:37:06+00
188009	t			98513dc	2017-05-10 08:37:10+00
188011	t	188009		8b531ae	2017-05-10 11:08:30+00
188013	f			909901f	2017-05-10 11:27:01+00
188015	t			fa978d0	2017-05-10 11:34:41+00
188017	t	188009		ecc277b	2017-05-10 11:31:18+00
188019	t			3035d43	2017-05-11 06:41:07+00
188069	t	187282	188020	05a5856	2017-05-11 18:37:59+00
190149	t	188019		efee8f9	2017-05-11 22:58:56+00
190753	t	188019	190150	efee8f9	2017-05-12 21:38:05+00
192875	t	188019	192272	efee8f9	2017-05-15 21:37:28+00
195041	t	188019	194433	483df2f	2017-05-20 19:23:15+00
195206	t			c8e1b61	2017-05-20 20:41:16+00

Run 195041 (second-to-last row) is public in Set 194433, and it was just a copy of Run 188019 from May 11 from within the prior ten days. Anything with a value in the copy of column is like that. Looks like that newest one, 195206, was processed in this PR about a day ago. Sorry if this is confusing! Many of the tradeoffs here are designed to get something recent-enough onto the OA page without unnecessarily re-processing upstream sources too frequently.

[stefanb]

Seeing so many records with the notion of "copy of" is a whole different story, more explanatory than just date being changed within the same, existing row, while keeping the linked data and logs old (as seen in the diff screenshot above).

Recent-enough sources could be skipped altogether, changing the date of an old import (even if just visually) does not look sane.

Perhaps the solution to confusion is to:

• show the date when original (the one being copied) was imported, so in case of id 195041 (which is a copy of 188019) show the date 2017-05-11 06:41:07+00 (can be without time, just date for brevity)
• make it clear that new ones will not be imported - until the most recent is at least x days old. (give it "fresh enough" badge or so)

Both could be achieved by improving the individual source page, making it show:

• more details about last successful import at top of the page:
  • Title / attribution / License (for SEO and legal reasons)
  • URL (as it is now)
  • date of last import (not date of copying an existing import)
  • until when it will be considered fresh, not due to re-import
  • a picture overview (preview.png same as with CI jobs), maybe even a link to or iframe with slippy map (Added frameset with slippymap preview #579)
• a table of all imports (with links to details, as it is now, just dates changed to the actual import date, not when the copy was made)

[migurski]

Great suggestions, Stefan! Would you be interested in making a contribution to the project? I could help you get set up with the code so that you could develop these changes directly.