Skip to content

rtipsy.pro: handle files larger than 2GB. - #3

Open
trquinn wants to merge 1 commit into
masterfrom
IDL_largefile
Open

rtipsy.pro: handle files larger than 2GB.#3
trquinn wants to merge 1 commit into
masterfrom
IDL_largefile

Conversation

@trquinn

@trquinn trquinn commented Sep 1, 2017

Copy link
Copy Markdown
Member

File size calculations were done with 32bit integers, which won't work for large files. Changed to use 64 "LL" ints.

@trquinn

trquinn commented Sep 1, 2017

Copy link
Copy Markdown
Member Author

This addresses issue #2. @mtremmel could you check that it works?

@mtremmel

mtremmel commented Sep 9, 2017

Copy link
Copy Markdown

So it still doesn't work. I tested and on these particular files (which are not extremely large) it seems that the header reading only works with "L", but still the end size is still very slightly off (1137874924 expected and 1137875008 is what it is). I have no idea what's going on with this... clearly a bigger issue is that L and LL seem like they are needed for different sized files which is extremely annoying.

@mtremmel

mtremmel commented Sep 9, 2017

Copy link
Copy Markdown

I can confirm that the header is being read correctly. The number of particles is accurate, but the sizes associated with each are off I think? Not sure why.

@mtremmel

mtremmel commented Sep 9, 2017

Copy link
Copy Markdown

On one of my runs (a larger one than the one where this is failing) it seems to work fine with "L" in the header and then comparing to "28LL+...". I am very confused.

@mtremmel

mtremmel commented Sep 9, 2017

Copy link
Copy Markdown

So in other words, the tipsy file from a smaller run is failing with the current fix and that of a larger file is working fine.

@trquinn

trquinn commented Sep 11, 2017

Copy link
Copy Markdown
Member Author

For the file that fails, what does it report? That is, after, "RTIPSY ERROR: Header and file size inconsistent Estimates: Header bytes: 28 or 32 (either is OK)", what are the numbers that it reports, and what should those numbers be, given the numbers in the header?

@mtremmel

Copy link
Copy Markdown

Here is an example:
RTIPSY ERROR: Header and file size inconsistent
Estimates: Header bytes: 28 or 32 (either is OK)
ngas: 1532498 bytes: 73559904
ndark: 29140809 bytes: 1049069124
nstar: 346497 bytes: 15245868
Actual File bytes: 1137875008 not one of: 1137874928 1137874924

@mtremmel

Copy link
Copy Markdown

rtipsy works fine on the previous and next snapshots of the same simulation. This is rtipsy as it is right now in this branch. On all prior steps it works fine. I think it fails on the following steps 2 more times. This must have something to do with either 1) the number of particles just barely crossing some threshold or 2) some kind of rounding error?

@trquinn

trquinn commented Sep 11, 2017

Copy link
Copy Markdown
Member Author

Would the problem file have been overwritten in a restart?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants