Skip to content

Running info on vector memory inputs return UNIX timestamps instead of ISO datetimes #4241

Closed
Closed
Running info on vector memory inputs return UNIX timestamps instead of ISO datetimes#4241
Assignees
maxrjones

Description

@weiji14
weiji14
opened on Sep 20, 2020

Description of the problem

UNIX timestamps (e.g. 1577836800) are returned instead of ISO datetimes (e.g. 2020-01-01T00:00:00) when running info on virtual tables. Originally noticed at GenericMappingTools/pygmt#619 (comment), check also #3396 where datetime inputs were originally accepted into GMT_Put_Vector.

Full script that generated the error

Sorry that I can't write C code, but given an input like so:

z time
10 2020-01-01T00:00:00
13 2020-01-02T00:00:00
12 2020-01-03T00:00:00
15 2020-01-04T00:00:00
14 2020-01-05T00:00:00

and then running info from PyGMT from the branch at GenericMappingTools/pygmt#619 (each column is passed into C API using GMT_Put_Vector)

import pygmt

# create data array 'table' and pass it into `info`
pygmt.info(table=table, V="d")

Full error message

gmtinfo [INFORMATION]: Processing input table data
gmtinfo [DEBUG]: gmtapi_init_import: Passed family = Data Table and geometry = Point|Line|Poly
gmtinfo (gmtapi_init_import): tried to free unallocated memory
gmtinfo [DEBUG]: gmtapi_init_import: Added 1 new sources
gmtinfo [DEBUG]: GMT_Init_IO: Returned first Input object ID = 0
gmtinfo [DEBUG]: gmtapi_init_export: Passed family = Data Table and geometry = Point|Line|Poly
gmtinfo [DEBUG]: Object ID 1 : Registered Data Table File /tmp/pygmt-gykf5n3t.txt as an Output resource with geometry Point|Line|Poly [n_objects = 2]
gmtinfo [DEBUG]: gmtapi_init_export: Added 1 new destination
gmtinfo [DEBUG]: GMT_Init_IO: Returned first Output object ID = 1
gmtinfo [DEBUG]: GMT_Begin_IO: Mode value 1 not considered (ignored)
gmtinfo [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Input
gmtinfo [DEBUG]: gmtapi_next_io_source: Selected object 0
gmtinfo [INFORMATION]: Reading Data Table from Input memory location via vector
gmtinfo [DEBUG]: GMT_Begin_IO: Input resource access is now enabled [record-by-record]
gmtinfo [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Output
gmtinfo [DEBUG]: gmtapi_next_io_source: Selected object 1
gmtinfo [INFORMATION]: Writing Data Table to file /tmp/pygmt-gykf5n3t.txt
gmtinfo [DEBUG]: GMT_Begin_IO: Output resource access is now enabled [record-by-record]
gmtinfo [DEBUG]: Number of numerical output columns has been set to 0
gmtinfo [DEBUG]: GMT_End_IO: Input resource access is now disabled
gmtinfo [DEBUG]: GMT_End_IO: Output resource access is now disabled
gmtinfo (GMT_gmtinfo): tried to free unallocated memory
gmtinfo (GMT_gmtinfo): tried to free unallocated memory
gmtinfo [DEBUG]: gmtlib_unregister_io: Unregistering object no 1 [n_objects = 1]
gmtinfo (gmtlib_free_tmp_arrays): tried to free unallocated memory

Actual outcome

<vector memory>: N = 5 <10/15> <1577836800/1578182400>

Expected outcome

<vector memory>: N = 5 <10/15> <2020-01-01T00:00:00/2020-01-05T00:00:00>

System information

  • Operating system: Linux
  • GMT version (gmt --version): 6.1.1

Activity

  1. PaulWessel commented on Sep 20, 2020

    @PaulWessel
    PaulWessel
    on Sep 20, 2020
    Contributor

    Hm, yes since what is returned is a text record I agree there is no good reason it should not be formatted as you expect here. If -C was used then you would get a numerical record and it would have UNIX time of course, but here it is text. I will try to have a look as to why it is not doing the formatting.

  2. PaulWessel commented on Sep 20, 2020

    @PaulWessel
    PaulWessel
    on Sep 20, 2020
    Contributor

    I would like to add another section to the DEVELOPER RESOURCES sidebar on debugging, for how to debug pyGMT via Xcode on macOS. Been a month since last time and cannot find all the step that @seisman helped with. Since I am a complete python rookie there is not much mental memory on even the most basics, so I need that recipe, which also had some issues with setting shared-dir and possibly library paths for all to work. Maybe @seisman could resurrect his instructions and I can turn it into a section in the developer RST?

  3. PaulWessel commented on Sep 20, 2020

    @PaulWessel
    PaulWessel
    on Sep 20, 2020
    Contributor

    And @joa-quim has given similar instructions for setting up GMT.jl so I can debug cases he finds directly in Xcode as well?

  4. weiji14 commented on Sep 20, 2020

    @weiji14
    weiji14
    on Sep 20, 2020
    MemberAuthor

    I would like to add another section to the DEVELOPER RESOURCES sidebar on debugging, for how to debug pyGMT via Xcode on macOS. Been a month since last time and cannot find all the step that @seisman helped with. Since I am a complete python rookie there is not much mental memory on even the most basics, so I need that recipe, which also had some issues with setting shared-dir and possibly library paths for all to work. Maybe @seisman could resurrect his instructions and I can turn it into a section in the developer RST?

    Issue is at #3778. I should probably start to learn C too at some point 😆

  5. seisman commented on Sep 20, 2020

    @seisman
    seisman
    on Sep 20, 2020
    Member

    The steps to debug PyGMT via Xcode was discussed in #3829.

    Here is a summary:

    1. Install PyGMT following the official instructions
    # add conda-forge channel
conda config --prepend channels conda-forge
# NOTE: the next step is different from the PyGMT official instructions, because we want to use the GMT dev version
conda create --name pygmt python=3.8 pip numpy pandas xarray netcdf4 packaging
# activate the pygmt envirionment
conda activate pygmt

# install pygmt in editable/development mode
cd pygmt
make install
    1. Compile GMT using Xcode
    2. Tell PyGMT where to find the GMT library by setting the environmental variable GMT_LIBRARY_PATH
    export GMT_LIBRARY_PATH=~/Gits/gmt/gmt/build/xcode/src/Debug
    1. Open Xcode
    2. Run a python console, attach the process id in xcode, and run PyGMT codes in the Python console and xcode will stop at the breakpoint.

  6. joa-quim commented on Sep 20, 2020

    @joa-quim
    joa-quim
    on Sep 20, 2020
    Member

    Issue is at #3778. I should probably start to learn C too at some point

    Please, please join the Force.

    Debugging from any of the environments (Matlab, Julia, Python) should all the same procedure.

    1. Open the shell (ML IDE, Julia or Py REPL), type the command that will trigger the program to debug, but HOLD ON
    2. Open the debugger an Attach to process (this is VS talk, Xcode does similar but on its own way). The process is either the Matlab running process or Julia, Py shells.
    3. Open the source code of the program to be debugged ans set break points.
    4. Go back to 1. and hit RETURN

  7. seisman commented on Sep 20, 2020

    @seisman
    seisman
    on Sep 20, 2020
    Member

    @weiji14 Perhaps we should merge GenericMappingTools/pygmt#619 first (mark the tests as xfail) and provide a minimal Python script to reproduce the issue. Your example above is incomplete, and @PaulWessel will need to rewrite a working script before debugging.

  8. joa-quim commented on Sep 20, 2020

    @joa-quim
    joa-quim
    on Sep 20, 2020
    Member

    Ofc, on Unix everything is more complicated 😈

  9. weiji14 commented on Sep 20, 2020

    @weiji14
    weiji14
    on Sep 20, 2020
    MemberAuthor

    Please, please join the Force.

    http://www.nooooooooooooooo.com/

    Ofc, on Unix everything is more complicated 😈

    Yeah, I'm on Linux which doesn't seem to have Xcode 🤷

    @weiji14 Perhaps we should merge GenericMappingTools/pygmt#619 first (mark the tests as xfail) and provide a minimal Python script to reproduce the issue. Your example above is incomplete, and @PaulWessel will need to rewrite a working script before debugging.

    Yep, give me a few minutes to finish up that PR.

  10. PaulWessel commented on Sep 20, 2020

    @PaulWessel
    PaulWessel
    on Sep 20, 2020
    Contributor

    Yes, it would be great if a snipped of Python code demonstrating the problem is submitted since that is not something I will know how to do. Same with Julia of course and @joa-quim usually posts a Julia command or mex command. We just want to lower the threshold for all of us to do these things. Even C-novices should be able to step in the the C library and at least detect where a crash happens - often that is enough of a clue for the C-team to figure things out.

  11. joa-quim commented on Sep 20, 2020

    @joa-quim
    joa-quim
    on Sep 20, 2020
    Member

    Good question for Linux. I guess that gcc should have an equivalent mechanism to Attach to process ID and from that on, the doing should be the same ... except that using gdb is an act of masochism.

  12. PaulWessel commented on Sep 20, 2020

    @PaulWessel
    PaulWessel
    on Sep 20, 2020
    Contributor

    On Linux, I have done lots of debugging using ddd, and even on macOS before Apple switched to clang and it became a bit harder (and I have not paid too much attention). But I dont know if you can connect to the process in ddd the same way we do in Xcode or Visual Studio.

  13. PaulWessel commented on Sep 20, 2020

    @PaulWessel
    PaulWessel
    on Sep 20, 2020
    Contributor

    ddd is just a graphical front-end to gdb which I assume can connect to other processes.

  14. PaulWessel commented on Sep 21, 2020

    @PaulWessel
    PaulWessel
    on Sep 21, 2020
    Contributor

    @joa-quim, do you mind posting the usual setup for a newbie to install GMT.jl from the developer version, including installing julia. I know you have some of that on your site but it would help me put the docs together if you can (like @seisman) cut/paste in here while I am thinking about this.

  15. weiji14 commented on Sep 21, 2020

    @weiji14
    weiji14
    on Sep 21, 2020
    MemberAuthor

    Ok, GenericMappingTools/pygmt#619 has been merged, and I'm starting to have a go at debugging using gdbgui (installed via pip install gdbgui). Will see if I can figure this out. Below is the Python code to run after following PyGMT installation instructions (pip install https://github.com/GenericMappingTools/pygmt/archive/master.zip), and launching Python:

    import pygmt
import pandas as pd

table = pd.DataFrame(
    data={
        "z": [10, 13, 12, 15, 14],
        "time": pd.date_range(start="2020-01-01", periods=5),
    }
)
output = pygmt.info(table=table, V="d")
print(output)
# <vector memory>: N = 5 <10/15> <1577836800/1578182400>

  16. 27 remaining items

  17. stale commented on Jan 23, 2021

    @stale
    staleBot
    on Jan 23, 2021

    This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions.

  18. stale
    added
    staleThis will not be worked on
    on Jan 23, 2021
  19. stale
    closed this as completedon Feb 2, 2021
  20. seisman
    reopened this on Feb 2, 2021
  21. maxrjones
    self-assigned this
    on Feb 11, 2021
  22. stale
    removed
    staleThis will not be worked on
    on Feb 11, 2021
  23. weiji14
    added a commit that references this issue on Feb 24, 2021

    Let pygmt.info load datetime columns into a str dtype array

    7b38109
  24. weiji14
    added a commit that references this issue on Mar 3, 2021

    Let pygmt.info load datetime columns into a str dtype array (#960)

    52dbd7a

  25. maxrjones commented on Mar 14, 2021

    @maxrjones
    maxrjones
    on Mar 14, 2021
    Member

    @weiji14 the output appears as you wanted it - is this because of workarounds implemented in PyGMT? Would you be able to provide an example that still demonstrates the bug for debugging purposes?

    import pygmt
import pandas as pd

table = pd.DataFrame(
    data={
        "z": [10, 13, 12, 15, 14],
        "time": pd.date_range(start="2020-01-01", periods=5),
    }
)
output = pygmt.info(table=table, V="d")
print(output)

#<vector memory>: N = 5 <10/15> <2020-01-01T00:00:00/2020-01-05T00:00:00>

  26. weiji14 commented on Mar 14, 2021

    @weiji14
    weiji14
    on Mar 14, 2021
    MemberAuthor

    Hmm, you might be right! I see that the these two units tests at https://github.com/GenericMappingTools/pygmt/blob/v0.3.1/pygmt/tests/test_info.py#L67-L107 are passing on GMT master (i.e. 6.2.0) now at the GMT Latest tests (https://github.com/GenericMappingTools/pygmt/runs/2104362367?check_suite_focus=true#step:14:324). Not sure when this got fixed 😅

    Just to double check, could you try this example out (passing in only one datetime column from a numpy array).

    import pandas as pd
import pygmt

table = pd.date_range(start="2020-01-01", periods=5).to_numpy()
output = pygmt.info(table=table, per_column=True, V="d")
print(output)
# ['2020-01-01T00:00:00' '2020-01-05T00:00:00']

  27. maxrjones commented on Mar 14, 2021

    @maxrjones
    maxrjones
    on Mar 14, 2021
    Member

    Hmm, you might be right! I see that the these two units tests at https://github.com/GenericMappingTools/pygmt/blob/v0.3.1/pygmt/tests/test_info.py#L67-L107 are passing on GMT master (i.e. 6.2.0) now at the GMT Latest tests (https://github.com/GenericMappingTools/pygmt/runs/2104362367?check_suite_focus=true#step:14:324). Not sure when this got fixed 😅

    Just to double check, could you try this example out (passing in only one datetime column from a numpy array).

    import pandas as pd
import pygmt

table = pd.date_range(start="2020-01-01", periods=5).to_numpy()
output = pygmt.info(table=table, per_column=True, V="d")
print(output)
# ['2020-01-01T00:00:00' '2020-01-05T00:00:00']

    Output looks good, I think:

    gmtinfo [INFORMATION]: Processing input table data
gmtinfo [DEBUG]: gmtapi_init_import: Passed family = Data Table and geometry = Point|Line|Poly
gmtinfo [DEBUG]: gmtapi_init_import: Added 1 new sources
gmtinfo [DEBUG]: GMT_Init_IO: Returned first Input object ID = 0
gmtinfo [DEBUG]: gmtapi_init_export: Passed family = Data Table and geometry = Point|Line|Poly
gmtinfo [DEBUG]: Object ID 1 : Registered Data Table File /var/folders/48/71lnqfxj77j8wrw6k0rtklv00000gq/T/pygmt-57pz3p76.txt as an Output resource with geometry Point|Line|Poly [n_objects = 2]
gmtinfo [DEBUG]: gmtapi_init_export: Added 1 new destination
gmtinfo [DEBUG]: GMT_Init_IO: Returned first Output object ID = 1
gmtinfo [DEBUG]: GMT_Begin_IO: Mode value 1 not considered (ignored)
gmtinfo [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Input
gmtinfo [DEBUG]: gmtapi_next_io_source: Selected object 0
gmtinfo [INFORMATION]: Reading Data Table from Input memory location via vector
gmtinfo [DEBUG]: GMT_Begin_IO: Input resource access is now enabled [record-by-record]
gmtinfo [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Output
gmtinfo [DEBUG]: gmtapi_next_io_source: Selected object 1
gmtinfo [INFORMATION]: Writing Data Table to file /var/folders/48/71lnqfxj77j8wrw6k0rtklv00000gq/T/pygmt-57pz3p76.txt
gmtinfo [DEBUG]: GMT_Begin_IO: Output resource access is now enabled [record-by-record]
gmtinfo [DEBUG]: GMT_End_IO: Input resource access is now disabled
gmtinfo [DEBUG]: GMT_End_IO: Output resource access is now disabled
==> 2 API Objects at end of GMTAPI_Garbage_Collection entry
--------------------------------------------------------
K.. ID RESOURCE.... FAMILY.... ACTUAL.... DIR... S O M L
--------------------------------------------------------
* 0  0 7fb7aa6283a0 Data Table Vector     Input  2 Y N 0
* 1  1            0 Data Table Data Table Output 2 Y N 1
--------------------------------------------------------
gmtinfo [DEBUG]: gmtlib_unregister_io: Unregistering object no 1 [n_objects = 1]
==> 1 API Objects at end of GMTAPI_Garbage_Collection exit
--------------------------------------------------------
K.. ID RESOURCE.... FAMILY.... ACTUAL.... DIR... S O M L
--------------------------------------------------------
* 0  0 7fb7aa6283a0 Data Table Vector     Input  2 Y N 0
--------------------------------------------------------
['2020-01-01T00:00:00' '2020-01-05T00:00:00']

  28. weiji14 commented on Mar 14, 2021

    @weiji14
    weiji14
    on Mar 14, 2021
    MemberAuthor

    Nice, I'll close this issue then and wait for GMT 6.2.0 to be released. Thanks everyone!

  29. weiji14
    closed this as completedon Mar 14, 2021

  30. maxrjones commented on Mar 14, 2021

    @maxrjones
    maxrjones
    on Mar 14, 2021
    Member

    Fixed by #4849

  31. sixy6e
    added a commit that references this issue on Dec 21, 2022

    Let pygmt.info load datetime columns into a str dtype array (GenericM…

    f57277d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions