BUG: fixing to allow unpickling of PY3 pickles from PY2 #14275

fersarr · 2019-08-14T15:50:16Z

Fixing to allow unpickling of PY3 pickles from PY2, particularly when the pickle contains structured dtypes with field names. Only applying to branch 1.16 as 1.17 and above are PY3 only

Example from gh-2407 by jmlarson1:

Write the pickle in PY3

import pickle
import numpy as np
my_datatype = np.dtype([('SPOT', np.float64)])
my_data = np.array([(6.0)], dtype=my_datatype)

# Pickle the data using maximum supported protocol in Py2
with open('py3_out', 'wb') as fid:
    pickle.dump(my_data, fid, protocol=2)

Reading the pickle from PY2:

import pickle
with open('py3_out', 'rb') as fid:
    loaded = pickle.load(fid)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 393, in load
    return format.read_array(fid)
  File "/path/lib/python2.7/dist-packages/numpy/lib/format.py", line 602, in read_array
    array = pickle.load(fp)
ValueError: non-string names in Numpy dtype unpickling

fersarr · 2019-08-14T16:23:13Z

It looks like the azure checks are complaining about code that wasn't modified? Am I reading this wrong?

seberg · 2019-08-14T16:48:21Z

I checked the travis failure right now. And maybe one or so is, but it is failing due to compiler warnings/errors that need to be fixed up.

numpy/core/src/multiarray/descriptor.c

fersarr · 2019-08-15T08:51:42Z

Thanks! I updated the code and now I am seeing this problem in the Linux_Pypi checks

++ tar --strip-components=1 -xf ../pypy.tar.bz2
tar: This does not look like a tar archive

bzip2: Compressed file ends unexpectedly;
	perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
	Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

tar: Child returned status 2
tar: Error is not recoverable: exiting now
##[error]Bash exited with code '1'.

Any ideas about that one?

mattip · 2019-08-15T09:39:41Z

That build uses a nightly PyPy build, and last night's build did not succeed. That failure can be ignored.

mattip · 2019-08-15T14:06:05Z

This needs a test. You can save the target with python3, add the binary bytes to the test, s = b'\x93NUMPY\x01\x00v\x00{\'descr\': \'|O\', \' ... then wrap the bytes in an np.load(io.BytesIO(s))

Maybe I didn;t understand the issue, but I think the example at the top of the PR will not work since numpy now stores object arrays with pickle protocol 3, which is not available on python2. The original issue was about unicode field names, is that what you wanted to fix?

fersarr · 2019-08-19T13:37:21Z

This needs a test. You can save the target with python3, add the binary bytes to the test, s = b'\x93NUMPY\x01\x00v\x00{\'descr\': \'|O\', \' ... then wrap the bytes in an np.load(io.BytesIO(s))

Maybe I didn;t understand the issue, but I think the example at the top of the PR will not work since numpy now stores object arrays with pickle protocol 3, which is not available on python2. The original issue was about unicode field names, is that what you wanted to fix?

Sure, will add a test.

About the protocol, you are right, I updated the example. I had copied it from gh-2407 but now I added the actual example I used for testing.

numpy/tests/test_reloading.py

fersarr · 2019-08-19T16:33:00Z

Seems there was an issue with the PR check continuous-integration/travis-ci/pr. How can I start it again?
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

mattip · 2019-08-20T06:31:20Z

Ahh, now the example is much clearer, thanks. ~~Note it will not close the file descriptors, you should always use open inside a context manager: with open(...) as fid: since people tend to indiscriminately copy-paste code from the internet.~~

numpy/core/src/multiarray/descriptor.c

charris · 2019-08-22T00:52:13Z

I kicked this off to 1.16.6.

mattip · 2019-08-22T06:01:11Z

I modified the example to use a context manager.

fersarr · 2019-08-23T16:40:04Z

Just pushed the refactored C code. Seems that there are some issue with the checks again, unless I am not reading it properly

fersarr · 2019-08-30T14:23:12Z

Just pushed the refactored C code. Seems that there are some issue with the checks again, unless I am not reading it properly

@mattip Hi! Sorry to insist, do I have a way to re-start the checks? The failures are not related to my changes I believe

mattip · 2019-08-30T14:28:18Z

This looks good to me. Anyone else want to take a look?

mattip · 2019-08-30T14:29:00Z

codecov is complaining since we only collect statistics on python3, and this is a fix for python2

eric-wieser · 2019-09-04T07:19:21Z

numpy/core/src/multiarray/descriptor.c

+                else if (PyUnicode_Check(name)) {
+                    // The field names of a structured dtype were pickled in PY3 as unicode strings
+                    // so, to unpickle them in PY2, we need to convert them to PY2 strings
+                    new_name = PyUnicode_AsEncodedString(name, "utf-8", "strict");


Doesn't this result in an an un-round-trippable field name? I think this encoding needs to match line 2795, else the following will fail:

Write in py3 with non-ascii field names

Read in py2

Write back out in py2

Read in py3

Choosing ascii as an encoding might be safest here, simply because its the intersection of latin1 and utf8, both of which we use elsewhere in compatibility code.

@fersarr - thoughts? Can we construct another test that will

add another bytestring result from the result of step 3

on python3 load the bytestring and compare it to the original

on python2 check that the result of the current test matches this second string

Hey Hey. Sure, I will give that a go as soon as I am back from holiday!

@eric-wieser @mattip I just pushed the new changes, hopefully now we are okay to merge. Thanks for all the help.

eric-wieser · 2019-09-22T00:23:48Z

numpy/tests/test_reloading.py

+'''
+
+    loaded = pickle.loads(saved_pickle_from_py2)
+    assert loaded == expected_data


Instead of running this conditionally on py3 only, should we change the test to the following?

if py3: assert loads(saved_pickle_from_py2) == expected_data else: # check that our string above is what we claim on py2 assert dumps(expected_data) == saved_pickle_from_py2

done! thanks. I used the idea for both tests

hmm @eric-wieser it seems doing that does not work on Windows and Pypy:

Pypy error
https://dev.azure.com/numpy/numpy/_build/results?buildId=5698&view=logs

=================================== FAILURES =================================== _____________ test_py3_can_load_py2_pickle_with_dtype_field_names ______________ def test_py3_can_load_py2_pickle_with_dtype_field_names(): # gh-2407 and PR #14275 # Roundtrip: Py3 should be able to load a pickle that was created in PY2 # after loading the saved_pickle (from PY3) in the test named # 'test_py2_can_load_py3_pickle_with_dtype_field_names' import numpy as np expected_dtype = np.dtype([('SPOT', np.float64)]) expected_data = np.array([(6.0)], dtype=expected_dtype) # Pickled under Python 2.7.16 with protocol=2 after it was loaded # by test 'test_py2_can_load_py3_pickle_with_dtype_field_names' saved_pickle_from_py2 = b'''\ \x80\x02cnumpy.core.multiarray\n_reconstruct\nq\x01cnumpy\nndarray\n\ q\x02K\x00\x85U\x01b\x87Rq\x03(K\x01K\x01\x85cnumpy\ndtype\nq\x04U\x02\ V8K\x00K\x01\x87Rq\x05(K\x03U\x01|NU\x04SPOTq\x06\x85q\x07}q\x08h\x06h\ \x04U\x02f8K\x00K\x01\x87Rq\t(K\x03U\x01<NNNJ\xff\xff\xff\xffJ\xff\xff\ \xff\xffK\x00tbK\x00\x86sK\x08K\x01K\x10tb\x89U\x08\x00\x00\x00\x00\x00\ \x00\x18@tb.\ ''' if sys.version_info[0] >= 3: # PY3 assert pickle.loads(saved_pickle_from_py2) == expected_data else: # check that the string above is what we claim on PY2 > assert pickle.dumps(expected_data, protocol=2) == saved_pickle_from_py2 E AssertionError: assert '\x80\x02cnum...q\x17tq\x18b.' == '\x80\x02cnump...0\x00\x18@tb.' E \x80\x02cnumpy.core.multiarray E _reconstruct E q\x01cnumpy E ndarray E - q\x02K\x00\x85q\x03U\x01bq\x04\x87q\x05Rq\x06(K\x01K\x01\x85q\x07cnumpy E ? ----- ----- ----- ^ ----- E + q\x02K\x00\x85U\x01b\x87Rq\x03(K\x01K\x01\x85cnumpy... E E ...Full output truncated (8 lines hidden), use '-vv' to show expected_data = array([(6.,)], dtype=[('SPOT', '<f8')]) expected_dtype = dtype([('SPOT', '<f8')]) np = <module 'numpy' from '/home/vsts/work/1/s/build/testenv/site-packages/numpy/__init__.pyc'> saved_pickle_from_py2 = '\x80\x02cnumpy.core.multiarray\n_reconstruct\nq\x01cnumpy\nndarray\nq\x02K\x00\x85U\x01b\x87Rq\x03(K\x01K\x01\x85cnum...U\x01<NNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK\x00tbK\x00\x86sK\x08K\x01K\x10tb\x89U\x08\x00\x00\x00\x00\x00\x00\x18@tb.' numpy/tests/test_reloading.py:96: AssertionError - generated xml file: /home/vsts/work/1/s/build/testenv/site-packages/junit/test-results.xml -

windows
https://dev.azure.com/numpy/numpy/_build/results?buildId=5698&view=logs

_____________ test_py3_can_load_py2_pickle_with_dtype_field_names _____________ def test_py3_can_load_py2_pickle_with_dtype_field_names(): # gh-2407 and PR #14275 # Roundtrip: Py3 should be able to load a pickle that was created in PY2 # after loading the saved_pickle (from PY3) in the test named # 'test_py2_can_load_py3_pickle_with_dtype_field_names' import numpy as np expected_dtype = np.dtype([('SPOT', np.float64)]) expected_data = np.array([(6.0)], dtype=expected_dtype) # Pickled under Python 2.7.16 with protocol=2 after it was loaded # by test 'test_py2_can_load_py3_pickle_with_dtype_field_names' saved_pickle_from_py2 = b'''\ \x80\x02cnumpy.core.multiarray\n_reconstruct\nq\x01cnumpy\nndarray\n\ q\x02K\x00\x85U\x01b\x87Rq\x03(K\x01K\x01\x85cnumpy\ndtype\nq\x04U\x02\ V8K\x00K\x01\x87Rq\x05(K\x03U\x01|NU\x04SPOTq\x06\x85q\x07}q\x08h\x06h\ \x04U\x02f8K\x00K\x01\x87Rq\t(K\x03U\x01<NNNJ\xff\xff\xff\xffJ\xff\xff\ \xff\xffK\x00tbK\x00\x86sK\x08K\x01K\x10tb\x89U\x08\x00\x00\x00\x00\x00\ \x00\x18@tb.\ ''' if sys.version_info[0] >= 3: # PY3 assert pickle.loads(saved_pickle_from_py2) == expected_data else: # check that the string above is what we claim on PY2 > assert pickle.dumps(expected_data, protocol=2) == saved_pickle_from_py2 E AssertionError: assert '\x80\x02cnum...0\x00\x18@tb.' == '\x80\x02cnump...0\x00\x18@tb.' E Skipping 90 identical leading characters in diff, use -v to show E - \x03(K\x01\x8a\x01\x01\x85cnumpy E ? ^^^^^^^^ E + \x03(K\x01K\x01\x85cnumpy E ? ^ E dtype E q\x04U\x02V8K\x00K\x01\x87Rq\x05(K\x03U\x01|NU\x04SPOTq\x06\x85q\x07}q\x08h\x06h\x04U\x02f8K\x00K\x01\x87Rq\t(K\x03U\x01<NNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK\x00tbK\x00\x86sK\x08K\x01K\x10tb\x89U\x08\x00\x00\x00\x00\x00\x00\x18@tb. expected_data = array([(6.,)], dtype=[('SPOT', '<f8')]) expected_dtype = dtype([('SPOT', '<f8')]) np = <module 'numpy' from 'C:\hostedtoolcache\windows\Python\2.7.16\x64\lib\site-packages\numpy\__init__.pyc'> saved_pickle_from_py2 = '\x80\x02cnumpy.core.multiarray\n_reconstruct\nq\x01cnumpy\nndarray\nq\x02K\x00\x85U\x01b\x87Rq\x03(K\x01K\x01\x85cnum...U\x01<NNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK\x00tbK\x00\x86sK\x08K\x01K\x10tb\x89U\x08\x00\x00\x00\x00\x00\x00\x18@tb.' C:\hostedtoolcache\windows\Python\2.7.16\x64\lib\site-packages\numpy\tests\test_reloading.py:96: AssertionError ------- generated xml file: D:\a\1\s\build\test\junit\test-results.xml -------- =========================== short test summary info ===========================

mattip · 2019-10-03T10:30:38Z

numpy/tests/test_reloading.py

+        assert pickle.loads(saved_pickle_from_py2) == expected_data
+    else:
+        # check that the string above is what we claim on PY2
+        assert pickle.dumps(expected_data, protocol=2) == saved_pickle_from_py2


It seems the actual byte stream produced on cpython linux 2.7 is not the same as the one produced on windows 2.7 nor the one produced on pypy 2.7. Rather than try to diagnose exactly why the stream is different, perhaps we could only assert they are identical when sys.platform.startswith('linux') and not IS_PYPY, after from numpy.testing import IS_PYPY. I verified that the saved_pickle_from_py2 correctly imports on those platforms, even though the roundtrip is not accurate. I don't want to dive into decoding the byte stream to find the difference.

Sounds good. Done!

@mattip @eric-wieser any other comments? can we merge?

numpy/tests/test_reloading.py

fixing to allow unpickling of PY3 pickles from PY2, particularly when the pickle containes structured dtypes with field names. see numpygh-2407

fersarr · 2019-10-18T16:34:58Z

@eric-wieser All your suggestions are done now :)

mattip · 2019-10-20T09:42:38Z

Thanks @fersarr

fersarr mentioned this pull request Aug 14, 2019

BUG: fixing to allow unpickling of PY3 pickles from PY2 #14274

Closed

charris added 01 - Enhancement 03 - Maintenance labels Aug 15, 2019

charris added this to the 1.16.5 release milestone Aug 15, 2019

mattip reviewed Aug 15, 2019

View reviewed changes

numpy/core/src/multiarray/descriptor.c Outdated Show resolved Hide resolved

fersarr force-pushed the 1.16_2 branch from 48ace75 to 4dce14a Compare August 15, 2019 08:28

fersarr force-pushed the 1.16_2 branch from 4dce14a to 48caa17 Compare August 19, 2019 15:05

fersarr commented Aug 19, 2019

View reviewed changes

numpy/tests/test_reloading.py Outdated Show resolved Hide resolved

fersarr force-pushed the 1.16_2 branch from 48caa17 to 244e878 Compare August 19, 2019 15:40

mattip reviewed Aug 20, 2019

View reviewed changes

numpy/core/src/multiarray/descriptor.c Outdated Show resolved Hide resolved

charris modified the milestones: 1.16.5 release, 1.16.6 Aug 22, 2019

fersarr force-pushed the 1.16_2 branch 2 times, most recently from 5cb5f63 to e19333c Compare August 23, 2019 14:06

eric-wieser reviewed Sep 4, 2019

View reviewed changes

fersarr force-pushed the 1.16_2 branch from e19333c to 34a897d Compare September 17, 2019 11:54

mattip requested a review from eric-wieser September 21, 2019 17:05

eric-wieser reviewed Sep 22, 2019

View reviewed changes

fersarr force-pushed the 1.16_2 branch 3 times, most recently from 1028f4c to 4418998 Compare September 30, 2019 10:20

mattip reviewed Oct 3, 2019

View reviewed changes

fersarr force-pushed the 1.16_2 branch from 4418998 to bf8f4df Compare October 4, 2019 14:26

mattip requested a review from eric-wieser October 5, 2019 17:30

eric-wieser reviewed Oct 14, 2019

View reviewed changes

numpy/tests/test_reloading.py Outdated Show resolved Hide resolved

eric-wieser reviewed Oct 14, 2019

View reviewed changes

numpy/tests/test_reloading.py Outdated Show resolved Hide resolved

eric-wieser reviewed Oct 14, 2019

View reviewed changes

numpy/tests/test_reloading.py Outdated Show resolved Hide resolved

BUG: fixing to allow unpickling of PY3 pickles from PY2

c548a14

fixing to allow unpickling of PY3 pickles from PY2, particularly when the pickle containes structured dtypes with field names. see numpygh-2407

fersarr force-pushed the 1.16_2 branch from bf8f4df to c548a14 Compare October 18, 2019 12:38

mattip requested a review from eric-wieser October 20, 2019 09:04

eric-wieser approved these changes Oct 20, 2019

View reviewed changes

mattip merged commit e9322e8 into numpy:maintenance/1.16.x Oct 20, 2019

eric-wieser mentioned this pull request Nov 25, 2019

dtype field names cannot be unicode (Trac #1814) #2407

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: fixing to allow unpickling of PY3 pickles from PY2 #14275

BUG: fixing to allow unpickling of PY3 pickles from PY2 #14275

fersarr commented Aug 14, 2019 •

edited by mattip

Loading

fersarr commented Aug 14, 2019

seberg commented Aug 14, 2019

fersarr commented Aug 15, 2019 •

edited

Loading

mattip commented Aug 15, 2019

mattip commented Aug 15, 2019

fersarr commented Aug 19, 2019

fersarr commented Aug 19, 2019

mattip commented Aug 20, 2019 •

edited

Loading

charris commented Aug 22, 2019

mattip commented Aug 22, 2019

fersarr commented Aug 23, 2019

fersarr commented Aug 30, 2019

mattip commented Aug 30, 2019

mattip commented Aug 30, 2019

eric-wieser Sep 4, 2019

mattip Sep 12, 2019

fersarr Sep 12, 2019

fersarr Sep 17, 2019

eric-wieser Sep 22, 2019 •

edited

Loading

fersarr Sep 30, 2019 •

edited

Loading

fersarr Sep 30, 2019 •

edited

Loading

mattip Oct 3, 2019

fersarr Oct 4, 2019 •

edited

Loading

fersarr Oct 14, 2019

fersarr commented Oct 18, 2019 •

edited

Loading

mattip commented Oct 20, 2019

BUG: fixing to allow unpickling of PY3 pickles from PY2 #14275

BUG: fixing to allow unpickling of PY3 pickles from PY2 #14275

Conversation

fersarr commented Aug 14, 2019 • edited by mattip Loading

fersarr commented Aug 14, 2019

seberg commented Aug 14, 2019

fersarr commented Aug 15, 2019 • edited Loading

mattip commented Aug 15, 2019

mattip commented Aug 15, 2019

fersarr commented Aug 19, 2019

fersarr commented Aug 19, 2019

mattip commented Aug 20, 2019 • edited Loading

charris commented Aug 22, 2019

mattip commented Aug 22, 2019

fersarr commented Aug 23, 2019

fersarr commented Aug 30, 2019

mattip commented Aug 30, 2019

mattip commented Aug 30, 2019

eric-wieser Sep 4, 2019

Choose a reason for hiding this comment

mattip Sep 12, 2019

Choose a reason for hiding this comment

fersarr Sep 12, 2019

Choose a reason for hiding this comment

fersarr Sep 17, 2019

Choose a reason for hiding this comment

eric-wieser Sep 22, 2019 • edited Loading

Choose a reason for hiding this comment

fersarr Sep 30, 2019 • edited Loading

Choose a reason for hiding this comment

fersarr Sep 30, 2019 • edited Loading

Choose a reason for hiding this comment

mattip Oct 3, 2019

Choose a reason for hiding this comment

fersarr Oct 4, 2019 • edited Loading

Choose a reason for hiding this comment

fersarr Oct 14, 2019

Choose a reason for hiding this comment

fersarr commented Oct 18, 2019 • edited Loading

mattip commented Oct 20, 2019

fersarr commented Aug 14, 2019 •

edited by mattip

Loading

fersarr commented Aug 15, 2019 •

edited

Loading

mattip commented Aug 20, 2019 •

edited

Loading

eric-wieser Sep 22, 2019 •

edited

Loading

fersarr Sep 30, 2019 •

edited

Loading

fersarr Sep 30, 2019 •

edited

Loading

fersarr Oct 4, 2019 •

edited

Loading

fersarr commented Oct 18, 2019 •

edited

Loading