BUG: Convert data elements when dtype=str in Series constructor with … #18795

reidy-p · 2017-12-15T17:08:36Z

…int/float list

closes Series constructor skips dtype=str conversion for list data #16605
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Not sure if my solution is correct but it seems to resolve the issue and pass the tests.

jreback · 2017-12-15T17:11:31Z

pandas/core/series.py

+        # If not empty convert the data to dtype
+        if not isna(data).all():
+            data = np.array(data, dtype=dtype)
+


u can set copy =False i think here
and just use .astype() which copies

A lot of tests started failing when I replaced

data = np.array(data, dtype=dtype)

with

data = np.array(data, copy=False).astype(dtype)

Do you know why? Is it related to the fact that dtype= is used only for upcasting but .astype(dtype) is used for downcasting?

yeah this is numpy stringifying things. I think this might be ok if you follow this with copy=False, IOW this will copy things, so don't need to copy again.

codecov · 2017-12-15T18:54:19Z

Codecov Report

❗ No coverage uploaded for pull request base (master@07d8c2d). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master   #18795   +/-   ##
=========================================
  Coverage          ?   91.62%           
=========================================
  Files             ?      154           
  Lines             ?    51410           
  Branches          ?        0           
=========================================
  Hits              ?    47106           
  Misses            ?     4304           
  Partials          ?        0

Flag	Coverage Δ
#multiple	`89.49% <100%> (?)`
#single	`40.84% <100%> (?)`

Impacted Files	Coverage Δ
pandas/core/series.py	`94.83% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07d8c2d...11542e9. Read the comment docs.

jreback · 2017-12-15T23:30:47Z

pandas/tests/series/test_constructors.py

@@ -142,6 +142,22 @@ def test_constructor_list_like(self):
            result = Series(obj, index=[0, 1, 2])
            assert_series_equal(result, expected)

+    @pytest.mark.parametrize('input_vals, dtype, expected', [
+        ([1, 2, 3], 'str', ['1', '2', '3']),


can you add other dtypes here: strings, datetimes, datetime w/tz, period, timedelta, categorical (with int), categorical (with float), categorical (with str), interval. throw in a nan or two on the floats.

(you can make it an array of len 2 to type a bit less if it helps). could also parametrize this I think (IOW an astype of str of a list should be quiv of str on each element

I added some more dtypes to the tests. I'm not sure how to add a test for categorical, though, because Series(..,dtype=str) throws a ValueError for a Categorical but .astype(str) doesn't:

In [1]: pd.Series(pd.Categorical([1.0, 2.0, np.nan])).astype(str) Out[1]: 0 1.0 1 2.0 2 nan dtype: object In [2]: pd.Series(pd.Categorical([1.0, 2.0, np.nan]), dtype=str) Out[2]: ValueError: cannot specify a dtype with a Categorical unless dtype='category'

jreback · 2017-12-18T12:36:28Z

pandas/core/series.py

+        # If not empty convert the data to dtype
+        if not isna(data).all():
+            data = np.array(data, dtype=dtype)
+


yeah this is numpy stringifying things. I think this might be ok if you follow this with copy=False, IOW this will copy things, so don't need to copy again.

jreback · 2017-12-18T12:38:00Z

pandas/tests/series/test_constructors.py

+    ])
+    def test_constructor_list_str(self, input_vals):
+        # GH 16605
+        # Ensure that data elements are converted to strings when


the original issue also covers DataFrame, I am not sure this fix will handle this. Can you add a test for that as well?

Yeah, the original issue covers DataFrame but shows that it works as expected. For example, from the original issue:

In [1]: int_list = [1, 2, 3] In [2]: s3 = pd.Series(int_list, dtype=str) # Expect str but get int In [3]: print(type(s3[0])) Out[3]: <class 'int'> In [4]: f3 = pd.DataFrame(int_list, dtype=str) # Expect str and get str In [5]: print(type(f3.iloc[0, 0])) Out[5]: <class 'str'>

But I can try to add similar DataFrame tests if they don't exist already.

are things covered for DataFrame? ping when ready.

I couldn't find any tests that directly covered the issue so I added a similar test in tests/frame/test_dtypes.py

…int/float list

pep8speaks · 2017-12-19T21:30:58Z

Hello @reidy-p! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on December 19, 2017 at 21:34 Hours UTC

jreback · 2017-12-21T15:18:39Z

thanks @reidy-p

jreback requested changes Dec 15, 2017

View reviewed changes

jreback added Bug Dtype Conversions Unexpected or buggy dtype conversions Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Dec 15, 2017

jreback requested changes Dec 15, 2017

View reviewed changes

reidy-p force-pushed the series_dtype_str branch from dae1c35 to f3fd1d9 Compare December 16, 2017 16:43

jreback requested changes Dec 18, 2017

View reviewed changes

reidy-p force-pushed the series_dtype_str branch from f3fd1d9 to 697385e Compare December 18, 2017 21:42

reidy-p added 4 commits December 19, 2017 21:30

BUG: Convert data elements when dtype=str in Series constructor with …

3c2d669

…int/float list

add more dtypes to tests

bb5847a

add copy=False

decb776

add DataFrame tests

732f1f8

reidy-p force-pushed the series_dtype_str branch from 697385e to 732f1f8 Compare December 19, 2017 21:30

PEP8

11542e9

jreback added this to the 0.22.0 milestone Dec 20, 2017

jreback approved these changes Dec 20, 2017

View reviewed changes

jreback merged commit 4a2d55b into pandas-dev:master Dec 21, 2017

reidy-p deleted the series_dtype_str branch December 21, 2017 16:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Convert data elements when dtype=str in Series constructor with … #18795

BUG: Convert data elements when dtype=str in Series constructor with … #18795

reidy-p commented Dec 15, 2017 •

edited

Loading

jreback Dec 15, 2017

reidy-p Dec 16, 2017

jreback Dec 18, 2017

codecov bot commented Dec 15, 2017 •

edited

Loading

jreback Dec 15, 2017

reidy-p Dec 16, 2017

jreback Dec 18, 2017

jreback Dec 18, 2017

reidy-p Dec 18, 2017 •

edited

Loading

jreback Dec 19, 2017

reidy-p Dec 19, 2017

pep8speaks commented Dec 19, 2017 •

edited

Loading

jreback commented Dec 21, 2017

BUG: Convert data elements when dtype=str in Series constructor with … #18795

BUG: Convert data elements when dtype=str in Series constructor with … #18795

Conversation

reidy-p commented Dec 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 15, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reidy-p Dec 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pep8speaks commented Dec 19, 2017 • edited Loading

Comment last updated on December 19, 2017 at 21:34 Hours UTC

jreback commented Dec 21, 2017

reidy-p commented Dec 15, 2017 •

edited

Loading

codecov bot commented Dec 15, 2017 •

edited

Loading

reidy-p Dec 18, 2017 •

edited

Loading

pep8speaks commented Dec 19, 2017 •

edited

Loading