EA: fillna should accept same type #32414 #43230

Bhavay-2001 · 2021-08-26T15:24:50Z

A unit test has been added to check the validity of pd.Categorical() . This PR is with respect to the GitHub issue no. 32414.
Thanks

closes EA: fillna should accept same type #32414

pep8speaks · 2021-08-26T15:24:53Z

Hello @Bhavay192! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file pandas/tests/series/methods/test_fillna.py:

Line 683:5: E304 blank lines found after function decorator

Comment last updated at 2021-11-20 10:33:09 UTC

Bhavay-2001 · 2021-08-26T15:26:22Z

I request all the members to please review my PR and please tell if any changes needs to done. Thanks

phofl

Your test seems broken, e.g. It is not passing the ci.

also you have a linting error. You could install pre-commit to run the Checks locally.

See https://pandas.pydata.org/pandas-docs/stable/development/contributing.html for further information

Bhavay-2001 · 2021-08-27T05:50:20Z

hey @phofl , thanks for reviewing. I couldn't understand where my test is failing. It would be really helpful if you could help me understand in a little more detail as I'm quite new to open source. Hope you understand. Also I checked the pandas documentation but couldn't find anything that suits to my test.

phofl · 2021-08-27T12:09:09Z

I was talking about https://pandas.pydata.org/docs/development/contributing_codebase.html# and your Pep 8 warning.

Is your test passing locally? If yes you can check https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=64906&view=logs&j=404760ec-14d3-5d48-e580-13034792878f&t=f81e4cc8-d61a-5fb8-36be-36768e5c561a&l=59 to get the stacktrace

Bhavay-2001 · 2021-08-28T18:21:25Z

hey @phofl , thanks for suggesting me the failure. I opened the link in you comments . It is quite difficult to interpret where my test is failing but by seeing the logs, it makes me feel that the code is failing with the TypeError that i have raised. Please cross check my code once and tell me if could you find my mistake. Also i ran the test locally on my laptop using the unittest module of python, soo there might be some changes with pytest. Please see if you can help me with the error. Thanks

phofl · 2021-08-28T20:31:13Z

Running them with unittest is not helpful, since we are using a lot of pytest functionality. Please run them with pytest then your test will fail locally too

Bhavay-2001 · 2021-08-30T09:07:03Z

hey @phofl, thanks for replying. Based on your comment I ran my test locally on my machine with pytest module and I have updated the code with the changes that were causing the error. Hope soo that it shall now pass all the tests and doesn't conflict with the rest of code. Please review the updated code. Thanks

phofl

Please use our pr template for your pull request header to reference the issue and check the remaining tasks

Edit: Test is still failing

phofl · 2021-08-30T10:31:03Z

pandas/tests/series/methods/test_fillna.py

+        data = ["A", "B", np.nan, np.nan, "C"]
+        ser = Series(Categorical(data, categories=["A", "B", "C"]))
+
+        # msg = "Element not present in categories. Cannot be filled in series."


Is this relevant? If not please delete

Okay I will delete the commented part

phofl · 2021-08-30T10:31:05Z

pandas/tests/series/methods/test_fillna.py

+        #     ser.fillna("D")
+
+        exp = Series(Categorical(expected_output, categories=["A", "B", "C"]))
+        result = ser.fillna(fill_value)


This does not cover the issue. We need to check with a categorical fill_value too.

I have defined the the fill_value in the pytest parameterized section. There I have provided a fill_value . Do we need to explicitly define that categroical value here??

Could you please ellaborate on this a little more ?? I find it difficult to understand

You are not testing everything mentioned in the issue. @mroeschke provided a code snippet there which should be covered here

In [22]: cat = pd.Categorical(["A", "B", None, "A"])
...: ser = pd.Series(cat).fillna("B")

In [23]: >>> filled = cat.fillna(ser)

In [24]: >>> cat.fillna(filled)
Out[24]:
['A', 'B', 'B', 'A']
Categories (2, object): ['A', 'B']

hey @phofl, this was the code snippet as being provided by @mroeschke, I have tried to add the same thing. Soo i'm asking should I add "B" instead of that fill_value??
I saw other tests too and they did the same thing, do you want me to explicitly declare "B"

okay @phofl , I have checked with categorical_fillna too. I will add the updated code tomorrow positively. I think with that issue will be resolved. Thanks

phofl · 2021-08-30T10:31:20Z

pandas/tests/series/methods/test_fillna.py

+        # with pytest.raises(TypeError, match=msg):
+        #     ser.fillna("D")
+
+        exp = Series(Categorical(expected_output, categories=["A", "B", "C"]))


Please call expected

Okay i will do the necessary changes.

Bhavay-2001 · 2021-09-03T13:43:57Z

hey @phofl, I have updated the pull request. Please review it once.

phofl · 2021-09-03T19:25:20Z

Tests are still failing

Bhavay-2001 · 2021-09-04T07:16:29Z

I'm unable to understand why they are still failing. I made all the necessary changes that you said, made a complete test out of that code snippet and its working fine on my machine and also I tried to design the test in the style of the other tests in that file. Still its failing .

MarcoGorelli · 2021-09-04T08:24:47Z

and its working fine on my machine

Pretty sure the code, as it's written, won't pass - could you show what you ran and what your output is please? Perhaps you have some commits you haven't pushed yet?

Bhavay-2001 · 2021-09-04T08:44:07Z

hey @MarcoGorelli, thanks for replying. I tried to run only the function I have written on my machine without the class. As i made the class and ran it showed some error and I'm unable to intrepret it.
I will attach the my code and the output here. Please review it.

Code

Output

Bhavay-2001 · 2021-09-04T08:47:10Z

However, If I try to run test_fillna.py complete file on my machine, It passes all the 39 tests, but fails on only 1 test. I'm adding below the output of that too.

MarcoGorelli · 2021-09-04T10:00:47Z

Try running pytest pandas/tests/series/methods/test_fillna.py -k test_series_fill, that'll reproduce the error you see in CI

(you may need to replace / with \ if you're on Windows)

jreback · 2021-09-04T14:29:48Z

pandas/tests/series/methods/test_fillna.py

+        ],
+    )
+
+    def test_series_fill(fill_value, expected_output):


can you rename to tests_fillna_categorical

yaa i will rename it.

jreback · 2021-09-04T14:31:37Z

pandas/tests/series/methods/test_fillna.py

+        exp_ser = Series(exp)
+        result = ser.fillna(fill_value)
+        filled = cat.fillna(fill_value)
+        tm.assert_almost_equal(result, exp_ser)


can you use
tm.assert_series_equal(result_ser, exp_ser)

and
tm.assert_categorical_equal(result_cat, exp_cat)

and rename things a bit, this is very hard to read

yaa i surely rename it. Thanks for checking that out.

Bhavay-2001 · 2021-09-04T15:54:59Z

Hey @MarcoGorelli, I tested the complete test_fillna.py in my machine, it shows no errors. However, on running the above command that you told, it gives just an import error in a seperate file and not in test_fillna.py. Now, what should i do next??

MarcoGorelli · 2021-09-04T15:56:07Z

can you paste your command and output please?

Bhavay-2001 · 2021-09-04T16:20:01Z

Yes, my testing_fillna.py is same as the test_fillna.py and i ran it. it shows the following command

MarcoGorelli · 2021-09-04T16:24:27Z

However, on running the above command that you told, it gives just an import error in a seperate file and not in test_fillna.py

can you paste command and output of this?

Bhavay-2001 · 2021-09-04T16:29:42Z

github-actions · 2021-10-05T00:03:00Z

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

mroeschke · 2021-10-31T00:57:52Z

Appears this PR has been dormant for a while and is still failing in the CI so closing. If interested in continuing, please merge master, address related comments and we can reopen.

Bhavay-2001 · 2021-11-12T05:59:07Z

Hey, I was a bit busy soo couldn't contribute to it. I'm thinking of working on this PR now.

Bhavay-2001 · 2021-11-12T06:48:38Z

Hey, my function is just working fine I believe. The problem is coming with other functions. I have tested my function by commenting out the error-causing functions and all the tests have passed. Please open the PR, soo that I can discuss on this further. Thanks

MarcoGorelli · 2021-11-12T08:32:49Z

sure, reopened

Bhavay-2001 · 2021-11-13T11:57:10Z

Hey @MarcoGorelli, May I show u my code ?? Cause the problem comes here that the test is failing for other codes in the file. If I run my code alone on a separate file it just works fine. Soo may I??

MarcoGorelli · 2021-11-13T12:10:11Z

feel free to paste your code (copy and paste it rather than showing a screenshot)

Bhavay-2001 · 2021-11-18T16:59:16Z

Hey @MarcoGorelli , sorry for late replying. I will surely paste my code sample here so that u can have a look. Please give me some time.

Bhavay-2001 · 2021-11-19T06:34:30Z

def test_fillna_categorical(self, fill_value, expected_output):
        # GH32414
        data = ["A", "B", np.nan, np.nan, "C"]
        cat = Categorical(data, categories=["A", "B", "C"])
        ser = Series(cat)
        exp_cat = Categorical(expected_output, categories=["A", "B", "C"])
        exp_ser = Series(exp_cat)
        result_ser = ser.fillna(fill_value)
        filled = cat.fillna(fill_value)
        tm.assert_almost_equal(result_ser, exp_ser)
        tm.assert_almost_equal(filled, exp_cat)

Bhavay-2001 · 2021-11-19T06:35:54Z

@MarcoGorelli , this is my code sample. Please review it soo that I can merge it. Thanks

MarcoGorelli · 2021-11-19T09:31:18Z

Please show your whole file - I don't believe that that one passes because self would be undefined.
Please also show what you ran, and the output

Bhavay-2001 · 2021-11-19T11:10:49Z

Hey @MarcoGorelli, soo I ran the testing_datatype.py file in another file and here was the result.

Bhavay-2001 · 2021-11-19T11:13:24Z

this was after I commented out all the error causing functions. Thanks

Bhavay-2001 · 2021-11-20T16:38:01Z

Please if anyone can review my code and see what is causing the problem cause I can't really find the mistake due to which it is not passing all the tests. Any help will be appreciated. Thanks

MarcoGorelli · 2021-11-20T16:49:48Z

show your whole testing_datatype.py file
you need to run the test in your clone of pandas, within your pandas-dev virtual environment

Bhavay-2001 · 2021-11-20T17:07:15Z

He @MarcoGorelli , how can I show you my whole testing_datatype.py file?? Should I paste the code here??? And also how can I run the tests in my pandas-dev virtual environment??

MarcoGorelli · 2021-11-20T17:19:26Z

Trim the file down so it only has this test - try to isolate the issue

Regarding pytest, you just need to:

activate your virtual environment - if you use conda, that's conda activate pandas-dev
run the test, like you've done

see https://pandas.pydata.org/pandas-docs/stable/development/contributing_codebase.html?highlight=pytest#test-driven-development-code-writing

In general, I'd suggest you read through the contributing guide, it sounds like there might be a bit more preparation needed before we can pick this PR up https://pandas.pydata.org/pandas-docs/stable/development/index.html

jreback · 2022-01-16T17:58:59Z

closing as stale

Bhavay-2001 added 2 commits August 26, 2021 20:46

TST Providing unit test to snippet GH32414

c46b201

TST Providing unit test to snippet GH#32414

02db9f0

phofl requested changes Aug 26, 2021

View reviewed changes

alimcmaster1 added the Testing pandas testing functions or related to the test suite label Aug 26, 2021

mroeschke mentioned this pull request Aug 27, 2021

TST Providing unit test to snippet GH32414 #43136

Closed

1 task

Updated the test_fillna file based on GH43230

b872815

phofl requested changes Aug 30, 2021

View reviewed changes

Bhavay-2001 added 2 commits August 30, 2021 20:02

Updated with required changes

49b693c

Updated the changes

edf6af7

jreback changed the title ~~TST Providing unit test to snippet GH32414~~ EA: fillna should accept same type #32414 Sep 4, 2021

jreback added ExtensionArray Extending pandas with custom dtypes or arrays. Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate labels Sep 4, 2021

jreback requested changes Sep 4, 2021

View reviewed changes

github-actions bot added the Stale label Oct 5, 2021

mroeschke closed this Oct 31, 2021

MarcoGorelli reopened this Nov 12, 2021

test_fillna.py updated

12ab6aa

jreback closed this Jan 16, 2022

EA: fillna should accept same type #32414 #43230

EA: fillna should accept same type #32414 #43230

Conversation

Bhavay-2001 commented Aug 26, 2021 • edited by mroeschke Loading

pep8speaks commented Aug 26, 2021 • edited Loading

Comment last updated at 2021-11-20 10:33:09 UTC

Bhavay-2001 commented Aug 26, 2021

phofl left a comment

Choose a reason for hiding this comment

Bhavay-2001 commented Aug 27, 2021

phofl commented Aug 27, 2021

Bhavay-2001 commented Aug 28, 2021

phofl commented Aug 28, 2021

Bhavay-2001 commented Aug 30, 2021

phofl left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bhavay-2001 commented Sep 3, 2021

phofl commented Sep 3, 2021

Bhavay-2001 commented Sep 4, 2021

MarcoGorelli commented Sep 4, 2021

Bhavay-2001 commented Sep 4, 2021

Bhavay-2001 commented Sep 4, 2021

MarcoGorelli commented Sep 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bhavay-2001 commented Sep 4, 2021

MarcoGorelli commented Sep 4, 2021

Bhavay-2001 commented Sep 4, 2021

MarcoGorelli commented Sep 4, 2021

Bhavay-2001 commented Sep 4, 2021

github-actions bot commented Oct 5, 2021

mroeschke commented Oct 31, 2021

Bhavay-2001 commented Nov 12, 2021

Bhavay-2001 commented Nov 12, 2021

MarcoGorelli commented Nov 12, 2021

Bhavay-2001 commented Nov 13, 2021

MarcoGorelli commented Nov 13, 2021

Bhavay-2001 commented Nov 18, 2021

Bhavay-2001 commented Nov 19, 2021 • edited by MarcoGorelli Loading

Bhavay-2001 commented Nov 19, 2021

MarcoGorelli commented Nov 19, 2021

Bhavay-2001 commented Nov 19, 2021

Bhavay-2001 commented Nov 19, 2021

Bhavay-2001 commented Nov 20, 2021

MarcoGorelli commented Nov 20, 2021

Bhavay-2001 commented Nov 20, 2021

MarcoGorelli commented Nov 20, 2021 • edited Loading

jreback commented Jan 16, 2022

Bhavay-2001 commented Aug 26, 2021 •

edited by mroeschke

Loading

pep8speaks commented Aug 26, 2021 •

edited

Loading

phofl left a comment •

edited

Loading

MarcoGorelli commented Sep 4, 2021 •

edited

Loading

Bhavay-2001 commented Nov 19, 2021 •

edited by MarcoGorelli

Loading

MarcoGorelli commented Nov 20, 2021 •

edited

Loading