Skip to content

df.interpolate limit_direction= does not work without limit=x setting #16282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
WBare opened this issue May 8, 2017 · 2 comments · Fixed by #16429
Closed

df.interpolate limit_direction= does not work without limit=x setting #16282

WBare opened this issue May 8, 2017 · 2 comments · Fixed by #16429
Labels
Bug Numeric Operations Arithmetic, Comparison, and Logical operations
Milestone

Comments

@WBare
Copy link
Contributor

WBare commented May 8, 2017

the interpolate limit_direction setting should work even without a limit=x setting.

When interpolate is called without a limit=x setting, it always (apparently) uses the limit_direction setting default of 'forward' even when it is explicitly set to 'backward' or 'both'

The code below will show 3 DataFrames.

  • The first shows the initial before interpolation.
  • The second shows the interpolation done correctly in the backward direction.
  • The third shows the dataframe done incorrectly in the 'forward' direction even though the limit_direction='backward' was used.

The docs do not mention a linkage between the two settings, but if this is expected behavior (I hope not), it would be oh so nice to raise an exception like the existing error for a bad limit_direction setting.

Many thanks for all your work!

Example Code:

import pandas as pd
import numpy as np

dfMain = pd.DataFrame({
    'a': [0, 1, np.NAN, 3, 4],
    'b': [np.NaN, 1, np.NaN, 3, 4],
    'c': [0 , 1, 2, 3, np.NaN]})

print("initial df")
print(dfMain)

print("backwards with limit=1 (correct result):")
print(dfMain.interpolate(limit_direction='backward', limit=1))

print("backwards no longer works when limit is removed (it is going forward):")
print(dfMain.interpolate(limit_direction='backward'))

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.0.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-75-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 34.4.1
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: 2.9.4
boto: 2.45.0
pandas_datareader: 0.2.1
None
initial df

@TomAugspurger
Copy link
Contributor

Thanks for the report, want to take a look at where it's going wrong? Maybe somewhere around

def interpolate(self, method='pad', axis=0, index=None, values=None,

@TomAugspurger TomAugspurger added Bug Numeric Operations Arithmetic, Comparison, and Logical operations labels May 8, 2017
@TomAugspurger TomAugspurger added this to the 0.20.2 milestone May 8, 2017
@WBare
Copy link
Contributor Author

WBare commented May 9, 2017

I've got a fix for this and the associated enhancement for limit_direction='inside' GH16284.

The issue was inside missing.py interpolate_1d

I have tested myself, and I'm running the test suite now, but since I'm not familiar with the code base, someone really should carefully review the code. It is actually fairly simple, but I don't know if there are other dependencies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Numeric Operations Arithmetic, Comparison, and Logical operations
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants