ENH/BUG: usecols does not raise an exception when col index is out of bounds.

[ghost]

import pandas as pd

df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
df.to_csv('test.csv', index=False)

# Should raise ValueError, since there is no col at index 10
pd.read_csv('test.csv', header=0, usecols=[0, 10], engine='python')

print('Silently ran to completion.')

---

Output

Silently ran to completion.

Problem description

Silently passing incorrect column indices (result of some arithmetic) could potentially lead to a debugging nightmare.

A possible solution is to check if the largest value in usecols is larger than the number of columns at any level.

# Something along the lines of
if any(max(col_indices) >= len(column) for column in columns):
    raise ValueError('...')
columns = [[n for i, n in enumerate(column) if i in col_indices]
           for column in columns]
self._col_indices = col_indices

If the idea is to accommodate different levels having different number of columns, maybe it would be better if usecols allowed a sequence of sequences (one for each level).

Note:

• A similar problem exists with the C engine, although the result is an unexpected column full of nulls.

• The issue seems to be handled correctly when index is written to file.

Edit:

• The issue occurs when the length of usecols equals the number of columns in dataframe.

Expected Output

ValueError

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.18.0-16-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 4.0.2
pip: 18.1
setuptools: 40.6.3
Cython: 0.29.2
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: 1.8.2
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 3.0.2
openpyxl: 2.5.12
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.1.2
lxml: 4.2.5
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[WillAyd]

I suppose this is related to #17301 though that only covers labels and not indices.

I think it makes sense to raise on this; investigation and PRs would certainly be welcome

[WillAyd]

Had to revert this change as it conflicted with usage in #25623

I personally think this is still desirable to validate but that behavior needs to be deprecated first

[phofl]

Deprecated in #41130

[Vondoe79]

I have a similar issue in this conversation where I have a function - "read_files()" - which takes 2 arguments (a file_name & a list). I then traverse my project directory using a "For loop" to fetch the target files and feed them to the "read_files()" function. But I am constantly getting this error: FutureWarning: Defining usecols with out of bounds indices is deprecated and will raise a ParserError in a future version. df_22 = pd.read_excel(file_name, sheet_name='Sheet1', skiprows=1, nrows=22, usecols=range(1, 23))

I have posted an excerpt of my code StackOverflow with the same data file at - https://stackoverflow.com/questions/73263189/pandas-excel-data-read-with-incorrect-output-no-getting-all-the-tabular-data-fr?noredirect=1#comment129454561_73263189. However, I have gotten any workable solution yet. I have even tried the solution offered by @ghost from March 2019, but still can't get my code to work.

I, therefore, welcome any further ideas/suggestions from this forum on what I may be missing.