Merge remote-tracking branch 'upstream/master' into argsort_labelling_index_sorted

Author: attack68

doc/source/reference/io.rst

@@ -13,6 +13,7 @@ Pickling
    :toctree: api/
 
    read_pickle
+   DataFrame.to_pickle
 
 Flat file
 ~~~~~~~~~
@@ -21,6 +22,7 @@ Flat file
 
    read_table
    read_csv
+   DataFrame.to_csv
    read_fwf
 
 Clipboard
@@ -29,30 +31,41 @@ Clipboard
    :toctree: api/
 
    read_clipboard
+   DataFrame.to_clipboard
 
 Excel
 ~~~~~
 .. autosummary::
    :toctree: api/
 
    read_excel
+   DataFrame.to_excel
    ExcelFile.parse
 
+.. currentmodule:: pandas.io.formats.style
+
+.. autosummary::
+   :toctree: api/
+
+   Styler.to_excel
+
+.. currentmodule:: pandas
+
 .. autosummary::
    :toctree: api/
    :template: autosummary/class_without_autosummary.rst
 
    ExcelWriter
 
+.. currentmodule:: pandas.io.json
+
 JSON
 ~~~~
 .. autosummary::
    :toctree: api/
 
    read_json
-   json_normalize
-
-.. currentmodule:: pandas.io.json
+   to_json
 
 .. autosummary::
    :toctree: api/
@@ -67,13 +80,40 @@ HTML
    :toctree: api/
 
    read_html
+   DataFrame.to_html
+
+.. currentmodule:: pandas.io.formats.style
+
+.. autosummary::
+   :toctree: api/
+
+   Styler.to_html
+
+.. currentmodule:: pandas
 
 XML
 ~~~~
 .. autosummary::
    :toctree: api/
 
    read_xml
+   DataFrame.to_xml
+
+Latex
+~~~~~
+.. autosummary::
+   :toctree: api/
+
+   DataFrame.to_latex
+
+.. currentmodule:: pandas.io.formats.style
+
+.. autosummary::
+   :toctree: api/
+
+   Styler.to_latex
+
+.. currentmodule:: pandas
 
 HDFStore: PyTables (HDF5)
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -101,13 +141,15 @@ Feather
    :toctree: api/
 
    read_feather
+   DataFrame.to_feather
 
 Parquet
 ~~~~~~~
 .. autosummary::
    :toctree: api/
 
    read_parquet
+   DataFrame.to_parquet
 
 ORC
 ~~~
@@ -138,6 +180,7 @@ SQL
    read_sql_table
    read_sql_query
    read_sql
+   DataFrame.to_sql
 
 Google BigQuery
 ~~~~~~~~~~~~~~~
@@ -152,6 +195,7 @@ STATA
    :toctree: api/
 
    read_stata
+   DataFrame.to_stata
 
 .. currentmodule:: pandas.io.stata
 

doc/source/reference/style.rst

@@ -35,11 +35,14 @@ Style application
    Styler.apply
    Styler.applymap
    Styler.format
+   Styler.hide_index
+   Styler.hide_columns
    Styler.set_td_classes
    Styler.set_table_styles
    Styler.set_table_attributes
    Styler.set_tooltips
    Styler.set_caption
+   Styler.set_sticky
    Styler.set_properties
    Styler.set_uuid
    Styler.clear

doc/source/user_guide/style.ipynb

@@ -152,7 +152,7 @@
     "\n",
     "Before adding styles it is useful to show that the [Styler][styler] can distinguish the *display* value from the *actual* value. To control the display value, the text is printed in each cell, and we can use the [.format()][formatfunc] method to manipulate this according to a [format spec string][format] or a callable that takes a single value and returns a string. It is possible to define this for the whole table or for individual columns. \n",
     "\n",
-    "Additionally, the format function has a **precision** argument to specifically help formatting floats, an **na_rep** argument to display missing data, and an **escape** argument to help displaying safe-HTML. The default formatter is configured to adopt pandas' regular `display.precision` option, controllable using `with pd.option_context('display.precision', 2):`\n",
+    "Additionally, the format function has a **precision** argument to specifically help formatting floats, as well as **decimal** and **thousands** separators to support other locales, an **na_rep** argument to display missing data, and an **escape** argument to help displaying safe-HTML or safe-LaTeX. The default formatter is configured to adopt pandas' regular `display.precision` option, controllable using `with pd.option_context('display.precision', 2):`\n",
     "\n",
     "Here is an example of using the multiple options to control the formatting generally and with specific column formatters.\n",
     "\n",
@@ -167,9 +167,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "df.style.format(precision=0, na_rep='MISSING', \n",
+    "df.style.format(precision=0, na_rep='MISSING', thousands=\" \",\n",
     "                formatter={('Decision Tree', 'Tumour'): \"{:.2f}\",\n",
-    "                           ('Regression', 'Non-Tumour'): lambda x: \"$ {:,.1f}\".format(x*-1e3)\n",
+    "                           ('Regression', 'Non-Tumour'): lambda x: \"$ {:,.1f}\".format(x*-1e6)\n",
     "                          })"
    ]
   },
@@ -179,9 +179,11 @@
    "source": [
     "### Hiding Data\n",
     "\n",
-    "The index can be hidden from rendering by calling [.hide_index()][hideidx], which might be useful if your index is integer based.\n",
+    "The index and column headers can be completely hidden, as well subselecting rows or columns that one wishes to exclude. Both these options are performed using the same methods.\n",
     "\n",
-    "Columns can be hidden from rendering by calling [.hide_columns()][hidecols] and passing in the name of a column, or a slice of columns.\n",
+    "The index can be hidden from rendering by calling [.hide_index()][hideidx] without any arguments, which might be useful if your index is integer based. Similarly column headers can be hidden by calling [.hide_columns()][hidecols] without any arguments.\n",
+    "\n",
+    "Specific rows or columns can be hidden from rendering by calling the same [.hide_index()][hideidx] or [.hide_columns()][hidecols] methods and passing in a row/column label, a list-like or a slice of row/column labels to for the ``subset`` argument.\n",
     "\n",
     "Hiding does not change the integer arrangement of CSS classes, e.g. hiding the first two columns of a DataFrame means the column class indexing will start at `col2`, since `col0` and `col1` are simply ignored.\n",
     "\n",
@@ -1403,7 +1405,9 @@
    "source": [
     "### Sticky Headers\n",
     "\n",
-    "If you display a large matrix or DataFrame in a notebook, but you want to always see the column and row headers you can use the following CSS to make them stick. We might make this into an API function later."
+    "If you display a large matrix or DataFrame in a notebook, but you want to always see the column and row headers you can use the [.set_sticky][sticky] method which manipulates the table styles CSS.\n",
+    "\n",
+    "[sticky]: ../reference/api/pandas.io.formats.style.Styler.set_sticky.rst"
    ]
   },
   {
@@ -1412,20 +1416,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "bigdf = pd.DataFrame(np.random.randn(15, 100))\n",
-    "bigdf.style.set_table_styles([\n",
-    "    {'selector': 'thead th', 'props': 'position: sticky; top:0; background-color:salmon;'},\n",
-    "    {'selector': 'tbody th', 'props': 'position: sticky; left:0; background-color:lightgreen;'}  \n",
-    "])"
+    "bigdf = pd.DataFrame(np.random.randn(16, 100))\n",
+    "bigdf.style.set_sticky(axis=\"index\")"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Hiding Headers\n",
-    "\n",
-    "We don't yet have any API to hide headers so a quick fix is:"
+    "It is also possible to stick MultiIndexes and even only specific levels."
    ]
   },
   {
@@ -1434,7 +1433,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "df3.style.set_table_styles([{'selector': 'thead tr', 'props': 'display: none;'}])  # or 'thead th'"
+    "bigdf.index = pd.MultiIndex.from_product([[\"A\",\"B\"],[0,1],[0,1,2,3]])\n",
+    "bigdf.style.set_sticky(axis=\"index\", pixel_size=18, levels=[1,2])"
    ]
   },
   {
@@ -1524,6 +1524,17 @@
     "![Excel spreadsheet with styled DataFrame](../_static/style-excel.png)\n"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Export to LaTeX\n",
+    "\n",
+    "There is support (*since version 1.3.0*) to export `Styler` to LaTeX. The documentation for the [.to_latex][latex] method gives further detail and numerous examples.\n",
+    "\n",
+    "[latex]: ../reference/api/pandas.io.formats.style.Styler.to_latex.rst"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

doc/source/whatsnew/v1.3.0.rst

@@ -138,6 +138,7 @@ which has been revised and improved (:issue:`39720`, :issue:`39317`, :issue:`404
  - Added the option ``styler.render.max_elements`` to avoid browser overload when styling large DataFrames (:issue:`40712`)
  - Added the method :meth:`.Styler.to_latex` (:issue:`21673`), which also allows some limited CSS conversion (:issue:`40731`)
  - Added the method :meth:`.Styler.to_html` (:issue:`13379`)
+ - Added the method :meth:`.Styler.set_sticky` to make index and column headers permanently visible in scrolling HTML frames (:issue:`29072`)
 
 .. _whatsnew_130.enhancements.dataframe_honors_copy_with_dict:
 
@@ -986,6 +987,7 @@ Indexing
 ^^^^^^^^
 - Bug in :meth:`Index.union` and :meth:`MultiIndex.union` dropping duplicate ``Index`` values when ``Index`` was not monotonic or ``sort`` was set to ``False`` (:issue:`36289`, :issue:`31326`, :issue:`40862`)
 - Bug in :meth:`CategoricalIndex.get_indexer` failing to raise ``InvalidIndexError`` when non-unique (:issue:`38372`)
+- Bug in :meth:`IntervalIndex.get_indexer` when ``target`` has ``CategoricalDtype`` and both the index and the target contain NA values (:issue:`41934`)
 - Bug in :meth:`Series.loc` raising a ``ValueError`` when input was filtered with a Boolean list and values to set were a list with lower dimension (:issue:`20438`)
 - Bug in inserting many new columns into a :class:`DataFrame` causing incorrect subsequent indexing behavior (:issue:`38380`)
 - Bug in :meth:`DataFrame.__setitem__` raising a ``ValueError`` when setting multiple values to duplicate columns (:issue:`15695`)
@@ -1025,6 +1027,7 @@ Indexing
 - Bug in :meth:`PeriodIndex.get_loc` failing to raise a ``KeyError`` when given a :class:`Period` with a mismatched ``freq`` (:issue:`41670`)
 - Bug ``.loc.__getitem__`` with a :class:`UInt64Index` and negative-integer keys raising ``OverflowError`` instead of ``KeyError`` in some cases, wrapping around to positive integers in others (:issue:`41777`)
 - Bug in :meth:`Index.get_indexer` failing to raise ``ValueError`` in some cases with invalid ``method``, ``limit``, or ``tolerance`` arguments (:issue:`41918`)
+- Bug when slicing a :class:`Series` or :class:`DataFrame` with a :class:`TimedeltaIndex` when passing an invalid string raising ``ValueError`` instead of a ``TypeError`` (:issue:`41821`)
 
 Missing
 ^^^^^^^
@@ -1033,13 +1036,15 @@ Missing
 - Bug in :meth:`DataFrame.fillna` not accepting a dictionary for the ``downcast`` keyword (:issue:`40809`)
 - Bug in :func:`isna` not returning a copy of the mask for nullable types, causing any subsequent mask modification to change the original array (:issue:`40935`)
 - Bug in :class:`DataFrame` construction with float data containing ``NaN`` and an integer ``dtype`` casting instead of retaining the ``NaN`` (:issue:`26919`)
+- Bug in :meth:`Series.isin` and :meth:`MultiIndex.isin` didn't treat all nans as equivalent if they were in tuples (:issue:`41836`)
 
 MultiIndex
 ^^^^^^^^^^
 - Bug in :meth:`DataFrame.drop` raising a ``TypeError`` when the :class:`MultiIndex` is non-unique and ``level`` is not provided (:issue:`36293`)
 - Bug in :meth:`MultiIndex.intersection` duplicating ``NaN`` in the result (:issue:`38623`)
 - Bug in :meth:`MultiIndex.equals` incorrectly returning ``True`` when the :class:`MultiIndex` contained ``NaN`` even when they are differently ordered (:issue:`38439`)
 - Bug in :meth:`MultiIndex.intersection` always returning an empty result when intersecting with :class:`CategoricalIndex` (:issue:`38653`)
+- Bug in :meth:`MultiIndex.difference` incorrectly raising ``TypeError`` when indexes contain non-sortable entries (:issue:`41915`)
 - Bug in :meth:`MultiIndex.reindex` raising a ``ValueError`` when used on an empty :class:`MultiIndex` and indexing only a specific level (:issue:`41170`)
 - Bug in :meth:`MultiIndex.reindex` raising ``TypeError`` when reindexing against a flat :class:`Index` (:issue:`41707`)
 
@@ -1079,6 +1084,7 @@ I/O
 - Bug in the conversion from PyArrow to pandas (e.g. for reading Parquet) with nullable dtypes and a PyArrow array whose data buffer size is not a multiple of the dtype size (:issue:`40896`)
 - Bug in :func:`read_excel` would raise an error when pandas could not determine the file type even though the user specified the ``engine`` argument (:issue:`41225`)
 - Bug in :func:`read_clipboard` copying from an excel file shifts values into the wrong column if there are null values in first column (:issue:`41108`)
+- Bug in :meth:`DataFrame.to_hdf` and :meth:`Series.to_hdf` raising a ``TypeError`` when trying to append a string column to an incompatible column (:issue:`41897`)
 
 Period
 ^^^^^^
@@ -1138,6 +1144,8 @@ Groupby/resample/rolling
 - Bug in :class:`DataFrameGroupBy` aggregations incorrectly failing to drop columns with invalid dtypes for that aggregation when there are no valid columns (:issue:`41291`)
 - Bug in :meth:`DataFrame.rolling.__iter__` where ``on`` was not assigned to the index of the resulting objects (:issue:`40373`)
 - Bug in :meth:`.DataFrameGroupBy.transform` and :meth:`.DataFrameGroupBy.agg` with ``engine="numba"`` where ``*args`` were being cached with the user passed function (:issue:`41647`)
+- Bug in :class:`DataFrameGroupBy` methods ``agg``, ``transform``, ``sum``, ``bfill``, ``ffill``, ``pad``, ``pct_change``, ``shift``, ``ohlc`` dropping ``.columns.names`` (:issue:`41497`)
+
 
 Reshaping
 ^^^^^^^^^
@@ -1160,6 +1168,7 @@ Reshaping
 - Bug in :func:`to_datetime` raising an error when the input sequence contained unhashable items (:issue:`39756`)
 - Bug in :meth:`Series.explode` preserving the index when ``ignore_index`` was ``True`` and values were scalars (:issue:`40487`)
 - Bug in :func:`to_datetime` raising a ``ValueError`` when :class:`Series` contains ``None`` and ``NaT`` and has more than 50 elements (:issue:`39882`)
+- Bug in :meth:`Series.unstack` and :meth:`DataFrame.unstack` with object-dtype values containing timezone-aware datetime objects incorrectly raising ``TypeError`` (:issue:`41875`)
 - Bug in :meth:`DataFrame.melt` raising ``InvalidIndexError`` when :class:`DataFrame` has duplicate columns used as ``value_vars`` (:issue:`41951`)
 
 Sparse
@@ -1206,6 +1215,7 @@ Other
 - Bug in :class:`Series` backed by :class:`DatetimeArray` or :class:`TimedeltaArray` sometimes failing to set the array's ``freq`` to ``None`` (:issue:`41425`)
 - Bug in creating a :class:`Series` from a ``range`` object that does not fit in the bounds of ``int64`` dtype (:issue:`30173`)
 - Bug in creating a :class:`Series` from a ``dict`` with all-tuple keys and an :class:`Index` that requires reindexing (:issue:`41707`)
+- Bug in :func:`pandas.util.hash_pandas_object` not recognizing ``hash_key``, ``encoding`` and ``categorize`` when the input object type is a :class:`DataFrame` (:issue:`41404`)
 
 .. ---------------------------------------------------------------------------
 

doc/source/whatsnew/v1.4.0.rst

@@ -167,7 +167,7 @@ Missing
 
 MultiIndex
 ^^^^^^^^^^
--
+- Bug in :meth:`MultiIndex.reindex` when passing a ``level`` that corresponds to an ``ExtensionDtype`` level (:issue:`42043`)
 -
 
 I/O