deps: use pandas-gbq to determine schema in `load_table_from_dataframe` #2095

tswast · 2024-12-26T22:28:55Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Towards go/pandas-gbq-and-bigframes-redundancy
🦕

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast · 2024-12-27T17:31:57Z

tests/unit/test_client.py::TestClientUpload::test_load_table_from_dataframe_w_higher_scale_decimal128_datatype looks like a real failure and might need a change in pandas-gbq.

…176126-pandas-gbq

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast · 2025-01-03T19:37:24Z

I've tested locally with googleapis/python-bigquery-pandas#844 and that PR should fix the remaining system test.

(venv) ➜  python-bigquery git:(b323176126-pandas-gbq) pytest tests/unit/test_client.py::TestClientUpload::test_load_table_from_dataframe_w_higher_scale_decimal128_datatype
==================================================== test session starts ====================================================
platform linux -- Python 3.12.6, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/swast/src/github.com/googleapis/python-bigquery
configfile: pyproject.toml
collected 1 item                                                                                                            

tests/unit/test_client.py .                                                                                           [100%]

===================================================== 1 passed in 0.08s =====================================================

pyproject.toml

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast · 2025-01-22T14:46:27Z

test_load_table_from_dataframe_w_partial_schema_extra_types

Looks like a real failure (the warning message changed slightly in pandas-gbq). I'll try to update the test to be a bit more robust.

tswast · 2025-01-23T17:24:55Z

@Linchin @chalmerlowe I believe this is finally ready for review. The failing checks are for infrastructure / Kokoro problems for a new setup, possibly?

google/cloud/bigquery/_pandas_helpers.py

tswast · 2025-01-27T15:37:32Z

tests/system/test_pandas.py

@@ -1259,7 +1259,7 @@ def test_upload_time_and_datetime_56(bigquery_client, dataset_id):
    df = pandas.DataFrame(
        dict(
            dt=[
-                datetime.datetime(2020, 1, 8, 8, 0, 0),
+                datetime.datetime(2020, 1, 8, 8, 0, 0, tzinfo=datetime.timezone.utc),


Not sure is we should have defined what happens when you mix and match naive datetime (DATETIME in BQ) with datetime + timezone (TIMESTAMP in BQ). Since the test expects TIMESTAMP, I update this to be a datetime + timezone.

We still mix/match timezones, so I think that part is important to make sure we end up returning UTC values.

chalmerlowe

By and large, this looks good.
I have a question that might be a potential blocker.

google/cloud/bigquery/_pandas_helpers.py

Co-authored-by: Chalmer Lowe <[email protected]>

tswast · 2025-01-28T15:24:04Z

pyproject.toml

@@ -74,6 +74,9 @@ bqstorage = [
 ]
 pandas = [
  "pandas >= 1.1.0",
+  "pandas-gbq >= 0.26.1; python_version >= '3.8'",


Note: this means we are testing without pandas-gbq on Python 3.7.

…176126-pandas-gbq

tswast · 2025-02-05T16:36:57Z

@chalmerlowe In 9a5656b I have explicitly uninstalled pandas-gbq in at least one unit and system test version. This should be robust to when we drop Python 3.7 and 3.8. I have verified the pip freeze output locally to confirm that pandas-gbq is not present in the sessions where we uninstall it.

chalmerlowe

LGTM.
Approved.

…e` (#2095) * feat: use pandas-gbq to determine schema in `load_table_from_dataframe` * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix some unit tests * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * bump minimum pandas-gbq to 0.26.1 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * drop pandas-gbq from python 3.7 extras * relax warning message text assertion * use consistent time zone presense/absense in time datetime system test * Update google/cloud/bigquery/_pandas_helpers.py * Update google/cloud/bigquery/_pandas_helpers.py Co-authored-by: Chalmer Lowe <[email protected]> * remove pandas-gbq from at least 1 unit test and system test session --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <[email protected]>

* Initial batch of changes to remove 3.7 and 3.8 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * more updates to remove 3.7 and 3.8 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * updates samples/geography/reqs * updates samples/magics/reqs * updates samples/notebooks/reqs * updates linting * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * updates conf due to linting issue * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * updates reqs.txt, fix mypy, lint, and debug in noxfile * Updates owlbot to correct spacing issue in conf.py * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * updates owlbot imports * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * removes kokoro samples configs for 3.7 & 3.8 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * removes owlbots attempt to restore kokoro samples configs * removes kokoro system-3.8.cfg * edits repo sync settings * updates assorted noxfiles for samples and pyproject.toml * update test-samples-impl.sh * updates install_deps template * Edits to the contributing documentation * deps: use pandas-gbq to determine schema in `load_table_from_dataframe` (#2095) * feat: use pandas-gbq to determine schema in `load_table_from_dataframe` * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix some unit tests * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * bump minimum pandas-gbq to 0.26.1 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * drop pandas-gbq from python 3.7 extras * relax warning message text assertion * use consistent time zone presense/absense in time datetime system test * Update google/cloud/bigquery/_pandas_helpers.py * Update google/cloud/bigquery/_pandas_helpers.py Co-authored-by: Chalmer Lowe <[email protected]> * remove pandas-gbq from at least 1 unit test and system test session --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <[email protected]> * Feat: Adds foreign_type_info attribute to table class and adds unit tests. (#2126) * adds foreign_type_info attribute to table * feat: Adds foreign_type_info attribute and tests * updates docstrings for foreign_type_info * Updates property handling, especially as regards set/get_sub_prop * Removes extraneous comments and debug expressions * Refactors build_resource_from_properties w get/set_sub_prop * updates to foreign_type_info, tests and wiring * Adds logic to detect non-Sequence schema.fields value * updates assorted tests and logic * deps: updates required checks list in github (#2136) * deps: updates required checks list in github * deps: updates snippet and system checks in github to remove 3.9 * changes the order of two items in the list. * updates linting * reverts pandas back to 1.1.0 * Revert changes related to pandas <1.5 * Revert noxfile.py changes related to pandas <1.5 * Revert constraints-3.9 changes related to pandas <1.5 * Revert test_query_pandas.py changes related to pandas <1.5 * Revert test__pandas_helpers.py changes related to pandas <1.5 * Revert test__versions_helpers.py changes related to pandas <1.5 * Revert tnoxfile.py changes related to pandas <1.5 * Revert test__versions_helpers.py changes related to pandas <1.5 * Revert test_table.py changes related to pandas <1.5 * Update noxfile changes related to pandas <1.5 * Update pyproject.toml changes related to pandas <1.5 * Update constraints-3.9.txt changes related to pandas <1.5 * Update test_legacy_types.py changes related to pandas <1.5 * Updates magics.py as part of reverting from pandas 1.5 * Updates noxfile.py in reverting from pandas 1.5 * Updates pyproject.toml in reverting from pandas 1.5 * Updates constraints.txt in reverting from pandas 1.5 * Updates test_magics in reverting from pandas 1.5 * Updates test_table in reverting from pandas 1.5 * Updates in tests re: reverting from pandas 1.5 * Updates pyproject to match constraints.txt * updates pyproject.toml to mirror constraints * remove limit on virtualenv * updates owlbot.py for test-samples-impl.sh * updates to owlbot.py * updates to test-samples-impl.sh * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * further updates to owlbot.py * removes unneeded files * adds presubmit.cfg back in --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Tim Sweña (Swast) <[email protected]>

feat: use pandas-gbq to determine schema in load_table_from_dataframe

b8ff031

tswast requested review from a team as code owners December 26, 2024 22:28

tswast requested a review from chalmerlowe December 26, 2024 22:28

product-auto-label bot added the size: m Pull request size is medium. label Dec 26, 2024

blunderbuss-gcf bot assigned yirutang Dec 26, 2024

product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Dec 26, 2024

🦉 Updates from OwlBot post-processor

04a9f11

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast and others added 5 commits December 27, 2024 12:56

fix some unit tests

be2dd1b

Merge remote-tracking branch 'origin/b323176126-pandas-gbq' into b323…

6a13cab

…176126-pandas-gbq

🦉 Updates from OwlBot post-processor

1606cd6

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge remote-tracking branch 'origin/main' into b323176126-pandas-gbq

2ae1121

🦉 Updates from OwlBot post-processor

e90cdfb

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast mentioned this pull request Jan 3, 2025

fix: ensure BIGNUMERIC type is used if scale > 9 in Decimal values googleapis/python-bigquery-pandas#844

Merged

4 tasks

tswast commented Jan 3, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

tswast added 2 commits January 7, 2025 09:56

Merge remote-tracking branch 'origin/main' into b323176126-pandas-gbq

61f035b

bump minimum pandas-gbq to 0.26.1

29f336a

tswast changed the title ~~feat: use pandas-gbq to determine schema in load_table_from_dataframe~~ deps: use pandas-gbq to determine schema in load_table_from_dataframe Jan 7, 2025

tswast requested a review from Linchin January 7, 2025 15:58

gcf-owl-bot bot and others added 4 commits January 7, 2025 15:59

🦉 Updates from OwlBot post-processor

aff7021

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'main' into b323176126-pandas-gbq

ce368d9

Merge remote-tracking branch 'origin/main' into b323176126-pandas-gbq

9b7583e

drop pandas-gbq from python 3.7 extras

f0ef613

tswast added 3 commits January 22, 2025 16:46

Merge remote-tracking branch 'origin/main' into b323176126-pandas-gbq

8f46e3d

relax warning message text assertion

f4c1a4f

use consistent time zone presense/absense in time datetime system test

76b7890

tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 23, 2025

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 23, 2025

tswast commented Jan 27, 2025

View reviewed changes

google/cloud/bigquery/_pandas_helpers.py Outdated Show resolved Hide resolved

Update google/cloud/bigquery/_pandas_helpers.py

d382e13

tswast commented Jan 27, 2025

View reviewed changes

chalmerlowe reviewed Jan 28, 2025

View reviewed changes

google/cloud/bigquery/_pandas_helpers.py Outdated Show resolved Hide resolved

Update google/cloud/bigquery/_pandas_helpers.py

672ae33

Co-authored-by: Chalmer Lowe <[email protected]>

tswast commented Jan 28, 2025

View reviewed changes

tswast added 4 commits February 3, 2025 12:42

Merge branch 'main' into b323176126-pandas-gbq

155528c

Merge remote-tracking branch 'origin/main' into b323176126-pandas-gbq

5a5e1b4

remove pandas-gbq from at least 1 unit test and system test session

9a5656b

Merge remote-tracking branch 'origin/b323176126-pandas-gbq' into b323…

1fab458

…176126-pandas-gbq

tswast mentioned this pull request Feb 18, 2025

catch pyarrow.lib.ArrowTypeError for augment_schema #2129

Open

Merge branch 'main' into b323176126-pandas-gbq

008aea3

chalmerlowe approved these changes Feb 18, 2025

View reviewed changes

tswast enabled auto-merge (squash) February 18, 2025 22:35

tswast merged commit 7603bd7 into main Feb 21, 2025
15 of 23 checks passed

tswast deleted the b323176126-pandas-gbq branch February 21, 2025 17:45

release-please bot mentioned this pull request Feb 21, 2025

chore(main): release 3.30.0 #2119

Merged

This was referenced Mar 4, 2025

March 03, 2025 kitta65/bq-extension-vscode#678

Closed

March 03, 2025 kitta65/prettier-plugin-bq#614

Closed

March 03, 2025 kitta65/bq2cst#480

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deps: use pandas-gbq to determine schema in `load_table_from_dataframe` #2095

deps: use pandas-gbq to determine schema in `load_table_from_dataframe` #2095

tswast commented Dec 26, 2024

tswast commented Dec 27, 2024

tswast commented Jan 3, 2025

tswast commented Jan 22, 2025

tswast commented Jan 23, 2025

tswast Jan 27, 2025

chalmerlowe left a comment

tswast Jan 28, 2025

tswast commented Feb 5, 2025

chalmerlowe left a comment

deps: use pandas-gbq to determine schema in load_table_from_dataframe #2095

deps: use pandas-gbq to determine schema in load_table_from_dataframe #2095

Conversation

tswast commented Dec 26, 2024

tswast commented Dec 27, 2024

tswast commented Jan 3, 2025

tswast commented Jan 22, 2025

tswast commented Jan 23, 2025

tswast Jan 27, 2025

Choose a reason for hiding this comment

chalmerlowe left a comment

Choose a reason for hiding this comment

tswast Jan 28, 2025

Choose a reason for hiding this comment

tswast commented Feb 5, 2025

chalmerlowe left a comment

Choose a reason for hiding this comment

deps: use pandas-gbq to determine schema in `load_table_from_dataframe` #2095

deps: use pandas-gbq to determine schema in `load_table_from_dataframe` #2095