refactor: Decorate api methods that require total ordering #802

TrevorBergeron · 2024-06-20T17:21:55Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

tswast · 2024-06-26T15:22:49Z

bigframes/core/indexes/base.py

@@ -424,6 +433,8 @@ def dropna(self, how: typing.Literal["all", "any"] = "any") -> Index:
        return Index(result)

    def drop_duplicates(self, *, keep: str = "first") -> Index:
+        if keep is not False:
+            enforce_ordered(self, "duplicated")


Is this a typo? I assume it should reflect the public API name used, right?

Suggested change

enforce_ordered(self, "duplicated")

enforce_ordered(self, "drop_duplicates")

tswast · 2024-06-26T15:24:13Z

bigframes/core/indexes/base.py

@@ -30,6 +30,7 @@
 import bigframes.core.expression as ex
 import bigframes.core.ordering as order
 import bigframes.core.utils as utils
+from bigframes.core.validate import enforce_ordered, requires_strict_ordering


Import modules, not methods.

https://google.github.io/styleguide/pyguide.html#22-imports

There are only a few exceptions: https://google.github.io/styleguide/pyguide.html#2241-exemptions

tswast · 2024-06-26T15:25:55Z

bigframes/core/window_spec.py

+
+    @property
+    def row_bounded(self):
+        # relevant for determining if window requires total ordering for determinism.


Nit: This could be transformed into a docstring. I like that you shared the "why" but I'd also like a little more explanation on what row_bounded means.

changed to docstring

tswast · 2024-06-26T15:26:52Z

bigframes/dataframe.py

@@ -61,6 +61,7 @@
 import bigframes.core.indexes as indexes
 import bigframes.core.ordering as order
 import bigframes.core.utils as utils
+from bigframes.core.validate import enforce_ordered, requires_strict_ordering


Import packages not methods.

tswast · 2024-06-26T15:27:30Z

bigframes/core/groupby/__init__.py

@@ -27,6 +27,7 @@
 import bigframes.core.blocks as blocks
 import bigframes.core.ordering as order
 import bigframes.core.utils as utils
+from bigframes.core.validate import requires_strict_ordering


Import packages not methods.

tswast · 2024-06-26T15:29:50Z

bigframes/core/validate.py

Nit: I'd prefer a noun for a package name. How about validations.py?

tswast · 2024-06-26T15:30:22Z

bigframes/operations/aggregations.py

@@ -42,6 +42,11 @@ def uses_total_row_ordering(self):
    def can_order_by(self):
        return False

+    @property
+    def order_independent(self):
+        """Whether the output of the operator depends on the ordering of input rows. Navigation functions are a notable case."""


Let's keep the title line short.

Suggested change

"""Whether the output of the operator depends on the ordering of input rows. Navigation functions are a notable case."""

"""True if the output of the operator depends on the ordering of input rows.

Navigation functions are a notable case.

"""

tswast · 2024-06-26T15:32:55Z

bigframes/operations/aggregations.py

@@ -78,6 +83,11 @@ def name(self) -> str:
    def arguments(self) -> int:
        ...

+    @property
+    def order_independent(self):
+        # Almost all aggregation functions are order independent, excepting array and string agg


Let's make this a docstring too.

Suggested change

# Almost all aggregation functions are order independent, excepting array and string agg

"""True if results don't depend on the order of the input.

Almost all aggregation functions are order independent, excepting ``array_agg`` and ``string_agg``.

"""

tswast · 2024-06-26T15:33:29Z

bigframes/series.py

@@ -43,6 +43,7 @@
 import bigframes.core.ordering as order
 import bigframes.core.scalar as scalars
 import bigframes.core.utils as utils
+from bigframes.core.validate import enforce_ordered, requires_strict_ordering


import package

tswast · 2024-06-26T15:35:36Z

tests/system/small/test_unordered.py

+        pytest.param(
+            lambda x: x.cumsum(),
+            id="cumsum",
+            marks=pytest.mark.xfail(raises=bigframes.exceptions.OrderRequiredError),


Since all of these tests fail with the same exception, prefer with pytest.raises(...) in the test function body. Ideally we check for the function name in the error message with match= as well.

bigframes/dataframe.py

bigframes/series.py

TrevorBergeron added 5 commits May 20, 2024 23:53

refactor: label all apis that use total ordering

0220ef5

Merge remote-tracking branch 'github/main' into unordered_mode

22cef3c

reorganize flags

db91e4b

Merge remote-tracking branch 'github/main' into unordered_mode

c3a364a

fix config reference and add another test

01888e5

product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels Jun 20, 2024

TrevorBergeron added 3 commits June 20, 2024 17:49

fix broken attribute reference

e54cec0

add _session property to groupby objects

32bceac

restore session_tokyo fixture

d1a123e

TrevorBergeron marked this pull request as ready for review June 20, 2024 20:25

TrevorBergeron requested review from a team as code owners June 20, 2024 20:25

TrevorBergeron requested a review from chelsea-lin June 20, 2024 20:25

blunderbuss-gcf bot assigned GarrettWu Jun 20, 2024

TrevorBergeron requested review from tswast and removed request for chelsea-lin June 20, 2024 20:26

TrevorBergeron added 2 commits June 20, 2024 21:17

add docstring for OrderRequiredError

d745ca2

add _session property to index object

cac44aa

GarrettWu removed their assignment Jun 21, 2024

TrevorBergeron added 2 commits June 25, 2024 17:18

handled methods where only some parameterizations need total order

bfcdeb9

Merge remote-tracking branch 'github/main' into unordered_mode

6ef878d

TrevorBergeron force-pushed the unordered_mode branch from 844f21c to 6ef878d Compare June 25, 2024 17:19

fix inverted validation

c37c683

tswast requested changes Jun 26, 2024

View reviewed changes

TrevorBergeron added 4 commits June 26, 2024 16:41

Merge remote-tracking branch 'github/main' into unordered_mode

809833a

import validations module and not functions

d933bda

mark some analytic ops as order independent

c09f5f6

rename validate.py to validations.py

c999a14

product-auto-label bot added the size: l Pull request size is large. label Jun 26, 2024

product-auto-label bot removed the size: m Pull request size is medium. label Jun 26, 2024

docstring for WindowSpec.row_bounded

50895de

TrevorBergeron requested a review from tswast June 26, 2024 17:16

Merge branch 'main' into unordered_mode

0b50a86

tswast approved these changes Jun 28, 2024

View reviewed changes

tswast added 2 commits June 28, 2024 09:50

Apply suggestions from code review

5726019

Merge branch 'main' into unordered_mode

3899eea

tswast enabled auto-merge (squash) June 28, 2024 14:51

tswast merged commit ec5b068 into main Jun 28, 2024
23 checks passed

tswast deleted the unordered_mode branch June 28, 2024 15:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: Decorate api methods that require total ordering #802

refactor: Decorate api methods that require total ordering #802

TrevorBergeron commented Jun 20, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

tswast Jun 26, 2024

TrevorBergeron Jun 26, 2024

	enforce_ordered(self, "duplicated")
	enforce_ordered(self, "drop_duplicates")

-        # Almost all aggregation functions are order independent, excepting array and string agg
+        """True if results don't depend on the order of the input.
+        Almost all aggregation functions are order independent, excepting ``array_agg`` and ``string_agg``.
+        """

refactor: Decorate api methods that require total ordering #802

refactor: Decorate api methods that require total ordering #802

Conversation

TrevorBergeron commented Jun 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment