values() query on ForeignObject discards additional columns

[Jacob Walls]

For a multi-column field using ForeignObject as modeled ​here, .values("user") will discard columns after the first, and values("user", "integer") will select the second column of user into the integer namespace.

Here's a failing test for tests/composite_pk/test_values.py:

    def test_foreign_object_values(self):
        Comment.objects.create(id=1, user=self.user_1, integer=42)
        values = list(Comment.objects.values("user", "integer"))
        self.assertEqual(values[0]["integer"], 42)

======================================================================
FAIL: test_foreign_object_values (composite_pk.test_values.CompositePKValuesTests.test_foreign_object_values)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/.../django/tests/composite_pk/test_values.py", line 217, in test_foreign_object_values
    self.assertEqual(values[0]["integer"], 42)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 1 != 42

And a similar rough test for foreign_objects/tests.py that fails on Django 4.2, to remove the composite primary key from discussion. (If this model had more fields, this values query would be more realistic):

class ForeignObjectValues(TestCase):
    def test_foreign_object_values(self):
        from .models import Customer
        customer_1 = Customer.objects.create(customer_id=1, company="a")
        customer_2 = Customer.objects.create(customer_id=1, company="b")
        company_a_customers = Customer.objects.filter(company="a").values("address", "company")
        self.assertSequenceEqual(
            Customer.objects.exclude(company__in=[cust["company"] for cust in company_a_customers]),
            [customer_2],  # Fails, all companies included
        )

[Simon Charette]

On interesting dilemma we'll have to figure out here is what exactly to return when a foreign object composed of more than one from_fields is selected through values and values_list.

With composite primary keys we've made usages of values("pk") turned into values(Tuple(*pk_field_attnames)) and not values(*pk_field_attnames) so we should be coherent here IMO which means that we likely want something like

tests/composite_pk/test_values.py

@@ -3 +3 @@
 
 from django.test import TestCase
 
-from .models import Post, Tenant, User
+from .models import Comment, Post, Tenant, User
 
 
 class CompositePKValuesTests(TestCase):
@@ -210 +210 @@
                     {"pk": self.user_3.pk, "id": self.user_3.id},
                 ),
             )
+
+    def test_foreign_object_values(self):
+        Comment.objects.create(id=1, user=self.user_1, integer=42)
+        values = list(Comment.objects.values("user", "integer"))
+        self.assertEqual(values[0]["user"], (self.user_1.tenant_id, self.user_1.id))
+        self.assertEqual(values[0]["integer"], 42)

In other words I suggest we double down on making sure references to composite fields (the only two we support today is composite pk and foreign objects with multiple fields) in values and values_list return tuples instead of flattening them as we've taken this approach for CompositePrimaryKey (which is now part of the public API) and that ForeignObject is still private and has been broken since its introduction in this regard.