Eager Loading a relation that calls another relation returns incorrect results

[allandantasdev]

Laravel Version

11.7.0

PHP Version

8.3.7

Database Driver & Version

PostgreSQL 15.7 and MySQL 8.0.37

Description

When eager loading a model relationships, the results differ from when they are lazy loaded.
This issue occurs whenever a Relation is called inside the definition of another Relation, but only when eager loading is used on the main one.

After investigation I realized that the cause lies in Illuminate\Database\Eloquent\Builder@eagerLoadRelation:

 // First we will "back up" the existing where conditions on the query so we can
 // add our eager constraints.
 $relation = $this->getRelation($name);
 $relation->addEagerConstraints($models);
 
 // Then we will merge the wheres that were on the
 // query back to it in order that any where conditions might be specified.
 $constraints($relation);

Which calls the Illuminate\Database\Eloquent\Builder@getRelation method:

 // We want to run a relationship query without any constraints so that we will
 // not have to remove these where clauses manually which gets really hacky
 // and error prone. We don't want constraints because we add eager ones.
 $relation = Relation::noConstraints(function () use ($name) {
     try {
         return $this->getModel()->newInstance()->$name();
     } catch (BadMethodCallException) {
         throw RelationNotFoundException::make($this->getModel(), $name);
     }
 });

Which gets to the root cause of the problem in Illuminate\Database\Eloquent\Relations\Relation@noConstraints:

$previous = static::$constraints;

 static::$constraints = false;

 // When resetting the relation where clause, we want to shift the first element
 // off of the bindings, leaving only the constraints that the developers put
 // as "extra" on the relationships, and not original relation constraints.
 try {
 return $callback();
 } finally {
 static::$constraints = $previous;
 }

The method Illuminate\Database\Eloquent\Relations\Relation@noConstraints is called during eager loading and uses a boolean attribute to manage the constraints. However, this flag is static and seems to be causing the where clauses of other relations to be omitted, leading to incorrect results.

Steps To Reproduce

1. Create the following schema

        // 0001_01_01_000000_create_users_table.php
        Schema::create('categories', function (Blueprint $table) {
            $table->bigIncrements('id');
            $table->unsignedBigInteger('user_id');
            $table->foreign('user_id')->references('id')->on('users');
            $table->text('name');
            $table->timestamps();
        });
        Schema::create('examples', function (Blueprint $table) {
            $table->bigIncrements('id');
            $table->unsignedBigInteger('category_id');
            $table->foreign('category_id')->references('id')->on('categories');
            $table->text('name');
            $table->boolean('restricted');
            $table->timestamps();
        });

2. Create the following seeder:

        $user = User::factory()->create();

        $categories = [
            Category::query()->create(['user_id' => $user->id, 'name' => 'Category 1']),
            Category::query()->create(['user_id' => $user->id, 'name' => 'Category 2']),
            Category::query()->create(['user_id' => $user->id, 'name' => 'Category 3']),
        ];

        Example::insert([
            ['category_id' => $categories[0]->id, 'name' => 'Example 1', 'restricted' => false],
            ['category_id' => $categories[1]->id, 'name' => 'Example 2', 'restricted' => true],
            ['category_id' => $categories[2]->id, 'name' => 'Example 3', 'restricted' => false],
            ['category_id' => $categories[2]->id, 'name' => 'Example 4', 'restricted' => false],
            ['category_id' => $categories[2]->id, 'name' => 'Example 5', 'restricted' => true],
        ]);

        User::factory()->create(); // another user just for demonstration

3. Create a scope in the Example model:

class Example extends Model
{
    // ...
    /**
     * The authenticated user should only have access to not restricted Examples
     * or to the examples he owns.
     */
    public function scopeHasAccess(Builder $query, ?User $user = null): Builder
    {
        return $query->where(
            fn ($query) => $query->where('restricted', false)
                ->when(
                    $user !== null,
                    fn($query) => $query->orWhereIn('category_id', $user->categories->pluck('id'))
                )
        );
    }
}

4. Add the following relations to the models:

class User extends Authenticatable
{
    // ...
    public function categories(): HasMany
    {
        return $this->hasMany(Category::class);
    }
}

class Category extends Model
{
    // ...
    public function examples(): HasMany
    {
        return $this->hasMany(Example::class);
            ->hasAccess(Auth::user());
     }
}

5. Authenticate:

    Auth::login(User::find(1));
    // Auth::login(User::find(2));

6. Fetch all categories with their respective examples

        dump('Authenticated user: '.Auth::user()->id);
        Category::get()->each(
            fn(Category $category) => dump(sprintf('- %s: %d examples', $category->name, $category->examples->count()))
        );
        
// Authenticated user: 1
// - Category 1: 1 examples"
// - Category 2: 1 examples"
// - Category 3: 3 examples"
// ----------------------------
// Authenticated user: 2"
// - Category 1: 1 examples"
// - Category 2: 0 examples"
// - Category 3: 2 examples"

7. Execute the code again but eager loading the examples relation:

        dump('Authenticated user: '.Auth::user()->id);
        Category::with('examples')->get()->each(
            fn(Category $category) => dump(sprintf('- %s: %d examples', $category->name, $category->examples->count()))
        );
        
// Authenticated user: 1
// - Category 1: 1 examples"
// - Category 2: 1 examples"
// - Category 3: 3 examples"
// ----------------------------
// Authenticated user: 2"
// - Category 1: 1 examples"
// - Category 2: 1 examples"
// - Category 3: 3 examples"

• Expected behavior:
  The fetched relations should be consistent regardless of whether they are lazy or eager loaded.

• Actual behavior:

  • Without eager loading: The user of id 2 has access to 3 examples (correct)
  • With eager loading: The user of id 2 has access to 5 examples (wrong)

[crynobone]

Hey there, thanks for reporting this issue.

We'll need more info and/or code to debug this further. Can you please create a repository with the command below, commit the code that reproduces the issue as one separate commit on the main/master branch and share the repository here?

Please make sure that you have the latest version of the Laravel installer in order to run this command. Please also make sure you have both Git & the GitHub CLI tool properly set up.

laravel new bug-report --github="--public"

Do not amend and create a separate commit with your custom changes. After you've posted the repository, we'll try to reproduce the issue.

Thanks!

[allandantasdev]

Here it is: allandantasdev/laravel-bug-report-51825@ce9741a

[allandantasdev]

Additional info:

• This is the query generated by Auth::user()->categories->pluck('id') at Example@scopeHasAccess:24 while lazy loading
  

• And is the same query generated after trying to eagerloading Category::with('examples'):
  

[github-actions]

Thank you for reporting this issue!

As Laravel is an open source project, we rely on the community to help us diagnose and fix issues as it is not possible to research and fix every issue reported to us via GitHub.

If possible, please make a pull request fixing the issue you have described, along with corresponding tests. All pull requests are promptly reviewed by the Laravel team.

Thank you!

[Tofandel]

I found a quick and dirty solution, in the constructor of Relation (beware that it breaks some other cases)

   public function __construct(Builder $query, Model $parent)
    {
        $this->query = $query;
        $this->parent = $parent;
        $this->related = $query->getModel();

        $this->addConstraints();

        static::$constraints = true; // This
    }

I've been working on getting a proper fix, but there doesn't seem to be a straight path forward as changing one thing breaks another one, this will likely require some debug_backtrace to fix this without breaking some test cases

[crynobone]

I believe #52461 have fixed this issue, please open a new issue if you still face the problem

[Tofandel]

Sorry, that PR doesn't address this issue at all, I was only talking about how using static causes this kind of issues

You can reopen it

[marius-mcp]

        // We want to run a relationship query without any constrains so that we will
        // not have to remove these where clauses manually which gets really hacky
        // and error prone. We don't want constraints because we add eager ones.
        $relation = Relation::noConstraints(function () use ($name) {
            try {
                return $this->getModel()->newInstance()->$name();
            } catch (BadMethodCallException $e) {
                throw RelationNotFoundException::make($this->getModel(), $name);
            }
        });

@allandantasdev this makes sense because the conditions might imply columns from the model (other than the foreign key), that will not be in the query for the related model.

the solution might be to add the constraints on the collection of related models based on each model

    protected function eagerLoadRelation(array $models, $name, Closure $constraints)
    {
        // First we will "back up" the existing where conditions on the query so we can
        // add our eager constraints. Then we will merge the wheres that were on the
        // query back to it in order that any where conditions might be specified.
        $relation = $this->getRelation($name);

        $relation->addEagerConstraints($models);

        $constraints($relation);
///////////////////////////////////////////////// HERE
        // Once we have the results, we just match those back up to their parent models
        // using the relationship instance. Then we just return the finished arrays
        // of models which have been eagerly hydrated and are readied for return.
        return $relation->match(
            $relation->initRelation($models, $name),
            $relation->getEager(), $name
        );
    }

[macropay-solutions]

Sorry for the previous replies. Now we understood the real issue.
This is called

$user->categories->pluck('id')

while the

static::$constraints

is false because of the eager load of the relation in which it is called, resulting in all categories being retrieved not only the categories from that user.

[macropay-solutions]

@Tofandel @allandantasdev

If the Relation had this function(which is doable via macros as static function):

    /**
     * Run a callback with constraints enabled on the relation.
     *
     * @param  \Closure  $callback
     * @return mixed
     */
    public static function yesConstraints(Closure $callback)
    {
        $previous = static::$constraints;

        static::$constraints = true;

        try {
            return $callback();
        } finally {
            static::$constraints = $previous;
        }
    }

then the scope or condition could be written like this:

class Example extends Model
{
    // ...
    /**
     * The authenticated user should only have access to not restricted Examples
     * or to the examples he owns.
     */
    public function scopeHasAccess(Builder $query, ?User $user = null): Builder
    {
        return $query->where(
            fn ($query) => $query->where('restricted', false)
                ->when(
                    $user !== null,
                    function ($query) use ($user) {
                          $userCategories = $user->relationLoaded('categories') ?
                              $user->categories :
                              Relation::yesConstraints(function () use ($user) {
                                  try {
                                      return $user->categories();
                                  } catch (BadMethodCallException $e) {
                                      throw RelationNotFoundException::make($user, 'categories');
                                  }
                              });

                        return $query->orWhereIn('category_id', $userCategories->pluck('id');); // also, this can be written with a sub select and the issue is avoided in that way
                    }
                )
        );
    }
}

Can this solution be embedded in laravel somehow so the user does not need to handle it in the scope or relation definition?

[macropay-solutions]

UPDATE: #51825 (comment)

[Tofandel]

@macropay-solutions I already tried this solution but it breaks some special cases

The withConstraints approach might be the easieast workaround to get into the core

[macropay-solutions]

@Tofandel Your code always set static:$constraints = true; #51825 (comment)

Our suggestion sets it only once (with the previous value not with hard codded true) at the construct's end and not in that finally clause.

But if you say it breaks special cases, we believe you.

[marius-mcp]

@Tofandel can you please share those special cases?

[Tofandel]

It will fail on those kind of relations because the Relation constructor is called twice in there and so it restores constraints too early

    public function price_without_key_in_aggregates()
    {
        return $this->hasOne(HasOneOfManyTestPrice::class, 'user_id')->ofMany(['published_at' => 'MAX']);
    }

    public function price_with_shortcut()
    {
        return $this->hasOne(HasOneOfManyTestPrice::class, 'user_id')->latestOfMany(['published_at', 'id']);
    }
    
    
    public function teamMatesWithPendingRelation()
    {
        return $this->through($this->ownedTeams())
            ->has(fn (Team $team) => $team->members());
    }

[Tofandel]

Just run the vendor/bin/phpunit tests on the repo with your changes and see how it goes, likely it will be very unreliable, I doubt where is the only method that needs this

[marius-mcp]

@Tofandel @macropay-solutions There is another case that would be uncovered:

public function relationName(): HasMany|HasManyThrough
{
    if ($this->exists && $this->children()->exist()) { // construct of Relation is called on eager loading relations in an existing model which I remember had issues also
        return $this->hasManyThrough...; // construct of Relation is called
    }

    return $this->hasMany...; // construct of Relation is called
}

[macropay-solutions]

@marius-mcp
Good catch. Then the whole Relation::noConstraints logic from \Illuminate\Database\Eloquent\Builder::getRelation is not fitting in...

[macropay-solutions]

@Tofandel

Another solution that sadly can't be implemented via macros for older versions:

\Illuminate\Database\Eloquent\Relations\Relation

    protected static ?string $noConstraintsForRelationName = null;

    public function __construct(Builder $query, Model $parent)
    {
        $this->query = $query;
        $this->parent = $parent;
        $this->related = $query->getModel();

        /** @see self::noConstraints */
        if (
            '' !== (string)static::$noConstraintsForRelationName
            && '' !== (string)$parent->nowEagerLoadingRelationNameWithNoConstraints
        ) {
            /**   1st execution is for ExampleModel $exampleModel on 'rel' relation
                 with nowEagerLoadingRelationNameWithNoConstraints = 'rel'
                           and with $noConstraintsForRelationName = 'rel'
             */
            //           2nd execution is for UserModel $userModel on 'categories' relation
            //     with nowEagerLoadingRelationNameWithNoConstraints = null
            //                and with $noConstraintsForRelationName = 'rel'

            //    1st execution is for ExampleModel $exampleModel on 'children' relation
            //    with nowEagerLoadingRelationNameWithNoConstraints = null
            //              and with $noConstraintsForRelationName = 'rel'
            /**   2nd execution is for ExampleModel $exampleModel on 'rel' relation
                with nowEagerLoadingRelationNameWithNoConstraints = 'rel'
                          and with $noConstraintsForRelationName = 'rel'
             */
            // 3rd execution is for UserModel $userModel on 'categories' relation
            //    with nowEagerLoadingRelationNameWithNoConstraints = null
            //       and with $noConstraintsForRelationName = 'rel'
            static::$constraints =
                static::$noConstraintsForRelationName !== $parent->nowEagerLoadingRelationNameWithNoConstraints;
        }

        $this->addConstraints();
    }

    /**
     * Run a callback with constraints disabled on the relation based on relationName.
     */
    public static function noConstraints(\Closure $callback, ?string $relationName = null): mixed
    {
        $previous = static::$constraints;
        $previousNoConstraintsForRelationName = static::$noConstraintsForRelationName;

        if ('' !== (string)$relationName) {
            static::$noConstraintsForRelationName = $relationName;
        } else {
            static::$constraints = false;
        }

        try {
            return $callback();
        } finally {
            static::$constraints = $previous;
            static::$noConstraintsForRelationName = $previousNoConstraintsForRelationName;
        }
    }

\Illuminate\Database\Eloquent\Builder

    /**
     * Get the relation instance for the given relation name (for eager loading)
     *
     * @param  string  $name
     * @return \Illuminate\Database\Eloquent\Relations\Relation
     */
    public function getRelation($name)
    {
        // We want to run a relationship query without any constrains so that we will
        // not have to remove these where clauses manually which gets really hacky
        // and error prone. We don't want constraints because we add eager ones.
        $relation = Relation::noConstraints(function () use ($name) {
            try {
                $model = $this->getModel()->newInstance();
                $model->nowEagerLoadingRelationNameWithNoConstraints = $name;

                return $model->$name();
            } catch (BadMethodCallException) {
                throw RelationNotFoundException::make($this->getModel(), $name);
            }
        }, $name);

        $nested = $this->relationsNestedUnder($name);

        // If there are nested relationships set on the query, we will put those onto
        // the query instances so that they can be handled after this relationship
        // is loaded. In this way they will all trickle down as they are loaded.
        if (count($nested) > 0) {
            $relation->getQuery()->with($nested);
        }

        return $relation;
    }

\Illuminate\Database\Eloquent\Concerns\HasRelationships

    public ?string $nowEagerLoadingRelationNameWithNoConstraints = null;

The definition of the relation:

    public function productsValueScope(): HasManyThrough
    {
        return $this->products()->where(
            fn ($query) => $query->where('value', '>',  10)->when(
                true,
                fn($query) => $query->orWhereIn('id', Operation::query()->where('id', 2)->first()->children->pluck('id')->toArray())
                )
        );
    }

RESULTS:

GET page=1&withRelations[0]=productsValueScope&limit=2

        -- Operation::query()->where('id', 2)->first()
        "0.48 ms, sql: select * from `operations` where `id` = 2 limit 1",

        -- ->children->pluck('id')
        "0.55 ms, sql: select * from `operations` where `operations`.`parent_id` = 2 and `operations`.`parent_id` is not null",

        -- count
        "0.56 ms, sql: select count(*) as aggregate from `operations`",

        -- fetch the list
        "1 ms, sql: select * from `operations` order by `created_at` desc limit 2 offset 0",

        -- eager load
        "0.63 ms, sql: select * from `operations` where `id` = 2 limit 1",
        "0.56 ms, sql: select * from `operations` where `operations`.`parent_id` = 2 and `operations`.`parent_id` is not null",
        "1.15 ms, sql: select distinct `products`.*, `operations_products_pivot`.`operation_id` as `laravel_through_key` from `products` inner join `operations_products_pivot` on `operations_products_pivot`.`product_id` = `products`.`id` where `operations_products_pivot`.`operation_id` is null and (`value` > 10 or `id` in (3)) and `operations_products_pivot`.`operation_id` in (3748917, 3748918)"

@crynobone we will not open a MR or PR so feel free to do it if you want to fix the issue.

[marius-mcp]

@macropay-solutions

Another solution that sadly can't be implemented via macros for older versions:

The good thing is that if the relations are defined with

if ($this->exists && {{other relations calls}}) {

or no other relations are called after the relation is instantiated, then this bug is not affecting older versions.

For the ->load( function call on the existing model, the fix is to not use it until this gets in the core.

So there is a way of avoiding it.

I tested the change. Works also for me.

[jkpeyi]

@allandantasdev Hello

Are you sure this code is correct ?

{
    // ...
    public function examples(): HasMany
    {
        return $this->hasMany(Example::class);  // this ends by a semicolon , so how the ->hasAccess() will work ?
            ->hasAccess(Auth::user());  
     }
}```

[marius-mcp]

@jkpeyi that is just a typo.

[macropay-solutions]

FYI
We will not be issuing a merge request or pull request with this solution.
We shared the solution to help the community but only if the community wants to be helped.

It would be pity to see here "closing this issue because it is too old" or something similar just like we saw happening in the past with other bugs that were not solved.

Please do not bury this under the carpet.