Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure most recent metadata is served despite duplicate version 0's #268

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Ensure most recent metadata is served despite duplicate version 0's #268

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4fe3700
Added ordering by createdOn
May 30, 2023
fe6916f
Added unit testing
May 31, 2023
96e4394
Only run test on old schema
Jun 1, 2023
9bef470
Fixed sorting
Jun 1, 2023
82d4ff4
Remove print
Jun 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions dao-impl/ebean-dao/src/main/java/com/linkedin/metadata/dao/EbeanLocalDAO.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -892,6 +892,8 @@ private List<EbeanMetadataAspect> batchGetUnion(@Nonnull List<AspectKey<URN, ? e
}
}

sb.append(" ORDER BY createdOn DESC");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs to be inside the For loop. Basically you want

//   SELECT * FROM metadata_aspect WHERE urn = 'urn0' AND aspect = 'aspect0' AND version = 0 ORDER BY createdOn DESC limit 1
//   UNION ALL
//   SELECT * FROM metadata_aspect WHERE urn = 'urn0' AND aspect = 'aspect1' AND version = 0 ORDER BY createdOn DESC limit 1

does it make sense?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought through this as well and believed it should be outside because in the case where you're running multiple SELECT's, we want the final result to be ordered, not just each subsection.


final Query<EbeanMetadataAspect> query = _server.findNative(EbeanMetadataAspect.class, sb.toString());

for (int i = 1; i <= params.size(); i++) {
Expand Down
23 changes: 23 additions & 0 deletions dao-impl/ebean-dao/src/test/java/com/linkedin/metadata/dao/EbeanLocalDAOTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3094,6 +3094,29 @@ public void testNewSchemaExactMatchEmptyArray() {
}
}

@Test
public void testDuplicityOrdering() {
if (_schemaConfig == SchemaConfig.OLD_SCHEMA_ONLY) {
_server.execute(Ebean.createSqlUpdate(readSQLfromFile(GMA_DROP_ALL_SQL)));
_server.execute(Ebean.createSqlUpdate(readSQLfromFile("gma-create-all-duplicity.sql")));

EbeanLocalDAO<EntityAspectUnion, FooUrn> dao = createDao(FooUrn.class);
FooUrn urn = makeFooUrn(1);
String aspectName = ModelUtils.getAspectName(AspectFoo.class);
AspectFoo v1 = new AspectFoo().setValue("foo");
AspectFoo v0 = new AspectFoo().setValue("bar");

// introduce two v0's
addMetadataWithAuditStamp(urn, aspectName, 0, v1, 1234L, "fakeCreator", "fakeImpersonator");
addMetadataWithAuditStamp(urn, aspectName, 0, v0, 0L, "fakeCreator", "fakeImpersonator");

// when retrieving a v0-duplicate record, the returned value should be the LATEST (timestamp=1234L)
AspectFoo actual = dao.get(AspectFoo.class, urn).get();
System.out.println(actual);
jphui marked this conversation as resolved.
Show resolved Hide resolved
assertEquals(actual, v1);
}
}

@Nonnull
private EbeanMetadataAspect getMetadata(Urn urn, String aspectName, long version, @Nullable RecordTemplate metadata) {
EbeanMetadataAspect aspect = new EbeanMetadataAspect();
Expand Down
32 changes: 32 additions & 0 deletions dao-impl/ebean-dao/src/test/resources/gma-create-all-duplicity.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
create table metadata_id (
namespace varchar(255) not null,
id bigint not null,
constraint uq_metadata_id_namespace_id unique (namespace,id)
);

create table metadata_aspect (
urn varchar(500) not null,
aspect varchar(200) not null,
version bigint not null,
metadata clob not null,
createdon timestamp not null,
createdby varchar(255) not null,
createdfor varchar(255),
constraint pk_metadata_aspect primary key (urn,aspect,version,createdon)
);

create table metadata_index (
id bigint auto_increment not null,
urn varchar(500) not null,
aspect varchar(200) not null,
path varchar(200) not null,
longval bigint,
stringval varchar(500),
doubleval double,
constraint pk_metadata_index primary key (id)
);

create index idx_long_val on metadata_index (aspect,path,longval,urn);
create index idx_string_val on metadata_index (aspect,path,stringval,urn);
create index idx_double_val on metadata_index (aspect,path,doubleval,urn);
create index idx_urn on metadata_index (urn);