Planning for 7.1 and/or 8.0 next

[tychonievich]

The steering committee today had a conversation about proposals to make more flexible records for several current substructures: PLAC (location), SOUR (citation), and the event/fact types. Some similar points could be made of non-record proposals like a revised NAME structure.

We can technically add these to 7.1, either as "pick one of the two", like we have for NOTE vs SNOTE; or as a "include both" pointer substructure to the current non-pointer structures, like we have for name parts (GIVN and so on). A disadvantage of "pick one of the two" is that it could be some applications understand only one of the two and lose data. A disadvantage of "standardized form" is significantly increased file size and duplicate data that could get out of sync, especially if interacting with an application that does not properly implement that structure.

We could also wait until 8.0 and remove the old structures, replacing them with records. This would force applications to adopt the new model in order to handle any 8.0 file. One of the six steering committee members asked if this meant we should consider skipping 7.1 and jumping to 8.0. (for context: prior to 7.0, a breaking change in GEDCOM was released every 2–3 years)

• One view: 8.0 needs to be years out and we shouldn't think about that yet.
• One view: 7.0 is not different enough from 5.5.1 to cause some applications to see a big win. We should add something big.
• One view: we should send a questionnaire/survey to applications we know how to contact, with an outline of the pros and cons of the options. This seemed valuable to all six of us.
• One view: add the features in 7.1, picking one of the two options above (weak preference for the "include both" instead of "pick one" option). This means we can stop recommending using particular extensions, and also pave the way for 8.0 later. Several agreed.
• One view: we should draft out what both the 7.1 and 8.0 versions would look like and then discuss. Making two changes is (7.0→7.1→8.0) is extra work to avoid if we can, so if the 7.1 solution has problems we should wait until 8.0.
• One view: we could develop 7.1 and 8.0 in parallel, supporting both the 7.x and 8.x standard concurrently for some time.

We didn't arrive at a final conclusion, but are leaning toward reaching out to application developers, and meantime working on what these would look like in 7.1 and/or 8.0, expecting that this work will be modifiable for the other if there's consensus on which we should do.

We welcome the opinions of the community here on this topic as well. In this discussion we're more interested in the process than in the specific inclusion of those three record types; there will no doubt be much back-and-forth on the nuances of those, which we will have but don't want to distract from the version progression discussion here.

[Norwegian-Sardines]

If a concept such as a new citation record or some kind of new place record structure can be included in v7.1 that does not adversely affect the use of the v7.0 concepts then having both (with the caveat that in v8.0 the old way will be depreciated) we should look into doing that!

Concentrating on v8.0 but pushing down some concepts to v7.1 that are compatible can be valuable toward Move GEDCOM forward with a way to get better compliance for v8.0 because we moved a little in that direction with v7.1.

A questionnaire would only be valuable after a well defined new approach has been well thought out and fully developed showing how much better the new concept is for genealogist and data creation. I worry that a survey to some application developers would color their opinions of potential changes toward maintaining the status quo (requiring less redesign and changes to their application) but not actually moving GEDCOM to a better place in international/multi-cultural/multi-lingual/historical centric data exchange.

I think v7.0 is an easy move for many application because they don’t really support v5.5.1 and I don’t think we removed anything that significant from v5.5.1 already used by them. It is probably more of a big deal for application that are closer to 100% v5.5.1 and may need more significant changes to maintain their close to 100% compatibility, while also excepting back release GEDCOM files.

[mother10]

This will end up in a long post again, I am sorry for that.

First a reaction on the previous 2 posts:
There is 1 thing that to me seems very important. Make sure that whatever is done cannot create a mess.
By that I mean the way people now use extensions. In an earlier post of me, I showed how many extensions there are, and I am sure I missed a lot. In GEDCOM, extensions were implemented (Version 5.3 draft), to give users the possibility to add things that were not specified in GEDCOM at a given time. But that should be defined in a future GEDCOM version.

But the whole thing exploded (kind of) and now we face an almost impossible amount of extensions, some meaning the same, under different names, others only used maybe once. Each developer has his own ideas of what should be in GEDCOM and added extensions for that, without thinking of what other programs should do with it.

Now in the previous posts in this thread, (as far as i understand) some people want to add 2 versions of a definition/structure (temporarely) in the following version, to later be removed in the version after that.
I would say: dont do that, if something has to change do it in one blow, not in 1 versions next to eachother and later remove 1.
Dont give developers (me included!!!) the freedom to choose how to do something, as (Dutch expression) "when you give them 1 finger, they grab the whole hand"
And you might create more mess.

About the views: yes make the next version different enough, add 1 or more big things.

The questionaire: Same thing, developers will look at their own program and choose accordingly, just as Norwegian-Sardines said. They will not think about whats best for GEDCOM.

Next version: Most important, make it worthwile. Whether that be 7.1 or 8.0, add enough changes, and or important changes so it becomes interesting to follow. And at the same time it will show developers the direction GEDCOM will take, and kind of warns them that if they want to stay aboard, they will have to follow.
Define a priority for what will change. And add things in the next version with the highest priority. Why? because the version after that will be further away and there will be things people want very much to be added.
Also see below.

Now my additions:

From reading up and down the GEDCOM spec, and my experiances with how GEDCOM is implemented in the 2 programs I use, and how developers react on questions about how to do things, I more and more get the feeling GEDCOM might need more strict rules. And well the sooner the better.
I searched the latest 7.0.14 for the word "recommend", and it gave me 53 times that word, so there are a lot of places in the GEDCOM specification, that leave the user/developer a choise in how to do things.
To me it sounded a bit like a law book saying "it is recommended you do not kill someone".
I am joking a bit, but maybe the very next GEDCOM, should be way more strict!
Take out a lot of "recommendings", and maybe change some "{0:1}" into "{1:1}" to force people to put in the information and make the GEDCOM more understandable.

For instance the header, that is allowed to be:

0 HEAD
1 GEDC
2 VERS 7.0.14

Now how usefull a header is this 3-line thing? But this is according to the spec am I right?

I would say this should be the mandatory minimum:

0 HEAD
1 GEDC
2 VERS 7.0.14
1 SOUR
2 NAME
2 CORP
3 ADDR
4 CITY
4 CTRY
3 EMAIL
1 DEST
1 DATE
2 TIME
1 SUBM
1 COPR
1 LANG
1 PLAC
2 FORM

There are lots of problems when there is no PLAC in the header, I am one of the many that struggle with this.
Also, if a developer creates a program, he should give minimum info about name city email and such of his program.
DEST should be there too. For Dest maybe we could add
1 DEST
2 VERS
Where VERS is same as with GEDC. That would mean: any program following that GEDCOM version, then names are not so necessary.

And why is there only 1 NOTE possible in the header?
One of the programs I use, has the possibility to have many remarks concerning the whole tree (not just 1 entity).
Do I need to put them all in 1 NOTE?
All other places where NOTEs are allowed, have the possibility to have {0:M} for the NOTE structure, why not the Header?

Further about the Header, give it a CREA and a CHAN, after all it is possible to change the PLAC of a GEDCOM and maybe other header info. So have that reflected by these 2 tags.
Programs can convert a GEDCOM into another version, give the possibility to reflect that.

So there should be more things in the next version that are mandatory in my opinion.
If you want to join the Olympics you have to obey the rules!
Same here!

There are more things but I will put that in a separate Post.

[cdhorn]

@tychonievich my personal opinion is that there has been low adoption of the 7.0 standard because it doesn't offer enough new features to make it worth while. Yes, there is some support, but from what I can tell none of the big players like Ancestry, MyHeritage or FindMyPast support it. I'm not even sure FamilySearch itself supports it yet, do they?

So I would argue you should focus on an 8.0 version of the standard that brings more to the table in the hope that gains some traction.

But I would go even further to also argue you should all step back and rethink everything.

Why? A few reasons.

First, what is possible with computers today far exceeds what was possible when Gedcom was first envisioned and designed. The amount of data a machine can handle is orders of magnitude larger. The amount of research data someone has easy access to is just immense compared to the old days when nothing was online and very little indexed. And today trees may not be the work of any one genealogist but a group of people working together in a collaborative manner online.

Second, while it has served the community well over the past 3 or 4 decades, it has long been recognized to have a number of underlying design assumptions that impose limitations on overall usefulness. Indeed I think there is an argument to be made that it has held back innovention in the field of genealogical applications as most developers and companies allow themselves to be constrained by the implicit data model it was constructed around.

So I think everyone should step back and ask with all that has been learned over the years what could be done to enable a whole new generation of applications with enhanced functionality? What would benefit the community as a whole that might serve as a good foundation and standard for the next 20 or 30 or 40 years?

Here are some ideas to think about...

• Families are just a group of people bound by biological or other relationships are they not? Why a unique top level item and not a generic top level group construct that could be used to record so much more? Not just for tracking the traditional nuclear family, but non-traditional family structures, mixed households, and perhaps the civic and social organizations people belonged to as well?

• Make places and events top level objects so they can be shared across everything. Most applications do this with places already, and some like Gramps do with events as well.

• Repositories are just a special type of place where collections of records happen to reside, are they not? If place is a top level object then why bother with a separate repository object at all?

• Add a top level object for items so the history of ownership for family heirlooms, bibles, pictures and other things can be recorded over time. Yes, some of those things may also be sources in their own right, but a hope chest or set of china passed through a family for 5 generations certainly is not a source record.

• Support hierarchies for the top level places, events, sources and groups. Support events and facts not just for people but for places, events, sources, groups and items as any of them could be the subjects of research in their own right.

• Add support for tags associated with any top level object. Note Ancestry is already using their own _MTTAG extension to export enumerated and custom tags, and they also use _MTCAT to export the tag categories.

• Thinking about that, perhaps add support for exchanging event and fact categories as well so added custom events or facts end up in the right category in a target application if it chooses to support it.

• Provide support for contributor/submitter/agent as a top level object, with support for tracking down to the smallest detail they may have added/changed/removed and when, so information from trees developed collaborately can be preserved.

• Support identifying events/facts that are preferred or are the conclusions of research work so people can choose to record all of the events/facts they find but identify the ones to be used for calculations, charts, and reports.

Think about what could be, make big changes refactoring everything if needed, and lay the groundwork for a new generation of applications.

Will it be perfect? Of course not, nothing is. People will likely disagree on many things. That is okay, I think if you get 80% of it right it could serve the community well for a long time. And indeed it probably would enable use cases outside of traditional genealogy.

Will developers and companies adopt it if you do such a thing? Perhaps, perhaps not, but I would like to think so.

Anyway, those are my thoughts on the matter and I felt I should share them.

[Norwegian-Sardines]

Thanks for your thoughts on this subject!
Here are my thoughts on what you said!
1.

“First, what is possible with computers today far exceeds what was possible when Gedcom was first envisioned and designed. The amount of data a machine can handle is orders of magnitude larger. The amount of research data someone has easy access to is just immense compared to the old days when nothing was online and very little indexed. And today trees may not be the work of any one genealogist but a group of people working together in a collaborative manner online.”

As a technologist I agree for the most part here, but we have to always remember that much of the user base of genealogy programs are using under-powered desktop computer with under-powered DBMS engines. In my community we have a large club of retired people that have older computers and small drives. Most users are limited by money and time, so their databases are generally small. On the other hand, I currently use an application that is completely online and not Ancestry/MyHeritage/Find My Past. Our database model does align with GEDCOM to some degree for speed of GEDCOM import/export, but also adds a lot of other records and relationships that are not found in GEDCOM. The goal of the product is to allow any program’s GEDCOM to load and produce 100% accurate GEDCOM output. Any changes to GEDCOM at a design level would be welcome!

“Second, while it has served the community well over the past 3 or 4 decades, it has long been recognized to have a number of underlying design assumptions that impose limitations on overall usefulness. Indeed I think there is an argument to be made that it has held back innovention in the field of genealogical applications as most developers and companies allow themselves to be constrained by the implicit data model it was constructed around.”

I agree to some level but also disagree as well! Most applications do not implicitly use the GEDCOM data model as their database model but do organize their data in ways that may follow GEDCOM. I have limited experience with the actual database model used by a large number of applications but the few I do know are of two types! 1) Use all elements of the GEDCOM data model as a starting point, but add additional record types, indexes and data elements to support their user interface and additional functions. 2) Use some of the of the elements found in the GEDCOM model, but add additional record types, indexes and data elements to support their user interface and additional functions. NOTE: most applications I’ve seen add additional elements to their database (record types, indexes, etc.) to support functions they support that GEDCOM does not or because their DB Engine needs them to speed data retrieval or data functionality. As far as innovation, I’m all in favor of innovation, what if anything do you have in mind that has not already been discussed in this forum?

“Families are just a group of people bound by biological or other relationships are they not? Why a unique top level item and not a generic top level group construct that could be used to record so much more? Not just for tracking the traditional nuclear family, but non-traditional family structures, mixed households, and perhaps the civic and social organizations people belonged to as well?”

I agree that the term “Family” should be revisited and probably renamed. I’ve seen GEDCOM (and associated software) used to document other genetic relationships such as dog and horse genetics. Family groups don’t always have the same meaning in all cultures and historically the term “Family” could mean different things. However, eliminating the term “Family” (Fam_Record) and substituting a different term may not be enough. If I was designing a database to support personal relationships between people, I would not have a “family” record but would use my background in manufacturing to create a Bill Of Materials (BOM) relationship. The biggest drawback to this is that all relationship types (i.e. Bio-Father/Bio-Mother/Adopt-Father/Adopt-Mother/GodParent/Clan Member/Sibbling/NameSake/Foster/slave/paster/doctor/neighbor/cousin/???) would need to be defined (enumerated) in the GEDCOM so that all applications, users, producers and receivers of a GEDCOM know the use case for each type of term. Cultural, international and historical uses could make this list very large and impossible to get right!

“Make places and events top level objects so they can be shared across everything. Most applications do this with places already, and some like Gramps do with events as well”

We are actually working toward this right now with “place”! Some issues with “place” as a shared record type include how to define a place in; a) historical terms, b) present day terms, c) as entered in the source, d) cultural terms, and e) language and localization values. I’ve seen some interpretations of a “shared place record” used by software programs and they are not complete and do not resolve the issues and needs noted above!

“Repositories are just a special type of place where collections of records happen to reside, are they not? If place is a top level object then why bother with a separate repository object at all?”

Yes and no! While a repository may contain “place” information, it is more than just a place where an event happened, it is used in creating a bibliography and full valid citations. It also carries information about the repository (Name, address, email, URL, contact information, etc.) that is not relevant to a place where an event occurred.

“Add a top level object for items so the history of ownership for family heirlooms, bibles, pictures and other things can be recorded over time. Yes, some of those things may also be sources in their own right, but a hope chest or set of china passed through a family for 5 generations certainly is not a source record.”

I agree with the need for “providence” information as it pertains to objects recorded and used in the GEDCOM. This is not a “Source of the Object” as some people want to suggest, the Source_Record should not be used to reflect who gave me the object, users of GEDCOM need to understand that the “Source_Record is used exclusively for documenting the “Source Artifact” not the colloquial use of the term source as in “who gave me the item”. A “providence structure” needs to be created!

“Support hierarchies for the top level places, events, sources and groups. Support events and facts not just for people but for places, events, sources, groups and items as any of them could be the subjects of research in their own right.”

“Add support for tags associated with any top level object. Note Ancestry is already using their own _MTTAG extension to export enumerated and custom tags, and they also use _MTCAT to export the tag categories.”

“Thinking about that, perhaps add support for exchanging event and fact categories as well so added custom events or facts end up in the right category in a target application if it chooses to support it.”

I’m not sure I completely understand these points, but they may be covered by the requirement for all terms and extensions not part of the GEDCOM standard to have an extTag in the HEAD record where GEDCOM transmission tell the receiving application where they can find documentation explaining the extension. (see section 1.5.1 in the GEDCOM document.) The current GEDCOM is not well “Normalized” for a shared event or place. Source Artifacts are already normalized, but what I think you are referring to are “Shared Citations” which are very different than “Shared Sources”! I’ve written a paper on Citations and developed a potential solution, but we have not yet had formal discussions regarding these concepts!

“Provide support for contributor/submitter/agent as a top level object, with support for tracking down to the smallest detail they may have added/changed/removed and when, so information from trees developed collaborately can be preserved.”

The HEAD record already supports a “Submitter” value for the entire GEDCOM transmission and is also found at the Record Level instance as well. The submitter’s information includes ways to contact them. I think more detail at the data element level is important but hard to implement. Each element in the database would need a change control attribute with before and after data or the transmitted GEDCOM could show before and after images of the entire GEDCOM but that would double its size. This is complicated by the fact that the GEDCOM transmitted may be “an original” but contain information that is different than what the receiver already has in their database. The transmitter could have no knowledge of the receiver’s data and therefore the receiver may still need to resolve changes and differences as part of the data merge. Your need for change control concept in a GEDCOM is best used for collaborative work. Collaboration may be best handled by not using GEDCOM sharing at all but by using collaborative specific software based on the internet or through networked computers and databases!

“Support identifying events/facts that are preferred or are the conclusions of research work so people can choose to record all of the events/facts they find but identify the ones to be used for calculations, charts, and reports.”

Preferred events/facts are already supported in the GEDCOM documentation with this statement:

“The collection of substructures is partially ordered. Substructures with the same structure type are in a fixed order, but substructures with different structure types may be reordered. The order of substructures of a single type indicates user preference, with the first substructure being the most-preferred value, unless a different meaning is explicitly indicated in the structure’s definition.”

In my opinion this could be done better with an indicator that would “cement” which event/fact (“fact”) was the preferred one, but it is not technically necessary! As far as an indication for “conclusive” vs “speculative” data, this need goes beyond the actual “fact” but individual parts of the “fact” which include, but not limited to, fact date, place, cause, relationship connections, name, restriction indicators, etc.

Thank you for your insight and comments.

I personally think that any changes to GEDCOM must in some way be conducive to the adoption by all genealogy software companies. The transfer protocol has limited value if companies don’t support its use and force their user base to be tied to one software program without the ability to use other software either because some other software has a better User Interface, or Better Functions, or at the very worse their current software goes “belly up” and they could possibly lose the work they have done! We have already seen with the current state of GEDCOM that many companies don’t support GEDCOM very well causing data to be lost or misinterpreted by the receiver! This latter point is the reason I became involved in GEDCOM many many years ago!

[webtrees-pesz]

Ken wrote: “I personally think that any changes to GEDCOM must in some way be conducive to the adoption by all genealogy software companies. The transfer protocol has limited value if companies don’t support its use and force their user base to be tied to one software program without the ability to use other software either because some other software has a better User Interface, or Better Functions, or at the very worse their current software goes “belly up” and they could possibly lose the work they have done! We have already seen with the current state of GEDCOM that many companies don’t support GEDCOM very well causing data to be lost or misinterpreted by the receiver! This latter point is the reason I became involved in GEDCOM many many years ago!” I can only fully agree with this statement! GEDCOM 7 was released over 3 years ago, but only a few applications support the current standard. I am not aware of any corresponding support from the large databases. Even Familysearch does not use it, as far as I know. What is the reason for this? In my opinion, one of the reasons is that the implementation of the standard on the market is not checked and ensured by the issuing organization. The providers of many applications market their software as GEDCOM-compatible, but often do not adhere to the standard. A seal of approval or a certificate could stem this uncontrolled growth and promote the spread of GEDCOM 7. At the time of GEDCOM 5.5, it was possible to register genealogy programs with corresponding certification. In my opinion, this is a good instrument for all parties involved: For Familysearch to ensure and promote the standard, for genealogy software companies an advertising argument, for users a criterion for loss-free data exchange. Best regards and all the best for 2025 Peter

[cdhorn]

@tychonievich regarding shared events, I have started looking at some of the more popular desktop genealogy programs other than Gramps and see RootsMagic, Legacy Family Tree, Ancestral Quest and Heredis all appear to support them as well. So this is clearly an area where the standard should be enhanced to properly support them.

I see Legacy Family Tree also supports tags, referring to them as hashtags, and Heredis does referring to them as flags. So this too is a feature more programs have or are adopting that the standard should be enhanced to properly support.

[tychonievich]

Thanks for this note. I understand that they are commonly exported using UID (v7) or _UID (v5.5.1) now, and agree that making a more straightforward implementation is appropriate.

[Norwegian-Sardines]

@cdhorn Yes, shared events (and attributes too) would be a great addition. GEDCOM has never been fully “normalized” and this is one of the areas that need this feature.

What exactly do hashtags do in Legacy? Why are they valuable and a feature to add to GEDCOM?

[cdhorn]

@Norwegian-Sardines I don't have an installed copy of Legacy but based on glance at documentation they appear to be the same as "tags" in Ancestry.com and Gramps and "flags" in Heredis. They're just a means of tagging particular objects/data to group them for reporting or other purposes. It seems from docs that RootsMagic has a "groups" construct they use for that purpose as well.

[Norwegian-Sardines]

Thanks, I’m still not sure what “tagging” does, or for that matter color coding that some software talks about. Some of these concepts could be out of the scope of a genealogy “data transfer” protocol and should be more application specific and only maintained within that specific application!