Grid data export API

[mshabarov]

Description

A new Java API for easier export of the Vaadin Grid content.

Data export is a quite popular use case especially for table-like data components: Grid, GridPro, Crud, TreeGrid and various add-on based on them. Vaadin Directory has several solutions.

When exporting Grid contents, one may need:

• the rows data, i.e. actual Grid content;
• static parts - headers and footers, empty state text, column settings, e.g. columns ordering;
• Grid customisations, including end user's customisations - selection, styling, sorting and filtering if a report requires other values that are rendered on the web page;
• some extra formatting or conversion that is not rendered on web page, but needed for the report, e.g. erasing sensitive data in the report, modifying some cells values.

All of these information is usually extracted through the different APIs, which is quite a long list: DataView, DataProvider, DataCommunicator, Renderer, ValueProvider, Element, which makes data export be a challenge for beginner users and even tricky for advanced users, because it requires a good knowledge of the listed classes and sometimes even hacky codes using reflection. Thus, a single facade interface would make sense that hides the complexity and could extract the necessary data for exporting needs.

This issue focuses on the way of getting necessary data from Grid, whereas the exact data exporting formats and downloading mechanism are out of scope of this. However, the export API should be optimised for most common cases (PDF, CSV, JSON/XML, Excel ?), but make it possible to use it for other formats, see for example Vaadin Grid Exporter add-on

Use cases

As a Vaadin application developer
I want an easy way to get the data from Grid through a single API, i.e. a single method or class
So that I can export the data rows, static information and user settings without a preliminary knowledge of Grid internals and various kinds of API.

[1] Use case example: I want to export the Grid data in a format that needs only pure data without formatting (e.g. CSV or JSON/XML), I don't need any data conversion nor any customisations, I have no expertise nor I want to dig into Render or DataProvider specifics, just print what I see on my web page as a plain text array.
[2] Advanced use case: I want to export with the actual renderings and formatting (or with some partial formatting), e.g. to the PDF or styled Excel format. Also I may want a restricted version of my Grid, i.e. less columns and only selected rows, with keeping as much styles as possible and filling custom rendered columns with component images, so they look same beauty as on the web page.
[3] Clone case: I want to clone my Grid entirely or partially, e.g. if I want to have original Grid on the left and same styled Grid, but with no data, to which I can drag and drop items from the original grid.

Acceptance criteria

0. Grid exporter gives all necessary extracted data that may be needed for report, but it doesn't help with generating exact type of report. Instead, it's a project code that choose the appropriate way of export. However, Vaadin provides 1-2 most used production-ready examples (e.g. CSV, Excel) that one can copy-paste and use.
1. Data to be exported:
   • Grid Content: All rows and columns, respecting current filters and sorting.
   • Handle Custom Renderers: whenever possible, default renderer handlers are applied, e.g. Button->getText(), Icon->getAttribute("icon"), CheckBox->"Yes"/"No", and additionally the API provides a callback that takes an item or component and returns an extracted plain text, then the report maker can extract a plain text from it and style it in it's own way, or use some html-to-image tools , e.g. this or that.
   • Data Conversion: Apply specific data conversion, e.g. editing/deleting cell values.
2. Static data:
   • Headers and footers, including multi-header case.
   • Styling: border, background and text colors, width, spaces and other styles that are important in the report document.
3. Customisation Options:
   • Selected Rows: ability to export only user-selected rows.
   • Columns: Option to filter the columns or manually exclude some columns, e.g. ID, buttons.
4. Grid data exporter takes the following inputs:
   • whether to export all rendered or only selected rows
   • columns to be excluded
   • custom renderer handlers
   • custom column values converters
   • custom filtering and sorting (?)
5. Grid data exporter gives the following outputs:
   • a sorted and filtered collection of rows that has either plain text or report-specific generic type
   • headers and footers
   • empty state text
   • styles
   • columns order
6. Clone of the entire Grid, including data and settings parts.

API example

GridExporter<Contact> exporter = grid.createExporter(optionalTitle)
        .withSelectedRows() // all visible rows by default
        .withExcludedColumns(grid.getColumnByKey("id")) // all columns by default
        .withExcludedFooters() // footers are added by default
        .withColumnExtractor(grid.getColumnByKey("company"),
                isCompany -> isCompany ? "Company" : "Person")
        // .withColumnExtractor(grid.getColumnByKey("company"),
        //      component -> component.getValue() ?
        //          new Image("/images/company.png) :
        //          new Image("/images/person.png))
        .withColumnConverter(grid.getColumnByKey("cardNumber"),
                cardNumber -> cardNumber == null ?
                        "N/A" : maskCardNumber(cardNumber))
        .export();

List<Row> rows = exporter.getRows();
for (Row row : rows) {
    String cellAsText = row.getAsText();
    Optional<Image> cellAsImage = row.getAsObject();
}

// not clear what exactly should be collected here
// perhaps some shortcuts for most used styles of Grid, like
// grid.getStyle().get("border");
GridStyles gridStyles = exporter.getStyles();

// These are directly available in Grid methods,
// but can be added to the exporter for consistency:
List<HeaderRow> headerRows = exporter.getHeaderRows();
List<FooterRow> footerRows = exporter.getFooterRows();
String text = exporter.getEmptyStateText();
List<Grid.Column<Contact>> columns = exporter.getColumns();

General criteria

• APIs reviewed

• Design

• Performance

• UX/DX tests in Alpha

• Documentation:

• How to test?

• Limitations:

Security

• Grid data export API should not allow to export more data than the current user is allowed to see on the web page.

[mstahv]

Exporting a Grid to CSV is indeed quite tricky at the moment—good effort! 👍

That said, I'm not convinced that introducing new APIs and interfaces is the right solution here. Instead, I believe we should focus on improving and completing the existing ones. To be honest, there are already too many "custom Vaadin concepts" around Grid and data binding, and some implementation details are leaking through, leading to confusion among users.

I think exporting Grid content should be achievable using a basic, generic API that would likely be useful for other cases as well.

One long-standing challenge has been the lack of getRows()/getItems(). We've had get(Generic)DataView for a while now, which somewhat addresses this, but it's not very discoverable. If users call setItems(), they naturally look for something related to "items" when trying to retrieve them.

Header and footer rows, on the other hand, are fairly easy to discover.

A bigger issue in the API is extracting what is actually rendered in data cells. The current Column interface seems like the right place for this, but it provides no real functionality. The Renderer concept exists, but again, it’s another custom abstraction that isn't particularly helpful (to this direction). Here’s an example of how I managed to make use of it 🤮:

https://github.com/mstahv/wmoc-result-helper/blob/main/src/main/java/org/orienteering/wmoc/resulthelperui/SymmetricDifferenceView.java#L136-L189

Exposing a more discoverable and easy-to-use API to access generated cell content could largely solve the problem that users face. It doesn’t need to be perfect—nothing ever is—but simply having a way to retrieve a raw HTML/text representation or a component of the cell could be sufficient.

Intuitively I'd start to investigate adding something like this to Column:

CellType getCellType(); // CellType: TEXT, HTML, COMPONENT
public Object getCellContent(T);

[TatuLund]

Use case is rather common in business applications, but implementation details differ a lot.

• Export all data vs. filtered data
• Export data in visible sorting vs. original order
• Export all column vs. currently visible columns
• Export columns in the visible order vs. in the order of the data model

Component renderer is a challenge as it can generate content that does not fit certain file formats. Possibly component renderer should have an API to produce text alternative. This text could be used also for accessibility purposes and if we have getter for that text, it could be used in data export.