Cheatsheet

Cheatsheet

Name	Usage	Description
SWAP	swap <column1> <column2>	Swaps the column names of two columns.
ENCODE	encode <base32	base64
XPATH	xpath <column> <destination> <xpath>	Extract a single XML element or attribute using XPath.
GENERATE-UUID	generate-uuid <column>	Populates a column with a universally unique identifier (UUID) of the record.
LOWERCASE	lowercase <column>	Changes the column values to lowercase.
WRITE-AS-CSV	write-as-csv <column>	Writes the records files as well-formatted CSV
PARSE-AS-PROTOBUF	parse-as-protobuf <column> <schema-id> <record-name> [version]	Parses column as protobuf encoded memory representations.
HASH	hash <column> <algorithm> [<encode=true	false>]
JSON-PATH	json-path <source> <destination> <json-path-expression>	Parses JSON elements using a DSL (a JSON path expression).
MASK-NUMBER	mask-number <column> <pattern>	Masks a column value using the specified masking pattern.
TEXT-DISTANCE	text-distance <method> <column1> <column2> <destination>	Calculates a text distance measure between two columns containing string.
PARSE-XML-TO-JSON	parse-xml-to-json <column> [<depth>]	Parses a XML document to JSON representation.
PARSE-AS-HL7	parse-as-hl7 <column> [<depth>]	Parses <column> for Health Level 7 Version 2 (HL7 V2) messages; <depth> indicates at which point JSON object enumeration terminates.
FIND-AND-REPLACE	find-and-replace <column> <sed-expression>	Finds and replaces text in column values using a sed-format expression.
RENAME	rename <old> <new>	Renames an existing column.
PARSE-AS-AVRO	parse-as-avro <column> <schema-id> <json	binary> [version]
FILL-NULL-OR-EMPTY	fill-null-or-empty <column> <fixed-value>	Fills a value of a column with a fixed value if it is either null or empty.
SET-TYPE	set-type <column> <type>	Converting data type of a column.
RTRIM	rtrim <column>	Trimming whitespace from right side of a string.
INVOKE-HTTP	invoke-http <url> <column>[,<column>] <header>[,<header>]	[EXPERIMENTAL] Invokes an HTTP endpoint, passing columns as a JSON map (potentially slow).
COLUMNS-REPLACE	columns-replace <sed-expression>	Modifies column names in bulk using a sed-format expression.
SEND-TO-ERROR	send-to-error <condition>	Send records that match condition to the error collector.
SET-RECORD-DELIM	set-record-delim <column> <delimiter> [<limit>]	Sets the record delimiter.
SET-VARIABLE	set-variable <variable> <expression>	Sets the value for a transient variable for the record being processed.
SET-CHARSET	set-charset <column> <charset>	Sets the character set decoding to UTF-8.
WRITE-AS-JSON-OBJECT	write-as-json-object <dest-column> [<src-column>[,<src-column>]	Creates a JSON object based on source columns specified. JSON object is written into dest-column.
KEEP	keep <column>[,<column>*]	Keeps the specified columns and drops all others.
CUT-CHARACTER	cut-character <source> <destination> <type> <range	indexes>
SPLIT-TO-ROWS	split-to-rows <column> <separator>	Splits a column into multiple rows, copies the rest of the columns.
XPATH-ARRAY	xpath-array <column> <destination> <xpath>	Extract XML element or attributes as JSON array using XPath.
FAIL	fail <condition>	Fails when the condition is evaluated to true.
INCREMENT-VARIABLE	increment-variable <variable> <value> <expression>	Wrangler - A interactive tool for data cleansing and transformation.
PARSE-AS-XML	parse-as-xml <column>	Parses a column as XML.
PARSE-AS-FIXED-LENGTH	parse-as-fixed-length <column> <width>[,<width>*] [<padding-character>]	Parses fixed-length records using the specified widths and padding-character.
CHANGE-COLUMN-CASE	change-column-case lower	upper
SPLIT-EMAIL	split-email <column>	Split a email into account and domain.
URL-ENCODE	url-encode <column>	URL encode a column value.
WRITE-AS-JSON-MAP	write-as-json-map <column>	Writes all record columns as JSON map.
MASK-SHUFFLE	mask-shuffle <column>	Masks a column value by shuffling characters while maintaining the same length.
DROP	drop <column>[,<column>*]	Drop one or more columns.
DECODE	decode <base32	base64
SPLIT	split <source> <delimiter> <new-column-1> <new-column-2>	[DEPRECATED] Use 'split-to-columns' or 'split-to-rows'.
PARSE-AS-SIMPLE-DATE	parse-as-simple-date <column> <format>	Parses a column as date using format.
DIFF-DATE	diff-date <column1> <column2> <destination>	Calculates the difference in milliseconds between two Date objects.Positive if <column2> earlier. Must use 'parse-as-date' or 'parse-as-simple-date' first.
INDEXSPLIT	indexsplit <source> <start> <end> <destination>	[DEPRECATED] Use the 'split-to-columns' or 'parse-as-fixed-length' directives instead.
PARSE-AS-AVRO-FILE	parse-as-avro-file <column>	parse-as-avro-file <column>.
FILTER-ROW-IF-TRUE	filter-row-if-true <condition>	[DEPRECATED] Filters rows if condition is evaluated to true. Use 'filter-rows-on' instead.
SPLIT-URL	split-url <column>	Split a url into it's components host,protocol,port,etc.
FORMAT-DATE	format-date <column> <format>	Formats a column using a date-time format. Use 'parse-as-date` beforehand.
QUANTIZE	quantize <source> <destination> <[range1:range2)=value>,[<range1:range2=value>]*	Quanitize the range of numbers into label values.
PARSE-AS-EXCEL	parse-as-excel <column> [<sheet number	sheet name>]
PARSE-AS-DATE	parse-as-date <column> [<timezone>]	Parses column values as dates using natural language processing and automatically identifying the format (expensive in terms of time consumed).
TABLE-LOOKUP	table-lookup <column> <table>	Uses the given column as a key to perform a lookup into the specified table.
FILTER-ROWS-ON	filter-rows-on empty-or-null-columns <column>[,<column>*]	Filters row that have empty or null columns.
TRIM	trim <column>	Trimming whitespace from both sides of a string.
URL-DECODE	url-decode <column>	URL decode a column value.
FLATTEN	flatten <column>[,<column>*]	Separates array elements of one or more columns into indvidual records, copying the other columns.
UPPERCASE	uppercase <column>	Changes the column values to uppercase.
CATALOG-LOOKUP	catalog-lookup <catalog> <column>	Looks-up values from pre-loaded (static) catalogs.
PARSE-AS-LOG	parse-as-log <column> <format>	Parses Apache HTTPD and NGINX logs.
LTRIM	ltrim <column>	Trimming whitespace from left side of a string.
EXTRACT-REGEX-GROUPS	extract-regex-groups <column> <regex-with-groups>	Extracts data from a regex group into its own column.
PARSE-AS-CSV	parse-as-csv <column> <delimiter> [<header=true	false>]
FILTER-ROW-IF-MATCHED	filter-row-if-matched <column> <regex>	[DEPRECATED] Filters rows if the regex is matched. Use 'filter-rows-on' instead.
PARSE-AS-JSON	parse-as-json <column> [<depth>]	Parses a column as JSON.
SET COLUMN	set column <column> <jexl-expression>	Sets a column by evaluating a JEXL expression.
STEMMING	stemming <column>	Apply Porter Stemming on the column value.
COPY	copy <source> <destination> [<force=true	false>]
SET-COLUMN	set-column <column> <expression>	Sets a column the result of expression execution.
SPLIT-TO-COLUMNS	split-to-columns <column> <regex>	Splits a column into one or more columns around matches of the specified regular expression.
CLEANSE-COLUMN-NAME	cleanse-column-names	Sanatizes column names: trims, lowercases, and replaces all but [A-Z][a-z][0-9].with an underscore ''.
SET COLUMNS	set columns <columm>[,<column>*]	Sets the name of columns, in the order they are specified.
TITLECASE	titlecase <column>	Changes the column values to title case.
MERGE	merge <column1> <column2> <new-column> <separator>	Merges values from two columns using a separator into a new column.
TEXT-METRIC	text-metric <method> <column1> <column2> <destination>	Calculates the metric for comparing two string values.
SET FORMAT	set format csv <delimiter> <skip empty lines>	[DEPRECATED] Parses the predefined column as CSV. Use 'parse-as-csv' instead.
FORMAT-UNIX-TIMESTAMP	format-unix-timestamp <column> <format>	Formats a UNIX timestamp using the specified format
FILTER-ROW-IF-NOT-MATCHED	filter-row-if-not-matched <column> <regex>	Filters rows if the regex does not match
FILTER-ROW-IF-FALSE	filter-row-if-false <condition>	Filters rows if the condition evaluates to false