Releases · wader/fq

21 Sep 15:19

wader

v0.13.0

9857323

v0.13.0 Latest

Latest

Changes

New format decoders midi, negentropy, tap and txz, see below for details.

Add byte_array bits format. #992

# by default binary values are UTF-8 strings
$ fq -V -n '[1,2,3] | tobytes'
# with -o bits_format=byte_array they are byte arrays instead
"\u0001\u0002\u0003"
$ fq -cV -n -o bits_format=byte_array '[1,2,3] | tobytes'
[1,2,3]

# data is a string
$ fq '.program_headers[] | select(.type=="interp")' /bin/ls
{"align":1,"data":"/lib/ld-musl-x86_64.so.1\u0000", ...}
# data is a byte array
$ fq -cV -o bits_format=byte_array '.program_headers[] | select(.type=="interp")' /bin/ls
{"align":1,"data":[47,108,105,98,47,108,100,45,109,117,115,108,45,120,56,54,95,54,52,46,115,111,46,49,0], ...}

Go 1.22 or later is now required to build. This is to keep up with build requirements of some dependencies. #1011
gojq changes from upstream: #1000
- Implement add/1 function
- Improve time functions to accept fewer element arrays
- Fix uri and urid formats not to convert a space between a plus sign

Format changes

matroska Updated to latest specification. #1010
midi MIDI decoder added by @twystd. #1004
negentropy Negentropy message decoder added by @fiatjaf. #1012
tap and txz TAP and TXZ tape format for ZX Spectrum computers added by @mrcook. #975

Changelog

64b0523 Add PBS tidbit 8 episode
3105b7d Added to probe group
d73c663 Update docker-golang to 1.22.6 from 1.22.5
790aa8a Update docker-golang to 1.23.0 from 1.22.6
ea3294f Update docker-golang to 1.23.1 from 1.23.0
34b2cc3 Update github-go-version to 1.22.6 from 1.22.5
8bb841f Update github-go-version to 1.23.0 from 1.22.6
2c38cf1 Update github-golangci-lint to 1.60.1 from 1.59.1
f9dff5c Update github-golangci-lint to 1.60.2 from 1.60.1
ccf78c7 Update github-golangci-lint to 1.60.3 from 1.60.2
23df925 Update gomod-creasty-defaults to 1.8.0 from 1.7.0
9e29682 Update gomod-ergochat-readline to 0.1.3 from 0.1.2
01b292d Update gomod-golang-x-crypto to 0.26.0 from 0.25.0
da6dd9e Update gomod-golang-x-net to 0.28.0 from 0.27.0
de3fa19 Update make-golangci-lint to 1.60.1 from 1.59.1
f6d0279 Update make-golangci-lint to 1.60.2 from 1.60.1
e01cff0 Update make-golangci-lint to 1.60.3 from 1.60.2
4299883 Update make-golangci-lint to 1.61.0 from 1.60.3
ee404b2 all: re-run the test --update
a241774 all: update tests for tap support
2c0b528 ci: Add CGO_ENABLED=0 to make sure we dont need it
550bcc2 cli: Add go version to version string
a195b3f doc: Regenerate
be9a916 doc: Update formats.svg
db0dfb1 doc: Update formats.svg
66345fe doc: regenerat formats.md for tzx/tap updates
f16f6dc fq: Update golang.org/x/exp and github.com/gomarkdown/markdown
7b6d853 go,lint: Update to go 1.22 and fix some lint warnings
3626787 gojq: Update fq fork
824e51e gojq: Update fq fork
fc74f68 intepr: Revert binary value each
3f3183a intepr: Support each for binary values
7bf1129 interp: Add bits_format=byte_array support
52e6e1b matroska: Update spec
fdde568 midi: (partly) fixed SMPTEOffsetEvent gap
a1385ca midi: Decoded EndOfTrack and ProgramChange events
3276641 midi: Decoded KeySignature, NoteOn and Controller events
109719f midi: Decoded MIDIChannelPrefix, MIDIPort and SequencerSpecific metaevents
cbd4a8a midi: Decoded NoteOff event
0226fc6 midi: Decoded PolyphonicPressure, ChannelPressure and PitchBend MIDI events
b2e71a3 midi: accomodated MIDI event running status
0915f75 midi: added 'data' field to EndOfTrack event
473394b midi: added event type to events to simplify query by event
91fa547 midi: added example queries to the test data
ce02d6e midi: added length field to TimeSignature struct
b5f2bda midi: added localised Makefile (cf. #1004 (comment))
7950dd6 midi: added midi to the TestFormats all.fqtest list (cf. #1004 (comment))
80b9343 midi: added new test and MIDI files
57adef4 midi: added sample queries to midi.md
0e0a669 midi: added tests for format 0, format 1 and format 2 MIDI files
e99d9f6 midi: asserted bytes left for MIDI events
d7ec38a midi: basic test set
a41e123 midi: cleaned up SysExContinuation and SysExEscape logic
a7d0cde midi: cleaned up and simplied event decoding logic
8995496 midi: cleaned up event decoding logic
0b4be89 midi: cleaned up unknown chunk logic
3809ddb midi: combined metaevent status + event bytes
b433998 midi: decoded SysEx events
8b236a1 midi: decoded TimeSignature meta event
f11dfaf midi: decoded TrackName meta event
9099a3e midi: decoded chunk tags as FieldUTF8
4fe27f5 midi: decoded remaining text metaevents
1bdeeb6 midi: decoded tempo metaevent
59b1faa midi: decorated 'delta' field with running tick
4fac4c6 midi: discarded unknown chunks
9dfcb96 midi: experimenting with struct metaevent data
2a65036 midi: fixed fuzzing errors
f1b888b midi: fixed gap in SequencerSpecificEvent
4bb3292 midi: fixed key signature map
f424936 midi: fixed lint warning (cf. #1004 (comment))
2f2070e midi: fixed lint warnings
87c80f5 midi: fixes for PR comments
9f057b6 midi: fixes for PR comments
9c7f7f9 midi: fixes for PR comments:
ad0a06d midi: initial import
2970fb1 midi: lowercased event names
3ed9889 midi: mapped SMPTE framerates
54a0cf1 midi: mapped SysEx event types
34fca40 midi: mapped manufacturer IDs to strings
0ef3304 midi: mapped note numbers to note names
763d142 midi: moved 'dev only' Makefile to workdir and removed it from .gitignore (cf. #1004 (comment))
5c89d7d midi: moved MIDI running status and Casio flag to context struct
c5637f0 midi: partly fixed gaps in SysExMessage
7fd9ad2 midi: removed debug leftovers (cf. #1004 (comment))
462ae15 midi: removed superfluous AssertLeastBytesLeft (cf. #1004 (comment))
17bac77 midi: removed superfluous uint64 cast (cf. #1004 (comment))
dad4a91 midi: replace d.BytesLen(1) with d.U8()
578b7e7 midi: restructured event decoding to decode length and struct fields (cf. #1004 (comment))
befdf1f midi: reworked SysEx events as struct with length field
ea3e089 midi: reworked decoding to expect an MThd header as the first chunk (cf. #1004 (comment))
e940f47 midi: reworked metaevent decoding for PR comments
a337ff0 midi: reworked sysex and metaevent 'struct' events to decode as byte arrays
cee51ec midi: reworked to include delta in event
a3a0a06 midi: simplifed and cleaned up MIDI 'fq' tests:
c8d9397 midi: snake-cased event types and event names (cf. #1004 (comment))
3966d5b midi: tightened up status/event decoding logic (cf. #1004 (comment))
2350afe midi: updated help text to use snake-case event names (cf. #1004 (comment))
49bd7c2 negentropy: add format.
e3c7b1a negentropy: another test from a different source, another doc entry, comments on source of test samples.
3853387 negentropy: const modes, RFC3339 dates and synthetic timestamps.
8f9a005 negentropy: timestamp description as UTC.
113ca63 tap: add DefaultInArg
e526caf tap: add README to tests directory
90de619 tap: add a read_one_block test
e3c3d92 tap: add author to tap.md
633160b tap: handle unoffical block types and minor improvements
e5be55c tap: remove Probe group
7816fe1 tap: remove format.TAP_In and update tzx
e345528 tap: rename block_type to type
d384930 tap: revert using an array of bytes
c93dc8b tap: verify checksum values
81b2304 tzx: Add suport for ZX Spectrum TZX and TAP files
6ac69e2 tzx: add README to tests directory
88b6ded tzx: add author to tzx.md
b27017c tzx: add readme instructions for bits_format=byte_array
fc350f3 tzx: change data fields to array of bytes for easier JSON parsing
d3a5b8d tzx: revert using an array of bytes
ffb5eb3 tzx: use jq-ish values for the mapped symbols in block type 0x18

Contributors

mrcook, fiatjaf, and twystd

Assets 9

15 Jul 09:57

wader

v0.12.0

cdba38d

v0.12.0

REPL word navigation fix and jpeg DHT decoding, otherwise mostly maintenance release to update dependencies.

Changes

Update readline package to fix issue with left/right word jump in REPL. Thanks @slingamn. #969
Update of version of golang and other dependencies.

Format changes

jpeg
- Decode DHT paramaters. Thanks @matmat. #934
- Fix EOI description. #932

Changelog

94cfbc6 Update docker-golang to 1.22.3 from 1.22.2
ab09d3c Update docker-golang to 1.22.4 from 1.22.3
c6dd0ed Update docker-golang to 1.22.5 from 1.22.4
9ff7da1 Update github-go-version to 1.22.3 from 1.22.2
1ff5a3f Update github-go-version to 1.22.4 from 1.22.3
e33c6c6 Update github-go-version to 1.22.5 from 1.22.4
a5de74c Update github-golangci-lint to 1.58.0 from 1.57.2
a59ba2a Update github-golangci-lint to 1.58.1 from 1.58.0
f6d7235 Update github-golangci-lint to 1.58.2 from 1.58.1
44e2385 Update github-golangci-lint to 1.59.0 from 1.58.2
6383626 Update github-golangci-lint to 1.59.1 from 1.59.0
61f81fb Update gomod-BurntSushi/toml to 1.4.0 from 1.3.2
12f3320 Update gomod-ergochat-readline to 0.1.1 from 0.1.0
6b1cc87 Update gomod-ergochat-readline to 0.1.2 from 0.1.1
14ada50 Update gomod-golang-x-crypto to 0.23.0 from 0.22.0
f7cbf84 Update gomod-golang-x-crypto to 0.24.0 from 0.23.0
384e4c2 Update gomod-golang-x-crypto to 0.25.0 from 0.24.0
cabb67e Update gomod-golang-x-net to 0.25.0 from 0.24.0
c55e106 Update gomod-golang-x-net to 0.26.0 from 0.25.0
e625fcb Update gomod-golang-x-net to 0.27.0 from 0.26.0
586cf14 Update gomod-golang-x-term to 0.20.0 from 0.19.0
7566fd9 Update gomod-golang-x-term to 0.21.0 from 0.20.0
41ff984 Update gomod-golang-x-term to 0.22.0 from 0.21.0
42730d7 Update gomod-golang/text to 0.15.0 from 0.14.0
8bc1a20 Update gomod-golang/text to 0.16.0 from 0.15.0
3a683b6 Update make-golangci-lint to 1.58.0 from 1.57.2
7714fcf Update make-golangci-lint to 1.58.1 from 1.58.0
aef47df Update make-golangci-lint to 1.58.2 from 1.58.1
0cd90ce Update make-golangci-lint to 1.59.0 from 1.58.2
7147674 Update make-golangci-lint to 1.59.1 from 1.59.0
6db6a54 build,test: Ignore some files to make ./... work
175661d doc: Add kodsnack 585 - Polymorfisk JSON
ebf063d doc: Cleanup and improve texts a bit
b818923 doc: Fix function indent
6f2b599 doc: Include format description per format
40f38a5 doc: Reorganize and cleanup function descriptions
ad2c032 goreleaser: Update action and fix deprecation warning
6e13b4b jpeg: Add parsing of DHT parameters
b482556 jpeg: Fix EOI description

Contributors

slingamn and matmat

Assets 9

14 Apr 10:55

wader

v0.11.0

d30755b

v0.11.0

New iNES/NES 2.0 ROM decoder (thanks @mlofjard) and basic JPEG 2000 format support. jq language improvements and fixes from gojq. And as always various decoder improvements and fixes.

Changes

Add string_truncate option to configure how to truncate long strings when displaying a decode value tree. dd, dv etc set truncate length to zero to not truncate. #919
gojq updates from upstream:
- Implement ltrim, rtrim, and trim functions
- Fix object construction with duplicate keys ({x:0,y:1} | {a:.x,a:.y})
- Fix halt and halt_error functions to stop the command execution immediately
- Fix variable scope of binding syntax ("a" as $v | def f: $v; "b" as $v | f)
- Fix ltrimstr and rtrimstr functions to emit error on non-string input
- Fix nearbyint and rint functions to round ties to even
- Improve parser to allow reduce, foreach, if, try-catch syntax as object values
- Remove pow10 in favor of exp10, define scalbn and scalbln by ldexp
Fix issue using decode value with ltrimstr/rtrimstr.

Format changes

fit
- Skip array fields on pre read messages. #878
- Fixed subfield referencing fields below self in message. #877
jp2c New JPEG 2000 codestream decoder. #928
icc_profile Strip whitespace in header and tag strings. #912
mp4
- Add jp2c, jp2h, ihdr jP JPEG 2000 related boxes support. #928
- Add thmb box support. #897
- Turns out for qt brand hdlr component name might be zero bytes. #896
nes New iNES/NES 2.0 ROM decoder (thanks @mlofjard). #893

Changelog

f7b067b Fixed subfield referencing fields below self in message
9aa99b4 Update docker-golang to 1.22.1 from 1.22.0
0afb5b5 Update docker-golang to 1.22.2 from 1.22.1
2657988 Update github-go-version to 1.22.1 from 1.22.0
33c9391 Update github-go-version to 1.22.2 from 1.22.1
a577c39 Update github-golangci-lint to 1.56.2 from 1.56.1
82d96cf Update github-golangci-lint to 1.57.0 from 1.56.2
72b4569 Update github-golangci-lint to 1.57.1 from 1.57.0
14aeab0 Update github-golangci-lint to 1.57.2 from 1.57.1
735256b Update gomod-golang-x-crypto to 0.20.0 from 0.19.0
043f067 Update gomod-golang-x-crypto to 0.21.0 from 0.20.0
15a7060 Update gomod-golang-x-crypto to 0.22.0 from 0.21.0
85f60df Update gomod-golang-x-net to 0.22.0 from 0.21.0
77c000e Update gomod-golang-x-net to 0.23.0 from 0.22.0
daba6b5 Update gomod-golang-x-net to 0.24.0 from 0.23.0
ba9ecb5 Update gomod-golang-x-term to 0.18.0 from 0.17.0
b2aa59f Update gomod-golang-x-term to 0.19.0 from 0.18.0
1c24f64 Update make-golangci-lint to 1.56.2 from 1.56.1
94e8086 Update make-golangci-lint to 1.57.0 from 1.56.2
4f55b6a Update make-golangci-lint to 1.57.1 from 1.57.0
a3b63b1 Update make-golangci-lint to 1.57.2 from 1.57.1
208b3e6 chore: fix function name in comment
621d7f2 decode: Align some heavily used structs to save space
ee2ee24 decode: Cleanup io panic a bit
e741ca7 doc: Add ImHex to related tools
36e8287 doc: Regenerate after nes and new ansisvg
225fd50 fit: Skip array fields on pre read messages
7500a8b fq: Sort formats
bf7fa07 fq: Use go 1.20 and cleanup
e267040 gojq: Update fq fork
f5fd587 gojq: Update fq fork
ed68511 icc_profile: Strip whitespace in header and tag strings
c8f9cdc interp: Add string_truncate option
0db671f interp: Add todo test for eval error in path
ebffb3b jp2c: Add jpeg2000 codestream format
79992b3 jp2c: Fail probe faster
63f7d79 jp2c: Support probe
b542ff1 lint: More linters and some fixes
c6165c0 mod: go get non-bump tracked modules
1784c43 mp4,avi: Trim spaces for type
2ea70c4 mp4: Add thmb box support
4f90a2e mp4: Decode jP box
70b1b0d mp4: Decode uinf box
87b6c4d mp4: Fix jp2 test
8009b6f mp4: JPEG200 boxes jp2h and ihdr
ed3a126 mp4: Turns out for qt brand hdlr component name might be zero bytes
f3b5404 nes: Add support for iNES/NES 2.0 ROM files
80bccc9 opus,vorbis: More sym snake_case
08df7f4 pkg/cli/test_exp.sh: use command -v
8705273 pssh_playready: Use snake_case sym values
aaa43df test: Replace pmezard/go-difflib with go's internal diff

Contributors

mlofjard

Assets 9

11 Feb 16:17

wader

v0.10.0

8ebe5da

v0.10.0

Adds support for various LevelDB formats (thanks @mikez) and Garmin Flexible and Interoperable Data Transfer format (FIT) (thanks @mlofjard).

And as usual some small fixes and dependency updates.

Changes

On macOS fq now reads init script from ~/.config/fq in addition to ~/Library/Application Support/fq. #871
Switch readline module from own fork to https://github.com/ergochat/readline #854
Updated gojq fork. Notable changes from upstream below. #844
- Fix pre-defined variables to be available in initial modules
- Fix object construction with duplicate keys

Format changes

aac_frame Decode instance tag and common window flag. #859

fit Add support for Garmin Flexible and Interoperable Data Transfer decoder. Thanks @mlofjard #874

This format is used by various GPS tracking devices to record location, speed etc.
Example of converting position information to KML:

# to_kml.jq
# convert locations in a fit structure to KML
def to_kml:
  ( [ .data_records[].data_message as {position_lat: $lat, position_long: $long}
    | select($lat and $long)
    | [$long, $lat, 0]
    | join(",")
    ]
  | join(". ")
  | { kml: {"@xmlns":"http://earth.google.com/kml/2.0"
    , Document: {Placemark: {LineString: {coordinates: .}}}}
    }
  | to_xml({indent:2})
  );

Usage:

# -L to add current directory to library path
# -r for raw string output
# 'include "to_ml";' to include to_kml.jq
# to_kml calls to_kml function
$ fq -L . -r 'include "to_kml"; to_kml' file.fit > file.kml

hevc_sps Fix some incorrect profile_tier_level decoding. #829
html Fix issue parsing elements including SOLIDUS "/". #870
- Upstream issue golang/go#63402
mpeg_es Support ES_ID_Inc and decode descriptors for IOD tags
leveldb_descriptor, leveldb_log, leveldb_table Add support for LevelDB. Thanks @mikez #824
- This format is used by many database backends and applications like Google chrome.
pcapng Decode all section headers instead of just the first. #860
png Fix incorrect decoding of type flags. #847
hevc_sps Fix incorrect decoding of profile_tier_level. #829
tls Fix field name typos. #839
mp4
- Don't try decode samples for a track that has an external reference. #834
- Use box structure instead of track id to keep track for sample table data. #833
- ctts box v0 sample offset seems to be signed in practice but not in spec. #832
webp Decode width, height and flags for lossless WebP. #857

Changelog

6c3914a Add DFDL acronym to README.md
43c7693 Update docker-golang to 1.21.5 from 1.21.4
836af78 Update docker-golang to 1.21.6 from 1.21.5
f9e1e75 Update docker-golang to 1.22.0 from 1.21.6
54da5ef Update ergochat-readline to 81f0f2329ad3 from cca60bf24c9f
bed912c Update github-go-version to 1.21.5 from 1.21.4
3f48e3b Update github-go-version to 1.21.6 from 1.21.5
5bfb2bb Update github-go-version to 1.22.0 from 1.21.6
888b0be Update github-golangci-lint to 1.56.0 from 1.55.2
283380b Update github-golangci-lint to 1.56.1 from 1.56.0
be79c19 Update gomod-golang-x-crypto to 0.16.0 from 0.15.0
6aba928 Update gomod-golang-x-crypto to 0.17.0 from 0.16.0
561fed9 Update gomod-golang-x-crypto to 0.18.0 from 0.17.0
b5388ea Update gomod-golang-x-crypto to 0.19.0 from 0.18.0
0200a4e Update gomod-golang-x-net to 0.19.0 from 0.18.0
2e22e1b Update gomod-golang-x-net to 0.20.0 from 0.19.0
40bcb19 Update gomod-golang-x-net to 0.21.0 from 0.20.0
2a1d970 Update gomod-golang-x-term to 0.16.0 from 0.15.0
310c78e Update gomod-golang-x-term to 0.17.0 from 0.16.0
536583c Update make-golangci-lint to 1.56.0 from 1.55.2
5369576 Update make-golangci-lint to 1.56.1 from 1.56.0
e51c746 aac_frame: CPE: Decode instance tag and common window flag
f5f8e93 bson: Fix jq style a bit
d0a1b30 bump readline to 0.1.0-rc1
b56258c bump,readline: Add bump config and update to latest master
0070e38 bump,readline: Skip bump until not rc
63e0aa3 doc: Fix weird wording in README.md
e5fd1eb doc: Generate svgs with new ansisvg
55470de doc: Update docs to include fit format
c47756d doc: Update svgs after ansisvg update
54c6f0c fit: Added support for "invalid" value checking.
46dbf5b fit: Added support for ANT+ FIT format (used by Garmin devices)
6219d57 fit: Added support for dynamic subfields
33e5851 fit: Fix field casing to snake_case. Misc cleanup.
76307e4 fit: Formatted date/time description for timestamp fields
8862280 fit: Made long/lat present as float64
788088f fit: Show crc as hex, lower case "invalid" and some style harmonization
a07ce11 fq: Relicense float80 code to MIT
5829c6b gojq: Update fq fork
d155c80 gojq: Update fq fork
a793d10 goreleaser: Fix deprecated options
919e079 hevc_sps: Fix some incorrect profile_tier_level decoding
69ae765 interp: Fix crash when using line_bytes=0
2176600 interp: Support ~/.config/fq as fallback on macOS
fb910bd ldb: first draft
efc59a8 ldb: uncompression support
41f27a1 leveldb: add torepr for descriptor
2df0f0f leveldb: add log and descriptor decoders
b05aa99 leveldb: address PR comments
2f5f183 leveldb: decode unfragmented .log files further; fix UTF8 decoding
e826f09 leveldb: fix Errorf arguments
4283091 leveldb: fix all.fqtest failures
287ed36 leveldb: fix metaindex keys, refactoring, and jq syntax per PR
8665df5 leveldb: fix table's data blocks' internal keys decoding
08e3d2d leveldb: improve stringify by preallocating result
3a396e1 leveldb: improve log documentation
1ba8dec leveldb: in some properties, change spaces to underscores
e735cea leveldb: propagate error
07ad940 leveldb: rename "suffix" to "sequence_number_suffix"
78a3e94 leveldb: rename functions and add comments
cc0d5a8 leveldb: update docs
fe1099b leveldb: updates per PR comments
0d06e0a mp4: Don't decode samples if track has external data reference
d2c5ce5 mp4: Use box structure instead of track id to keep track for sample table data
aadf26f mp4: ctts v0 sample_offset seems to be signed in practice
fca55b2 mpeg_es: Support ES_ID_Inc and decode descriptors for IOD tags
e3af467 pcapng: Decode all section headers, clenaup descriptions
38b4412 png: Type flags were off-by-one bit
fb1c625 readline,bump: Fix version compare link
41226f4 readline: Switch to ergochat/readline
cd572d3 readline: Update to 0.1.0 and add bump config
7906f33 test: Support to more common -update flag
3b7cc1f tls: Fix field name typos
b0421df webp: Decode width, height and flags for lossless webp

Contributors

mikez and mlofjard

Assets 9

25 Nov 17:53

wader

v0.9.0

1482145

v0.9.0

Changes

Make synthetic values not look like decode values with a zero range. #777
Bit ranges are now displayed using exclusive end notation to be more consistent. For example if a field at byte offset 5 is 8 bit in size using the <byte>[.<bits>] notation it is now shown as 5-6, before it was shown as 5-5.7. See usage documentation for more examples. #789
Improve colors when using a light background. Thanks @adedomin for reporting. #781
Better from_jq error handling. Thanks @emanuele6 for reporting. #788

Updated gojq fork. Notable changes from upstream below. #808

Adds pick/1 function. Outputs value and nested values for given input expression.

# pick out all string values from a ELF header
$ fq '.header | pick(.. | select(type == "string"))' /bin/ls
{
  "ident": {
    "data": "little_endian",
    "magic": "\u007fELF",
    "os_abi": "sysv",
    "pad": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
  },
  "machine": "x86_64",
  "type": "dyn"
}
# bonus tip:
# if you only need to pick one level then jq has a shortcut syntax
$ fq '.header | {type, machine}' /bin/ls
{
  "machine": "x86_64",
  "type": "dyn"
}

Implements comment continuation with backslash.

Updated gopacket to 1.2.1. Notable changes from upstream below. #815
- fix(ip4defrag): allow final fragment to be less than 8 octets by @niklaskb
- refactor: don't fill empty metadata slots by @smira
- refactor: optimize port map to switch statement by @smira

Decoder changes

avi

Some general clean up fixes. Thanks to Marion Jaks at mediathek.at for testing and motivation.
Add extended chunks support and decode_extended_chunks option. This is trailing chunks used by big AVI files. #786

Add type, handler, compression (video) and format_tag (audio) per stream. #775

$ fq '.streams[0]' file.avi
      │00 01 02 03 04 05 06 07│01234567│.streams[0]{}: stream
0x1680│      00 00 00 01 67 f4│  ....g.│  samples[0:3]:
0x1688│00 0d 91 9b 28 28 3f 60│....((?`│
0x1690│22 00 00 03 00 02 00 00│".......│
*     │until 0x2409.7 (3464)  │        │
      │                       │        │  type: "vids"
      │                       │        │  handler: "H264"
      │                       │        │  compression: "H264"

Properly use sample size field when decoding samples. #776

exif (and tiff)
- Handle broken last next ifd offset by treating it as end marker. #804
gzip
- Correctly handle multiple members. Thanks @TomiBelan for the bug report and assistance. #795
- Now gzip is modelled as a struct with a members array and a uncompressed field that is the concatenation of the uncompressed members.
macho
- Properly respect endian when decoding some flag fields. #796
- Move formatted timestamp to description so that numeric value is easier to access. #797
matroska
- Support decoding EBML date type. #787
protobuf
- No need to use synthetic fields for string and bytes. #800
webp
- Refactor to use common RIFF decoder and also decode VP8X, EXIF, ICCP and XMP chunks. #803

zip Better timestamp support and fixes

Fix incorrect MSDOS time/date decoding and add extended timestamp support. Also remodel time/date to be a struct with raw values, components and a synthetics UTC unixtime guess. Thanks @TomiBelan for the bug report and assistance. #793

$ fq '.local_files[] | select(.file_name == "a").last_modification' file.zip
    │00 01 02 03 04 05 06 07│01234567│.local_files[3].last_modification{}:
0xd0│            81 01      │    ..  │  fat_time: 0x181
    │                       │        │  second: 2 (1)
    │                       │        │  minute: 12
    │                       │        │  hour: 0
0xd0│                  73 53│      sS│  fat_date: 0x5373
    │                       │        │  day: 19
    │                       │        │  month: 11
    │                       │        │  year: 2021 (41)
    │                       │        │  unix_guess: 1637280722 (2021-11-19T00:12:02)

Changelog

b702218 Update docker-golang to 1.21.2 from 1.21.1
d704711 Update docker-golang to 1.21.3 from 1.21.2
c31fc87 Update docker-golang to 1.21.4 from 1.21.3
861487d Update github-go-version to 1.21.2 from 1.21.1
d766356 Update github-go-version to 1.21.3 from 1.21.2
caef93c Update github-go-version to 1.21.4 from 1.21.3
de7fdae Update github-golangci-lint to 1.55.0 from 1.54.2
60edf97 Update github-golangci-lint to 1.55.1 from 1.55.0
534a2c8 Update github-golangci-lint to 1.55.2 from 1.55.1
906bc3b Update gomod-golang-x-crypto to 0.14.0 from 0.13.0
8d4d18d Update gomod-golang-x-crypto to 0.15.0 from 0.14.0
f108194 Update gomod-golang-x-net to 0.16.0 from 0.15.0
5381f38 Update gomod-golang-x-net to 0.17.0 from 0.16.0
14fe728 Update gomod-golang-x-net to 0.18.0 from 0.17.0
1011f19 Update gomod-golang/text to 0.14.0 from 0.13.0
527aad6 Update gomod-gopacket to 1.2.0 from 1.1.1
0c22c79 Update make-golangci-lint to 1.55.0 from 1.54.2
5f06364 Update make-golangci-lint to 1.55.1 from 1.55.0
36576a5 Update make-golangci-lint to 1.55.2 from 1.55.1
d703321 avi: Add extended chunks support and option
55521bb avi: Add stream type constants
5196554 avi: Add type, handler, format_tag and compreession per stream
0f225c3 avi: Add unused field for extra indx chunk space
df085b9 avi: Handle stream sample size
c7ec18d avi: Increase sample size heuristics to 32bit stereo
a745b12 avi: More correct strf chunk extra data
9b10e59 avi: Only use sample size heuristics if there is no format
23ae4d9 decode,interp: Make synthetic values more visible and not act as decode values
5abf151 doc: Remove spurious backtick
02b3527 exif,tiff: Handle broken last next ifd offset by treating it as end marker
dc376f3 gojq: Update rebased fq fork
ac276ee gzip: Correctly handle multiple members
45a8dd9 interp: Better from_jq error handling
051a70b interp: Change bit ranges to use exclusive end
29e7541 interp: Fix infinite recursion when casting synthetic raw value into a jq value
c28163f interp: Improve colors when using light background
797c7d9 macho: Move timestamp string to description
71a5fc9 macho: Respect endian when decoding flags
1d14ea5 matroska: Decode ebml date type
b24ed16 mod: Update golang.org/x/exp and github.com/gomarkdown/markdown
5e2e49e protobuf: No need for synthetic for string and bytes value
6034c70 webp,avi,wav,aiff: Trim RIFF id string
9e58067 webp: Refactor to use riff code and decode VP8X, EXIF, ICCP and XMP chunks
a83cac6 zip: Fix incorrect time/date, add extended timestamp and refactor

Contributors

smira, TomiBelan, and 3 other contributors

Assets 9

27 Sep 16:52

wader

v0.8.0

3988399

v0.8.0

Fix handling of shadowing order for _<name> keys, 3 new decoders caff, moc3 and opentimestamps, some smaller fixes and improvements.

In other jq news jq 1.7 was finally released 5 years since the last release! also happy to now be part of the jq maintainance team.

Changes

New decoders caff, moc3 and opentimestamps. See below for details.
Fix shadowing of underscore prefixed keys (_<name>) for text formats like json, yaml etc. #757
This happenned because fq has a bunch of internal underscore prefixed "extra" keys that is used for various things and these had priority even when there already existed a "value" key with same name.
```
$ fq -n '`{"_format": 123}` | fromjson | ._format'
```
Now 123, before "json".
```
$ fq -n '`{}` | fromjson | ._missing'
```
Now null, before error
Rename --null/nul-output to --raw-output0 and also clarify that NUL and new lines are outputted after and not between each output.
This is to be in sync with jq (jqlang/jq#2684). #736
Updated gojq fork with fixes from upstream:
- Improved error messages for indices, setpath, delpaths
- Add abs function
- Change behavior of walk with multiple outputs
- Change zero division to produce an error when dividend is zero
- Fix empty string repeating with the maximum integer
- Fix string multiplication by zero to emit empty string
- Remove deprecated leaf_paths function
Fix split in combination with binary to not include separator. #767

Decoder changes

caff Add archive format decoder. Thanks @Ronsor #747
- CAFF is an archive format usually found with the extensions .cmo3 and .can3 used by Live2D Cubism.
id3v2 Handle W000-WZZZ and W00-WZZ URL frames. #758
matroska Update spec and regenerate. #737
moc3 Add Live2D Cubism MOC3 decoder. Thanks @Ronsor #747
- MOC3 is a format for 2D rigged puppets, somewhat like Flash.
mp3_frame_xing Detect lame ext more similar to ffmpeg and mediainfo. #763
mp4
- Decode sgpd (Sample group definition box) entries. Thanks Sergei Kuzmin @ksa-real #707
- Decode cslg (Composition to decode timeline mapping) box. #754
- Decode emsg (Event message) and id3v2 message data. #755
- Nicer trimmed major brand for ftyp. #723
opentimestamps Add OpenTimestamps decoder. Thanks @fiatjaf #769

Changelog

4031082 Update docker-golang to 1.20.6 from 1.20.5
6daa0aa Update docker-golang to 1.20.7 from 1.20.6
8bd7b6d Update docker-golang to 1.21.0 from 1.20.7
bff668c Update docker-golang to 1.21.1 from 1.21.0
8e705aa Update github-go-version to 1.20.6 from 1.20.5
3828b42 Update github-go-version to 1.20.7 from 1.20.6
c09d50a Update github-go-version to 1.21.0 from 1.20.7
30b27a5 Update github-go-version to 1.21.1 from 1.21.0
104c3bd Update github-golangci-lint to 1.54.0 from 1.53.3
7906a46 Update github-golangci-lint to 1.54.1 from 1.54.0
31de3f9 Update github-golangci-lint to 1.54.2 from 1.54.1
8394729 Update gomod-golang-x-crypto to 0.12.0 from 0.11.0
ebb71e2 Update gomod-golang-x-crypto to 0.13.0 from 0.12.0
c8aae66 Update gomod-golang-x-net to 0.13.0 from 0.12.0
a46ee65 Update gomod-golang-x-net to 0.14.0 from 0.13.0
07069a5 Update gomod-golang-x-net to 0.15.0 from 0.14.0
79432e7 Update gomod-golang/text to 0.12.0 from 0.11.0
2f8ebf1 Update gomod-golang/text to 0.13.0 from 0.12.0
1fa14a0 Update make-golangci-lint to 1.54.0 from 1.53.3
fc4101d Update make-golangci-lint to 1.54.1 from 1.54.0
4e20e04 Update make-golangci-lint to 1.54.2 from 1.54.1
013cc2f caff: eliminate gaps and specify unused fields
6a3fecd caff: include uncompressed bits for proper decompressed entries that can't be decoded as a format
da41a8d caff: initial implementation
23e660f caff: minor formatting changes
fa11572 caff: obfuscation key is a signed integer, add test data
29084e3 caff: remove dead code
4dd0f6d caff: run go fmt
b3759de caff: run go fmt
cc58c4b caff: update doc/formats.md
d5345f0 cli: Rename --null/nul-output to --raw-output0
c0352f2 decode,interp: Don't shadow _key and error on missing _key
44f0060 dev,jq: Reformat jq code to look more the same
9cd1d0f dev: Move examples and snippets to wiki
f15f9bc doc,moc3,caff: Add author and regenerate docs
406f392 doc: Move up and update differences jq section a bit
8edef78 docker: Change to bookworm
56fec2a elf: Fix broken static and segfault tests
fa3dba1 gojq: Update fq fork
0cefc46 golangci: Fix gosec aliasing warnings
0d01476 gomod: Update x/exp and gomarkdown
c503bc1 html: Add forgotten re test
0efe5a2 id3v2: Handle W000-WZZZ,W00-WZZ URL frames
a614c9d interp: split: Correctly split binary
3af0462 luajit: file null terminator: raw bits, validate
c07673a matroska: Update spec and regenerate
441fcd0 moc3, caff: update tests and README
f7eb027 moc3: Fix field order in blend_shape_keyform_bindings structure, version detection in count_info
03ba71b moc3: add support for version 5
d3073c6 moc3: add test data for new version 5
ce40fd1 moc3: consistency - scales array contains value elements, not scale elements
fac1e68 moc3: count_info: extra space is reserved, not normal alignment/padding
e424e29 moc3: eliminate gaps and properly handle padding, fix version 5 format decoding
092662e moc3: initial implementation
3caf34e moc3: nicer tree structure, use more meaningful names for array elements
20f02e7 moc3: remove dead code
6d10a25 moc3: update certain array element names, explicitly mark unused or reserved space
833b063 moc3: update test data
14f233d moc3: update tests
c4e8644 mod: Update golang.org/x/exp and github.com/gomarkdown/markdown
0699c80 mp3_frame_xing: Detect lame ext more similar to ffmpeg and mediainfo
e50028a mp4,mpeg_es: Decode iods box and MP4_IOD_Tag OD
312d807 mp4: Decode cslg box
bedd719 mp4: Decode emsg box
97194ad mp4: Nicer major brand and handle some qt brand short strings better
cc8e6f1 opentimestamps: abstract away file digest sizes and support sha1, ripemd160 and keccac256.
64a4ff2 opentimestamps: account for unknown attestation types.
912f411 opentimestamps: add help text.
cef5faa opentimestamps: add parser.
1aa557d opentimestamps: add tests.
5e7c01a opentimestamps: address comments and improve things.
976a756 opentimestamps: one last make doc.
0a22a32 opentimestamps: satisfy linter.
456a6a4 protobuf_widevine: Make protection_scheme constants less magic

Contributors

fiatjaf, Ronsor, and ksa-real

Assets 9

07 Jul 09:44

wader

v0.7.0

d15a41f

v0.7.0

Added LuaJIT bytecode decoder by @dlatchx, otherwise mostly small things. Been busy with nice weather and helping our getting jq development and maintenance back on track.

Changes

Better performance of binary arrays when they only include 0-255 numbers or strings. #704

Make tovalue on binary, in addition decode value binary, respect bits_format options. #677

# uses -V to do tovalue to get a hex string
# uses -r to output "raw" string
$ fq -o bits_format=hex -Vr -n '[1,0xff,3] | tobytes'
01ff03

Updated gojq fork with fixes from upstream: #679
- Improved error messages
- @urid URI format function

Decoder changes

luajit Add LuaJIT bytecode decoder. Thanks @dlatchx #709
mp4 Improved sample group definition box sgpd entries decoder. Thanks @ksa-real #707
mp4 Improved user metadata udta structure decoding #676
wav Decode bext chunk. #712

Changelog

47b9060 Improved README.md
d02b70f Update README.md
64e17f0 Update docker-golang to 1.20.5 from 1.20.4
6faed67 Update github-go-version to 1.20.5 from 1.20.4
b9fce9b Update github-golangci-lint to 1.53.1 from 1.52.2
ff4048c Update github-golangci-lint to 1.53.2 from 1.53.1
76e0e17 Update github-golangci-lint to 1.53.3 from 1.53.2
8e75dc9 Update gomod-BurntSushi/toml to 1.3.2 from 1.2.1
6dc0746 Update gomod-golang-x-crypto to 0.10.0 from 0.9.0
98351ff Update gomod-golang-x-crypto to 0.11.0 from 0.10.0
939d98c Update gomod-golang-x-net to 0.11.0 from 0.10.0
660ca03 Update gomod-golang-x-net to 0.12.0 from 0.11.0
36ef2a2 Update gomod-golang/text to 0.10.0 from 0.9.0
0eb6557 Update gomod-golang/text to 0.11.0 from 0.10.0
a079b73 Update gomod-gopacket to 1.1.1 from 1.1.0
c3e104b Update make-golangci-lint to 1.53.1 from 1.52.2
7c1da0e Update make-golangci-lint to 1.53.2 from 1.53.1
47ea6cf Update make-golangci-lint to 1.53.3 from 1.53.2
fd2cb6f doc: Fix broken link in README
db2e621 go fmt
38cb829 gojq: Update rebased fq fork
41f40b7 interp: Add to binary fast path for arrays with only 0-255 numbers and strings
b2c0e5f interp: Make binary also respect bits_format
b24063b luajit: *.fqtest: add comments for generating .luac from source
bdf158b luajit: add luajit.md
93c9696 luajit: add to probe group
32300a3 luajit: check binary.Read() error
a83576a luajit: clarify description
751ee5a luajit: explain LuaJITDecodeKNum, fix negative in bug
3561c08 luajit: fallbackUintMapSymStr
5d9a08c luajit: fix regression: (u64 vs i64)
64c11be luajit: improve debuginfo decoding
1afdf8b luajit: initial support
29ab66b luajit: lowercase flags
e44f5c0 luajit: magic number: raw bits, check with assert
23b9eea luajit: make doc
715f850 luajit: opcodes: implement scalar.UintMapper
c3a123a luajit: remove unecessary dependency
64c92da luajit: remove unused variable
52ce818 luajit: split in smaller decode functions
441d246 luajit: standardize field names (key/value/type ect.)
eb819dd luajit: tests: improve coverage
dd594f4 luajit: tests: rename lua source file
c42fb9e luajit: typo
08ae661 luajit: use UTF8 strings
1da8069 mp4: udta: Improve length/lang box probe and support empty value
e869d8a sgpd box entries parsing
8c75509 wav: Decode bext chunk

Contributors

dlatchx and ksa-real

Assets 9

16 May 14:06

wader

v0.6.0

4335c30

v0.6.0

Adds decoders for PostgreSQL btree, control and heap files. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka

Adds new option skip gaps and output binary as hex string.

Make bits/bytes formats work a bit more intuitive.

Bug fixes to to_hex, to_base64 and trim functions.

Bug fixes and additions to bson, bitcoin_transaction, mp4, id3v2, html, and matroska formats.

Changes

bits,bytes now are real binaries and not raw decode value. This means they behave more like you would expect. #666

# build your own strings(1)-like tool:
# scan matches range in a binary using a regexp and output ranges as new binaries
# \w\s looks for whitespace and alpha/numeric characters
# uses `...` raw string literal to not have to escape
# select/test to only include strings containing "thread"
# dd to display with no truncation
$ fq -d bytes 'scan(`[\w\s]{16,}`) | select(test("thread")) | dd' file.mp4
     │00 01 02 03 04 05 06 07│01234567│
0x250│36 20 6c 6f 6f 6b 61 68│6 lookah│.: raw bits 0x250-0x262.7 (19)
0x258│65 61 64 5f 74 68 72 65│ead_thre│
0x260│61 64 73               │ads     │
     │00 01 02 03 04 05 06 07│01234567│
0x260│            31 20 73 6c│    1 sl│.: raw bits 0x264-0x273.7 (16)
0x268│69 63 65 64 5f 74 68 72│iced_thr│
0x270│65 61 64 73            │eads    │

to_hex,to_base64 now correctly handles raw decode values, before the raw bits would be turned into codepoints and then binary UTF-8 possibly introducing invalid codepoints (0xfffd). Thanks @Rogach #672
```
$ fq -r '.uncompressed | to_hex' file.gz
f6f2074cf77d449d

# with the change to add hex bits format you can also do this now
$ fq -Vr -o bits_format=hex .uncompressed file.gz
f6f2074cf77d449d
```
tovalue Now output a "deep" jq value, before it was shallowly a jq value which could be confusing, ex tovalue | .header could be a decode value.
New option skip_gaps for -V/tovalue can be used to filter out gap fields when represented as JSON. Gaps are bit ranges that no decoder added any field for. #649
- Can for example be used to remove trailing data that will end up as gap0 etc.
- Note that gaps in an array that gets filtered out might affect index numbers.

Add bits_format=hex to represent raw bits as hex string in JSON. #673

# output decode tree, JSON with binaries as strings and JSON with binaries as hex strings
$ fq '.packets[0].packet | ., tovalue, tovalue({bits_format:"hex"})' file.pcap
    │00 01 02 03 04 05 06 07│01234567│.packets[0].packet{}: (ether8023_frame)
0x28│8c 85 90 74 b8 3b      │...t.;  │  destination: "8c:85:90:74:b8:3b" (0x8c859074b83b)
0x28│                  e8 de│      ..│  source: "e8:de:27:c8:9a:6e" (0xe8de27c89a6e)
0x30│27 c8 9a 6e            │'..n    │
0x30│            08 06      │    ..  │  ether_type: "arp" (0x806) (Address Resolution Protocol)
0x30│                  00 01│      ..│  payload: raw bits
0x38│08 00 06 04 00 01 e8 de│........│
0x40│27 c8 9a 6e c0 a8 01 01│'..n....│
0x48│00 00 00 00 00 00 c0 a8│........│
0x50│01 e6                  │..      │
{
  "destination": "8c:85:90:74:b8:3b",
  "ether_type": "arp",
  "payload": "\u0000\u0001\b\u0000\u0006\u0004\u0000\u0001\ufffd\ufffd'Țn\ufffd\ufffd\u0001\u0001\u0000\u0000\u0000\u0000\u0000\u0000\ufffd\ufffd\u0001\ufffd",
  "source": "e8:de:27:c8:9a:6e"
}
{
  "destination": "8c:85:90:74:b8:3b",
  "ether_type": "arp",
  "payload": "0001080006040001e8de27c89a6ec0a80101000000000000c0a801e6",
  "source": "e8:de:27:c8:9a:6e"
}

Explicit use of d/display on a binary will now always hexdump, this feels more intuitive. Before it could output raw binary as display is used as an implicit output function. Implicit (you don't mention d at all) can still output raw binary. #665

# outputs raw binary (if stdout is not a tty)
$ fq -n '[1,2,3] | tobytes' | xxd
00000000: 0102 03

# outputs hexdump (even if stdout is not a tty) as we explicitly use d
$ fq -n '[1,2,3] | tobytes | d' | cat
   |00 01 02 03 04 05 06 07|01234567|
0x0|01 02 03|              |...|    |.: raw bits 0x0-0x2.7 (3)

trim can now handle multi-line strings. #668
Help texts originated from markdown are now rendered a bit more nicely and compact. #661
Fix weirdly rendered monospace font in demo SVG when using Japanese locale and Chrome on macOS. Thanks @acevif helping with investigation. #655 #662
Fix a bunch of typos. Thanks @kianmeng and @castilma. #644 #670

Decoder changes

bits,bytes Is a proper binary not a raw decode value. #666
bitcoin_transaction Properly decode witness items array. #671 Thanks @Rogach
bson Add javascript, decimal128, minkey and maxkey support. Fix decoding of datetime and use the correct size type for binary and document size. Thanks Matt Dale @matthewdale. #650
id3v2
- Add WXXX frame support. #656
- Add CTOC flags support. #657
html Is now probeable. #667
matroska Fallback raw is probe fail or file_data #645
mp4
- Better description for terminator atom (is a QuickTime thing) #651
- ctts,infe,iinf,trun more proper decoding based on version. #643
- Use correct QuickTime epoch for timestamps. #674
pg_btree,pg_control,pg_heap Add PostgreSQL btree, control and heap support. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka. #415

Changelog

d4f8dfa Add flavour arg to postgres parser
3c6ea87 Add heap infomask flags parser to PostgreSQL
0107d12 Add pgproee14 flavour to postgres
9a96da8 Add postgres pg_control parser
3b81d99 DBState enum for postgres
51878dc PostgreSQL heap page parser implememtation.
6ed0263 PostgreSQL: accept only normal item pointers
ffc08cf PostgreSQL: add heap for pgpro14
b4ae1d5 PostgreSQL: add pg_control for pgproee 12
a653710 PostgreSQL: add pg_control to pgpro14
d9de2d4 PostgreSQL: fix
96a86e2 PostgreSQL: fixes
96a96a5 PostgreSQL: heap impl for pgproee 12
6618e76 PostgreSQL: heap impl for version 11
ce9ae76 PostgreSQL: heap tuples implementation
ffd7c9b PostgreSQL: implement pgproee 14
850dc60 PostgreSQL: lp_flags format
f5278f3 PostgreSQL: pg_control for ver 12
07056c2 PostgreSQL: pg_control impl for pgproee 10
972c5a3 PostgreSQL: pg_control, pgheap impl for pgproee13
d148727 PostgreSQL: pgheap impl for pgproee10
2d3884a PostgreSQL: pgproee11 heap impl
d8b891c PostgreSQL: pgproee11 pg_control impl
b722b21 PostgreSQL: ref
fdb3b3e PostgreSQl: fix offset
621c4c4 PostgreSQl: heap impl for version 13
c273a6c PostgreSQl: pg_control impl for version 13
01b380e PostgrreSQl heap decode refactoring
bebdfa9 Try to implement pgwal - fail.
c9d3b8f Update docker-golang to 1.20.4 from 1.20.3
2a5927b Update github-go-version to 1.20.4 from 1.20.3
c405afd Update gomod-golang-x-crypto to 0.9.0 from 0.8.0
ed0cd6d Update gomod-golang-x-net to 0.10.0 from 0.9.0
deaf5ef WalLevel for postgres
f069ddc [WIP] initial attempt to add postgres
2b1bdfb add icu version mapper
cf1e7b2 add pgpro11 for postgres
9570d4d add pgpro12 postgres
7bf6b11 add pgpro13 heap
2b3035f add pgpro13 to postgres
f56c72d add postgres tests for mem, cpu profiling
2ee01f7 allow to change FillGaps in decoder
a3361e7 bitcoin: fix witness item structs
222cd88 bits,bytes: Behave as binary instead of raw decode value
9a982d0 bson: add BSON test file generator module and correct BSON format docs
40630d3 bson: fix doc formatting and add author info
2017ff8 bson: support all non-deprecated types and fix int/uint bugs
b08ef00 decode,interp: Refactor format groups into a proper struct
af68511 dev,doc Clarify some dev docs and rename launch.json to be a template
97c952b doc: Add some more examples
88be3a7 doc: Hopefully fix svg fixed font issue
dd4fa26 doc: fix typos
b0e4da2 fix non-ascii characters handling in to_hex and to_base64 functions
a4a332b formats: Clenaup naming a bit
b3b6cd0 gzip.go: fix typo in variablename: delfate
2c505fe help,markdown: Rewrote and made text rendering nicer
e2eb667 html: Add to probe group
d010dce id3v2: Add WXXX (desc/url) frame support
f237db2 id3v2: Decode CTOC flags
684a083 interp,decode: Support decode group argument
8a468f4 interp: Add hex bits format
c512713 interp: Add skip_gaps option for tovalue/-V
033498b interp: Don't output raw binary if display is called explicitly
ee66fec interp: Make tovalue output behave as jq value
d5ae116 interp: trim: Add multi-line support
8d317dd lsn mapper
fcd7fbc mappers for postgres
8941b13 matroska: file_data: Fallback to raw if probe fails
7adc1e7 mp4: Better description for QuickTime terminator atom
493848a mp4: Use correct epoch for quicktime timestamps
3c6d31b mp4: ctts,infe,iinf,trun: More ISOMFF version handling
d6f785c pcap: Add forgotten help test
8cf83fb pg_control implementation
441ecee pgpro version mapper
5ff6273 postgres: add additional checks in pg_heap
4676590 postgres: add argument to calc page's check sum correctly
7fd4109 postgres: add btree index tests
1aa08e3 postgres: add btree, pg_control to how_to.md
2423f86 postgres: add how_to.md - how to generate test files for postgres
97bbc22 postgres: add page arg in pg_btree, change args names in pg_heap
6aed238 postgres: add pg_wal for pgproee11 as copy of postgres14
08eb303 postgres: add postgres format docs, refactoing postgres flavours
90386a6 postgres: add postgres.md to format
09c42c3 postgres: add state to wal struct
c2591ac postgres: add test data with specific values
de68785 postgres: add test files
4db1284 postgres: add tests
3c7ed5d postgres: add test...

Contributors

mizhka, kianmeng, and 5 other contributors

Assets 9

13 Apr 15:47

wader

v0.5.0

23d980d

v0.5.0

Mostly a bug fix release but adds -V for easy JSON output.

Changes

Add -V argument to default output JSON instead of decode tree in case of decode value. #385 Thanks @peterwaller-arm for reminding me to merge this.

# default in case of decode value is to show a hexdump tree
$ fq '.headers | grep_by(.id=="TSSE").text' file.mp3
    │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
0x0c│                           4c 61 76│         Lav│.headers[0].frames[0].text: "Lavf58.76.100"
0x18│66 35 38 2e 37 36 2e 31 30 30 00   │f58.76.100. │

# with -V an implicit "tovalue" is done
$ fq -V '.headers | grep_by(.id=="TSSE").text' file.mp3
"Lavf58.76.100"

# and in combination with -r will for strings output a "raw string" without quotes
# for other types like number, object, array etc -r makes not difference (same as jq)
$ fq -Vr '.headers | grep_by(.id=="TSSE").text' file.mp3
Lavf58.76.100

As a side note -V can be used with binary type also. Then the binary data will be interpreted as UTF-8 and turned into a string.

# trailing null terminator ends up as codepoint zero `\u0000`
$ fq -V '.headers | grep_by(.id=="TSSE").text | tobytes' file.mp3
"Lavf58.76.100\u0000"

# with -r null terminator and a new line is outputted
$ fq -Vr '.headers | grep_by(.id=="TSSE").text | tobytes' file.mp3 | hexdump -C
00000000  4c 61 76 66 35 38 2e 37  36 2e 31 30 30 00 0a     |Lavf58.76.100..|
0000000f

# in contrast raw binary output has no new line separator
$ fq '.headers | grep_by(.id=="TSSE").text | tobytes' doc/file.mp3 | hexdump -C
00000000  4c 61 76 66 35 38 2e 37  36 2e 31 30 30 00        |Lavf58.76.100.|
0000000e

Fix issue using decode value in object passed as argument to internal function. #638

# this used to fail but now works
fq '.tracks[0].samples[10] | avc_au({length_size: <decode value>})' file.mp4

Some typo fixes. Thanks @retokromer and @peterwaller-arm

Decoder changes

aiff Basic AIFF decoder added. #614
matroska Update to latest specification. #640
msgpack Fix bug decoding some fixstr lengths. #636 Thanks @schmee for reporting.

Changelog

4ad1cce Update docker-golang to 1.20.3 from 1.20.2
f7dca47 Update github-go-version to 1.20.3 from 1.20.2
c960893 Update github-golangci-lint to 1.52.0 from 1.51.2
0a6b46c Update github-golangci-lint to 1.52.1 from 1.52.0
c4eb67d Update github-golangci-lint to 1.52.2 from 1.52.1
19140a6 Update gomod-creasty-defaults to 1.7.0 from 1.6.0
6e5df72 Update gomod-golang-x-crypto to 0.8.0 from 0.7.0
6c4aebf Update gomod-golang-x-net to 0.9.0 from 0.8.0
f13cc97 Update gomod-golang/text to 0.9.0 from 0.8.0
e2af57e Update gomod-gopacket to 1.1.0 from 1.0.0
a63fd68 Update make-golangci-lint to 1.52.0 from 1.51.2
d3d1f0e Update make-golangci-lint to 1.52.1 from 1.52.0
f0b0845 Update make-golangci-lint to 1.52.2 from 1.52.1
dc4a82e aiff: Add basic decoder
c5f6809 decode,fuzz,dev: Move recoverable error check to recoverfn.Run
980ecdb decode: Add float 80 reader
a6c4db7 decode: Cleanup old unused help system code
87e5bb1 fix typo
0b6ef2a golangci-lint: Disable revive unused-parameter and update for new default config
427ce78 interp: Add --value-output/-V option to do tovalue before output
9a1ef84 interp: Allow and convert JQValues:s (ex decode value) in function arg objects
3dd2c61 interp: Fix input completion regression in sub-REPLs
5415bfc interp: Make completion work again
2a2b64d matroska: Update ebml specification
82da99c msgpack: Add str, array and object type tests
97360d6 msgpack: fixstr length field is 5 bits
ffc66db readline: remove direct access to (*Instance).Config
e1b0231 wav: Cleanup avi leftovers

Contributors

retokromer, schmee, and peterwaller-arm

Assets 9

08 Mar 16:49

wader

v0.4.0

29bb15a

v0.4.0

TLS decode and decryption, better streaming matroska/webm support, support raw IP in PCAP and bug fixes.

Changes

Fix panic when interrupting big JSON output. #573
Support passing options (-o name=value) to nested decoders. #589
- Allows for example to pass keylog to a TLS decoder inside a PCAP file or to tell a container decoders to not decode samples inside a ZIP file etc.
Exit with error if -o name=@path fails to read file at path. #597

Decoder changes

id3v2 Properly decode CTOC subframes. #606
matroska
- Now supports streaming matroska and webm better (master elements with unknown size). #576 #581
- Add decode_samples option. #574
- Spec update and clean up of symbols and descriptions. #580
pcap,pcapng Support raw IPv4 and IPv6 link frames. #599 #590

tls Add Transport layer security decoder and decryption. #603

Supports TLS 1.0, 1.1, 1.2 and some SSL 3.0.
Decodes records and most messages and extensions.
Can decrypt most common cipher suites if a keylog is provided. See documentation for list of supported ciphers suites.

# show first 50 bytes of decrypted client/server TLS application data stream
# -o [email protected] is used to read keylog from a file
# first .stream is TCP stream, second .stream the application data stream
$ fq -o [email protected] '.tcp_connections[0].["client", "server"].stream.stream | tobytes[0:50] | dd' file.pcap
    │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
0x00│47 45 54 20 2f 64 75 6d 70 2f 6c 6f│GET /dump/lo│.: raw bits 0x0-0x31.7 (50)
0x0c│67 20 48 54 54 50 2f 31 2e 31 0d 0a│g HTTP/1.1..│
0x18│48 6f 73 74 3a 20 69 6e 77 61 64 65│Host: inwade│
0x24│72 2e 63 6f 6d 0d 0a 55 73 65 72 2d│r.com..User-│
0x30│41 67                              │Ag          │
    │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
0x00│48 54 54 50 2f 31 2e 31 20 32 30 30│HTTP/1.1 200│.: raw bits 0x0-0x31.7 (50)
0x0c│20 4f 4b 0d 0a 41 63 63 65 70 74 2d│ OK..Accept-│
0x18│52 61 6e 67 65 73 3a 20 62 79 74 65│Ranges: byte│
0x24│73 0d 0a 43 6f 6e 74 65 6e 74 2d 4c│s..Content-L│
0x30│65 6e                              │en          │

# show first TLS record from server
$ fq '.tcp_connections[0].server.stream.records[0] | d' file.pcap
    │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│.tcp_connections[1].server.stream.records[0]{}: record
0x00│16                                 │.           │  type: "handshake" (22) (valid)
0x00│   03 03                           │ ..         │  version: "tls1.2" (0x303) (valid)
0x00│         00 40                     │   .@       │  length: 64
    │                                   │            │  message{}:
0x00│               02                  │     .      │    type: "server_hello" (2)
0x00│                  00 00 3c         │      ..<   │    length: 60
0x00│                           03 03   │         .. │    version: "tls1.2" (0x303)
    │                                   │            │    random{}:
0x00│                                 86│           .│      gmt_unix_time: 2249760024 (2041-04-16T21:20:24Z)
0x0c│18 9d 18                           │...         │
0x0c│         19 92 33 c2 21 ce 4f 97 30│   ..3.!.O.0│      random_bytes: raw bits
0x18│28 98 b3 fd 1e 15 f4 36 bb e9 14 f4│(......6....│
0x24│67 61 66 79 d5 3f 06               │gafy.?.     │
0x24│                     00            │       .    │    session_id_length: 0
    │                                   │            │    session_id: raw bits
0x24│                        c0 2f      │        ./  │    cipher_suit: "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" (0xc02f)
0x24│                              00   │          . │    compression_method: "null" (0x0)
0x24│                                 00│           .│    extensions_length: 20
0x30│14                                 │.           │
    │                                   │            │    extensions[0:2]:
    │                                   │            │      [0]{}: extension
0x30│   ff 01                           │ ..         │        type: "renegotiation_info" (65281)
0x30│         00 01                     │   ..       │        length: 1
0x30│               00                  │     .      │        data: raw bits
    │                                   │            │      [1]{}: extension
0x30│                  00 10            │      ..    │        type: "application_layer_protocol_negotiation" (16)
0x30│                        00 0b      │        ..  │        length: 11
0x30│                              00 09│          ..│        serer_names_length: 9
    │                                   │            │        protocols[0:1]:
    │                                   │            │          [0]{}: protocol
0x3c│08                                 │.           │            length: 8
0x3c│   68 74 74 70 2f 31 2e 31         │ http/1.1   │            name: "http/1.1"

# use ja3.jq to calculate ja3 TLS fingerprint
# https://github.com/wader/fq/blob/master/format/tls/testdata/ja3.jq
$ fq -L path/to/ja3 'include "ja3"; pcap_ja3' file.pcap
[
  {
    "client_ip": "192.168.1.193",
    "client_port": 64126,
    "ja3": "771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-16-22-23-49-13-43-45-51-21,29-23-30-25-24,0-1-2",
    "ja3_digest": "bc29aa426fc99c0be1b9be941869f88a",
    "server_ip": "46.101.135.150",
    "server_port": 443
  }
]

toml Fail faster to speed up probe. Could in some cases read the whole file before failing. Thanks @0-wiz-0 for report. #594
zip Properly decode EOCD record in zip64 files. Thanks @0-wiz-0 for report and spec interpretation. #586 #596
xml Fail faster to speed up probe. Could in some cases read the whole file before failing. Thanks @0-wiz-0 for report. #594

Changelog

0581ece Update docker-golang to 1.20.1 from 1.20.0
72870a5 Update docker-golang to 1.20.2 from 1.20.1
02e573a Update github-go-version to 1.20.1 from 1.20.0, 1.20.0, 1.20.0
c513088 Update github-go-version to 1.20.2 from 1.20.1
ce26372 Update github-golangci-lint to 1.51.1 from 1.51.0
75bfdda Update github-golangci-lint to 1.51.2 from 1.51.1
b1d9306 Update gomod-golang-x-crypto to 0.6.0 from 0.5.0
c03d3cc Update gomod-golang-x-crypto to 0.7.0 from 0.6.0
2430fba Update gomod-golang-x-net to 0.6.0 from 0.5.0
dd8ab79 Update gomod-golang-x-net to 0.7.0 from 0.6.0
80a0744 Update gomod-golang-x-net to 0.8.0 from 0.7.0
97643b9 Update gomod-golang/text to 0.7.0 from 0.6.0
e7168b9 Update gomod-golang/text to 0.8.0 from 0.7.0
36df57e Update make-golangci-lint to 1.51.1 from 1.51.0
70e08fa Update make-golangci-lint to 1.51.2 from 1.51.1
50d26ec colorjson: Handle encoding error value
5c8e115 colorjson: Refactor to option struct
8e0dde0 decode: Support multiple format args and some rename and refactor
a1bb630 doc,fq: Improve cli help and some cleanup
156aeec doc: Add FOSDEM 2023 talk
3e0ebaf doc: Run make doc
3cc8383 gojq: Update fq fork
dec433f help,markdown: Fix double line breaks when converting to text
c75a83c help: Show default option value as JSON
cc52a44 id3v2: Decode subframes for CTOC and add struct for headers
dc79a73 interp,json: Move error handling to colorjson
73db658 interp: Exit with error if -o name=@path fails to be read, also document
c8666ee ipv4_packet,ipv6_packet,sll_packet,sll2_packet: Support ipv4/ipv6 link frames and pass correct in arg
b60acec matroska: Add decode_samples option
9aaf2dd matroska: Add unknown size test and add description to ebml header
a8d0bf4 matroska: Assume master with unknown size has ended if a valid parent is found
0d14d7b matroska: Handle unknown size for non-master types a bit better
c890a28 matroska: Update spec and make refs in descriptions look nicer
6c03245 pcap,pcapng,ipv4,ipv6: Support raw link type (ipv4 or ipv6)
d4ea663 pcap: Add ipv4 fragments tcp test
f50bd6e readline: Update fq fork
9852f56 tls: Add TLS 1.0, 1.1, 1.2 decode and decryption
56edb59 toml,xml: Fail fast on invalid content
5228fdd zip: Correctly look for and decode both zip32/64 EOCD record
bdd6718 zip: Correctly peek for zip64 EOCD

Contributors

path and 0-wiz-0

Assets 9

Releases: wader/fq

v0.13.0

Changes

Format changes

Changelog

Contributors

v0.12.0

Changes

Format changes

Changelog

Contributors

v0.11.0

Changes

Format changes

Changelog

Contributors

v0.10.0

Changes

Format changes

Changelog

Contributors

v0.9.0

Changes

Decoder changes

Changelog

Contributors

v0.8.0

Changes

Decoder changes

Changelog

Contributors

v0.7.0

Changes

Decoder changes

Changelog

Contributors

v0.6.0

Changes

Decoder changes

Changelog

Contributors

v0.5.0

Changes

Decoder changes

Changelog

Contributors

v0.4.0

Changes

Decoder changes

Changelog

Contributors