Releases · wader/fq

02 Feb 10:47

wader

v0.3.0

bafc8e3

v0.3.0

Bug fix release, no new features mostly due to holidays and busy with other things (some jq related!).

Also been preparing for a talk about fq this weekend at FOSDEM 2023.

Changes

TCP reassembly is now less strict about invalid TCP options. Turns out some options might end up wrong in packet captures due to hardware acceleration etc. For example it seems to be common that TCP segments end up larger than configured connection MSS. Now PCAP:s with those kinds of TCP segments should be reassembled correctly.
REPL now handles the del key properly. Before it could in some cases cause the output to be ignored.

Decoder changes

mp3 Add option for max unknown bits to handle more mis-probing. Default to 50%
mp4
- ftyp set minor description to date for "qt" files
- tkhd decode enabled, preview, etc flags
- udta Handle case wit box type is key and value rest of box
- sgpd,sbgp Change grouping type to a string as it seems to be what it is in practice.
tcp_segment Decode all standard options, MSS, Window scale, timestamp etc. Rename "maxseg" to "mss".

Changelog

8702e1d Update docker-golang to 1.19.5 from 1.19.4
a7f37d7 Update docker-golang to 1.20.0 from 1.19.5
826d9a5 Update github-go-version to 1.19.5 from 1.19.4, 1.19.4, 1.19.4
d338c8b Update github-go-version to 1.20.0 from 1.19.5, 1.19.5, 1.19.5
ad4919a Update github-golangci-lint to 1.51.0 from 1.50.1
e8ecbf9 Update gomod-golang/text to 0.6.0 from 0.5.0
f1057b9 Update make-golangci-lint to 1.51.0 from 1.50.1
ca27e42 doc: Add _parent for decode values and clenaup doc a bit
b04a650 flac_picture,mpeg: Fix trailing ")" typo in map sym and description
57144b2 github-action: Use quotes becase yaml (1.20 -> 1.2)
0aa6e3e gojq: Update rebased fq fork
7855b35 gomod: Update non-bump tracked mods and add bump config
6e17de3 goreleaser: Use name_template instead of deprecated archive replacements
8b49b42 interp: Wrap Binary in decodeValue to fix prompt issue with bits/bytes format
2d82c05 mp3: Add max_unknown option to fail decode if too much unknown bits
f386a51 mp4: Decode qt minor verison as YYYY.MM description
3555dc6 mp4: Decode tkhd flags
c3e3b3e mp4: Decode udta metadata boxes without meta box
c49012d mp4: sgpd,sbgp: Change grouping_type to a string
6340365 mp4: udta: Handle box with value rest of box
55ef7a4 readline: Update fq fork to fix draw issue when using del key
1eb5e50 tcp: Ignore TCP option check for now as it seems unreliable in dumps
62e2cef tcp_segment: Decode standard options and rename maxseg to mss

Assets 9

23 Dec 13:35

wader

v0.2.0

63b5828

v0.2.0

This ended up being a release to cleanup old sins in the decoder internals and change some defaults how binary values work with JSON and string functions.

It also adds a new Time Zone Information Format decoder tzif (Thanks Takashi Oguma @bitbears-dev) and a new Apple BookmarkData decoder apple_bookmark decoder (Thanks David McDonald @dgmcdona). Also a new function from_ns_keyed_archiver was added to convert NSKeyedArchiver encoded objects into JSON.

A possible breaking change is that now all from/to prefix functions now has a from_/to_ prefix, ex: from_mp3 instead of frommp3. There are some few exceptions to this. Note that the functions named just be the format name, ex mp3 are still around.

In other fq related news jq-lsp got some fixed and additions and seems to work fine with neovim. It's also possible to use jq-lsp with vscode using vscode-jq.

Changes

All functions that had a from/to prefix now has the prefix from_/to_. This is to be easier to read and more consistent, there are still some exceptions like tovalue, torepr, tobytes etc but in general anything that does not deal with primitive types is now snake_case. #535
Change default bit_formats option value (how raw bits values are represented in JSON) from snippet to string. snippet meant truncated bits as base64. Now all bits are included as a UTF-8 string. The string will be binary safe (not lose any data) when used internally in fq but will lose data when represented in JSON as some bytes can't be encoded as UTF-8. #499

Don't auto convert to binary for string/regexp functions, turned out this is very confusing. Now you have to manually use tobytes etc to convert to binary value. #540

# This used to not work as test/1 would convert decode values to the source bytes
# (0x00 0x00 0x00 0x01) in this case. Now the jq value (symbolic in this case) will
# be used instead. You can do ".test | tobytes" to get old behavior.
#
# find all types with a "mdta." prefix
$ fq -o line_bytes=10 'grep_by(.type | test(`^mdta\.`))' file.mp4
     │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[0]{}: box
0x528│      00 00 00 1c            │  ....    │  size: 28
0x528│                  00 00 00 01│      ....│  type: "mdta.title" ("\x00\x00\x00\x01")
0x532│00 00 00 14 64 61 74 61 00 00│....data..│  boxes[0:1]:
0x53c│00 01 00 00 00 00 74 65 73 74│......test│
     │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[1]{}: box
0x546│00 00 00 25                  │...%      │  size: 37
0x546│            00 00 00 02      │    ....  │  type: "mdta.encoder" ("\x00\x00\x00\x02")
0x546│                        00 00│        ..│  boxes[0:1]:
0x550│00 1d 64 61 74 61 00 00 00 01│..data....│
0x55a│00 00 00 00 4c 61 76 66 35 39│....Lavf59│
0x564│2e 32 37 2e 31 30 30│        │.27.100│  │

Fix panic when cancel (ctrl-c etc) before interpreter is executing. Thanks @pldin601 for reporting. #495
Fix error using JQValue:s in assign/update paths, ex .[<JQValue here>] = 123 #509
Rename fields added for bit-ranges not used by a decoder from unknown# to gap#. "unknown" is probably a useful field name in some formats and "gap" describe better what it is. #500
Big decode API internals refactor to split scalars types into their own go types so they can store per type specific values. This also opens up for more ways to make fq both faster and more memory efficient. It also makes the decode API more type safe and makes it possible to experiment with decode DLS that uses chained methods etc. #523

Decoder changes

apple_bookmark New Apple BookmarkData decoder. Thanks David McDonald @dgmcdona. #493

bplist

Fix decoding of UID types
Adds a lost_and_found array with unused values
Fix an endian issue for unicode strings
Add NSKeyedArchiver to JSON helper function from_ns_keyed_archiver, see bplist docs for details on how to use it. Thanks David McDonald @dgmcdona. #493

# decode bplist, from_ns_keyed_archiver converts to JSON plist and then into object data as JSON, find app bookmarks keys and expand them as bookmark data and convert to represented JSON, and finally build path to applications
$ fq -r 'from_ns_keyed_archiver | (.. | .Bookmark? // empty) |= (apple_bookmark | torepr) | .. | .target_path? // empty | join("/")' recentapps.sfl2
System/Applications/Utilities/Terminal.app
Applications/Spotify.app
System/Applications/Calculator.app
System/Applications/Preview.app
Applications/Alacritty.app
Applications/DB Browser for SQLite.app
System/Applications/System Preferences.app
System/Library/CoreServices/Applications/Directory Utility.app
System/Applications/Utilities/Activity Monitor.app
Applications/Safari.app

tzif new Time Zone Information Format decoder. Thanks Takashi Oguma @bitbears-dev. #498

mp4

Map mdta metadata namespace and key names for ilst child boxes. #521

# create object with all ilst key/value pairs
$ fq 'grep_by(.type=="ilst").boxes | map({key: .type, value: .boxes[0].data}) | from_entries' file.mp4
{
  "mdta.encoder": "Lavf59.27.100",
  "mdta.title": "test"
}
# query specific value
$ fq -r 'grep_by(.type=="mdta.encoder").boxes[0].data | tovalue' file.mp4
Lavf59.27.100

Support sidx version 1. #506
Add description and symbolic values for traf sample flags, makes it easier to see and query for I-frames etc. #514

# which boxes has depends_on flags
$ fq 'grep_by(.sample_depends_on) | parent' fragmented.mp4

Support PNG codec mapping. #492
Decode pdin boxes. #524
Decode hnti boxes. #513

mp3_tags Add VBRI support and split into into mp3_frame_xing and mp3_frame_vbri decoders. #525

Changelog

7fa8b63 Add related file format projects to README
4fdb736 Update docker-golang to 1.19.4 from 1.19.3
519eff6 Update github-go-version to 1.19.4 from 1.19.3, 1.19.3, 1.19.3
2a91d29 Update gomod-golang/text to 0.5.0 from 0.4.0
cb15b37 added checks to prevent infinite looping and recursion
c244533 added some sfl2 test files to bplist package
7d13cf7 adds flag parsing to applebookmark
71b17d0 apple bookmarkdata decoder initial commit
8f39ef6 bplist: Harmonize ns_keyed_archive jq style a bit
cba72db bplist: added overload for from_ns_keyed_archiver jq func
129b4b7 bplist: doc: update docs to reflect changes to ns_keyed_archiver
9dab3c6 bplist: minor fix to from_ns_keyed_archiver
448c3ef bplist: update docs with from_ns_keyed_archiver reference, add error case to function
a9047c0 bplist: updates from_ns_keyed_archiver to do automatic torepr based on format detection
4a28e44 changes decoder package name from bookmark to apple_bookmark
d0b044c converts to snake_case and refactors decode helper
d199793 created stack type
e77f776 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
a85da29 decode: Make FieldFormat usage more consistent
9b81d4d decode: More type safe API and split scalar into multiple types
3ec0ba3 decode: add ns_keyed_archiver, restructure apple decoder into apple package
330d5f7 decode: apple_bookmark: simplifies flag decoding
93f2aa5 decode: change PosLoopDetector to use generics
7e98b53 decode: fix type on defer function call, test: add loop.fqtest
a873819 decode: fixes endian of unicode strings
f747873 decode: implements lost and found for unreferenced objects
b45f9fa decode: improve stack push/pop
a162e07 decode: minor change to method receiver name
3232f9c decode: moves PosLoopDetector into its own package
7c9504c decode: moves macho decoder to apple package
7083467 decode: remove dead code from ns_keyed_archiver
7ab4466 decode: remove unused field from decoder, unused parens from torepr
bdb8166 decode: removed unnecessary struct
98eab8c decode: rename parameter for consistency
04379df decode: revert decode.D back, place posLoopDetector in apple_bookmark
7fb674b decode: unexport methods
fa368bb decode: updates all.go with correct macho path
0287ffa decoding well but torepr needs work
42debe5 dev,doc,make: Cleanup makefile and have proper targets for *.md and *.svg
423bab9 dev,test: Use jqtest code from jqjq for jq tests
6fc84a8 doc,dev: Add more usage and dev tips
2fc16ae doc: Add some padding margin to formats table to make it less likely to cause git conflicts
62f377c doc: fixes snippet for recursive bookmark searching
22064f5 doc: remake
4aad2fd doc: remake
b872b1a doc: remake
1e1fc55 fixed one more snake_case letter
d0b76ca fixes broken test and removes long link from markdown body
5146f28 fixes broken test for all.fqtest
253033c fixes broken uid parsing in plist decoder
f535ad3 fixes spacing in jq files
64351e8 fixes tests and adds torepr test
c7d00b8 fixes unknown bit ranges
8f930aa forgot to add bookmark.jq in last commit
164e527 gojq: Update rebased fq fork
6c86945 gojq: Update rebased fq fork
578b84d interp,display: Add workaround for go 1.18 when escaping 0x7f
42d9f2c interp,help: Properly count line length when breaking on whole words
8d69f1f interp: Change default bits_format=string
6c229d7 interp: Don't auto convert to binary for string functions, is just confusing
568afff interp: Fix panic when trigger before any context has been pushed
e3ae144 interp: Rename to/from functions to to_/from_
ba88a68 interp: mimic jq: if expr arg is given read stdin even if tty
9bd65f9 migrates tests to per-sample files
f7d7a49 missed a letter on last commit - converting to snake_case
2f37cb5 mod: Up...

Contributors

pyldin601, dgmcdona, and bitbears-dev

Assets 9

24 Nov 15:34

wader

v0.1.0

a97d37e

v0.1.0

Adds avi decoder and replace raw with more convenient bits and bytes format. Otherwise mostly small updates and bug fixes.

Increase minor version. fq does not use sementic versioning (yet) but it's probably a good idea to start increase minor version when adding features to be able to do patch releases.

In other fq related news:

I gave a talk about fq at No Time To Wait 6 a conference about open media, standardization, and audiovisual preservation.
While prototyping writing decoders directly in jq for fq I ended up implementing jq in jq. Still thinking and working on how to do decoders in jq.

Changes

Replace raw format with bits and bytes format that decodes directly to a binary with different unit size.

$ echo -n 'hello' | fq -d bytes '.[-3:]' > last_3_bytes
$ echo 'hello' | fq -d bytes '.[1]'
101
$ echo 'hello' | fq -c -d bits '[.[range(8)]]'
[0,1,1,0,1,0,0,0]

Decoder changes

avc_au Support annexb format (used in AVI). #476

avi Add AVI (Audio Video Interleaved) decoder. #476

# extract samples for stream 1
$ fq '.streams[1].samples[] | tobytes' file.avi > stream01.mp3

bits Replaces raw but is a binary using bit units. #485
bytes Replaces raw but is a binary using byte units. #485
bplist
- Fix signed interger decoding. #451 @dgmcdona
- Use correct size for references and check for infinite loops. #454 @dgmcdona
flac_frame Correctly decode zero escape sample size. #461
id3v2 Fix decoding of COMM and TXXX with missing null terminator. #468
matroska Updated to latest specification. #455
mp3_frame Use frame size calculation from spec instead of own as it seems to not work in some cases. #480
mp3_frame_tags Replaces xing and also decodes "lame extensions" for both Xing and Info. #481
raw Removed. #485
wav More codec symbol names and now shares RIFF code with AVI decoder. #476
yaml Fix type panic for large intergers. #462

Changelog

7b6492e Improve README.md a bit, one more demo and move up usage
4e06962 Update docker-golang to 1.19.2 from 1.19.1
e033449 Update docker-golang to 1.19.3 from 1.19.2
f3f2648 Update github-go-version to 1.19.2 from 1.19.1, 1.19.1, 1.19.1
003197e Update github-go-version to 1.19.3 from 1.19.2, 1.19.2, 1.19.2
453963d Update github-golangci-lint to 1.50.1 from 1.50.0
56dcb3a Update gomod-BurntSushi/toml to 1.2.1 from 1.2.0
101b280 Update gomod-golang/text to 0.3.8 from 0.3.7
d80f12c Update gomod-golang/text to 0.4.0 from 0.3.8
753927b Update make-golangci-lint to 1.50.1 from 1.50.0
4d8dd5c adds check for recursion in decodeReference, adds test to verify fix
b7c4576 adds necessary cast
46b7ab3 adds test to verify fix
4ee7dd8 changes Errorf to Fatalf on infinite loops
41b2d1a cli: Better decode error help
7254b0f decode,elf,fuzz: TryBytesRange error on negative size
bafd1f5 decode,fuzz: Signed integer (S) read require at least one bit
2a86d32 doc,rtmp,pcap,markdown: Add more examples
0b9b017 doc: Add gomarkdown to license/dependencies
4bfd9d8 doc: Add link to nttw6 presentation video and slides
fb1a91a drop indented else block per lint
4dd594c fixes bad path in test output
f9a1d3f fixes calculation of floating point lengths
236fbc1 fixes reference calculation to use reference size from trailer
ac86f93 fixes signed integer parsing
fb2a2b9 flac,fuzz: Fatal error on negative partition sample count
7859be1 flac_frame: Properly decode zero escape sample size
7cb2a6c fuzz: gotip not needed anymore
cef4245 fuzz: make fuzz GROUP=mp4 to fuzz one group
413d425 gofmt
349d949 gojq: Update rebased fq fork
450f584 gojq: Update rebased fq fork
d8641ab gomod: Update modules that lack bump config
f66e224 id3v2: In the wild COMM and TXXX frame might not have a null terminator
b09d611 makes dictionary key type checking more sensible
d07b2ee markdown,fuzz: Update gomarkdown
646f32d matroska: Fix path tests and make _tree_path more robust
e748079 matroska: Update spec and regenerate
1c7d325 mod: Update ones without bump config
2de8753 mp3_frame: Fix issue calc frame size for some configs
c3a0686 mp3_frame_tags: Refactor and rename xing format to mp3_frame_tags
d75748d mp4: Decode more sample flags
c93301f raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary
b08e25c removes unnecessary cast
2b3adbe renames test data file
0cf46e1 wav,avi,avc_au: Add avi decoder and refactor wav decoder
2606916 yaml,fuzz: gojq.Normalize value to fix type panic

Contributors

dgmcdona

Assets 9

04 Oct 16:36

wader

v0.0.10

8997faa

v0.0.10

Changes

Add bplist Apple Binary Property List decoder. Thanks David McDonald @dgmcdona #427
Add markdown decoder. #422
Fix panic when interrupting (ctrl-c) JSON output (fq tovalue file etc), #440
Fix issue using debug (and some other native go iterator functions) inside path(...), which is used by assign (... = ...) expressions etc. #439
Fix issue and also make toactual and tosym work more similar to tovalue. #432
Fix issue with unknown fields (gaps found after decoding) where one continuous gap could end up split into two of more unknown fields. #431

More format documentation and also nicer help output. Also now all documentation is in markdown format. #430 #422

# or help(matroska) in the REPL
$ fq -h matroska
matroska: Matroska file decoder

Decode examples
===============

  # Decode file as matroska
  $ fq -d matroska . file
  # Decode value as matroska
  ... | matroska

Lookup element using path
=========================

  $ fq 'matroska_path(".Segment.Tracks[0)")' file.mkv

Get path to element
===================

  $ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv

References
==========
- https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
- https://matroska.org/technical/specs/index.html
- https://www.matroska.org/technical/basics.html
- https://www.matroska.org/technical/codec_specs.html
- https://wiki.xiph.org/MatroskaOpus

Decoder changes

ar Allow empty integer strings. For example owner id can be an empty string. #428
bitcoin_blkdat Assert that there is a header. As the format is part of the probe group this speeds up probing. #402

bplist Add Apple Binary Property List decoder.

$ fq '.objects.entries[0] | .key, .value' Info.plist
    │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].key{}:
0x32│               5c            │     \    │  type: "ascii_string" (5) (ASCII encoded string)
0x32│               5c            │     \    │  size_bits: 12
    │                             │          │  size: 12
0x32│                  43 46 42 75│      CFBu│  value: "CFBundleName"
0x3c│6e 64 6c 65 4e 61 6d 65      │ndleName  │
    │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].value{}:
0x1ea│         5f                  │   _      │  type: "ascii_string" (5) (ASCII encoded string)
0x1ea│         5f                  │   _      │  size_bits: 15
0x1ea│            10               │    .     │  large_size_marker: 1 (valid)
0x1ea│            10               │    .     │  exponent: 0
0x1ea│               18            │     .    │  size_bigint: 24
    │                             │          │  size: 24
0x1ea│                  41 70 70 6c│      Appl│  value: "AppleProResCodecEmbedded"
0x1f4│65 50 72 6f 52 65 73 43 6f 64│eProResCod│
0x1fe│65 63 45 6d 62 65 64 64 65 64│ecEmbedded│

Supports torepr

$ fq torepr.CFBundleName Info.plist
"AppleProResCodecEmbedded"

elf

More robust decoding when string tables are missing. #417

# extract entry opcodes and disassemble with ndisasm
$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' \
| ndisasm -b 64 -
00000000  B03C              mov al,0x3c
00000002  66BF0600          mov di,0x6
00000006  0F05              syscall

Now decodes program header notes. #421

markdown Add decoder. Is used in fq to render CLI help. #422

# array with all level 1 and 2 headers
$ fq -d markdown '[.. | select(.type=="heading" and .level<=2)?.children[0]]' README.md
[
  "fq",
  "Usage",
  "Presentations",
  "Install",
  "TODO and ideas",
  "Development and adding a new decoder",
  "Thanks and related projects",
  "License"
]

matroska Add support for sample lacing. Used by FLAC samples etc. #404

mp4 More codec names and also use official names from mp4ra.org. #424

# show details of first two track in file
$ fq -o line_bytes=10 '.tracks[0,1]' big_buck_bunny.mp4
        │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[0]{}: track
0x00910a│20 68 10 01 a0 40 0e 20 8c 1b│ h...@. ..│  samples[0:1295]:
0x009114│c2 2b 99 09 84 42 60 a8 c4 60│.+...B`..`│
*       │until 0x541697.7 (5473678)   │          │
        │                             │          │  id: 1
        │                             │          │  data_format: "mp4a" (MPEG-4 Audio)
        │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[1]{}: track
0x00a5e6│                           00│         .│  samples[0:1440]:
0x00a5f0│00 00 0c 06 00 07 8b 71 b0 00│.......q..│
0x00a5fa│00 03 00 40 80 00 00 00 15 06│...@......│
*       │until 0x540959.7 (5464939)   │          │
        │                             │          │  id: 2
        │                             │          │  data_format: "avc1" (Advanced Video Coding)

html Handle leading doc type and processing directives. #414

Changelog

a77cec9 Added documentation and tests, fixed bad date parsing
d784db6 Adds support for Apple Binary Plist, version 00
5711f29 Code fixes from PR, still need to add tests and testdata
6b04f2d Documentation cleanup
bcccde2 Fixes and embeds documentation
ebae938 Fixes bug in integer parsing
368d183 Size check on nBits to save memory
84ca101 Update docker-golang from 1.19.0 to 1.19.1
c47c386 Update github-go-version from 1.19.0, 1.19.0, 1.19.0 to 1.19.1
816169b Update github-golangci-lint to 1.50.0 from 1.49.0
21f2980 Update make-golangci-lint to 1.50.0 from 1.49.0
5f61994 adds function for decoding fixed sized arrays
f08f44f ar: Integer strings might be empty
004406d bitcoin_blkdat,bitcoin_block: Make sure there is a header if blkdat
421b2b3 bplist: Fix unknown field for singletons and add torepr tests
16b0121 bplist: Make torepr convert to values
fe64530 csv: Add tsv and header example
cb3dc80 decode,tar: Add scalar description and Try* helpers
a6429ff decode: Remove RangeSorted flag as we can decide on array/struct instead
a468684 deps: Manual update ones not using bump
a7a101c doc,help: Nicer format help and move help tests into each format
725ab1b doc,html,xml: Add more documentation and examples
abd19ed doc: Fix format sections a bit
0fdc03a doc: Fix some incorrect example prompts
5382d46 elf: Basic program header notes decoding
12105d8 elf: Treat missing string tables as empty to be more robust
3deceee fixes from PR comments
226a9a3 generics: Use more from x/exp
404b170 gojq: Update fq fork
376f0eb gojq: Update rebased fq fork
87b2c6c help,doc: Use markdown for format documentation again
8016352 html: Handle html with leading doctype etc
768df30 interp,decode: For struct use map to lookup field
c4219d6 interp: Fix interrupt panic for cli eval
00ee10a interp: Make to{actual,sym} behave similar to tovalue
00a5066 markdown: Add decoder
7749e1b matroska: Add proper lacing support
20a1537 mp4: Fix data_format typo
2655ba0 mp4: More codec names (from mp4ra.org)
7cd43b4 perfomance: increase performance by map usage
6a6fec5 range,decode: Use own range sort impl to speed up a bit
0f35fe4 ranges,decode: Correctly skip empty ranges when adding unknown fields
ea81efe readline: Update rebased fq fork
369f401 removed unneccessary type conversions
3198602 removed unused return type
7d86534 sortex: Package with type safe sort helpers
808202f test: Skip go test with -race by default
12836ab updates fqtest
1e47f4f updates tests post integer-bug fix
3d8ea1d updates torepr for data type
1385b5d wasm: Add some documentation
d6316d5 wav: Decode smpl chunk

Contributors

dgmcdona

Assets 9

25 Aug 15:51

wader

v0.0.9

6bbfc66

v0.0.9

Changes

New wasm WebAssembly Binary Format decoder by Takashi Oguma @bitbears-dev

# show part of code_section
$ fq '.sections[4].content.code.x[0].code.e | d' add.wasm
    │00 01 02 03 04 05 06 07 08 09│0123456789│.sections[4].content.code.x[0].code.e[0:4]:
    │                             │          │  [0]{}: in
0x3c│                           20│          │    opcode: "local.get" (0x20)
0x46│01                           │.         │    x: 1 (valid)
    │                             │          │  [1]{}: in
0x46│   20                        │          │    opcode: "local.get" (0x20)
0x46│      00                     │  .       │    x: 0 (valid)
    │                             │          │  [2]{}: in
0x46│         6a                  │   j      │    opcode: "i32.add" (0x6a)
    │                             │          │  [3]{}: in
0x46│            0b               │    .     │    opcode: "end" (0xb)

# count opcode usage
$ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' add.wasm
{
  "end": 1,
  "i32.add": 1,
  "local.get": 2
}

# list exports and imports
$ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' add.wasm
{
  "export": [
    "memory",
    "add"
  ],
  "import": []
}

Decode value display now shows address bar on new format or buffer. Should make it easier to spot changes and read hex and ASCII view. #365

Examples of PCAP with different formats and TCP stream buffers:

...
       │00 01 02 03 04 05 06 07 08 09│0123456789│      packet{}: (ether8023_frame)
0x00668│   00 0a 95 67 49 3c         │ ...gI<   │        destination: "00:0a:95:67:49:3c" (0xa9567493c)
0x00668│                     00 c0 f0│       ...│        source: "00:c0:f0:2d:4a:a3" (0xc0f02d4aa3)
0x00672│2d 4a a3                     │-J.       │
0x00672│         08 00               │   ..     │        ether_type: "ipv4" (0x800) (Internet Prot...
       │00 01 02 03 04 05 06 07 08 09│0123456789│        payload{}: (ipv4_packet)
0x00672│               45            │     E    │          version: 4
...
       │00 01 02 03 04 05 06 07 08 09│0123456789│          payload{}: (tcp_segment)
0x00686│               00 50         │     .P   │            source_port: "http" (80) (World Wide ...
...
       │                             │          │  ipv4_reassembled[0:0]:
       │                             │          │  tcp_connections[0:1]:
       │                             │          │    [0]{}: tcp_connection
       │                             │          │      client{}:
       │                             │          │        ip: "192.168.69.2"
       │                             │          │        port: 34059
       │                             │          │        has_start: true
       │                             │          │        has_end: true
       │                             │          │        skipped_bytes: 0
       │00 01 02 03 04 05 06 07 08 09│0123456789│
  0x000│47 45 54 20 2f 74 65 73 74 2f│GET /test/│        stream: raw bits
  0x000│65 74 68 65 72 65 61 6c 2e 68│ethereal.h│
  *    │until 0x1bc.7 (end) (445)    │          │
...

Add --unicode-output/-U argument to force use of Unicode characters to improve output readability. #377
- For example useful when piping to less and you want fancy unicode and colors:
  
  fq -cU d file | less -r
to_entries now preserves struct field order. #340
Experimental `raw string` literal support. Work the same as golang raw string literals. Useful in REPL when pasting things etc but should probably be avoided in jq scripts. #371
Properly fail lexing of invalid binary, octal and hex number literals. #371
REPL completion now include all functions. Before some functions with multiple argument counts were skipped. #375
Switch to new gopacket fork with speedup and bug fixes. Remove SLL2 workarounds in fq. #386

Decoder changes

csv Correctly handle decode values when tocsv normalize to strings. Before array and object ended up being JSON serialized to strings. #341
- Normalize to strings is done so that non-string scalars can be used:
```
$ fq -n '[[1,true,null,"a"]] | tocsv'
"1,true,,a\n"
```
dns DNS over UDP format was accidentally used to probe TCP streams #366
elf Remove redundant program_header struct
flac
- Add 32 bit samples support. #378 Thanks @ktmf01
- Properly decode/checksum samples in partitions with raw samples. #379 Thanks @ktmf01
  
  Now successfully decodes all test cases from https://github.com/ietf-wg-cellar/flac-test-files
jsonl Add decoder. Decodes JSON lines. There is also fromjsonl and tojsonl. #374
macho
- Split FAT Macho decoding into macho_fat format which also fixed handling of file offsets in sections. #362
- Decode symbol and string sections. #352
matroska Remove new lines in descriptions. Messes up tree. #369
mp3_frame
- Support LSF (low sampling frequency) frames. #376
- Skip trying to figure out what main data is for current frame and not. Was incorrect and doing it properly probably requires hoffman decoding. #376
pcap Support files with nanosecond precision. Has a different magic. #382
prores_frame Add basic decoder. Decodes container and fram header. #396 Thanks @Doctor-love for test files
tar Fix regression when decoding number fields. Now ok again to be empty string. #372
wasm Add WebAssembly Binary Format decoder. #383 Thanks to Takashi Oguma @bitbears-dev
- Decodes to a tree following the WASM binary grammar specification
yaml Fail on trailing data. Before it succeeded with the last value. #373
zip
- Don't require PK header as there seems to be zip files with prepended data. #359
- Correctly limit amount of backwards search for EOCD (end of content directory). #363
xml Correctly handle decode values when toxml normalize to strings. Before array and object ended up being JSON serialized to strings. #341
xml
- Change attribute prefix to @ instead of - and make it an option attribute_prefix. #401
- Skip default namespace in element names. #389
- Always include attributes and children even when empty in array mode. Makes it a lot easer to work with as you can assume .[1] will be attributes and so on. #357
- Normalize to strings is done so that non-string scalars can be used:
```
$ fq -nr '{a: {"@boolean": true, "@number": 123, "@null": null}} | toxml'
<a boolean="true" null="" number="123"></a>
```
- Allow and ignore trailing <?procinstr?> and improve trailing data error message. #368
- Correctly sort if any #seq is found and also properly sort negative #seq. #384

Changelog

0cd846a *extra: Rename extra to just ex and refactor to use generics
fb583e2 Add 32 bps FLAC to test
c1d5b2b Add sample size entry to list for 32bps flac streams
3f209c4 Fix decoding of FLAC raw entropy partition
25061ac Update docker-golang from 1.18.4 to 1.18.5
0de2c90 Update docker-golang from 1.18.5 to 1.19.0
7b8d95b Update github-go-version from 1.18.4, 1.18.4, 1.18.4 to 1.18.5
103991f Update github-go-version from 1.18.5, 1.18.5, 1.18.5 to 1.19.0
4255b87 Update github-golangci-lint from 1.47.2 to 1.47.3
198305e Update github-golangci-lint from 1.47.3 to 1.48.0
fa9fec3 Update github-golangci-lint from 1.48.0 to 1.49.0
f579e9c Update make-golangci-lint from 1.47.2 to 1.47.3
c8069d2 Update make-golangci-lint from 1.47.3 to 1.48.0
004eb56 Update make-golangci-lint from 1.48.0 to 1.49.0
abcc736 add ULEB and SLEB to known words for spell check
9238251 ci: Skip -race for windows and macos
913f578 columnwriter,dump: Add Column interface and refactor into BarColumn and MultiLineColumn
5d9ffea decode,scalar: Map empty string also else sym might ends up nil
326dada decode: Add LEB128 readers
502f451 decode: Refactor to use scalar type assert helper
840292b decode: Simplify compound range sort behaviour
15f7c67 dev,fuzz: Add some useful retrigger snippets
46dca8c dns: Don't use dns (udp) format for tcp also
c233215 dns: Rename isTCP to hasLengthHeader
ed42478 doc,interp: Update and add more examples
f247edb doc: Update README demo a bit with new features
3613b6d elf: Remove redundant program_header struct
8a19978 flac: Make gen script generate correct fqtest files
2bfbe9a flac_frame: Cleanup some dev lefterovers and todos
64b2365 fqtest: Run tests in parallell
af35b28 gojq: Preserve keys order for to_entries when used with JQValue
804ad1e gojq: Update fq fork
add3dcf gojq: Update fq fork, fix scope argcount issue
d898732 gojq: Update fq fork, new scope function, rawstring, stricter integers
394717c gopacket: Switch/update to new fork, remove SLL2 hack
4eae7ff interp,doc: Add -R raw string slurp hint to -s help
d8792fd interp,dump: Correctly flush columns if data will be shown
29005c7 interp,dump: Show address bar for root, nested roots and on format change
c7559b5 interp: Add --unicode-output/-U to force use of unicode
9e447c9 interp: Use RegisterFS instead of format files
701c67c jsonl: Add decoder, also tojsonl encoder
bc6cffd lint,decode,fuzz:: Fix nilerr warnings, one real one should be ignored for now
3c21b05 lint: Fix ioutil deprecation, reformat for new doc standard
b2d4e6d macho: Decode c...

Contributors

Doctor-love, ktmf01, and bitbears-dev

Assets 9

24 Jul 12:13

wader

v0.0.8

055b029

v0.0.8

Changes

Add support for some common structured serialization formats: #284 #335

XML, toxml, fromxml options for indent, jq mapping variants (object or array) and order preservation
HTML, fromhtml options for indent, jq mapping variants (object or array) and order preservation
TOML, totoml, fromtoml
YAML, toyaml, fromyaml

jq-flavored JSON (optional key quotes and trailing comma) tojq, fromjq options for indent #284

# query a YAML file
$ fq '...' file.yml

# convert YAML to JSON
# note -r for raw string output, without a JSON string with JSON would outputted
$ fq -r 'tojson({indent:2})' file.yml

$ fq -nr '{hello: {world: "test"}} | toyaml, totoml, toxml, tojq({indent: 2})'
hello:
    world: test

[hello]
  world = "test"

<hello>
  <world>test</world>
</hello>
{
  hello: {
    world: "test"
  }
}
$ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a"'
b
$ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a" = "<test>" | toxml({indent: 2})'
<doc>
  <element a="&lt;test&gt;"></element>
</doc>

CSV, tocsv, fromcsv options for separator and comment character

$ echo -e  '1,2\n3,4' | fq -rRs 'fromcsv | . + [["a","b"]] | tocsv'
1,2
3,4
a,b

Add support for binary encodings
- Base64. tobase64, frombase64 options for encoding variants.
```
$ echo -n hello | base64 | fq -rRs 'frombase64 | tostring'
hello
```
- Hex string. tohex, fromhex

Add support for text formats

XML entities toxmlentities, fromxmlentities

URL tourl, fromurl

$ echo -n 'https://host/path/?key=value#fragment' | fq -Rs 'fromurl | ., (.host = "changed" | tourl)'
{
  "fragment": "fragment",
  "host": "host",
  "path": "/path/",
  "query": {
    "key": "value"
  },
  "rawquery": "key=value",
  "scheme": "https"
}
"https://changed/path/?key=value#fragment"

URL path encoding tourlpath, fromurlpath
URL encoding tourlencode, fromurlencode
URL query tourlquery, fromurlquery

Add support for common hash functions:
- MD4 tomd4
- MD5 tomd5
```
$ echo -n hello | fq -rRs 'tomd5 | tohex'
5d41402abc4b2a76b9719d911017c592
```
- SHA1 tosha1
- SHA256 tosha256
- SHA512 tosha512
- SHA3 224 tosha3_224
- SHA3 256 tosha3_256
- SHA3 384 tosha3_384
- SHA3 512 tosha3_512
Add support for common text encodings:
- ISO8859-1 toiso8859_1, fromiso8859_1
- UTF8 tutf8, fromutf8
- UTF16 toutf16, fromutf16
- UTF16LE toutf16le, fromutf16le
- UTF16BE toutf16be, fromutf16be
```
$ echo -n 00680065006c006c006f | fq -rRs 'fromhex | fromutf16be'
hello
```
Add group function, same as group_by(.) #299
Update/rebase readline dependency (based on @tpodowd chzyer/readline#207) #305 #308
- Less blinking/redraw in REPL
- Lots of small bug fixes
Update/rebase gojq dependency #247
- Fixes JQValue destructing issue (ex: <some object JQValue> as {$key})
Major rewrite/refactor how native function are implemented. Less verbose and less error-prone as now shared code takes care of type casting and some argument errors. #316
Add tojson($opts) that support indent option. tojson still works as before (no indent).
```
$ echo '{a: 1}' | fq -r 'tojson({indent: 2})'
{
  "a": 1
}
```
Rename --decode-file (will still work) to --argdecode be be more consistent with existing --arg* arguments. #309
On some decode error cases fq can now keep more of partial tree making it easier to know where it stopped #245
Build with go 1.18 #272

Decoder changes

bitcoin Add Bitcoin blkdat, block, transcation and script decoders #239
elf Use correct offset to dynamic linking string table #304

tcp Restructure into separate client/server objects and add skipped_bytes (number of bytes with known missing ACK), has_start (has first byte in stream) and has_end (has last byte in stream) per direction #251

Old:

      │00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection
      │                       │        │  source_ip: "192.168.69.2"
      │                       │        │  source_port: 34059
      │                       │        │  destination_ip: "192.168.69.1"
      │                       │        │  destination_port: "http" (80) (World Wide Web HTTP)
      │                       │        │  has_start: true
      │                       │        │  has_end: true
 0x000│47 45 54 20 2f 74 65 73│GET /tes│  client_stream: raw bits
 0x008│74 2f 65 74 68 65 72 65│t/ethere│
 *    │until 0x1bc.7 (end) (44│        │
 0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│  server_stream: raw bits
 0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│
 *    │until 0x191.7 (end) (40│        │

New:

      │00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection
      │                       │        │  client{}:
      │                       │        │    ip: "192.168.69.2"
      │                       │        │    port: 34059
      │                       │        │    has_start: true
      │                       │        │    has_end: true
      │                       │        │    skipped_bytes: 0
 0x000│47 45 54 20 2f 74 65 73│GET /tes│    stream: raw bits
 0x008│74 2f 65 74 68 65 72 65│t/ethere│
 *    │until 0x1bc.7 (end) (44│        │
      │                       │        │  server{}:
      │                       │        │    ip: "192.168.69.1"
      │                       │        │    port: "http" (80) (World Wide Web HTTP)
      │                       │        │    has_start: true
      │                       │        │    has_end: true
      │                       │        │    skipped_bytes: 0
 0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│    stream: raw bits
 0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│
 *    │until 0x191.7 (end) (40│        │

zip Add 64-bit support and add uncompress option #278
matroska Update and regenerate based on latest spec and also handle unknown ids better #291
mp4 Changes:
- Fix PSSH decode issue #283
- Add track for track_id references without tfhd box
- Makes it possible to see samples in fragments without having an init segment.
  Note it is possible to decode samples in a fragment file by concatenating the init and fragment file ex: cat init frag | fq ....
- Add senc box support #290
- Don't decode encrypted samples #311
- Add track_id to tracks #254
- Add fairplay PSSH system ID #310
- Properly handle trun data offset #294
- Skip decoding of individual PCM samples for now #268
- Add mvhd, tkhd, mdhd and mehd version 1 support #258
- Make sure to preserve sample table order #330
fairplay_spc Add basic FairPlay Server Playback Context decoder #310
avc_pps Correctly check for more rbsp data

Changelog

210940a Update docker-golang from 1.18.1 to 1.18.2
fbeabdc Update docker-golang from 1.18.2 to 1.18.3
51a414d Update docker-golang from 1.18.3 to 1.18.4
3017e8b Update github-go-version from 1.18.1, 1.18.1, 1.18.1 to 1.18.2
c597f7f Update github-go-version from 1.18.2, 1.18.2, 1.18.2 to 1.18.3
dd28392 Update github-go-version from 1.18.3, 1.18.3, 1.18.3 to 1.18.4
d10a361 Update github-golangci-lint from 1.45.2 to 1.46.0
75b5946 Update github-golangci-lint from 1.46.0 to 1.46.1
3ffa9ef Update github-golangci-lint from 1.46.1 to 1.46.2
4be8cb9 Update github-golangci-lint from 1.46.2 to 1.47.0
1b8f4be Update github-golangci-lint from 1.47.0 to 1.47.1
fc596a7 Update github-golangci-lint from 1.47.1 to 1.47.2
62be922 Update gomod-BurntSushi/toml from 1.1.0 to 1.2.0
5db7397 Update make-golangci-lint from 1.45.2 to 1.46.0
456742e Update make-golangci-lint from 1.46.0 to 1.46.1
0675711 Update make-golangci-lint from 1.46.1 to 1.46.2
3d69e9d Update make-golangci-lint from 1.46.2 to 1.47.0
2170925 Update make-golangci-lint from 1.47.0 to 1.47.1
c4199c0 Update make-golangci-lint from 1.47.1 to 1.47.2
02f00be Update usage.md
75169a6 asn1: Add regression test for range decode fix ##330
b0096bc avc_pps: Correct check if there is more rbsp data
5d67df4 avro_ocf: Fix panic on missing meta schema
417255b bitcoin: Add blkdat, block, transcation and script decoder
a6a9713 decode: Cleanup Try/ pairs
3ce660a decode: Keep decode tree on RangeFn error
c4dd518 decode: Make compound range sort optional
8bb4a6d decode: Range decode with new decoder to preserve bit reader
342612e dev: Cleanup linters and fix some unused args
78aa96b dev: Cleanup some code to fix a bunch of new linter warnings
3570f1f doc: Add more related tools
7aff654 doc: Clarify decode, slurp and spew args
0863374 doc: Correct bencode spec URL
10cc551 doc: Improve and cleanup text formats
b100611 doc: Typos and add note about Try* functions
c27646a doc: Update and shorten README.md a bit
b038872 doc: Use singular jq value to refer to jq value
a980656 doc: go 1.18 and improve intro text a bit
a64c28d dump: Skip JQValueEx if there are not options
40481f6 elf,fuzz: Error on too large string table
f66a359 elf: Use correct offset to dynamic linking string table
64f3e5c fairplay: Add basic SPC decoder and PSSH system id
cae288e format,intepr: Refactor json, yaml, etc into formats also move out related functions
e9d9f8a fq: Use go 1.18
377af13 fqtest: Cleanup path usage
2464ebc fuzz: Replace built tag wi...

Contributors

tpodowd

Assets 9

01 May 15:50

wader

v0.0.7

19617a6

v0.0.7

Changes

Format specific options
- Formats can now have own options
- Example to skip decoding of samples in a mp4 file use:
- fq -d decode_samples=false d file.mp4 or ... | mp4({decode_samples: false})
- To see supported options for a formats see formats documentation, use fq -h mp4 or help(mp4) in the REPL.
gojq fork rebase:
- Many performance improvements from upstream
- Assign to a JQValue will now shallowly turn it into a jq value and then be assigned.
- Refactor and rewrote large parts to make it easier to rebase and maintain in the future.

Decoder changes

amf0 Add Action Message Format 0 decoder #214
hevc_pps Add H.265/HEVC Picture Parameter Set decoder #210
hevc_sps Add H.265/HEVC Sequence Parameter Set decoder #210
hevc_vpc Add H.265/HEVC Video Parameter Set decoder #210
mp3 Add max_unique_header_config and max_sync_seek options #242
mp4 Simplify granule structure a bit #242
mp4 Add decode_samples and allow_truncate options #242
flac_frame Add bits_per_sample option #242
icmpv6 Add Internet Control Message Protocol v6 decoder #216
id3v2 Add v2.0 PIC support
ipv6_packet Add Internet protocol v6 packet decoder #216
macho Remove redundant arch struct level and cleanup some sym values #226
macho Add raw fields for section and encryption info #238
mp4 Add more HEIF boxes support #221
mpeg_pes Support MPEG1 #219
rtmp Add Real-Time Messaging Protocol decoder. Only plain RTMP for now. #214
matroska Symbol name cleanup #220
tcp Better port matching and make it possible to know if byte stream has start/end. #223
udp Better port matching #223

Changelog

010f643 Update docker-golang from 1.17.8 to 1.18.0
05096f5 Update docker-golang from 1.18.0 to 1.18.1
e5f61e2 Update github-go-version from 1.17.7, 1.17.7, 1.17.7 to 1.18.0
fdfc5c5 Update github-go-version from 1.18.0, 1.18.0, 1.18.0 to 1.18.1
4ea362e Update github-golangci-lint from 1.44.2 to 1.45.0
2a90485 Update github-golangci-lint from 1.45.0 to 1.45.2
d9195ac Update gomod-mapstructure from 1.4.3 to 1.5.0
cf88bc1 Update make-golangci-lint from 1.44.2 to 1.45.0
3a0799c Update make-golangci-lint from 1.45.0 to 1.45.2
34cbe48 amf0: Decode strings in more detail
b2a865e avc_sps: Add chroma format name mapping
b35b180 decode,format: Add d.FieldFormatOrRaw(Len)
f4480c6 decode,interp: Support for format specific options
5ff67e4 formats: Sym and field name cleanup to be more jq friendly
3c02992 github: Update action versions
02a97fa gojq: Rebase fq fork
2e24044 gojq: Rebase fq fork
518f6af gojq: Rebase fq fork
88f791e gojq: Rebase fq fork
8c91870 gojq: Rebase fq fork
adde8c7 gojq: Rebase fq fork
d79afeb gojq: Rebase fq fork
dd0d97e gojq: Rebase fq fork (speedup and fix range with JQValue)
afd724b gojq: Rebase fq fork. Fixes JQValue path tracking when iterating
74978c9 hevc: Add hevc_vps, hevc_sps and hevc_pps decoders
c020248 hevc_vpc,hevc_sps: Use same nameing for profile as in spec
09385c6 id3v2: Add 2.0 PIC support
9cb4b57 interp,cli: Handle ctrl-c properly
607202b interp: Don't truncate last display column
6f03471 interp: Paths with a array as root was missing start dot
dabad85 interp: Proper display column truncate
e8678ca interp: Remove opts refactor leftover
d376520 interp: Remove to*range pad argument and fix stdout padding issue
087d124 interp: Simpler and more efficient hexdump
21ad628 interp: dump: Show field name for compound values in arrays
e8dc711 ipv6,icmpv6: Add decoder
d6c31da macho: Add section and encryption_info raw data fields
5424eed macho: Cleanup syms and remove redundant fat_arch struct
f8d79a5 matroska: More sym cleanup
f34ebd8 mp4: Add more HEIF boxes
f8fd6b7 mp4: Add more HEIF boxes
39ba5c4 mpeg_pes: Support mpeg1 and some cleanup
d8aaf30 rtmp,amf0: Add decoders
788b0ac rtmp,amf0: Improve decoders, aac asc, chunk stream interrupt, fix amf0 ecma arrays
5d25bbc tcp,udp: Refactor and make port matching better

Assets 9

11 Mar 16:14

wader

v0.0.6

ad0e3f5

v0.0.6

Added macho decoder (thanks @Akaame), nicer REPL interrupt, error and prompt, add slurp/spew functions and explode for binary.

Added fq talk slides from Binary Tools Summit 2022 to README.md.

Changes

Major query rewrite refactor to share code for slurp-ish functions repl, slurp and future help system. #178
REPL improvements:
- Much improved eval and output interrupt. Should fix more or less all issue with un-interruptable long outputs. It is still possible to get "hangs" if some decode value ends up being expanded into a huge string etc. #191 #192
- Prompt paths now has colors support. #181
- Shows an arrow on parse error.
- Faster on multi inputs. #165
Speedup interpeter by skipping redundant includes. #172
gojq fork rebase: #179
- Fixes try ... | ... catch precedence issue.
- tonumber now supports non-base-10 numbers.
Add slurp/spew to collect outputs and outputs them later.
- 1,2,3 | slurp("a") collects, later do spew("a") to output them. Also a global array $a will be available. #178
Add explode for binary. #188
- "a" | tobits | explode return bits [0,1,1,0,0,0,0,1].
- "åäö" | tobytes | explode return utf8 bytes instead of codepoints [195,165,195,164,195,182].
Add optional sub topic to --help: #177
- Replace --formats with --help formats. #181
- Add --help options to see all default option values. #181
Remove var, use slupr instead.

Decoder changes

macho Add decoder. Thanks @Akaame #43
mp4 Support colr box. #176

Changelog

ee5e471 Update docker-golang from 1.17.7 to 1.17.8
ca04cc2 Update github-golangci-lint from 1.44.0 to 1.44.1
5c6e1d3 Update github-golangci-lint from 1.44.1 to 1.44.2
1b8e693 Update make-golangci-lint from 1.44.0 to 1.44.1
9d5ba82 Update make-golangci-lint from 1.44.1 to 1.44.2
cd2cbef decode: Some cleanup
9e4f264 dev: Add .jq-lsp.jq to add additional builtins for jq-lsp
c6a90cf doc,asn1_ber: Add more documentation
c53bd77 doc: Add bts2022 video
b97776c doc: Add fq bts2022 presentation
d334c2d doc: Add href in supported format list
c95b0d6 doc: Forgot make doc
a202df9 doc: Improve and fix some typos
9ec1d35 doc: Improve project description
758b2d0 doc: Regenerate after macho merge
920629f doc: Regenerate and fix macho section size
d3397cf doc: Tweak format diagram
d47e04c fixup! macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff
27e7615 format: Simplify torepr, no need for _f function
206dcd0 fuzz: Include more testdata seed files
be6f009 gojq: Rebase fq fork and add support for non-10 base for tonumber
33efb02 interp,repl: Add path and value colors to prompt
41551de interp,repl: Improved eval and output interrupt
dff7e7d interp: Cleanup binary regexp overloading and add explode
fe8183b interp: Color parse in jq
6f10745 interp: Fix interrupt regression after query rewrite refactor
f66f115 interp: Make _finally handle null, call fin once and last
eeb5915 interp: Make help output less wide
9dc59e5 interp: Move _is_decode_value to jq
0bc1171 interp: Move opts eval to options.jq
3f50bb9 interp: Rework formats and options help
03f450f interp: Skip redundant includes
c5918d2 macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff
5c97420 macho: TS string to UTC
04eae93 macho: add basic docs
5e95d1c macho: add cpuSubTypes
2638f41 macho: add darwin_amd64 test
5c5bd87 macho: add fqtest actualization
bf214d5 macho: add nolint suppression to const defs
333a324 macho: add scalar.Hex mapper to addr fields
a86e704 macho: add section type parsing
90b9463 macho: adopt plural-singular scheme for FieldStructArrayLoop
b78ed02 macho: barebones decoder impl
e199d21 macho: basic impl for ar and fat file parsing
66feebc macho: change parseFlags impl for ordered results
b5fe9ce macho: change registry description
20e5be3 macho: delete ar decoder code
efdd0bf macho: discard lc_ and lowercase command names
b0911af macho: docs review changes
a29bfca macho: expand filetypes and header flags
fb0654e macho: fix FieldUTF8NullFixedLen for segname
d1f093c macho: fix fat header decode bug
0d64892 macho: fix null in segname sectname
9eb71dc macho: generate doc via make doc
3991c51 macho: handle unknown lc_commands better
ef2919b macho: introduce arm and fat tests
98c9840 macho: linting changes for ar parse
1feb81c macho: little-endian to little_endian
141a8e8 macho: mach_header_X to header
9206d9d macho: magicToHex to scalar.Hex
2021b05 macho: make actual
70b84cd macho: ntools fix LC_MAIN fix
78699f3 macho: parse flags individually
4016ad0 macho: parse segment section flags
5a48cb3 macho: refactor prebound_dylib
2e7767c macho: remake docs
3334750 macho: reuse ar decoder
228757b macho: review fixes
5ee9a23 macho: review fixes
e3daee7 macho: simplify thread state decoder
70c9d51 macho: thread state visualization
a4789dc macho: timestamp mapper
2ccb808 macho: update test cases v to dv
74abe99 macho: update tests
12eb7cc macho: use FieldUTF8NullFixedLen
5f4ad41 macho: use FieldUTF8NullFixedLen for segname
f8690e6 mp4: Add colr box support
b157751 mp4: Reformat and use dv in test
0a043f9 repl,interp: Refactor repl and slurp
ca8cdad repl: Add comments and query from/to helper
9cb4205 repl: Correct error arrow position in color mode
e238f29 repl: Speedup multi input to sub-repl
56ae4a0 test: Make expect cli test more robust

Contributors

Akaame

Assets 9

13 Feb 22:58

wader

v0.0.5

ba94821

v0.0.5

Improved binary slicing and bit reading, avro_ocf decoder (thanks @xentripetal), asn1_ber decoder, renamed display aliases, new grep_by and paste function.

Changes

Big internal bit reader refactor. Now much more consistent code and fixes some issues reading and decoding of binary arrays and binary slices. #123
- Bit reading and IO have been moved to a bitio package.
- Non-simple bit reading have been move out of bitio to decode package.
- [0,1,1,0,0,1,1,0,0,1,1,1,0,0,0,1 | tobits] | tobytes | tostring returns "fq".
- [.frames[0], .frames[-1]] | mp3 decode mp3 based on first and last frame of other mp3.
Add grep_by that recursively selects using a filter condition and ignores errors. #102
- grep_by(.type == "trak") finds all objects where .type is "trak" (all mp4 track boxes).
- grep_by(tonumber | . >= 40 and . <= 100) find all numbers between 40 and 100.
- grep_by(format == "jpeg") find all jpegs.
Add paste function to read string from stdin util ^D (EOF). Can be used to paste in REPl etc. #143
- paste | frompem | asn1_ber | repl wait for PEM encoded text on stdin (paste and press ^D), decode it with asn1_ber and start a sub-REPL with the result.
- paste | fromjson decode pasted JSON.
- eval(paste) eval pasted jq expression.
Cleanup display aliases. Remove v and f, add da, dd, dv and ddv. #112
- d/d($opts) display value and truncate long arrays and buffers
- da/da($opts) display value and don't truncate arrays
- dd/dd($opts) display value and don't truncate arrays or buffers
- dv/dv($opts) verbosely display value and don't truncate arrays but truncate buffers
- ddv/ddv($opts) verbosely display value and don't truncate arrays or buffers
Refactor radix into toradix($base)/fromradix($base). #139
Remove number_to_bytes. Can be done with tobytes. #139
Change tobytes to zero pad most significant bits to byte alignment if needed. #133
Add tobytes/tobits variant that takes an argument to add extra padding. #133
- 0xf | tobytes 8 bit binary with last 4 bits set
- 0xf | tobytes(4) 32 bit binary with last 4 bits set
- 0xf | tobits(12) 12 bit binary with last 4 bits set
Rename fq type buffer to binary as it makes more sense. #133
Add topem/frompem to work with PEM encoding. #92
Add Windows scoop install. #137 Thanks @thushan
Add display, decode value, binary and binary array documentation. #118 #136 #133
Add decode API documentation. #149
Improved REPL completion for keys. #144
Add -o force=<bool> option that sets force decode option. Same as mp4({force: true}). #126

Decoder changes

avro_ocf Add decoder. #38 Thanks @xentripetal
- Full avro OCF support. Handles all primitive, complex, and logical types besides decimals.
- Able to handle deflate, snappy, and null codecs for blocks.
asn1_ber Add decoder. #92
- Also decodes CER and DER (X.690) but with no additional validation at the moment.
- Support all types but real type is currently limited to range for 64 bit interger/float.
- Has torepr support.
- No schema support.
aac_frame Only decode object types we know about. #130
mp3 Shorter sync find heuristics. #104
mp4 Add stz2 support
mp4 Add pnot (preview container) and jP (JPEG 2000) signature. #125

Also thanks to @Doctor-love for keeping things tidy.

Changelog

6fc1efd Add test case with all data types
ae4a624 Adds Windows Scoop instructions for fq.
4b809a7 Change avro codec to funcs
66ca1f1 Change tests to use new verbose syntax
7345b8c Cleanup
07ddf36 Cleanup for linting
4508241 Cleanup snappy
0909fb6 Comment on snappy decompression
21cfc70 Dates need to specify UTC too
75b8496 Fix lint
7a8e3ca Hook into registry, add codecs
251053e Initial pass on logical types
2605bce Lint and add basic doc
ee18407 Parse header using avro decoders. Still not certain this is the best idea. Will get opinions before finalizing.
ab50088 Polish of problem template and clarifying questions
27789f2 Regenerate docs
5a1d35e Remove redudant question and fix typo
31c4c0d Support snappy and deflate codecs
0300c95 Take heading off doc to match make doc format
6f57cdb Timestamps should be UTC
06085a2 Undo change to doc/file.mp4. I have no idea how this got changed in the first place? Maybe some macos shenanigans.
d137a72 Update docker-golang from 1.17.6 to 1.17.7
267e30e Update github-go-version from 1.17.6, 1.17.6 to 1.17.7
1e859cd Update github-golangci-lint from 1.43.0 to 1.44.0
16849c8 Update linting
d02d896 Update make-golangci-lint from 1.43.0 to 1.44.0
68e85a2 Use existing scalar description helper
3bab3d6 aac_frame: Only try decode object types we know about
0829c16 asn1_ber: Add decoder
0312c92 asn1_ber: Add more doc and multiple outputs for frompem
06245d1 binary,decode,doc: Rename buffer to binary and add some documentation
7c52153 bitio,decode: Refactor bitio usage and make buffer slicing more correct
0d74e87 bitio,doc: Even more cleanup
d854ed5 bitio: Cleanup documentation a bit
82aeb35 bitio: More doc cleanup
01ecde6 bump: Add snappy config
de64a99 cleanup some docs, change enum to mapper, error zigzag on more than 8 bytes
6cd1c38 decode,scalar: Add scalar.Str{Uint/Int/F}ToSym to parse numbers
4ab6381 decode: Add scalars args to FieldRootBitBuf
be71eb0 decode: Rename LenFn() to LimitedFn, add FramedFn and document
7bc2521 doc,interp: Add some example usages to cli help
8e47fb1 doc,matroska: Fix filesname in example
c15f528 doc: Add format links to format table
b86da7a doc: Add inital decoder API documentation
49c90f8 doc: Add macOS security notes and move supported format up a bit
06b67e4 doc: Add more license details
a8664ed doc: Add per format documentation
0955262 doc: Add snappy license
3630785 doc: Cleanup and add more decode value and binary documentation
710c29b doc: Color edges in diagram based on dest
f0ce717 doc: Document display and some more jq hints
b350468 doc: More decode API details and polish
6b51b06 doc: More display alias leftover fixes
dd3e40f doc: Unbreak formats_digaram.jq since radix change
c52a1a2 doc: Use f($a; $b) instead of jq f/2 notation
233d86a fq: Add arch and os to --version
b8efd8e fuzz: Fuzz all formats
e1bdfdf fuzz: List seed numbers and make it build again
6090b65 fuzz: Make it compile again and run one format per fuzz
aea4884 github: Add basic issue template
b55ca2c gojq: Rebase fq branch
47c978e goreleaser: Use zip for macos
8537117 id3v2: Should assert not validate magic
d6ca481 initial work for avro OCF files
ca68e6a interp: Add Platform() method to OS interface
e792598 interp: Add grep_by/1 to recursively match using a filter
0a1a561 interp: Add missing default opts for tovalue
48a19cb interp: Add paste function to allow pasting text into REPL etc
fc0aacb interp: Cleanup display aliases, now: d, da, dd, dv, ddv
bf7a483 interp: Fix handling of group decode error from stdin
26d9650 interp: Refactor radix* into toradix($base)/fromradix($base)
366f6b1 interp: Support force decode as -o force=true
77ab667 interp: Use absolute path in errors
c31ec2a interp: Use correct sym color
898dfec lint: Fix typeassert and case exhaustive warnings
d540116 make doc
bf170be make: Cleanup some not very used targets
8d2d88f mp3: Decrease max sync seek length between frames to 4k
d555c32 mp4,fuzz: Fatal error on infinite sgpd box entries
45b00aa mp4: Add stz2 support
092609b mp4: Add video preview (pnot) and JPEG 2000 (jP) signatures
febce5a mpeg_spu: Fatal error on infinite loop
c58ba28 mpeg_spu: Fatal error on unknown cmd
d1943da pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0
2ab395a protobuf: Add note about sub message decoding
af05381 repl,interp: Make stdio work during completion
bd9be2c repl: Fix completion of non-underscore extkeys
69c745d simplify scalar usage
778a1a4 zip: Assert signature not validate

(Some commits have been removed from list for clarity)

Contributors

thushan, Doctor-love, and xentripetal

Assets 9

21 Jan 14:38

wader

v0.0.4

04dd153

v0.0.4

Changes

Add a torepr/0 function that converts decoded value into what it represents.
Initially works for:bencode, cbor, bson and msgpack #74
Example usage:
fq torepr file.cbor
fq -i torepr file.cbor
fq torepr.field file.cbor
fq 'torepr | .field | ...' file.cbor
Add stderr/0 function for jq compatibility #58
Bitwise operators band, bor etc are now normal functions instead of operators. Was done to be syntax compatible with jq #61
Uses the same convention as jq math functions: Unary uses input 123 | bnot, more than one argument all as arguments band(123; 456)
Decode API now supports null values #81
Decode API now supports arbitrary large integers #79
TCP reassembly now supports streams with missing SYN/ACK #57
Update readline package to version with less dependencies #83
Make REPL prompt more jqish #70

Decoder changes

bencode Add decoder #64
cbor Add decoder #75
msgpack Add decoder #69
mp4 Much improved sample decode #82 #78
png Decode PLTE and tRNS chunks #59
tar Don't assume there is a end marker and support more than 2 blocks #86 #87

Also thanks to @Doctor-love for keeping things tidy.

Changelog

af8e7ef bencode: Add decoder
0b0f28e cbor: Add decoder
1383b41 decode,interp: Add arbitrary large integer support (BigInt)
548a065 decode,interp: Finish up nil value support
ff5c0b8 decode: Error on negative number of bits when reading numbers
cf8a50c decode: Use stable sort for values to not change order or values with same range start
b4694b6 doc,dev: Add some more decoder implementation help
0c1716b doc: Add alpine and go run
809210b doc: Add more dev tips
59b8803 doc: Document dev dependencies and related PRs/issues etc
6ca4767 doc: Improve formats graph a bit
8e9700d doc: Improve readme a bit and add torepr example
0cf486d elf: fix all-platforms naming typo
263f1ae flac: Don't allow zero subframe sample size
729a6ca formats: Sort and make lists less likely to cause collision
78c0775 fq: Embed version in source
aa7adb6 fq: Update version to 0.0.4
7461264 fuzz: Skip other tests when fuzzing
be0ef80 interp,fq: Make bit operators normal functions
a3cfcd0 interp: Add stderr again for jq compat
149cb3f interp: Add torepr/0 that converts decode value into what it reptresents
b3a0980 interp: Document bit opts funcs and add some error tests
8d10423 make: Fix quote issue in release script
4a1e859 mp4: Improved stsz handling
61bf2ce mp4: Refactor sample decode into something more sane
a6bf62c msgpack: Add decoder
edad481 num,mathextra: Rename num package to mathextra
bfc977b png: Decode PLTE and tRNS chunks and cleanup syms a bit
36d2891 readline: Update to verison with less deps
9770b00 repl: Make prompt for array and iter more jqish
ba1edef tar: Allow more than 2 zero end blocks at end
5921d76 tar: Don't assume there is a end marker
edd0ae1 tcp,flow: By default allow missing syn/ack for now

(Some commits have been removed from list for clarity)

Contributors

Doctor-love

Assets 9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes

Decoder changes

Changelog

Changes

Decoder changes

Changelog

Contributors

Changes

Decoder changes

Changelog

Contributors

Changes

Decoder changes

Changelog

Contributors

Changes

Decoder changes

Changelog

Contributors

Changes

Decoder changes

Changelog

Contributors

Changes

Decoder changes

Changelog

Changes

Decoder changes

Changelog

Contributors

Changes

Decoder changes

Contributors

Changes

Decoder changes

Changelog

Contributors

Releases: wader/fq

v0.3.0

Changes

Decoder changes

Changelog

v0.2.0

Changes

Decoder changes

Changelog

Contributors

v0.1.0

Changes

Decoder changes

Changelog

Contributors

v0.0.10

Changes

Decoder changes

Changelog

Contributors

v0.0.9

Changes

Decoder changes

Changelog

Contributors

v0.0.8

Changes

Decoder changes

Changelog

Contributors

v0.0.7

Changes

Decoder changes

Changelog

v0.0.6

Changes

Decoder changes

Changelog

Contributors

v0.0.5

Changes

Decoder changes

Contributors

v0.0.4

Changes

Decoder changes

Changelog

Contributors