Skip to content

v0.0.10
Compare
Choose a tag to compare
@wader wader released this 04 Oct 16:36
· 1405 commits to master since this release
v0.0.10
8997faa
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Changes

  • Add bplist Apple Binary Property List decoder. Thanks David McDonald @dgmcdona #427
  • Add markdown decoder. #422
  • Fix panic when interrupting (ctrl-c) JSON output (fq tovalue file etc), #440
  • Fix issue using debug (and some other native go iterator functions) inside path(...), which is used by assign (... = ...) expressions etc. #439
  • Fix issue and also make toactual and tosym work more similar to tovalue. #432
  • Fix issue with unknown fields (gaps found after decoding) where one continuous gap could end up split into two of more unknown fields. #431
  • More format documentation and also nicer help output. Also now all documentation is in markdown format. #430 #422 
    # or help(matroska) in the REPL
$ fq -h matroska
matroska: Matroska file decoder

Decode examples
===============

  # Decode file as matroska
  $ fq -d matroska . file
  # Decode value as matroska
  ... | matroska

Lookup element using path
=========================

  $ fq 'matroska_path(".Segment.Tracks[0)")' file.mkv

Get path to element
===================

  $ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv

References
==========
- https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
- https://matroska.org/technical/specs/index.html
- https://www.matroska.org/technical/basics.html
- https://www.matroska.org/technical/codec_specs.html
- https://wiki.xiph.org/MatroskaOpus

Decoder changes

  • ar Allow empty integer strings. For example owner id can be an empty string. #428
  • bitcoin_blkdat Assert that there is a header. As the format is part of the probe group this speeds up probing. #402
  • bplist Add Apple Binary Property List decoder. 
    $ fq '.objects.entries[0] | .key, .value' Info.plist
    │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].key{}:
0x32│               5c            │     \    │  type: "ascii_string" (5) (ASCII encoded string)
0x32│               5c            │     \    │  size_bits: 12
    │                             │          │  size: 12
0x32│                  43 46 42 75│      CFBu│  value: "CFBundleName"
0x3c│6e 64 6c 65 4e 61 6d 65      │ndleName  │
    │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].value{}:
0x1ea│         5f                  │   _      │  type: "ascii_string" (5) (ASCII encoded string)
0x1ea│         5f                  │   _      │  size_bits: 15
0x1ea│            10               │    .     │  large_size_marker: 1 (valid)
0x1ea│            10               │    .     │  exponent: 0
0x1ea│               18            │     .    │  size_bigint: 24
    │                             │          │  size: 24
0x1ea│                  41 70 70 6c│      Appl│  value: "AppleProResCodecEmbedded"
0x1f4│65 50 72 6f 52 65 73 43 6f 64│eProResCod│
0x1fe│65 63 45 6d 62 65 64 64 65 64│ecEmbedded│
    • Supports torepr
    $ fq torepr.CFBundleName Info.plist
"AppleProResCodecEmbedded"
  • elf
    • More robust decoding when string tables are missing. #417
      # extract entry opcodes and disassemble with ndisasm
$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' \
| ndisasm -b 64 -
00000000  B03C              mov al,0x3c
00000002  66BF0600          mov di,0x6
00000006  0F05              syscall
    • Now decodes program header notes. #421
  • markdown Add decoder. Is used in fq to render CLI help. #422 
    # array with all level 1 and 2 headers
$ fq -d markdown '[.. | select(.type=="heading" and .level<=2)?.children[0]]' README.md
[
  "fq",
  "Usage",
  "Presentations",
  "Install",
  "TODO and ideas",
  "Development and adding a new decoder",
  "Thanks and related projects",
  "License"
]
  • matroska Add support for sample lacing. Used by FLAC samples etc. #404
  • mp4 More codec names and also use official names from mp4ra.org. #424
    # show details of first two track in file
$ fq -o line_bytes=10 '.tracks[0,1]' big_buck_bunny.mp4
        │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[0]{}: track
0x00910a│20 68 10 01 a0 40 0e 20 8c 1b│ h...@. ..│  samples[0:1295]:
0x009114│c2 2b 99 09 84 42 60 a8 c4 60│.+...B`..`│
*       │until 0x541697.7 (5473678)   │          │
        │                             │          │  id: 1
        │                             │          │  data_format: "mp4a" (MPEG-4 Audio)
        │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[1]{}: track
0x00a5e6│                           00│         .│  samples[0:1440]:
0x00a5f0│00 00 0c 06 00 07 8b 71 b0 00│.......q..│
0x00a5fa│00 03 00 40 80 00 00 00 15 06│...@......│
*       │until 0x540959.7 (5464939)   │          │
        │                             │          │  id: 2
        │                             │          │  data_format: "avc1" (Advanced Video Coding)
  • html Handle leading doc type and processing directives. #414

Changelog

  • a77cec9 Added documentation and tests, fixed bad date parsing
  • d784db6 Adds support for Apple Binary Plist, version 00
  • 5711f29 Code fixes from PR, still need to add tests and testdata
  • 6b04f2d Documentation cleanup
  • bcccde2 Fixes and embeds documentation
  • ebae938 Fixes bug in integer parsing
  • 368d183 Size check on nBits to save memory
  • 84ca101 Update docker-golang from 1.19.0 to 1.19.1
  • c47c386 Update github-go-version from 1.19.0, 1.19.0, 1.19.0 to 1.19.1
  • 816169b Update github-golangci-lint to 1.50.0 from 1.49.0
  • 21f2980 Update make-golangci-lint to 1.50.0 from 1.49.0
  • 5f61994 adds function for decoding fixed sized arrays
  • f08f44f ar: Integer strings might be empty
  • 004406d bitcoin_blkdat,bitcoin_block: Make sure there is a header if blkdat
  • 421b2b3 bplist: Fix unknown field for singletons and add torepr tests
  • 16b0121 bplist: Make torepr convert to values
  • fe64530 csv: Add tsv and header example
  • cb3dc80 decode,tar: Add scalar description and Try* helpers
  • a6429ff decode: Remove RangeSorted flag as we can decide on array/struct instead
  • a468684 deps: Manual update ones not using bump
  • a7a101c doc,help: Nicer format help and move help tests into each format
  • 725ab1b doc,html,xml: Add more documentation and examples
  • abd19ed doc: Fix format sections a bit
  • 0fdc03a doc: Fix some incorrect example prompts
  • 5382d46 elf: Basic program header notes decoding
  • 12105d8 elf: Treat missing string tables as empty to be more robust
  • 3deceee fixes from PR comments
  • 226a9a3 generics: Use more from x/exp
  • 404b170 gojq: Update fq fork
  • 376f0eb gojq: Update rebased fq fork
  • 87b2c6c help,doc: Use markdown for format documentation again
  • 8016352 html: Handle html with leading doctype etc
  • 768df30 interp,decode: For struct use map to lookup field
  • c4219d6 interp: Fix interrupt panic for cli eval
  • 00ee10a interp: Make to{actual,sym} behave similar to tovalue
  • 00a5066 markdown: Add decoder
  • 7749e1b matroska: Add proper lacing support
  • 20a1537 mp4: Fix data_format typo
  • 2655ba0 mp4: More codec names (from mp4ra.org)
  • 7cd43b4 perfomance: increase performance by map usage
  • 6a6fec5 range,decode: Use own range sort impl to speed up a bit
  • 0f35fe4 ranges,decode: Correctly skip empty ranges when adding unknown fields
  • ea81efe readline: Update rebased fq fork
  • 369f401 removed unneccessary type conversions
  • 3198602 removed unused return type
  • 7d86534 sortex: Package with type safe sort helpers
  • 808202f test: Skip go test with -race by default
  • 12836ab updates fqtest
  • 1e47f4f updates tests post integer-bug fix
  • 3d8ea1d updates torepr for data type
  • 1385b5d wasm: Add some documentation
  • d6316d5 wav: Decode smpl chunk

Contributors

dgmcdona
Assets 9
Loading