Skip to content

v0.0.9
Compare
Choose a tag to compare
@wader wader released this 25 Aug 15:51
· 1497 commits to master since this release
v0.0.9
6bbfc66
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Changes

  • New wasm WebAssembly Binary Format decoder by Takashi Oguma @bitbears-dev
    # show part of code_section
$ fq '.sections[4].content.code.x[0].code.e | d' add.wasm
    │00 01 02 03 04 05 06 07 08 09│0123456789│.sections[4].content.code.x[0].code.e[0:4]:
    │                             │          │  [0]{}: in
0x3c│                           20│          │    opcode: "local.get" (0x20)
0x46│01                           │.         │    x: 1 (valid)
    │                             │          │  [1]{}: in
0x46│   20                        │          │    opcode: "local.get" (0x20)
0x46│      00                     │  .       │    x: 0 (valid)
    │                             │          │  [2]{}: in
0x46│         6a                  │   j      │    opcode: "i32.add" (0x6a)
    │                             │          │  [3]{}: in
0x46│            0b               │    .     │    opcode: "end" (0xb)
    # count opcode usage
$ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' add.wasm
{
  "end": 1,
  "i32.add": 1,
  "local.get": 2
}
    # list exports and imports
$ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' add.wasm
{
  "export": [
    "memory",
    "add"
  ],
  "import": []
}
  • Decode value display now shows address bar on new format or buffer. Should make it easier to spot changes and read hex and ASCII view. #365

    Examples of PCAP with different formats and TCP stream buffers: 
    ...
       │00 01 02 03 04 05 06 07 08 09│0123456789│      packet{}: (ether8023_frame)
0x00668│   00 0a 95 67 49 3c         │ ...gI<   │        destination: "00:0a:95:67:49:3c" (0xa9567493c)
0x00668│                     00 c0 f0│       ...│        source: "00:c0:f0:2d:4a:a3" (0xc0f02d4aa3)
0x00672│2d 4a a3                     │-J.       │
0x00672│         08 00               │   ..     │        ether_type: "ipv4" (0x800) (Internet Prot...
       │00 01 02 03 04 05 06 07 08 09│0123456789│        payload{}: (ipv4_packet)
0x00672│               45            │     E    │          version: 4
...
       │00 01 02 03 04 05 06 07 08 09│0123456789│          payload{}: (tcp_segment)
0x00686│               00 50         │     .P   │            source_port: "http" (80) (World Wide ...
...
       │                             │          │  ipv4_reassembled[0:0]:
       │                             │          │  tcp_connections[0:1]:
       │                             │          │    [0]{}: tcp_connection
       │                             │          │      client{}:
       │                             │          │        ip: "192.168.69.2"
       │                             │          │        port: 34059
       │                             │          │        has_start: true
       │                             │          │        has_end: true
       │                             │          │        skipped_bytes: 0
       │00 01 02 03 04 05 06 07 08 09│0123456789│
  0x000│47 45 54 20 2f 74 65 73 74 2f│GET /test/│        stream: raw bits
  0x000│65 74 68 65 72 65 61 6c 2e 68│ethereal.h│
  *    │until 0x1bc.7 (end) (445)    │          │
...
  • Add --unicode-output/-U argument to force use of Unicode characters to improve output readability. #377
    • For example useful when piping to less and you want fancy unicode and colors:

      fq -cU d file | less -r
  • to_entries now preserves struct field order. #340
  • Experimental `raw string` literal support. Work the same as golang raw string literals. Useful in REPL when pasting things etc but should probably be avoided in jq scripts. #371
  • Properly fail lexing of invalid binary, octal and hex number literals. #371
  • REPL completion now include all functions. Before some functions with multiple argument counts were skipped. #375
  • Switch to new gopacket fork with speedup and bug fixes. Remove SLL2 workarounds in fq. #386

Decoder changes

  • csv Correctly handle decode values when tocsv normalize to strings. Before array and object ended up being JSON serialized to strings. #341
    • Normalize to strings is done so that non-string scalars can be used: 
      $ fq -n '[[1,true,null,"a"]] | tocsv'
"1,true,,a\n"
  • dns DNS over UDP format was accidentally used to probe TCP streams #366
  • elf Remove redundant program_header struct
  • flac
  • jsonl Add decoder. Decodes JSON lines. There is also fromjsonl and tojsonl. #374
  • macho
    • Split FAT Macho decoding into macho_fat format which also fixed handling of file offsets in sections. #362
    • Decode symbol and string sections. #352
  • matroska Remove new lines in descriptions. Messes up tree. #369
  • mp3_frame
    • Support LSF (low sampling frequency) frames. #376
    • Skip trying to figure out what main data is for current frame and not. Was incorrect and doing it properly probably requires hoffman decoding. #376
  • pcap Support files with nanosecond precision. Has a different magic. #382
  • prores_frame Add basic decoder. Decodes container and fram header. #396 Thanks @Doctor-love for test files
  • tar Fix regression when decoding number fields. Now ok again to be empty string. #372
  • wasm Add WebAssembly Binary Format decoder. #383 Thanks to Takashi Oguma @bitbears-dev
  • yaml Fail on trailing data. Before it succeeded with the last value. #373
  • zip
    • Don't require PK header as there seems to be zip files with prepended data. #359
    • Correctly limit amount of backwards search for EOCD (end of content directory). #363
  • xml Correctly handle decode values when toxml normalize to strings. Before array and object ended up being JSON serialized to strings. #341
  • xml
    • Change attribute prefix to @ instead of - and make it an option attribute_prefix. #401
    • Skip default namespace in element names. #389
    • Always include attributes and children even when empty in array mode. Makes it a lot easer to work with as you can assume .[1] will be attributes and so on. #357
    • Normalize to strings is done so that non-string scalars can be used: 
      $ fq -nr '{a: {"@boolean": true, "@number": 123, "@null": null}} | toxml'
<a boolean="true" null="" number="123"></a>
    • Allow and ignore trailing <?procinstr?> and improve trailing data error message. #368
    • Correctly sort if any #seq is found and also properly sort negative #seq. #384

Changelog

  • 0cd846a *extra: Rename extra to just ex and refactor to use generics
  • fb583e2 Add 32 bps FLAC to test
  • c1d5b2b Add sample size entry to list for 32bps flac streams
  • 3f209c4 Fix decoding of FLAC raw entropy partition
  • 25061ac Update docker-golang from 1.18.4 to 1.18.5
  • 0de2c90 Update docker-golang from 1.18.5 to 1.19.0
  • 7b8d95b Update github-go-version from 1.18.4, 1.18.4, 1.18.4 to 1.18.5
  • 103991f Update github-go-version from 1.18.5, 1.18.5, 1.18.5 to 1.19.0
  • 4255b87 Update github-golangci-lint from 1.47.2 to 1.47.3
  • 198305e Update github-golangci-lint from 1.47.3 to 1.48.0
  • fa9fec3 Update github-golangci-lint from 1.48.0 to 1.49.0
  • f579e9c Update make-golangci-lint from 1.47.2 to 1.47.3
  • c8069d2 Update make-golangci-lint from 1.47.3 to 1.48.0
  • 004eb56 Update make-golangci-lint from 1.48.0 to 1.49.0
  • abcc736 add ULEB and SLEB to known words for spell check
  • 9238251 ci: Skip -race for windows and macos
  • 913f578 columnwriter,dump: Add Column interface and refactor into BarColumn and MultiLineColumn
  • 5d9ffea decode,scalar: Map empty string also else sym might ends up nil
  • 326dada decode: Add LEB128 readers
  • 502f451 decode: Refactor to use scalar type assert helper
  • 840292b decode: Simplify compound range sort behaviour
  • 15f7c67 dev,fuzz: Add some useful retrigger snippets
  • 46dca8c dns: Don't use dns (udp) format for tcp also
  • c233215 dns: Rename isTCP to hasLengthHeader
  • ed42478 doc,interp: Update and add more examples
  • f247edb doc: Update README demo a bit with new features
  • 3613b6d elf: Remove redundant program_header struct
  • 8a19978 flac: Make gen script generate correct fqtest files
  • 2bfbe9a flac_frame: Cleanup some dev lefterovers and todos
  • 64b2365 fqtest: Run tests in parallell
  • af35b28 gojq: Preserve keys order for to_entries when used with JQValue
  • 804ad1e gojq: Update fq fork
  • add3dcf gojq: Update fq fork, fix scope argcount issue
  • d898732 gojq: Update fq fork, new scope function, rawstring, stricter integers
  • 394717c gopacket: Switch/update to new fork, remove SLL2 hack
  • 4eae7ff interp,doc: Add -R raw string slurp hint to -s help
  • d8792fd interp,dump: Correctly flush columns if data will be shown
  • 29005c7 interp,dump: Show address bar for root, nested roots and on format change
  • c7559b5 interp: Add --unicode-output/-U to force use of unicode
  • 9e447c9 interp: Use RegisterFS instead of format files
  • 701c67c jsonl: Add decoder, also tojsonl encoder
  • bc6cffd lint,decode,fuzz:: Fix nilerr warnings, one real one should be ignored for now
  • 3c21b05 lint: Fix ioutil deprecation, reformat for new doc standard
  • b2d4e6d macho: Decode cmd symtab symbols
  • 725c8e8 macho: Split into macho/macho_fat, fix offset issue and add string decoding
  • 2e40738 matroska: Strip newlines in description
  • cf15661 mp3_frame: Add LSF support and fix incorrect main data handling
  • 74c7dc4 pcap: Add ns support and add header field
  • 8fc4353 prores_frame: Add basic container and frame header decoder
  • dc32ac0 script: Use strings.Builder to collect output
  • 0d44b93 tar: Some number fields can be empty
  • 545dac8 test: Update tests, go 1.19 uses \xff instead of \u00ff
  • ce43887 wasm: make doc
  • 074c22c wasm: add -timeout 20m for go test to workaround ci test fail
  • cd037c5 wasm: add comment to clarify lazy initialization
  • f73965d wasm: add wasm to probe list
  • 00869b3 wasm: avoid race condition
  • db8021c wasm: define and use constants for some insturctions
  • bcc0dfd wasm: fix comment format
  • 289ddf5 wasm: fix lint issues
  • 3fca7cc wasm: fix lint issues
  • cbb5a8e wasm: further simplification
  • 934ed9a wasm: initial version
  • e5cf173 wasm: make the godoc formatter happy
  • b0f3fec wasm: remove nolint:unparam which is no longer needed
  • e1691de wasm: remove unused function
  • ae4529c wasm: run golangci-lint run --fix
  • fead68d wasm: tidy up
  • 3298d18 wasm: to be able to probe
  • 2eb1750 wasm: update tests
  • d5d9e73 wasm: use FieldULEB128() / FieldSLEB128() defined in the upstream
  • 7401d14 wasm: use WRITE_ACTUAL=1 to generate .fqtest files
  • 2037b86 wasm: use map, not switch
  • ae08bf7 wasm: use s.ActualU() instead of s.Actual.(uint64)
  • 63f4a72 wasm: use scalar.UToSymStr for simplicity
  • 0ad5a8e wasm: use underscores for symbol values
  • fa20c74 xml,csv,interp: Handle JQValue when string normalizing
  • f4e0137 xml,html: Always include attrs and children in array mode
  • 9a5fcc8 xml: Allow trailing
  • 71900c2 xml: Correctly sort if one #seq is found and allow negative seq numbers
  • 716323c xml: Even more namespace fixes
  • f24d685 xml: Keep track of default namespace and skip it element names
  • 095e116 xml: Switch from "-" to "@" as attribute prefix and make it an option
  • 3623eac yaml: Error on trailing yaml/json
  • d607bee zip: Correctly limit max EOCD find
  • 19b7089 zip: Skip header assert as there are zip files with other things appended

Contributors

Doctor-love, ktmf01, and bitbears-dev
Assets 9
Loading