forked from arkime/arkime
-
Notifications
You must be signed in to change notification settings - Fork 3
/
CHANGELOG
380 lines (360 loc) · 16.3 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
0.11.1 2014/05/XX
- NOTICE: ES 0.90.12+, 1.1.x, 1.2.0 are support by this version.
ES 1.0 is NOT supported.
This is the LAST version to support 0.90.x
Restart viewer AFTER upgrading ES versions
- NOTICE: When upgrading your runes.sh for 1.x add a -d to the
command, ES no longer runs in background by default
- Parsers can register for session save events (issue #248)
- Fix compressES check with ES 1.x (issue #255)
- Show error for ip queries with regex or wildcard (issue #252)
- added session.segments and session.length (issue #254)
- support elasticsearch=http:// or https:// format (issue #249)
- Only libmagic the first 50 bytes
- users tab can now sort various tabs
- Turn of bloom filter for previous indexes if using db.pl expire
- Set threadpool search queue size to unlimited
- stats page works again with dynamic scripts disabled
- New db.pl rm-missing command (issue #242)
- Upgrade qtip2 to 2.2.0
- Mouse over view names shows expression (issue #220)
- Display SPI Data even if node is unavailabe (issue #219)
- Netflow plugin timestamp fixes (issue #241)
- Comma seperate list of elasticsearch hosts (issue #176)
- New includes directive (issue #144)
- Initial bigendian support in viewer (issue #259)
- List queries can now have wildcard and regex items.
example: http.uri = [term, w*ldcard, /.*regex/]
- freeSpaceG now supports a percentage
- Show up to 25 items of each SPI data field with a ...
to show more (issue #262)
- If a http header went across two packets the leading piece
would be chopped
0.11.0 2014/05/08
- BREAKING: elasticsearch 0.90.7 or newer required, recommend 0.90.12+,
1.x not supported yet
- BREAKING: node 0.10.20 or newer required, 0.11+ not supported yet
- BREAKING: Many of the older expression that were kept for backwards
compatibility no longer work
- BREAKING: All plugins need to be updated and rebuilt
- BREAKING: Glib 2.30 or newer is now required, short term workaround is
adding "#define G_VALUE_INIT { 0, { { 0 } } }" to moloch.h, but please upgrade
- BREAKING: switched to official elasticsearch javascript client,
npm update required (issue #222)
- Major internal fields refactoring
- Fields are now 'easy' to create, only need to change 2 places
- db.pl upgrade should be needed less often
- Plugins/Parsers can have their own sessionDetail UI
- New protocols, dns.status, dns.query.type, dns.query.class fields
- Fixed bug with http parser not capturing last query value
- http connecting is now mostly async for faster startup (issue #225)
- tagger loading is now mostly async for faster startup
- titleTemplate config option (issue #229)
- output buffers are now mmaped so they are more likely to be returned to OS
- free output buffers are now cached, controlled by maxFreeOutputBuffers
- More untagging, new fields http.method, http.statuscode, http.bodymagic
- More untagging, new fields email.bodymagic
- Start of viewer regression testing
- Fix reverse http header parsing
- simple mysql parser
- Fix smtp subject empty encoded sections
- Increase ES query timeout to 5 minutes
- simple postgresql parser
- More same src/dst ip fixes
- easybutton installs node 0.10.28 & ES 0.90.13 now
0.10.1 2014/03/30
- Status code not being set when . after mime data
- db.pl has simple mv/rm commands now
- Fixed all pagination (issue #192)
- multies tag fix (issue #205)
- New email.hasheader
- New packets.(src|dst), bytes.(src|dst), databytes.(src|dst) (issue #206)
- New payload8.(src|dst), payload.(hex|utf8), payload.(src|dst).(hex|utf8) (issue #209)
- pcapDir can now be a semicolon seperated list of directories, currently just
round robin is supported
- UI: Fix Search/Actions showing up on second line on page load
- capture now does memlock and max schedule priority on startup (issue #199)
- when yara is disabled dont retain extra data
- parse email user names
- antiSynDrop config option
- remove schedule priority change for now
- Changing memlock failure message to WARNING
- new pcapWriteMethod advanced setting, supports direct, thread, thread-direct now
- Change ES updates to support "script.disable_dynamic: false"
- DNS parsing improvements
- Deal with windows-1252 subject encoding better (issue #213)
- Tagger supports md5s
- Increased default pcap size to 8096
- Added viewHost and multiESHost
- Both Yara 1.x and 2.x now supported (issue #201)
- DNS status support (issue #218)
0.10.0 2013/12/31
- IMPORTANT: all parsers have been broken out into individual
shared libraries. It still isn't possible to easily add new
db fields yet, coming soon.
- parsersDir and pluginsDir can now be a list of directories
- jade 1.0 support (issue #194)
- webBasePath fix (issue #193)
- reverse socks support
- memory reduction
- fixed plugin and header sections when together not working
- fixed memory leak with GErrors
- support traffic to/from same ip better
- more capture tests
0.9.3 2013/12/17
- db.pl only open/closes indexes for pre version 12
- Custom date was broken for urls with no date param
- Non standard date param added to menu
- Http file parsing improvements
- ES health loaded on page load (issue #172)
- Session detail check boxes work multiple times again
- core fix with empty tagging plugin information
- multiple connections.csv files (issue #163)
- fixed view editing
- unique.txt tags fixed
- plugins can add fields
- start of capture regression tests
- SNI support (issue #157)
- lots of socks decoding improvements
- fixed socks memory leak
- smtp status code tagging (issue #180)
- added missing DNS qtypes
- tcp DNS support (issue #173)
- DNS MX support
- easybutton builds libpcap 1.5.1
- proxy content type correctly
- fixed viewer exit (issue #183)
- added unique email filenames
- src/dst raw view (issue #178)
- SMTP subject encoded parsing
- SMTP received header parsing (issue #175)
- Basic IMAP tagging (issue #186)
- Basic RDP tagging (issue #187)
- Better bad passwordSecret error message (issue #190)
- Upgrade d3 package
- smtp file finger printing (#174)
- include smtp user-agent header
0.9.2 2013/11/14
- BREAKING: nodejs 0.8 is no longer supported
- Upgrade d3 and cubism
- Fixed searches so numbers don't have to be quoted
- Fixed export hitting max number of stack frames
- Connections tab new UI
- Connections tab allows any field for src/dst
- More user settings
- Fixed unique.txt to deal with multi value fields
- viewer.js now uses forever-agent package to help multi
machine communication. (npm install required)
- easybutton installs node 0.10.20 now
- fixed race condition with tag lookup rate limiting
- expression ip.dst == ip:port wasn't working
- more max stack fixes
- users tab improvements (issue #152)
- New views concept (issue #146), created in settings tab
- settings tab improvements
- Ability to search for http.uri.path, http.uri.key,http.uri.value for
uri path, query string key, and query string value (parseQSValue must
be set to true)
- --dryrun doesn't use ES for anything now
- New session hash algorithm
- Token checking function now shared
- Fixed broken upload
- Change 'npm install' to 'npm update' everywhere
- New maxFileTimeM for time rotation (issue #158)
- Increased SMB decode buffer size
- Fixed SMB decode infinite loop
- Fixed expire bug with multi nodes on same machine and different traffic rates
- Added connections.csv (issue #163)
- Added unlock button to connections
- small resolution UI improvements (issue #159)
- sessionDetail cleanup
- Permalinks are faster (issue #162)
- Missing rir data would cause session detail to not open
- Reassembled IP frames > ~5k would cause session detail to not open
- Fixed right click issues (issue #169)
- New payload8.src, payload8.dst that saves the first 8 bytes of sessions
in hex
- New socks.user field (issue #167)
- Tagger supports CIDR and 1 level hostname lookups (issue #160)
- DHT tagging (issue #154)
- stylus > 0.39 fix
- javascript loop length "improvements"
- switch from forever-agent to keep-alive-agent, npm update required
- caTrustFile config option (issue #161, pull #171
- start of some javascript cleanup
- BREAKING: Upgrade to jquery 2.x, no more IE <= 8 support
- remove connect-timeout package requirement
- increase 2 minute http timeout to 10 minutes
- increase max session queried to 2 million
0.9.1 2013/10/03
- Make sure at least one stats record is written per run
- Display IRC channel in sessions view
- Fix right click on sessions view info column
- Fixed post increment issue in js0n code (issue #128)
- Fixed broken hourly rotateIndex in viewer (issue #130)
- Fixed broken settings page for other user (issue #126)
- Basic SMB tagging
- Basic ES query throttling
- Added missing ssh.ver from spigraph
- EXPERIMENTAL: Multi cluster search (issue #97)
- Fixed CSV not equal search queries with range fields (issue #132)
- BREAKING: To specify install dir with ./easybutton-build.sh use --dir
for example: ./easybutton-build.sh --dir /nids/moloch
- Can build with PFRING now, easybutton-build.sh has --pfring
or easybutton-single.sh asks
- Basic smb parsing, disable with parseSMB=false
- Basic socks4 and socks5 decoding
- rir lookups, configure with rirFile=ipv4-address-space.csv
https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv
- Netflowish CSV exporting from UI
- clean up db.pl some, rename rotate command to expire
- With custom date queries can now select bounded by
- New user setting for sessions sort order
- Fixed encoding issues
- New plugin pre save callback
- Fixed entirePcap not setting correct Content-Type
- New right click pivot option in spiview
0.9.0 2013/08/26
- 32bit fix for lpd/fpd
- easybutton now uses nodejs 0.10, 0.8 is still supported for now
- Work around for tcp seq number wrapping causing viewer exit
- dns parsing core fix
- switch to nonblocking pcap saves
- more debuging info on proxy failure
- Fixed bug when setting viewUrl
- Limit number of libnids errors (issue #115)
- Display possible reasons for libnids IP Header error
- Another domainless hostname fix (issue #116)
- Exports should be between 2x-5x faster
- Added actions menu for search/sessions
- Scrub and Delete actions, user must have remove right enabled (issue #119, issue #89)
- Add/Remove(remove right required) tags actions
- Hourly rotation (issue #122)
- unique.txt fixes (issue #123)
- Actions can be done on linked sessions (issue #120)
- SPI Graph auto refresh (issue #111)
- Better error handling for SPI data display (issue #109)
- List queries using [] syntax (issue #106)
- user prefs with timezone display (issue #95)
- Basic IRC searches
- Disk Queue stats display
0.8.7 2013/07/12
- Use recent versions of express which REQUIRES "npm install" in viewer directory
- Use recent version of jade which requires extra spaces, use "git diff -w" to
- Now index Host headers with and without port
- pcapng exporting with meta data
- Basic upload feature, doesn't support transfers of meta data yet
- addUser.js has better help and error reporting
- ES optimizations to use bool instead of and/or, also use regexp filter
instead of regexp query
- Changed ES stats shown to hopefully more useful ones
- Fixed viewer exit on empty data gzip decode
0.8.6 2013/06/20
- Deal with non data ES nodes
- Viewer prints error if it can't find pcapDir setting
- New setting dbFlushTimeout that controls how often we flush to ES
- New setting compressES that turns on compresesion to ES, requires
http.compression: true in elasticsearch yml file
- libnids was overreporting traffic, switch to libpcap stats,
bytes/sec and total bytes/sec in stats will be lower
- Fixed recent jade warnings
- Fixed openned export
- minor ui improvements
0.8.5 2013/06/14
- NOTICE: Requires at least 0.90.1 ES
- New export dialog that asks for filename and number selection
- spigraph shows health, decodes tags/ips, has sort by name
- spigraph/spiview show total counts
- upgrade to jvectormap 1.2.2 which fixes spigraph issues
- deal with 113 (SLL) pcap type
- header search and header cnt search didn't always work
- ignore case of trailing .pcap when processing a directory
- fixed bad bug with exporting large files corrupting pcap
- HTTP file decoding works better
- On exit ignore http queue limits
0.8.4 2013/05/28
- NOTICE: Last version to support 0.20 ES
- NOTICE: Changed some expressions, old versions are supported for now
email.ct* => email.content-type*
email.mv* => email.content-type*
email.id* => email.message-id*
email.ua* => email.x-mailer*
header* => http.hasheader*
ua* => http.user-agent*
http.ua* => http.user-agent*
- valgrind fixes and memory reduction
- New SPI Graph tab which lets you graph an expression per field
- Now possible to chose which http request, response and smtp headers
to index using headers-http-request, headers-http-response,
headers-email sections
- Session Graph now shows the full queried range instead of data
available range
- Fixed db.pl wipe error
- Added density to db.pl info
- Added override-ips config section that allows overriding of
country, tag, asn for ips and cidr ips
- Clean up add users UI a little, and clear fields on successful add
0.8.3 2013/05/09
- full text for uri is now available
- regex searches using == /REGEXHERE/
regex can be slow so be careful
- regex and wildcard searches full text instead of tokenized
- fixed bug with uri.cnt not be recorded
- filenumber generation rewritten, can now deal with
multiple instances running and other edge cases
- http body content is md5, although the encoded
and non encoded version will get different md5s
- detect when npm install needs to be run
- quoted strings and regex strings detect better
- new centerTime=time&timeWindow=minutes option to do +- views
- show tags names in unique views
- remember view setting for future session views
- DNS qclass and qtype tags
- Upgrade yara and glib version
0.8.2 2013/04/29
- Install ES 0.90
- fixed dropped packet stats
- netflow plugin (issue #27)
- memory capture improvements
- record capture memory in stats
- record filesize for offline pcap
- remove port from http host header (issue #63)
- db.pl prints more info by default, multiple -v even more
information, and new info command
- fixed viewer crashes if pcap can't be read (issue #67)
- minor css cleanup
- Display CERT info in session view
0.8.1 2013/04/19
- Should support nodejs 0.10.3, but still use 0.8.23 for now
- Support RAW link type pcap files
- renamed decode.js to pcap.js
- Setting spiDataMaxIndices to -1 allows all for spiview
- Log userId for requests
- fixed uri.cnt
- don't exit moloch-capture until all file creates finish
0.8.0 2013/04/17
- New SPI View tab, REQUIRES elasticsearch 0.90 RC2 or later
- config spiDataMaxIndices controls how many indices to run against since
spiview feature can cause elastic search to blowup memory.
- display date as year/mon/day
- Lots of UI cleanup, slighly less ugly as before hopefully
- 32 bit builds should work
- Fixed bug where status codes/http methods weren't always recorded
- New SMTP plugin callbacks, more to come
- offline capture reading should work better with old libpcap versions
- DB now stores full and tokenized version of user agents, ASNs, and cert info
- verify the config file has a defaults section
- display elastic search health for admin users on pages
- display elastic search stats on stats page
- display ip protocol friendly name
- display simple png view of raw session data and attachments on mouseover,
requires "npm install" in viewer directory
- new much more accurate world map [thanks Dave]
- fixed user name XSS issue [thanks z0mbiehunt3r]
- fixed many viewer exits
- timestamp display option in sessionDetail
- graph now uses seconds if less then 30 minutes and hours if more
then 5 days. This makes display faster
- Refactored how capture stores spi data in memory
- Refactored hash table code
- Added host.dns, host.http, host.email