Neolytics

Neolytics is a gem which allows you to analyze a section of your code using Neo4j. The data model used is described below and a number of example queries are given (you can suggest an idea for a new query).

There is also a ruby_code_analytics Rails application which can take data generated by neolytics and allow you to browse it with a nice UI.

Installation

Add this line to your application's Gemfile:

gem 'neolytics'

And then execute:

$ bundle

Or install it yourself as:

$ gem install neolytics

Usage

All you need to do to use Neolytics is

neo4j_session = Neo4j::Session.open(:server_db, 'http://neo4j:pass@localhost:7474')
# or
neo4j_session = Neo4j::Session.current

Neolytics.record_execution(neo4j_session) do
  # Code that you want to analyze here
end

During the code execution

Model:

Neolytics generates the following structures:

Abstract Syntax Trees

TracePoint execution tracking

Objects (including recursively browsed class and modules)

Links between these domains

TracePoints are linked to AST nodes:

TracePoints are linked to objects (arguments, return values, variable values, context objects):

Example queries:

Queries using the abstract syntax trees:

Get all assignments

  MATCH (a:ASTNode)
  WHERE a.type IN ['lvasgn', 'ivasgn', 'casgn', 'op_asgn']
  RETURN a.type, a.name, a.operator, a.expression
  ORDER BY a.type, a.name

Example Output

All defined methods:

  MATCH (def:ASTNode {type: 'def'})
  WITH def
  ORDER BY def.file_path, def.first_line
  LIMIT 50
  MATCH shortestPath((def)-[:HAS_PARENT*]->(class:ASTNode {type: 'class'}))
  RETURN def.file_path +':'+ def.first_line AS line, class.name AS class, def.name AS method

Example Output

Assignment-Branch-Condition (ABC) Metric

  MATCH (def:ASTNode {type: 'def'})

  OPTIONAL MATCH (def)<-[:HAS_PARENT*]-(assignment:ASTNode)
  WHERE assignment.type IN ['lvasgn', 'ivasgn', 'casgn', 'op_asgn']
  WITH def, count(assignment) AS a

  OPTIONAL MATCH (def)<-[:HAS_PARENT*]-(branch:ASTNode)
  WHERE branch.type = 'send'
  WITH def, a, count(branch) AS b

  OPTIONAL MATCH (def)<-[:HAS_PARENT*]-(condition:ASTNode)
  WHERE condition.type IN ['if', 'while', 'until', 'for', 'rescue', 'when', 'and', 'or']
  WITH def, a, b, count(condition) AS c

  RETURN def.name, def.file_path, def.first_line, a, b, c, sqrt((a*a) + (b*b) + (c*c)) AS abc
  ORDER BY abc DESC

Example Output

Cyclomatic Complexity

  MATCH (def:ASTNode {type: 'def'})
  OPTIONAL MATCH (def)<-[:HAS_PARENT*]-(condition:ASTNode)
  WHERE condition.type IN ['begin', 'if', 'while', 'until', 'for', 'rescue', 'when', 'and', 'or']
  RETURN def.name, def.file_path, def.first_line, count(condition)
  ORDER BY count(condition) DESC
  LIMIT 50

Example Output

All args / vars / statics / etc... in a class

  MATCH (:ASTNode {name: 'Numeric', type: 'class'})<-[:HAS_PARENT*]-(n:ASTNode)
  WHERE n.type IN ['def', 'arg', 'optarg', 'restarg', 'lvar', 'const', 'sym']
  RETURN n.type, n.expression, n.name
  ORDER BY n.type, n.expression

Example Output

TracePoint queries

Time spent by method

  MATCH (tp:TracePoint)
  WITH tp.path AS path, tp.lineno AS line, tp.defined_class AS class, tp.method_id AS method_id, sum(tp.execution_time) AS sum, count(tp) AS count
  ORDER BY sum(tp.execution_time) DESC
  RETURN path +':'+ line AS line, class +'#'+ method_id AS method, sum, count, sum / count AS average
  LIMIT 50

Example Output

Common ancestor

  MATCH
    (tp1:TracePoint {defined_class: 'ActiveSupport::Autoload', method_id: 'autoload_at'}),
    (tp2:TracePoint {defined_class: 'ActiveSupport::Inflector', method_id: 'underscore'})
  WITH tp1, tp2 LIMIT 1
  MATCH
    path1=(tp1)-[:HAS_PARENT*0..]->(common_ancestor),
    path2=(common_ancestor)<-[:HAS_PARENT*0..]-(tp2:TracePoint)
  RETURN tp1, tp2, common_ancestor, length(path1), length(path2)

Example Output

See all arguments and return values for methods

  MATCH (start_tp:TracePoint)
  WHERE start_tp.event = 'call'
  OPTIONAL MATCH (start_tp)-[argument_rel:RECEIVED_ARGUMENT]->(arg:Object)-[:IS_A]->(class:Object)
  WITH
    start_tp,
    collect(argument_rel.argument_name +': '+ arg.ruby_inspect +' ('+ class.ruby_inspect +')') AS arguments

  OPTIONAL MATCH (start_tp)<-[:STARTED_AT]-(return_tp:TracePoint)-[:RETURNED]->(o:Object)-[:IS_A]-(class:Object)
  RETURN
    start_tp.path +':'+ start_tp.lineno AS line,
    start_tp.defined_class +'#'+ start_tp.method_id AS method,
    arguments,
    o.ruby_inspect +' ('+ class.ruby_inspect +')' AS return_object
  ORDER BY start_tp.execution_index
  LIMIT 30

Example Output

Show all returns values from a particular superclass (i.e. Asking for Numeric gives Fixnum, Rational, BigDecimal, etc... types)

  MATCH (tp:TracePoint)-[:RETURNED]->(o:Object)-[:IS_A]-(class:Object)-[:HAS_SUPERCLASS*]->(superclass:Object {ruby_inspect: 'Numeric'})
  WHERE tp.event IN ['return', 'c_return']
  RETURN tp.path +':'+ tp.lineno AS line, tp.defined_class +'#'+ tp.method_id AS method, o.ruby_inspect +' ('+ class.ruby_inspect +')' AS return_value
  ORDER BY tp.execution_index

Example Output

Object space queries:

Show class hierarchy for Numeric class:

  MATCH (class:Object)-[:HAS_SUPERCLASS*]->(superclass:Object {ruby_inspect: 'Numeric'})
  RETURN *

Example Output:

Combination queries

Combining cyclomatic complexity and run-time metrics

  MATCH (tp:TracePoint)
  WITH sum(tp.execution_time) AS total_execution_time

  MATCH (node:ASTNode {type: 'def'})
  OPTIONAL MATCH (node)<-[:HAS_PARENT*]-(condition:ASTNode)
  WHERE condition.type IN ['begin', 'if', 'while', 'until', 'for', 'rescue', 'when', 'and', 'or']
  WITH node, count(condition) AS cyclomatic_complexity, total_execution_time

  MATCH (node)<-[:HAS_AST_NODE]-(tp:TracePoint)<-[:STARTED_AT]-(return_tp:TracePoint)

  WITH
    cyclomatic_complexity,
    total_execution_time,
    tp.path + ':' + tp.lineno + ' (' + return_tp.defined_class + '#' + return_tp.method_id + ')' AS method,
    count(tp) AS executions,
    sum(return_tp.execution_time) AS total_method_execution_time

  RETURN
    method,
    cyclomatic_complexity,
    executions,
    total_method_execution_time,
    100.0 * (total_method_execution_time / total_execution_time) AS percentage_of_total_time,
    total_method_execution_time / executions AS average_execution_time

  ORDER BY total_method_execution_time DESC

Example Output

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/neolytics. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.