Mnemosyne · Apache RAT · Tika · Solr

Point it at a tree.
Get a license report.

RAT on one JVM is too slow and prints nothing until it finishes. DRAT splits the tree by MIME type, runs RAT on chunks, and reduces the logs into one audit — while you watch. 

$ bin/drat go ~/src/northwind-sdk
How to run Source

A demo audit

Invented tree. Real product.

Nothing here is from a live repository. The listing is what a reduce looks like after drat go: licenses from RAT, MIME types from Tika, CSV under data/archive/rataggregate/.

northwind-sdk

1,062 files · crawl → index → map → reduce

Apache-2.0
748
MIT
188
BSD-3
91
Unknown
33
Binaries
2
text/x-java 412 text/x-python 280 application/javascript 188 application/xml 91 text/markdown 64 text/plain 27

The files that matter

RAT is for the ones it cannot stamp.

The 748 Apache headers are not the story. The 33 unknowns are. The report exists so a human can open those files and put a header on them.

Printed source with a blank header
No header. Coffee ring. This is an unknown.
Source listing that begins with an Apache 2.0 license header
Header present. RAT stamps it and moves on.
unknownsrc/legacy/Util.javatext/x-java
unknownvendor/shim.ctext/x-csrc
unknownscripts/once.shapplication/x-sh
unknownthird_party/hash.pytext/x-python

One command

go hides the MapReduce.

  1. crawlWalk the tree, Tika MIME
  2. indexSolr drat core
  3. mapRAT on k-sized MIME chunks
  4. reduceOne combined log + CSV
File folders standing in a rack

MIME partition

Java with Java. Python with Python.

Tika sorts first. RAT never sees a mixed pile. Chunk size is configurable. Exclude .git and target.

Small stacks of paper in a grid

Map

k files, one JVM, a log.

Each stack is a chunk. Incremental logs instead of one silent RAT process. Then the reducer concatenates them.

The stamp

A header is a mark on the page.

Apache RAT looks for a license notice in the file. DRAT is the workflow that lets that check finish on a tree RAT alone would sit on for hours.

Indigo rubber stamp on a printed listing

Watch it

OPSUI for the workflow. Proteus for Go and Reset. Solr cores drat and statistics.

Reset and go again

drat reset clears crawl state and keeps stats. dratseq walks many trees. --full-wipe drops stats too.

Exclude noise

drat go --exclude .git ~/src/northwind-sdk. Regex exclude file for the rest of the junk you do not want RAT to see.

Proteus

The console is a path and a button.

Proteus is the Vue 3 UI at /proteus/. You paste an absolute path, press Go, and the stack crawls. Reset is the other button. The phases stay in bin/drat. The audit above is what the reduce is for.

Proteus Vue 3 run dialog with path and Go
The actual run dialog. Path in, Go.

bin/drat

The operator path.

$ bin/oodt start
$ bin/drat go --exclude .git ~/src/northwind-sdk
$ bin/drat reset
$ bin/drat go ~/src/another-tree

One verb for the whole audit, reset before the next tree. oodt start is the Mnemosyne launcher — the name is unchanged on purpose.

Wiki