When it comes to pentests you sometimes have access to the source code (opensource applications or grey-/whitebox tests). It started using tools to help me analyze the code. So far I had only Java code to analyze, but it was quite interesting to see the different tools and the results. Let me write down my personal findings.
The tools
I tried 3 different tools: Sonarqube, PMD and SpotBugs (successor to FindBugs).
First problem – .class vs .java
SpotBugs is able to analyze .jar or .class files directly, while the other two need source files, so for Java we need disassemblers. While I like jd-gui for quickly looking into files, I found its results not really useable for source code analysis. cfr yielded much better results, but was veeeery slow sometimes.
The setup
PMD and SpotBugs can simply be downloaded and run, while Sonarqube is a docker container that needs some more tweaks. Especially importing your files is tricky, because it can not be done wrong the webinterface. You need to log in via WebUI, create the API key, log into the docker container, get sonar-scan, copy the files into the docker container and run sonar-scan to import them. Not painful, because it can be scripted, but much more work compared to simply running SpotBugs and opening your directory with .jar files. And here is the next issue:
Input formats and the tools
For my Java example the 3 tools require different input formats. My project is a single .jar file and the second test I did was a folder with some .jar files and some .class files.
SpotBugs I simply told the directory and it started unzipping the .jar file recursively and analyzing the .class files within. Awesome.
PMD needs .java files, so I had to recursively unzip the .jar files (the initial .jar contained other .jar files) and the reverse the .class files into .java files. Not that great.
SonarQube needs .java AND .class files. You can bypass the .class restriction, but it will produce more false-positives according to the developers. So I needed to put all .class and .java files in directories (/bin and /src) for SonarQube. Not that great.
Speed
I was very impressed by the speed of SpotBugs. It is close to instant on a normal business laptop. PMD was slower, but acceptable. SonarQube took some hours on the same hardware (Lenovo T480 laptop). Without having exact numbers PMD was about 5x slower than SpotBugs and SonarQube about 1000x slower.
SpotBugs clear winner. SonarQube clear loser.
Results
This may be the most important part – findings. I found all of the solutions to report many false positives, which is pretty bad. I would expect these tools to help me safe time. Instead it is stealing my time to check false positives. I mean .. in the time I used to check false positives I could have read the code ..
SonarQube did pretty good regarding findings. It found the most – but also the most false positives.
SpotBugs found a lot of bugs and false positives, too. Second most in my test. Good for a quick overview.
PMD comes last and mostly found issues in code quality, not security bugs. Something I was not looking for.
Personal Rating
Personally I would use SpotBugs in quick assessments and SonarQube if you have more time or need a deeper analysis. I will personally not use PMD as it doesnt perform better in any of my tests.