Bug 237211 - [NEW PORT] devel/ghidra: Software reverse engineering (SRE) framework
Summary: [NEW PORT] devel/ghidra: Software reverse engineering (SRE) framework
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Kurt Jaeger
URL:
Keywords: feature
Depends on:
Blocks:
 
Reported: 2019-04-11 20:24 UTC by Tamas Szakaly
Modified: 2019-07-01 06:52 UTC (History)
7 users (show)

See Also:
pi: maintainer-feedback+


Attachments
port archive (24.66 KB, text/plain)
2019-04-11 20:24 UTC, Tamas Szakaly
no flags Details
ghidra-9.0.4.shar (25.72 KB, text/plain)
2019-06-24 07:18 UTC, Tamas Szakaly
no flags Details
ghidra-9.0.4_v2.shar (26.30 KB, text/plain)
2019-06-26 06:37 UTC, Tamas Szakaly
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tamas Szakaly 2019-04-11 20:24:52 UTC
Created attachment 203605 [details]
port archive

Ghidra is a software reverse engineering (SRE) framework created and maintained by the National Security Agency Research Directorate. This framework includes a suite of full-featured, high-end software analysis tools that enable users to analyze compiled code on a variety of platforms including Windows, macOS, and Linux. Capabilities include disassembly, assembly, decompilation, graphing, and scripting, along with hundreds of other features. Ghidra supports a wide variety of processor instruction sets and executable formats and can be run in both user-interactive and automated modes. Users may also develop their own Ghidra plug-in components and/or scripts using Java or Python.

WWW: https://ghidra-sre.org
Comment 1 Kurt Jaeger freebsd_committer 2019-04-22 08:01:20 UTC
Did you test with poudriere ?

It failed on 12.0amd64 early in the build, see

https://people.freebsd.org/~pi/logs/devel__ghidra-120.txt
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2019-04-22 08:24:53 UTC
This should go in CATEGORIES=security, with devel as a secondary category
Comment 3 Tamas Szakaly 2019-04-22 09:20:10 UTC
(In reply to Kurt Jaeger from comment #1)
> Did you test with poudriere ?

I've only tested it with Synth, but I realize now that it was a mistake, since it allows network connections after the fetch phase. I will look into how to fetch dependencies separately from building with Gradle.

(In reply to Kubilay Kocak from comment #2)
> This should go in CATEGORIES=security, with devel as a secondary category

Yup, I also feel security would be a better category. I've used devel because radare2 is in devel, and wanted to be consistent.
Comment 4 Yuri Victorovich freebsd_committer 2019-05-17 03:47:00 UTC
Tamas,


Thank you for submitting this port.

I have several questions/comments.


What is the benefit of having the option selecting JDK11 vs. JDK12, why not just using once JDK?

GH_PROJECT isn't needed when it is the same as PORTNAME.


GH_TAGNAME= should be converted to DISTVERSIONPREFIX/DISTVERSION/DISTVERSIONSUFFIX


It fails to build in poudriere. The immediate reason is:
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/5.0/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing

FAILURE: Build failed with an exception.

* Where:
Build file '/wrkdirs/usr/ports/devel/ghidra/work/ghidra-Ghidra_9.0.2_build/Ghidra/Features/Python/build.gradle' line: 28

* What went wrong:
A problem occurred evaluating project ':Python'.
> Could not resolve all files for configuration ':Python:jython'.
   > Could not resolve org.python:jython-standalone:2.7.1.
     Required by:
         project :Python
      > Could not resolve org.python:jython-standalone:2.7.1.
         > Could not get resource 'https://repo.maven.apache.org/maven2/org/python/jython-standalone/2.7.1/jython-standalone-2.7.1.pom'.
            > Could not GET 'https://repo.maven.apache.org/maven2/org/python/jython-standalone/2.7.1/jython-standalone-2.7.1.pom'.
               > repo.maven.apache.org
      > Could not resolve org.python:jython-standalone:2.7.1.
         > Could not get resource 'https://jcenter.bintray.com/org/python/jython-standalone/2.7.1/jython-standalone-2.7.1.pom'.
            > Could not GET 'https://jcenter.bintray.com/org/python/jython-standalone/2.7.1/jython-standalone-2.7.1.pom'.
               > jcenter.bintray.com





The other concern that I have is that gradle should be called with --offline during build, otherwise it would be downloading things which isn't allowed.

Please see biology/gatk as an example which uses gradle and is, IMO, a good gradle-based port implementation.
Comment 5 Tamas Szakaly 2019-05-21 17:27:15 UTC
(In reply to Yuri Victorovich from comment #4)

Yuri,

Thank you for the feedback, and sorry for the late response. I was, and still am swamped with work, but my next week is free, and I will finally have the time to fix these issues.

To answer your questions:

> What is the benefit of having the option selecting JDK11 vs. JDK12, why not just using once JDK?
I didn't want to force JDK12 on somebody who might need JDK11 for whatever reason, and vice versa.

> GH_PROJECT isn't needed when it is the same as PORTNAME.
Yes, you are right. And I've just noticed I also have GH_TUPLE defined - I don't really remember, but probably wanted to switch to GH_TUPLE at one point, and forgot to the delete the other GH_ defines.

> GH_TAGNAME= should be converted to DISTVERSIONPREFIX/DISTVERSION/DISTVERSIONSUFF
Thanks, this makes the Makefile a bit tidier.

> It fails to build in poudriere. ...

> The other concern that I have is that gradle should be called with --offline during build, otherwise it would be downloading things which isn't allowed.

Yes, the poudriere build fails because the port tries to download dependencies from Maven and JCentral during the build phase. I had some ideas (https://twitter.com/sghctoma/status/1123103954227929088) on how to resolve this, but somehow I didn't thought about the simple solution biology/gatk uses (host the dependencies as a separate tarball). One question though: would it be OK if I host the dependency tarball on my GitHub? Or is there a way to get such files hosted on the FreeBSD distfile cache sites?

Regards,
Toma
Comment 6 Yuri Victorovich freebsd_committer 2019-05-21 18:31:04 UTC
(In reply to Tamas Szakaly from comment #5)

Please see the port science/opsin for how to solve the dependencies issue.

Yuri
Comment 7 Rodney W. Grimes freebsd_committer 2019-05-23 04:48:30 UTC
(In reply to Kubilay Kocak from comment #2)
No, this should go in devel, all devel tools are usable for security purposes, not all security tools are developer tools.

Reverse engineering tools are a class of developer tools, just because I am reverse engineering does not mean there is any secuirty concern at all.
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-23 13:18:46 UTC
(In reply to Rodney W. Grimes from comment #7)

I won't die in/on the CATEGORIES bikeshed/hill, but I'll just note that (ports) try to lean towards using devel as a category of last resort unless something is entirely and only development related, or there isn't somewhere more specific. There's plenty of development-only software in !devel categories.

I also think its safe to assume that people would go looking for this in security as a first port of call, but the "development" class is also taken care of in my initial suggestion to use 'devel' it as a secondary category.

But again, having said that, CATEGORIES=devel security is fine too, and this might be a good example of something that is 'more in devel' than it is 'security'.
Comment 9 Yuri Victorovich freebsd_committer 2019-06-15 00:09:22 UTC
Feedback timeout expired: 20 days, more than the threshold of 14 days.
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-15 00:44:23 UTC
@Tamas Please let us know that you 'can' provide an updated and tested patch within a reasonable timeframe
Comment 11 Yuri Victorovich freebsd_committer 2019-06-19 19:04:47 UTC
(In reply to Kubilay Kocak from comment #10)

What is the timeout policy for things like this?

I thought that it is 14 days, which has expired.
Comment 12 Kurt Jaeger freebsd_committer 2019-06-19 19:12:43 UTC
Yes, but some ports are more relevant than others. So having ghidra in the tree sounds like a valid reason to be liberal in feedback timeout values 8-}
Comment 13 Yuri Victorovich freebsd_committer 2019-06-19 19:16:21 UTC
(In reply to Kurt Jaeger from comment #12)

I was going to add it, but now I have to wait for another timeout to expire.
Comment 14 Kurt Jaeger freebsd_committer 2019-06-19 19:23:44 UTC
On which timeout do you have to wait ? Please, just proceed. No need to be
pedantic on those new ports.
Comment 15 Yuri Victorovich freebsd_committer 2019-06-19 19:27:09 UTC
(In reply to Kurt Jaeger from comment #14)

Ok, thanks!
Comment 16 Tamas Szakaly 2019-06-20 06:36:34 UTC
Hi,

Sorry for my absence, life got in the way.
I will upload a patch this weekend that fixes the issues mentioned here, and also updates the port to version 9.0.4.
Comment 17 Tamas Szakaly 2019-06-24 07:18:32 UTC
Created attachment 205307 [details]
ghidra-9.0.4.shar

I've uploaded a new .shar archive that addresses the issues mentioned in this thread, and also updates Ghidra to version 9.0.4.
To summarize the changes:

 - Changed CATEGORIES to "devel security".
 - Defined DISTVERSIONPREFIX and DISTVERSIONSUFFIX, which made GH_PROJECT/GH_TAGNAME unnecessary.
 - Fixed the Poudriere build issue. I've copied relevant parts from science/opsin to achieve this. Both ports recommended here as examples (science/opsin and biology/gatk) use the LOCAL master site to store Maven/Gradle dependencies, which refers to the FreeBSD distfile cache sites. I don't know where and how to ask for a distfile to be uploaded there, so the port currently uses my GitHub for this purpose (line #72).

Thanks for your patience, and please let me know if any other changes are required!
Comment 18 Kurt Jaeger freebsd_committer 2019-06-24 07:23:10 UTC
testbuilds@work
Comment 19 Kurt Jaeger freebsd_committer 2019-06-24 13:48:41 UTC
builds on 13-amd64, fails on 12.0 or 11.2/3-amd64.

build error seems to be gradle-related ?

See for example:

https://people.freebsd.org/~pi/logs/devel__ghidra-120-1561360943.txt
Comment 20 Kurt Jaeger freebsd_committer 2019-06-24 13:49:35 UTC
(In reply to Tamas Szakaly from comment #17)
Can you tell which FreeBSD version was used in the poudriere test build ?
Comment 21 Tamas Szakaly 2019-06-24 14:01:00 UTC
(In reply to Kurt Jaeger from comment #20)

The machine itself runs HardenedBSD 13-CURRENT, but the poudriere jail is 12.0-RELEASE-p6 amd64. I'll install a 12.0 VM to try to replicate this error.
Comment 22 Tamas Szakaly 2019-06-26 06:37:44 UTC
Created attachment 205345 [details]
ghidra-9.0.4_v2.shar

A closer look at the log file revealed that the error was caused by the Poudriere settings BUILD_AS_NON_ROOT=yes/PORTBUILD_USER=nobody. The build process tries to write a log file under the user's home directory (/nonexistent in case of nobody), which of course fails. I didn't run into this, because I've had CCACHE_DIR defined, which cancels BUILD_AS_NON_ROOT.

The solution was to override the "user.home" Java system variable in the Gradle build file. I've built this new version successfully in 11.2-RELEASE-p10, 12.0-RELEASE-p6 and 13.0-CURRENT jails with BUILD_AS_NON_ROOT enabled.
Comment 23 Kurt Jaeger freebsd_committer 2019-06-28 18:02:58 UTC
testbuilds@work
Comment 24 Yuri Victorovich freebsd_committer 2019-06-28 18:05:49 UTC
I am swamped this week. Will try to look into it this weekend.
Comment 25 Kurt Jaeger freebsd_committer 2019-06-28 20:23:20 UTC
testbuilds are fine on 11.2a, 11.3-rc3-a, 12.0, current 8-)

Yuri, if it's OK, I'd commit it before Q3 ?
Comment 26 Yuri Victorovich freebsd_committer 2019-06-28 20:29:29 UTC
Ok!
Comment 27 Kurt Jaeger freebsd_committer 2019-06-29 00:18:16 UTC
Committed, thanks very much! Cool project!
Comment 28 commit-hook freebsd_committer 2019-06-29 00:18:37 UTC
A commit references this bug:

Author: pi
Date: Sat Jun 29 00:17:52 UTC 2019
New revision: 505315
URL: https://svnweb.freebsd.org/changeset/ports/505315

Log:
  New port: devel/ghidra

  Ghidra is a software reverse engineering (SRE) framework created
  and maintained by the National Security Agency Research Directorate
  of the United States of America.

  This framework includes a suite of full-featured, high-end software
  analysis tools that enable users to analyze compiled code on a
  variety of platforms including Windows, macOS, and Linux. Capabilities
  include disassembly, assembly, decompilation, graphing, and scripting,
  along with hundreds of other features. Ghidra supports a wide variety
  of processor instruction sets and executable formats and can be run
  in both user-interactive and automated modes. Users may also develop
  their own Ghidra plug-in components and/or scripts using Java or
  Python.

  WWW: https://ghidra-sre.org/

  PR:		237211
  Submitted by:	Tamas Szakaly <sghctoma@gmail.com>
  Reviewed by:	koobs, yuri, rgrimes

Changes:
  head/devel/Makefile
  head/devel/ghidra/
  head/devel/ghidra/Makefile
  head/devel/ghidra/distinfo
  head/devel/ghidra/files/
  head/devel/ghidra/files/patch-GPL_CabExtract_build.gradle
  head/devel/ghidra/files/patch-GPL_DemanglerGnu_build.gradle
  head/devel/ghidra/files/patch-GPL_build.gradle
  head/devel/ghidra/files/patch-GPL_nativeBuildProperties.gradle
  head/devel/ghidra/files/patch-GhidraBuild_Skeleton_certification.manifest
  head/devel/ghidra/files/patch-GhidraBuild_Skeleton_os_freebsd64_README.txt
  head/devel/ghidra/files/patch-Ghidra_Features_Decompiler_build.gradle
  head/devel/ghidra/files/patch-Ghidra_Features_Decompiler_src_decompile_cpp_Makefile
  head/devel/ghidra/files/patch-Ghidra_Features_Decompiler_src_decompile_cpp_loadimage__bfd.hh
  head/devel/ghidra/files/patch-Ghidra_Features_Decompiler_src_decompile_cpp_types.h
  head/devel/ghidra/files/patch-Ghidra_Features_FunctionID_build.gradle
  head/devel/ghidra/files/patch-Ghidra_Framework_Generic_src_main_java_ghidra_framework_Platform.java
  head/devel/ghidra/files/patch-Ghidra_Framework_Utility_src_main_java_ghidra_framework_OperatingSystem.java
  head/devel/ghidra/files/patch-Ghidra_RuntimeScripts_Linux_support_launch.sh
  head/devel/ghidra/files/patch-build.gradle
  head/devel/ghidra/files/patch-gradleScripts_distribution.gradle
  head/devel/ghidra/files/patch-gradleScripts_ip.gradle
  head/devel/ghidra/pkg-descr
Comment 29 Tamas Szakaly 2019-07-01 06:52:40 UTC
(In reply to Kurt Jaeger from comment #27)

Thanks for committing, and thanks to everyone who helped!