The following sections provide information that may be useful for developers of this utility.
Conventional Commits & Versioning
Versioning is handled automatically by release-please-action based on Conventional Commits. Every commit to the main
branch should follow the Conventional Commits convention. Following are some examples; these can be combined in a single commit message (separated by empty lines), or you can have commit messages describing just a single fix or feature.
chore: Won't show up in changelog ci: Change to GitHub Actions workflow; won't show up in changelog docs: Change to documentation; won't show up in changelog fix: Some fix (#2) feat: New feature (#3) feat!: Some feature that breaks backward compatibility feat: Some feature BREAKING-CHANGE: No longer supports xyz
See the output of git log
to view some sample commit messages.
release-please-action
invoked from the GitHub CI workflow generates pull requests containing updated CHANGELOG.md
and version.txt
files based on these commit messages. Merging the pull request will result in a new release version being published by creating a GitHub release describing the changes.
Prerequisites & Considerations
As can be seen in the Technologies & frameworks section, this is no ordinary Java project. Some of these technologies and frameworks require special prerequisites, precautions and other considerations to be taken into account to prevent compilation issues and runtime errors, as described below.
IDE Setup
This project uses the following frameworks that may require some special setup in order to have your IDE compile this project without errors:
-
Lombok: Please see https://projectlombok.org/setup/overview for more information on how to add Lombok support to your IDE
-
Annotation processors: This project requires various annotation processors to be run, like the picocli annotation processor. Please see your IDE documentation on how to enable annotation processing
Incremental Compilation
Incremental compilation (for example in IDE or when doing a gradle build
without clean
) may leave behind old AOT artifacts causing exceptions when trying to run fcli
. This is especially the case when renaming or moving classes, which may result in exceptions like the below:
Message: Error loading bean [com.fortify.cli.ssc.rest.unirest.SSCUnirestRunner]: com/fortify/cli/rest/unirest/AbstractUnirestRunner ... Caused by: java.lang.NoClassDefFoundError: com/fortify/cli/rest/unirest/AbstractUnirestRunner ...
In this example, the AbstractUnirestRunner class was moved to a different package, but the now obsolete AOT-generated classes were still present. So, if at any time you see unexpected exceptions, please try to do a full clean/rebuild and run fcli
again.
Reflection
GraalVM Native Image needs to know ahead-of-time the reflectively accessed program elements; see GraalVM Native Image & Reflection for details.
In fcli, the following needs to be taken into account to avoid reflection issues in native fcli binaries:
-
All data classes that are serialized or deserialized using Jackson must be annotated with the
@Reflectable
annotation. For de-serialization, a public no-args constructor is required as well. To avoid any potential issues, it is highly recommend to have both@Reflectable
and@NoArgsConstructor
annotations on such classes, preferably on the same line for easy identification.-
Failure to do so will cause errors when serializing or deserializing data to or from classes that lack this annotation.
-
Usually this only affects a small group of commands or a single command, possibly even only under certain conditions, and as such any missing annotations may be difficult to detect.
-
Most notably, the following classes will need the
@Reflectable
annotation:-
Most
*Descriptor
classes. -
Classes holding REST request or response data.
-
Classes holding data stored in the fcli data folder, like configuration data classes.
-
-
If a command runs fine on a regular JVM but not when running as a native image, then quite likely this is caused by missing reflection configuration which may be fixed by adding the @Reflectable
annotation to the appropriate classes.
Gradle Wrapper
It is strongly recommended to build this project using the included Gradle Wrapper scripts; using other Gradle versions may result in build errors and other issues.
The Gradle build uses various helper scripts from https://github.com/fortify/shared-gradle-helpers; please refer to the documentation and comments in included scripts for more information.
Common Commands
All commands listed below use Linux/bash notation; adjust accordingly if you are running on a different platform. All commands are to be executed from the main project directory.
-
./gradlew tasks --all
: List all available tasks -
Build: (plugin binary will be stored in
build/libs
)-
./gradlew clean build
: Clean and build the project -
./gradlew build
: Build the project without cleaning -
./gradlew dist distThirdParty
: Build distribution zip and third-party information bundle
-
Documentation
Two types of documentation are automatically being generated; the standard repository documentation like README.md
and CONTRIBUTING.md
, and fcli user documentation (including manual pages). The following two sections describe the generation process in more detail.
Repository Documentation
Most or all of the *.md
and LICENSE.txt
files located in the repository root are generated automatically. Generation of CHANGELOG.md
is done by release-please-action
as described in the Conventional Commits & Versioning section. Generation of the other files is done by the doc-resources/update-repo-docs.sh
scripts, based on the templates provided in https://github.com/fortify/shared-doc-resources, combined with the repo-specific MarkDown files in the repository doc-resources
directory. For more information about this generation process, please see https://github.com/fortify/shared-doc-resources/blob/main/USAGE.md.
User Documentation
User documentation is generated automatically by the buildRoot/app/fcli-doc
project. Part of the documentation consists of manual pages in various formats, generated using picocli. Other user documentation is generated from the src/docs/asciidoc
directory in the before-mentioned project. This consists of static AsciiDoc files, used to generate the top-level contents of the fcli GitHub Pages site, and 'versioned' AsciiDoc files that describe the functionality provided by a particular fcli version.
Documentation can be generated using the Gradle distDocs
task, ending up in the dist
directory of the aforementioned project. The global dist
task includes the distDocs
task.
The GitHub Actions workflow defined in .github/workflows/ci.yml
is responsible for publishing the documentation:
-
The
build
job builds the documentation artifacts and archives them as artifacts -
The
release
job publishesdocs-html.zip
anddocs-manpage.zip
to the release artifacts (when building a release or development version) -
The
publishPages
job published the output of theasciidoctorJekyll
andasciidoctorGHPages
to the appropriate directories on the GitHub Pages site, and updates the version index in the Jekyll_data
directory (when building a release or development version)
All HTML-formatted documentation described above is generated using the doc-resources/templates/html5/document.html.erb
template. This template is based on the official AsciiDoctor template with various modifications. Based on the attributes provided in the relevant Gradle tasks:
-
For Jekyll output:
-
Add Jekyll front matter
-
Add a Jekyll include to include additional content in the HTML
<head>
section; mostly used for applying stylesheets -
Add a Jekyll include to include the site-wide banner and (version) navigation bar
-
-
For offline HTML output:
-
Add hardcoded custom styling
-
Add hardcoded banner and version bar
-
The offline HTML documentation is supposed to be self-contained, i.e., pages should render correctly, without having to extract the full contents, if users open any HTML file from docs-html.zip
. In particular, this means that styles and images need to be embedded inside the HTML files. Of course, links to other documentation files will not work unless the full zip-file is extracted.
For now, the hardcoded banner and navigation bar in the offline documentation is similar to the banner included by Jekyll. However:
-
Stylesheets and images are linked rather than being included in the HTML page, allowing for better browser cache utilization
-
The navigation bar in the offline documentation contains just a static version number, whereas the navigation bar in the online documentation allows for navigating to different versions
-
We can potentially add more advanced (navigation) functionalities in the online documentation
-
We can easily update the banner for the online documentation to have a new layout/styling, for example to apply OpenText styling; this will be automatically applied to all existing online documentation pages
Usually it shouldn’t be necessary to update the documentation contents for existing release versions. However, if necessary, and assuming the build.gradle file is compatible with older versions, potentially a command like the following can be used to regenerate the documentation for the given versions:
for v in 1.0.0 1.0.1 1.0.2 1.0.3 1.0.4 1.0.5 1.1.0 1.2.0 1.2.1 1.2.2; do (git restore . && git clean -fd && git checkout v$v && cp -r ../fcli-fork/doc-resources ../fcli-fork/build.gradle . && ./gradlew clean distDocs -Pversion=$v && mkdir -p ~/Downloads/fcli-docs/$v && cp build/dist/docs-html.zip ~/Downloads/fcli-docs/$v && cd ../fcli-pages/v$v && echo $pwd && rm -rf * && unzip ../../fcli/build/dist/docs-jekyll.zip && cd - && git restore . && git clean -fd); done
This command iterates over the given version numbers, regenerates the documentation for each version (using latest build.gradle
and doc-resources
), copies the docs-html.zip
to a separate directory for later upload to the corresponding release assets, and updates the GitHub Pages site, based on the following assumptions:
-
Current directory is a clone of the fcli repository
-
../fcli-fork
would contain the latest version ofdoc-resources
andbuild.gradle
-
../fcli-pages
would be a clone of the fcli repository with the gh-pages branch checked out
Note that project directory structure may change beteen fcli versions, possibly requiring extra work to make the above work without issues.
Code Style & Structure
Coding Conventions
Common Java coding conventions should be used for fcli source code, taking the following into consideration:
-
Indentation is done using 4 spaces; fcli source code should not contain tabs.
-
Use of System.out and System.err should be avoided, except for code that explicitly handles output.
-
Command output should be generated through the output framework provided in the common module.
-
The logging framework should be used for outputting warning messages for example.
-
If you use System.out for debugging, potentially commenting out or removing these statements once done with debugging, consider using the logging framework for debug logging. If you need this information for debugging, it may be useful to permanently have this information included in debug logs.
-
-
Avoid having commented out source code.
-
Use the 'Organize Imports` feature of the IDE to remove any unused imports.
-
Avoid having unused variables, methods, …
-
Avoid unsafe type conversions. In particular, when using Jackson for deserializing generic types, use
TypeReference
instead of the generic type class.
Package Structure
Most fcli command modules use the package structure described below. Note that there may be slight variations between product-specific modules that interact with a remote system, and fcli-specific modules like config
and tool
.
-
com.fortify.cli.<module>
-
Root package for the given module
-
-
com.fortify.cli.<module>._main.cli.cmd
-
Contains
<module>Commands
class listing all entity commands for the given module. -
May contain command implementations that operate at module-level rather than entity-level, like the
ConfigClearCommand
.
-
-
com.fortify.cli.<module>.<entity>
-
Root package for the given module entity
-
-
com.fortify.cli.<module>.<entity>.cli
-
Root package for Picocli-based code, like command implementations and mixins
-
-
com.fortify.cli.<module>.<entity>.cli.cmd
-
Contains the
<module><entity>Commands
class, listing all sub-commands for the given entity -
Contains the individual entity action command classes
-
Where appropriate, sub-packages may be used to group related action commands
-
-
com.fortify.cli.<module>.<entity>.cli.mixin
-
Contains classes used as Mixin classes, for example defining reusable options and parameters, which may be used by commands in the current entity but also by other entities.
-
May contain classes used as ArgGroups, but these should be used sparingly as noted in ArgGroup Annotations
-
Usually contains a
<module><entity>ResolverMixin
class, containing inner classes that allow for resolving one or more<entity>
instances based on command-line options and/or positional parameters. Each inner class name describes the provided functionality, likeRequiredOption
,OptionalOption
,RequiredPositionalParameter
, …
-
-
com.fortify.cli.<module>.<entity>.helper
-
Contains entity-related helper classes, for example for loading entity data, deleting entities, …
-
Contains
*Descriptor
classes that hold entity-related data -
Classes in this package should not contain any picocli-related functionality; they should be designed in such a way that they could potentially be used in non-picocli applications
-
-
com.fortify.cli.<module>._common.*
-
Root package for module-specific generic functionality, like connecting to, and authenticating with, the remote system, generic request/response transformations, … Please see existing modules for example structure and contents.
-
Implementing fcli Commands
The following sections provide information on implementing fcli commands.
Command Structure
In general, we try to adhere to the following fcli command structure:
fcli <module> <entity> <action>
-
<module>
represents either a product likessc
orfod
, or an fcli-specific module likeconfig
,state
,tool
orutil
. -
<entity>
represents the entity on which the<action>
sub-commands operate, likeapp
,appversion
orappversion-attribute
. Virtually every entity should have its own top-level command inside a<module>
, we usually don’t use nested entities likeapp→version→attribute
. -
<action>
represents the action to be taken on the<entity>
and is usually a verb likelist
,get
,set
,delete
,update
, …-
If there are multiple variants of a particular command,
<action>
may include a suffix after the verb, likedownload-by-id
anddownload-state
, orpurge-by-id
andpurge-by-date
. -
Aliases should be used to maintain backward compatibility if needed. For example, if there is already a
delete
command that deletes by id, and a new command for deleting by date needs to be added, the original command would be renamed todelete-by-id
with aliasdelete
.
-
Fcli commands should be atomic and specific in nature. Each command should only do one thing, and do it well. A clear example are the wait-for
commands that provide a lot of wait-related options, rather than having the wait-related options on the command that initiated the action that we’re waiting for.
In general, exclusive options that influence the outcome of a command are an indication that a command is not specific enough; in such cases you may want to consider having multiple variants of the same command as described above. For example, if you are considering a single purge
command with exclusive options --id <id>
and --older-than <date>
, then having more specific purge-by-id
and purge-by-date
or even purge-older-than
commands would be more appropriate.
As usual, there are some exceptions to this rule, in particular for commands that are consistently named across fcli modules. For example, login commands often allow for logging in with either user or token credentials; for consistency we just have a single login command that provides options for either approach.
In general, each container command should contain either only leaf commands, or only container commands. For example, the top-level <module>
command should usually only contain <entity>
container commands and no leaf commands, whereas <entity>
commands should usually only contain leaf commands. There are some exceptions to this rule though, for example if a command operates on all entities within a module, like the fcli config clear
command.
Command Implementation
Most or all product-specific leaf command implementations should have the following generic structure:
@Command(name = [Module]OutputHelperMixins.<Action>.CMD_NAME)
public class <Module><Entity><Action>Command extends <SuperClass> implements <CommonInterfaces> {
@Getter @Mixin private [Module]OutputHelperMixins.<Action> outputHelper;
// Options, positional parameters, other fields; see next sections for info on options and parameters
// Overrides for interfaces, for example methods generating the output data,
// record transformations, ...
@Override
public boolean isSingular() {
return <false if potentially returning multiple records, true if always returning single record>;
}
}
-
<Module>
: Corresponds to the module in which this class is located; for product-specific commands this would correspond to the product name likeSSC
orFoD
. -
[Module]
: Commands can useIOutputHelper
implementations from either the genericOutputHelperMixins
or the module-specific[Module]OutputHelperMixins
-
<Entity>
represents the entity that the command is operating on, likeApp
,AppVersion
,User
, … -
<Action>
: Represents the action performed by this command; should be one of the classes in the[Module]OutputHelperMixins
class, likeGet
,List
,Delete
, …-
Every action should have a corresponding inner class in either the generic
OutputHelperMixins
class, or the module-specific[Module]OutputHelperMixins
class. -
In general, only concrete command implementations should declare references to
*OutputHelperMixins
classes, both in the@Command
declaration and theoutputHelper
field. Defining anoutputHelper
field in an (abstract) superclass will result in any aliases defined on*OutputHelperMixins
inner classes not being applied to the concrete command implementations.
-
-
<SuperClass>
is usuallyAbstract<Module>OutputCommand
or one of its abstract subclasses. Most modules provideAbstract<Module>JsonNodeOutputCommand
andAbstract<Module>BaseRequestOutputCommand
as bases classes for commands that generateJsonNode
orHttpRequest
instances respectively. Indirectly, virtually all leaf commands should extend fromAbstractOutputCommand
. -
<CommonInterfaces>
is a list of interfaces that define how output is being generated and processed. Following are some commonly used interfaces; see JavaDoc for details on usage:-
IBaseRequestSupplier
: Supply a UnirestHttpRequest
instance for retrieving command output data. -
IJsonNodeSupplier
: Supply aJsonNode
instance representing the command output. -
IActionCommandResultSupplier
: Supply data for a result column to be included in the output, likeDELETED
,CREATED
, … -
IInputTransformer
: Allows for transforming the full JSON data before it is being processed for output. -
IRecordTransformer
: Allows for transforming individual records before they are being processed for output.
-
Leaf commands in non-product modules usually have a similar structure, but some details may be different. Container commands, i.e. commands that represent <module>
and <entity>
have a very different (easier) structure as they don’t have any actual functionality associated to them; please refer to existing fcli container commands to serve as an example.
Commands & Options
Commands (including aliases) and options should use kebab-case names, i.e., lower-case names with words separated by a dash.
-
Acceptable command/option names:
app
,appversion-attribute
-
Invalid command/option names:
App
,appversionAttribute
,appversion_attribute
Commands may have one or more aliases, and multiple names may be defined for options. Having multiple names for a single command or option may be useful for various reasons, for example:
-
For providing shorter names, resulting in less typing for the user.
-
To maintain backward compatibility when renaming an existing command or option
-
As a 'default' command or option name if there are multiple variants of the same command or option, for example
purge
as the an alias forpurge-by-id
to have this as the default 'purge' operation if there are also otherpurge-*
commands.
In general:
-
Each command and option should have at most two names; the full name and a shorter form. For options, the shorter form is usually a single-letter option.
-
Having more than two names is only allowed if needed for backward compatibility; the backward-compatible name(s) should be removed on the next major release.
-
Options should always have a full name to describe their meaning, short (single-letter) name is optional.
-
Single-letter options are preceded by
-
(single dash), multi-letter options are preceded by--
(double dash).
ArgGroup Annotations
It was decided that ArgGroup
annotations should be used sparingly (Issue #89). ArgGroup
annotations are mostly used for generic options, like logging, help, output and query options. In general, command-specific options should use ArgGroup
annotations only for defining exclusive options, not for creating a separate section in the help output.
Options vs Positional Parameters
It was decided that every fcli command should have at most one positional parameter definition to specify the primary entity id or name that the command operates on. For example, <entity> delete
commands can have a positional parameter for specifying the entity name or id to be deleted, usually matching the <entity>
command group that the command is located at. For example, an app delete
command would take a positional parameter for identifying the app
to be deleted. Potentially the positional parameter may be an array or collection, for example if the app delete
command allows for deleting multiple apps in a single operation.
For everything else, including parent entities, options should be used. As an example, the fcli ssc appversion-attribute set
command takes the parent entity (application version) as an option named --appversion
, whereas the attribute names and values to be set are taken as a positional parameter Map
instance.
Message Keys
In general, standard picocli conventions should be used for locating i18n message keys for options, positional parameters, command descriptions, and so on. In some cases, it may be necessary to configure explicit messages keys in picocli annotations, for example for options and parameters defined in command super-classes or mixins.
As an example, take the @Option
descriptionKey
attributes in the AbstractToolInstallCommand
, specifying a fixed description key for all sub-commands. This way, the option description only needs to be specified once in the resource bundle, whereas using the standard picocli conventions would potentially have resulted in requiring the option description to be repeated for every individual command that extends from AbstractToolInstallCommand
.
Ideally, all commands, options and positional parameters should have a proper description. The following Linux commands can be used to find missing descriptions for options and commands:
-
Check which options don’t have a description:
./gradlew build generateManpageAsciiDoc && grep -e "^\\*-.*::" -A 1 build/generated-picocli-docs/*.adoc | grep -e "^build.*-\s*$" -B 1
-
Check which commands don’t have a usage header (inheriting the header of the root command):
./gradlew build && java -jar build/libs/fcli.jar util all-commands list --include-parents | fgrep "Command-line interface for working with"
Where applicable, option and positional parameter descriptions should include references to other related fcli commands, in particular when these related commands are in a separate command tree. For example, available attribute names and values that can be specified on the fcli ssc appversion-attribute set
command can be found through the fcli ssc attribute-definition *
commands. Being in a separate command tree, this may not be obvious to users and as such should be documented on the fcli ssc appversion-attribute set
command.
Comparing this to the 'application name or id' to be passed to the fcli ssc app get
command; available application id’s can be found through the fcli ssc app list
command under the same app
parent command, so this doesn’t need to be documented as it should be obvious to users.