The following sections provide information that may be useful for developers of this utility. Note that this documentation is not always properly updated, so some details may be out of date and no longer apply. However, general concepts won’t change very often, so even with some outdated details, it’s still useful to review this documentation if you’re contributing to fcli. If you notice anything that’s incorrect or outdated, please let us know or update yourself.

Branches

The dev/v3.x branch is our primary branch for active fcli development. In general, new features/fixes/…​ should be developed in a dedicated branch or fcli fork, and eventually merged into the dev/v3.x branch by raising a Pull Request.

Once we are ready for releasing a new fcli version, the dev/v3.x branch will be merged into the rel/v3.x branch, which, based on conventional commit messages (see next section) will result in a release PR to be automatically raised; merging this PR will automatically build the fcli release version and publish the various release artifacts. Once released, the rel/v3.x branch needs to be merged back into the dev/v3.x branch to apply changelog and version updates.

In some cases, the process above may require a rewrite of the commit history on the dev/v3.x branch, as we may have a need to rewrite some commit messages to generate proper release notes. So, after an fcli release, you should always update/replace any branches/forks/local copies to apply the new commit history to avoid issues when merging new changes.

In general, the rel/v3.x branch shouldn’t be touched (other than as described above), except if we need to apply some (relatively urgent) bug fix at a time when the dev/v3.x branch is not yet ready for release. In general, this is only applicable if we’ve already merged some features to the dev/v3.x branch that still need more work before final release.

Apart from the branches listed above, there’s also a gh-pages branch that is used to publish documentation to https://fortify.github.io/fcli. Most of the files in this branch are either static or automatically generated during fcli builds, so other than changing documentation structure or performing cleanup (like removing documentation for outdated fcli versions), in general this branch shouldn’t be touched.

Conventional Commits & Versioning

rel/v3.x & dev/v3.x branches

Versioning is handled automatically by release-please-action based on Conventional Commits. Every commit to the dev/v3.x branch should follow the Conventional Commits convention, with those commit messages then being merged into the rel/v3.x branch to generate release notes. Following are some examples; these can be combined in a single commit message (separated by empty lines), or you can have commit messages describing just a single fix or feature.

chore: Won't show up in changelog

ci: Change to GitHub Actions workflow; won't show up in changelog

docs: Change to documentation; won't show up in changelog

fix: `fcli ssc issue list`: Fix ... (fixes #2)

fix: SSC `gitlab-sast-report` action: Fix ... (fixes #3)

feat: Add `fcli fod issue update` command ... (resolves #4)

feat: `fcli fod issue list`: Add `--app` option to allow for querying issues at application level (resolves #5)

feat!: Remove `fcli fod issue list` command; use `...` instead (resolves #6)

feat!: `fcli ssc issue list`: Remove `--xyz` option (resolves #7)

See the output of git log to view some sample commit messages. As shown above, commit messages usually list the fcli command and any options that were added/removed/changed in literal blocks (back quotes), and include the associated GitHub issue number to allow the issue to be automatically closed when the commit message is pushed/merged onto the dev/v3.x branch. If an existing command was updated, we first list the command name (may include wild cards, like +`fcli fod issue *+`, followed by a colon, followed by the description. If a command was added/removed, we prefix the command name with Add/Remove. This allows for nice (alphabetical) ordering of release note entries.

Note that such commit messages should only describe changes since the last fcli release, not changes within the current development release. For example, suppose you’ve developed a new feature that hasn’t been released yet, and then implement some fix for this unreleased feature, you shouldn’t use a fix: commit message but instead use a chore: or similar message. Otherwise, the changelog would contain entries for both the new feature and fixes to that feature, which doesn’t make sense.

release-please-action invoked from the GitHub CI workflow generates pull requests containing updated CHANGELOG.md and version.txt files based on these commit messages. Merging the pull request will result in a new release version being published by creating a GitHub release describing the changes.

Feature & other branches

Given that feature and other branches are usually squash-merged into the dev/v3.x branch, you can use any kind of commit messages, although to avoid any incorrect or duplicate messages appearing in the change log, it may be better to not use conventional commit messages as described above, but for example use something like proposed-feat: instead.

Please include the feat/fix messages as described in the previous section in a literal text block in your pull request, to allow for easy copy/paste into the squash commit.

Technologies & Frameworks

Following is a list of the main frameworks and technologies used by fcli:

  • picocli: Used for fcli command implementations; perform fcli actions, process command line options, generate usage information, generate command output, …

  • Jackson: Parse and generate data in JSON and other formats

  • GraalVM: Generate native images (native executables)

Prerequisites & Considerations

As can be seen in the Technologies & frameworks section, this is no ordinary Java project. Some of these technologies and frameworks require special prerequisites, precautions and other considerations to be taken into account to prevent compilation issues and runtime errors, as described below.

IDE Setup

This project uses the following frameworks that may require some special setup in order to have your IDE compile this project without errors:

  • Lombok: Please see https://projectlombok.org/setup/overview for more information on how to add Lombok support to your IDE

  • Annotation processors: This project requires various annotation processors to be run, like the picocli annotation processor. Please see your IDE documentation on how to enable annotation processing

Incremental Compilation (Java)

Incremental compilation (for example in IDE or when doing a gradle build without clean) may leave behind old AOT artifacts causing exceptions when trying to run fcli. This is especially the case when renaming or moving classes, which may result in exceptions like the below:

Message: Error loading bean [com.fortify.cli.ssc.rest.unirest.SSCUnirestRunner]: com/fortify/cli/rest/unirest/AbstractUnirestRunner
...
Caused by: java.lang.NoClassDefFoundError: com/fortify/cli/rest/unirest/AbstractUnirestRunner
...

In this example, the AbstractUnirestRunner class was moved to a different package, but the now obsolete AOT-generated classes were still present. So, if at any time you see unexpected exceptions, please try to do a full clean/rebuild and run fcli again.

Incremental Compilation (Resources)

Incremental compilation may sometimes result in incorrect resource file handling. For example, action zip files are being updated with updated action YAML files, rather than being fully replaced. This can sometimes cause incorrect behavior, like removed actions still being available, or updated actions not being properly replaced.

Reflection

GraalVM Native Image needs to know ahead-of-time the reflectively accessed program elements; see GraalVM Native Image & Reflection for details.

In fcli, the following needs to be taken into account to avoid reflection issues in native fcli binaries:

  • All data classes that are serialized or deserialized using Jackson must be annotated with the @Reflectable annotation. For de-serialization, a public no-args constructor is required as well. To avoid any potential issues, it is highly recommend to have both @Reflectable and @NoArgsConstructor annotations on such classes, preferably on the same line for easy identification.

    • Failure to do so will cause errors when serializing or deserializing data to or from classes that lack this annotation.

    • Usually this only affects a small group of commands or a single command, possibly even only under certain conditions, and as such any missing annotations may be difficult to detect.

    • Most notably, the following classes will need the @Reflectable annotation:

      • Most *Descriptor classes.

      • Classes holding REST request or response data.

      • Classes holding data stored in the fcli data folder, like configuration data classes.

If a command runs fine on a regular JVM but not when running as a native image, then quite likely this is caused by missing reflection configuration which may be fixed by adding the @Reflectable annotation to the appropriate classes.

Gradle Wrapper

It is strongly recommended to build this project using the included Gradle Wrapper scripts; using other Gradle versions may result in build errors and other issues.

The Gradle build uses various helper scripts from https://github.com/fortify/shared-gradle-helpers; please refer to the documentation and comments in included scripts for more information.

Common Commands

All commands listed below use Linux/bash notation; adjust accordingly if you are running on a different platform. All commands are to be executed from the main project directory.

  • ./gradlew tasks --all: List all available tasks

  • ./gradlew :fcli-core:fcli-app:dependencyUpdates: List potential dependency updates for main fcli executable

  • Build: (plugin binary will be stored in build/libs and or build/dist)

    • ./gradlew clean build: Clean and build the project

    • ./gradlew build: Build the project without cleaning

    • ./gradlew dist distThirdParty: Build distribution zip and third-party information bundle

Documentation

Two types of documentation are automatically being generated; the standard repository documentation like README.md and CONTRIBUTING.md, and fcli user documentation (including manual pages). The following two sections describe the generation process in more detail.

Repository Documentation

Most or all of the *.md and LICENSE.txt files located in the repository root are generated automatically. Generation of CHANGELOG.md is done by release-please-action as described in the Conventional Commits & Versioning section. Generation of the other files is done by the doc-resources/update-repo-docs.sh scripts, based on the templates provided in https://github.com/fortify/shared-doc-resources, combined with the repo-specific MarkDown files in the repository doc-resources directory. For more information about this generation process, please see https://github.com/fortify/shared-doc-resources/blob/main/USAGE.md.

User Documentation

User documentation is generated automatically by the fcli-other/fcli-doc project. Part of the documentation consists of manual pages in various formats, generated using picocli. Other user documentation is generated from the src/docs/asciidoc directory in the before-mentioned project. This consists of static AsciiDoc files, used to generate the top-level contents of the fcli GitHub Pages site, and 'versioned' AsciiDoc files that describe the functionality provided by a particular fcli version.

Documentation can be generated using the Gradle distDocs task, ending up in the dist directory of the aforementioned project. The global dist task includes the distDocs task.

The GitHub Actions workflow defined in .github/workflows/ci.yml is responsible for publishing the documentation:

  • The build job builds the documentation artifacts and archives them as artifacts

  • The release job publishes docs-html.zip and docs-manpage.zip to the release artifacts (when building a release or development version)

  • The publishPages job published the output of the asciidoctorJekyll and asciidoctorGHPages to the appropriate directories on the GitHub Pages site, and updates the version index in the Jekyll _data directory (when building a release or development version)

All HTML-formatted documentation described above is generated using the doc-resources/templates/html5/document.html.erb template. This template is based on the official AsciiDoctor template with various modifications. Based on the attributes provided in the relevant Gradle tasks:

  • For Jekyll output:

    • Add Jekyll front matter

    • Add a Jekyll include to include additional content in the HTML <head> section; mostly used for applying stylesheets

    • Add a Jekyll include to include the site-wide banner and (version) navigation bar

  • For offline HTML output:

    • Add hardcoded custom styling

    • Add hardcoded banner and version bar

The offline HTML documentation is supposed to be self-contained, i.e., pages should render correctly, without having to extract the full contents, if users open any HTML file from docs-html.zip. In particular, this means that styles and images need to be embedded inside the HTML files. Of course, links to other documentation files will not work unless the full zip-file is extracted.

For now, the hardcoded banner and navigation bar in the offline documentation is similar to the banner included by Jekyll. However:

  • Stylesheets and images are linked rather than being included in the HTML page, allowing for better browser cache utilization

  • The navigation bar in the offline documentation contains just a static version number, whereas the navigation bar in the online documentation allows for navigating to different versions

  • We can potentially add more advanced (navigation) functionalities in the online documentation

  • We can easily update the banner for the online documentation to have a new layout/styling, for example to apply OpenText styling; this will be automatically applied to all existing online documentation pages

Usually it shouldn’t be necessary to update the documentation contents for existing release versions. However, if necessary, and assuming the build.gradle file is compatible with older versions, potentially a command like the following can be used to regenerate the documentation for the given versions:

for v in 1.0.0 1.0.1 1.0.2 1.0.3 1.0.4 1.0.5 1.1.0 1.2.0 1.2.1 1.2.2; do (git restore . && git clean -fd && git checkout v$v && cp -r ../fcli-fork/doc-resources ../fcli-fork/build.gradle . && ./gradlew clean distDocs -Pversion=$v && mkdir -p ~/Downloads/fcli-docs/$v && cp build/dist/docs-html.zip ~/Downloads/fcli-docs/$v && cd ../fcli-pages/v$v && echo $pwd && rm -rf * && unzip ../../fcli/build/dist/docs-jekyll.zip && cd - && git restore . && git clean -fd); done

This command iterates over the given version numbers, regenerates the documentation for each version (using latest build.gradle and doc-resources), copies the docs-html.zip to a separate directory for later upload to the corresponding release assets, and updates the GitHub Pages site, based on the following assumptions:

  • Current directory is a clone of the fcli repository

  • ../fcli-fork would contain the latest version of doc-resources and build.gradle

  • ../fcli-pages would be a clone of the fcli repository with the gh-pages branch checked out

Note that project directory structure may change beteen fcli versions, possibly requiring extra work to make the above work without issues.

Code Style & Structure

See https://github.com/fortify/fcli/blob/dev/v3.x/.github/copilot-instructions.md

Implementing fcli Commands

The following sections provide information on implementing fcli commands.

Command Structure

In general, we try to adhere to the following fcli command structure:

fcli <module> <entity> <action>

  • <module> represents either a product like ssc or fod, or an fcli-specific module like config, state, tool or util.

  • <entity> represents the entity on which the <action> sub-commands operate, like app, appversion or appversion-attribute. Virtually every entity should have its own top-level command inside a <module>, we usually don’t use nested entities like app→version→attribute.

  • <action> represents the action to be taken on the <entity> and is usually a verb like list, get, set, delete, update, …​

    • If there are multiple variants of a particular command, <action> may include a suffix after the verb, like download-by-id and download-state, or purge-by-id and purge-by-date.

    • Aliases should be used to maintain backward compatibility if needed. For example, if there is already a delete command that deletes by id, and a new command for deleting by date needs to be added, the original command would be renamed to delete-by-id with alias delete.

Fcli commands should be atomic and specific in nature. Each command should only do one thing, and do it well. A clear example are the wait-for commands that provide a lot of wait-related options, rather than having the wait-related options on the command that initiated the action that we’re waiting for.

In general, exclusive options that influence the outcome of a command are an indication that a command is not specific enough; in such cases you may want to consider having multiple variants of the same command as described above. For example, if you are considering a single purge command with exclusive options --id <id> and --older-than <date>, then having more specific purge-by-id and purge-by-date or even purge-older-than commands would be more appropriate.

As usual, there are some exceptions to this rule, in particular for commands that are consistently named across fcli modules. For example, login commands often allow for logging in with either user or token credentials; for consistency we just have a single login command that provides options for either approach.

In general, each container command should contain either only leaf commands, or only container commands. For example, the top-level <module> command should usually only contain <entity> container commands and no leaf commands, whereas <entity> commands should usually only contain leaf commands. There are some exceptions to this rule though, for example if a command operates on all entities within a module, like the fcli config clear command.

Command Implementation

Most or all product-specific leaf command implementations should have the following generic structure:

@Command(name = [Module]OutputHelperMixins.<Action>.CMD_NAME)
public class <Module><Entity><Action>Command extends <SuperClass> implements <CommonInterfaces> {
    @Getter @Mixin private [Module]OutputHelperMixins.<Action> outputHelper;
    // Options, positional parameters, other fields; see next sections for info on options and parameters

    // Overrides for interfaces, for example methods generating the output data,
    // record transformations, ...

    @Override
    public boolean isSingular() {
        return <false if potentially returning multiple records, true if always returning single record>;
    }
}

  • <Module>: Corresponds to the module in which this class is located; for product-specific commands this would correspond to the product name like SSC or FoD.

  • [Module]: Commands can use IOutputHelper implementations from either the generic OutputHelperMixins or the module-specific [Module]OutputHelperMixins

  • <Entity> represents the entity that the command is operating on, like App, AppVersion, User, …​

  • <Action>: Represents the action performed by this command; should be one of the classes in the [Module]OutputHelperMixins class, like Get, List, Delete, …​

    • Every action should have a corresponding inner class in either the generic OutputHelperMixins class, or the module-specific [Module]OutputHelperMixins class.

    • In general, only concrete command implementations should declare references to *OutputHelperMixins classes, both in the @Command declaration and the outputHelper field. Defining an outputHelper field in an (abstract) superclass will result in any aliases defined on *OutputHelperMixins inner classes not being applied to the concrete command implementations.

  • <SuperClass> is usually Abstract<Module>OutputCommand or one of its abstract subclasses. Most modules provide Abstract<Module>JsonNodeOutputCommand and Abstract<Module>BaseRequestOutputCommand as base classes for commands that generate JsonNode or HttpRequest instances respectively, but note that this is legacy code; most commands should now extend directly from Abstract<Module>OutputCommand and generate an IObjectNodeProducer instance through one of the *Builder methods defined on AbstractOutputCommand, like simpleObjectNodeProducerBuilder or requestObjectNodeProducerBuilder. Indirectly, virtually all leaf commands should extend from AbstractOutputCommand.

  • <CommonInterfaces> is a list of interfaces that define how output is being generated and processed. Following are some commonly used interfaces; see JavaDoc for details on usage:

    • IBaseRequestSupplier (legacy; use RequestObjectNodeProducerBuilder instead): Supply a Unirest HttpRequest instance for retrieving command output data.

    • IJsonNodeSupplier (legacy; use SimpleObjectNodeProducerBuilder instead): Supply a JsonNode instance representing the command output.

    • IActionCommandResultSupplier: Supply data for a result column to be included in the output, like DELETED, CREATED, …​

    • IInputTransformer (legacy; configure directly on *ObjectNodeProducerBuilder): Hook to transform the full JSON data before it is processed. Applied once per input in AbstractObjectNodeProducer.

    • IRecordTransformer (legacy; configure directly on *ObjectNodeProducerBuilder): Hook to transform individual records. Applied to each record in AbstractObjectNodeProducer.

Leaf commands in non-product modules usually have a similar structure, but some details may be different. Container commands, i.e. commands that represent <module> and <entity> have a very different (easier) structure as they don’t have any actual functionality associated to them; please refer to existing fcli container commands to serve as an example.

Commands & Options

Commands (including aliases) and options should use kebab-case names, i.e., lower-case names with words separated by a dash.

  • Acceptable command/option names: app, appversion-attribute

  • Invalid command/option names: App, appversionAttribute, appversion_attribute

Commands may have one or more aliases, and multiple names may be defined for options. Having multiple names for a single command or option may be useful for various reasons, for example:

  • For providing shorter names, resulting in less typing for the user.

  • To maintain backward compatibility when renaming an existing command or option

  • As a 'default' command or option name if there are multiple variants of the same command or option, for example purge as the an alias for purge-by-id to have this as the default 'purge' operation if there are also other purge-* commands.

In general:

  • Each command and option should have at most two names; the full name and a shorter form. For options, the shorter form is usually a single-letter option.

  • Having more than two names is only allowed if needed for backward compatibility; the backward-compatible name(s) should be removed on the next major release.

  • Options should always have a full name to describe their meaning, short (single-letter) name is optional.

  • Single-letter options are preceded by - (single dash), multi-letter options are preceded by -- (double dash).

ArgGroup Annotations

It was decided that ArgGroup annotations should be used sparingly (Issue #89). ArgGroup annotations are mostly used for generic options, like logging, help, output and query options. In general, command-specific options should use ArgGroup annotations only for defining exclusive options, not for creating a separate section in the help output.

Options vs Positional Parameters

It was decided that every fcli command should have at most one positional parameter definition to specify the primary entity id or name that the command operates on. For example, <entity> delete commands can have a positional parameter for specifying the entity name or id to be deleted, usually matching the <entity> command group that the command is located at. For example, an app delete command would take a positional parameter for identifying the app to be deleted. Potentially the positional parameter may be an array or collection, for example if the app delete command allows for deleting multiple apps in a single operation.

For everything else, including parent entities, options should be used. As an example, the fcli ssc appversion-attribute set command takes the parent entity (application version) as an option named --appversion, whereas the attribute names and values to be set are taken as a positional parameter Map instance.

For fcli v4.0, we should consider using options only; it’s difficult to identify the proper FCLI_DEFAULT_* environment variable to use for positional parameters, and it’s simpler for both developers (no need to decide between option vs positional parameter) and users (although a bit more typing, users won’t need to look up whether a given command accepts positional parameter or option, for example for specifying application version).

Message Keys

In general, standard picocli conventions should be used for locating i18n message keys for options, positional parameters, command descriptions, and so on. In some cases, it may be necessary to configure explicit messages keys in picocli annotations, for example for options and parameters defined in command super-classes or mixins.

As an example, take the @Option descriptionKey attributes in the AbstractToolInstallCommand, specifying a fixed description key for all sub-commands. This way, the option description only needs to be specified once in the resource bundle, whereas using the standard picocli conventions would potentially have resulted in requiring the option description to be repeated for every individual command that extends from AbstractToolInstallCommand.

Ideally, all commands, options and positional parameters should have a proper description. The following Linux commands can be used to find missing descriptions for options and commands:

  • Check which options don’t have a description:
    ./gradlew build generateManpageAsciiDoc && grep -e "^\\*-.*::" -A 1 build/generated-picocli-docs/*.adoc | grep -e "^build.*-\s*$" -B 1

  • Check which commands don’t have a usage header (inheriting the header of the root command):
    ./gradlew build && java -jar build/libs/fcli.jar util all-commands list --include-parents | fgrep "Command-line interface for working with"

Where applicable, option and positional parameter descriptions should include references to other related fcli commands, in particular when these related commands are in a separate command tree. For example, available attribute names and values that can be specified on the fcli ssc appversion-attribute set command can be found through the fcli ssc attribute-definition * commands. Being in a separate command tree, this may not be obvious to users and as such should be documented on the fcli ssc appversion-attribute set command.

Comparing this to the 'application name or id' to be passed to the fcli ssc app get command; available application id’s can be found through the fcli ssc app list command under the same app parent command, so this doesn’t need to be documented as it should be obvious to users.

Fcli action-related development

Implementing fcli built-in actions

We have fairly extensive user documentation regarding fcli action development, including an action schema for IDE code assistance, and many existing actions that may be used as an example. Virtually all of this documentation also applies to internal actions, except for one exception: built-in actions should always use the dev schema.

With every GitHub workflow run (for any branch), the dev schema will be automatically updated to reflect any changes in the action model (syntax), allowing IDE code assistance to take these changes into account (you may need to restart your IDE to apply any schema changes). This allows for utilizing any new/updated action syntax while developing built-in actions.

Upon fcli release, the dev schema referenced from built-in actions will be automatically updated to reflect current action schema release version.

Action schema

Fcli action schemas are versioned independently from fcli; as not every fcli release will have schema changes, it doesn’t make much sense to republish an already existing schema with a new (fcli) version number. Having schema’s versioned independently allows for easily identifying when there have been any schema changes, and avoids users having to update the schema version used by their custom actions upon every fcli release.

The schema is automatically generated from the …​action.model.Action class structure located in fcli-common, and current action schema version is defined in the fcliActionSchemaVersion property in gradle.properties in the fcli root project. Upon each fcli release, the configured schema version will be published if it hasn’t been published yet upon earlier fcli releases.

If any changes are made to the action model source code, the fcliActionSchemaVersion property must be updated accordingly, taking into account proper major, minor, or patch version increase based on the type of changes; see comments in gradle.properties for details. If a given fcliActionSchemaVersion has been previously released, and any changes in action model are detected, the Gradle generateActionSchema task will fail as we won’t allow updating any previously released schema.

If the currently configured fcliActionSchemaVersion hasn’t been published yet, GitHub workflows will produce a warning stating that a new schema version will be released upon the next fcli release. This is to try to avoid situations like the following:

  1. Someone updates a property description and increases fcliActionSchemaVersion to the next patch release to avoid the generateActionSchema task from failing.

  2. Later on, someone makes a structural change to the action model, but forgets to update the fcliActionSchemaVersion property.

  3. Upon fcli release, we now release a schema patch version, whereas it should have been a minor or major version change.

Any suggestions on better ways to avoid such situations are welcome.