<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2026-10-26T15:07:04+00:00</updated><id>/feed.xml</id><title type="html">jammy.id</title><subtitle>The technology focused blog of James Hewitt, covering whatever he happens to be playing with at the time.</subtitle><author><name>James Hewitt</name></author><entry><title type="html">Resolving Out-of-Sync App Connect Resources in OpenShift GitOps</title><link href="/integration/app-connect-openshift-gitops-sync.html" rel="alternate" type="text/html" title="Resolving Out-of-Sync App Connect Resources in OpenShift GitOps" /><published>2026-10-26T00:00:00+00:00</published><updated>2026-10-26T00:00:00+00:00</updated><id>/integration/app-connect-openshift-gitops-sync</id><content type="html" xml:base="/integration/app-connect-openshift-gitops-sync.html"><![CDATA[<p>When managing IBM App Connect in containers (part of IBM Integration Software, formerly Cloud Pak for Integration) with OpenShift GitOps (Argo CD), newly deployed <code class="language-plaintext highlighter-rouge">IntegrationRuntime</code> and <code class="language-plaintext highlighter-rouge">Configuration</code> resources can immediately appear <code class="language-plaintext highlighter-rouge">OutOfSync</code>.</p>

<p>This occurs because the App Connect operator automatically mutates live Custom Resources (CRs) after creation by injecting default configurations and migrating inline data to secrets.</p>

<!--more-->

<blockquote>
  <p><strong>Note on Naming:</strong> Cloud Pak for Integration has been renamed to <strong>IBM Integration Software</strong>. While operator license annotations and fields (such as <code class="language-plaintext highlighter-rouge">use: CloudPakForIntegrationNonProductionFREE</code>) may still carry the legacy identifiers for backward compatibility, the platform itself is now IBM Integration Software.</p>
</blockquote>

<hr />

<h2 id="issue-1-integrationruntime-auto-injected-configurations">Issue 1: IntegrationRuntime Auto-Injected Configurations</h2>

<p>By default, an <code class="language-plaintext highlighter-rouge">IntegrationRuntime</code> enables both the admin REST interface and the MCP (Model Context Protocol) server. If you do not specify configuration objects for these features, the operator automatically provisions them and appends two entries to <code class="language-plaintext highlighter-rouge">spec.configurations</code> on the live cluster:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">&lt;runtime-name&gt;-mcp-ba-creds</code></li>
  <li><code class="language-plaintext highlighter-rouge">&lt;runtime-name&gt;-ir-adminssl</code></li>
</ul>

<h3 id="example">Example</h3>

<p>You declare this resource in your Git repository:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">appconnect.ibm.com/v1beta1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">IntegrationRuntime</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">greetingapi1</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">license</span><span class="pi">:</span>
    <span class="na">accept</span><span class="pi">:</span> <span class="no">true</span>
    <span class="na">license</span><span class="pi">:</span> <span class="s">L-RJGW-BUAA2R</span>
    <span class="na">use</span><span class="pi">:</span> <span class="s">CloudPakForIntegrationNonProductionFREE</span>
  <span class="na">barURL</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">https://gitea.example.com/barfiles/raw/branch/main/greetingapiv1.bar</span>
  <span class="na">configurations</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">barauth-empty</span>
  <span class="na">logFormat</span><span class="pi">:</span> <span class="s">basic</span>
  <span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">13.0.8"</span>
</code></pre></div></div>

<p>The operator mutates the live resource on the cluster:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">spec</span><span class="pi">:</span>
  <span class="na">configurations</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">barauth-empty</span>
    <span class="pi">-</span> <span class="s">greetingapi1-mcp-ba-creds</span>
    <span class="pi">-</span> <span class="s">greetingapi1-ir-adminssl</span>
</code></pre></div></div>

<p>Because Git only contains <code class="language-plaintext highlighter-rouge">barauth-empty</code>, Argo CD reports the resource as <code class="language-plaintext highlighter-rouge">OutOfSync</code>.</p>

<h3 id="solutions">Solutions</h3>

<ol>
  <li>
    <p><strong>Disable MCP if unused (Recommended for MCP):</strong>
If you are not using the MCP server, disable it explicitly in Git under <code class="language-plaintext highlighter-rouge">spec.mcp.runtime.disabled</code>:</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">spec</span><span class="pi">:</span>
  <span class="na">mcp</span><span class="pi">:</span>
    <span class="na">runtime</span><span class="pi">:</span>
      <span class="na">disabled</span><span class="pi">:</span> <span class="no">true</span>
</code></pre></div>    </div>
    <p>This prevents the operator from injecting <code class="language-plaintext highlighter-rouge">&lt;name&gt;-mcp-ba-creds</code>.</p>
  </li>
  <li>
    <p><strong>Handle the Admin SSL Configuration:</strong>
For <code class="language-plaintext highlighter-rouge">*-ir-adminssl</code>, you can either pre-create and declare your own admin SSL <code class="language-plaintext highlighter-rouge">Configuration</code> object in Git (recommended), or configure Argo CD to ignore the generated configuration in its diff calculation.</p>

    <p>To ignore the generated SSL configuration across all <code class="language-plaintext highlighter-rouge">IntegrationRuntime</code> resources, add a <code class="language-plaintext highlighter-rouge">resourceIgnoreDifferences</code> entry to your <code class="language-plaintext highlighter-rouge">ArgoCD</code> custom resource:</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">spec</span><span class="pi">:</span>
  <span class="na">resourceIgnoreDifferences</span><span class="pi">:</span>
    <span class="na">resourceIdentifiers</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">group</span><span class="pi">:</span> <span class="s">appconnect.ibm.com</span>
        <span class="na">kind</span><span class="pi">:</span> <span class="s">IntegrationRuntime</span>
        <span class="na">customization</span><span class="pi">:</span>
          <span class="na">jqPathExpressions</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="s1">'</span><span class="s">.spec.configurations[]</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">select(endswith("-ir-adminssl"))'</span>
</code></pre></div>    </div>

    <p>You can use similar configuration in a GitOps <code class="language-plaintext highlighter-rouge">Application</code> if you don’t control the whole GitOps installation. Use the same block, but under <code class="language-plaintext highlighter-rouge">spec.ignoreDifferences</code>.</p>
  </li>
</ol>

<hr />

<h2 id="issue-2-configuration-resources-mutating-specdata">Issue 2: Configuration Resources Mutating <code class="language-plaintext highlighter-rouge">spec.data</code></h2>

<p>When you create a <code class="language-plaintext highlighter-rouge">Configuration</code> resource that defines inline base64 data using <code class="language-plaintext highlighter-rouge">spec.data</code>, the operator extracts the payload into an automatically generated <code class="language-plaintext highlighter-rouge">Secret</code>, removes <code class="language-plaintext highlighter-rouge">spec.data</code> from the CR on the cluster, and sets <code class="language-plaintext highlighter-rouge">spec.secretName</code> and <code class="language-plaintext highlighter-rouge">spec.version</code>.</p>

<h3 id="example-1">Example</h3>

<p>You declare this in Git:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">appconnect.ibm.com/v1beta1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Configuration</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">barauth-empty</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">type</span><span class="pi">:</span> <span class="s">barauth</span>
  <span class="na">description</span><span class="pi">:</span> <span class="s">Empty BarAuth file for git access.</span>
  <span class="na">data</span><span class="pi">:</span> <span class="s">eyJhdXRoVHlwZSI6IkJBU0lDX0FVVEgiLCJjcmVkZW50aWFscyI6eyJ1c2VybmFtZSI6IiIsInBhc3N3b3JkIjoiIn19Cg==</span>
</code></pre></div></div>

<p>On the cluster, the operator updates it to:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">spec</span><span class="pi">:</span>
  <span class="na">type</span><span class="pi">:</span> <span class="s">barauth</span>
  <span class="na">description</span><span class="pi">:</span> <span class="s">Empty BarAuth file for git access.</span>
  <span class="na">secretName</span><span class="pi">:</span> <span class="s">barauth-empty-bpq5l</span>
  <span class="na">version</span><span class="pi">:</span> <span class="s">13.0.8.2-r1</span>
</code></pre></div></div>

<p>Argo CD flags the missing <code class="language-plaintext highlighter-rouge">spec.data</code> as a live diff.</p>

<h3 id="solutions-1">Solutions</h3>

<ol>
  <li>
    <p><strong>Create the Secret directly (Recommended):</strong>
Instead of embedding <code class="language-plaintext highlighter-rouge">spec.data</code> directly in the <code class="language-plaintext highlighter-rouge">Configuration</code> CR, manage the Kubernetes <code class="language-plaintext highlighter-rouge">Secret</code> explicitly with GitOps (leveraging Sealed Secrets, External Secrets, or SOPS for sensitive values) and reference it with <code class="language-plaintext highlighter-rouge">spec.secretName</code>:</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Secret</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">barauth-empty-secret</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">Opaque</span>
<span class="na">data</span><span class="pi">:</span>
  <span class="na">configuration</span><span class="pi">:</span> <span class="s">eyJhdXRoVHlwZSI6IkJBU0lDX0FVVEgiLCJjcmVkZW50aWFscyI6eyJ1c2VybmFtZSI6IiIsInBhc3N3b3JkIjoiIn19Cg==</span>
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">appconnect.ibm.com/v1beta1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Configuration</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">barauth-empty</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">type</span><span class="pi">:</span> <span class="s">barauth</span>
  <span class="na">description</span><span class="pi">:</span> <span class="s">Empty BarAuth file for git access.</span>
  <span class="na">secretName</span><span class="pi">:</span> <span class="s">barauth-empty-secret</span>
</code></pre></div>    </div>
  </li>
  <li>
    <p><strong>Ignore <code class="language-plaintext highlighter-rouge">spec.data</code> differences in Argo CD:</strong>
If you want to keep inline <code class="language-plaintext highlighter-rouge">spec.data</code> in Git, configure the <code class="language-plaintext highlighter-rouge">ArgoCD</code> CR to ignore <code class="language-plaintext highlighter-rouge">.spec.data</code> differences on <code class="language-plaintext highlighter-rouge">Configuration</code> resources:</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">spec</span><span class="pi">:</span>
  <span class="na">resourceIgnoreDifferences</span><span class="pi">:</span>
    <span class="na">resourceIdentifiers</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">group</span><span class="pi">:</span> <span class="s">appconnect.ibm.com</span>
        <span class="na">kind</span><span class="pi">:</span> <span class="s">Configuration</span>
        <span class="na">customization</span><span class="pi">:</span>
          <span class="na">jqPathExpressions</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="s">.spec.data</span>
</code></pre></div>    </div>
  </li>
</ol>]]></content><author><name>James Hewitt</name></author><category term="integration" /><category term="appconnect" /><category term="integration" /><category term="ace" /><category term="openshift" /><category term="gitops" /><category term="argocd" /><category term="cp4i" /><summary type="html"><![CDATA[When managing IBM App Connect in containers (part of IBM Integration Software, formerly Cloud Pak for Integration) with OpenShift GitOps (Argo CD), newly deployed IntegrationRuntime and Configuration resources can immediately appear OutOfSync. This occurs because the App Connect operator automatically mutates live Custom Resources (CRs) after creation by injecting default configurations and migrating inline data to secrets.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="/assets/app-connect-gitops-sync.jpg" /><media:content medium="image" url="/assets/app-connect-gitops-sync.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">IBM App Connect Enterprise containers using image volume mounts</title><link href="/integration/ace-image-volume.html" rel="alternate" type="text/html" title="IBM App Connect Enterprise containers using image volume mounts" /><published>2026-08-18T00:00:00+00:00</published><updated>2026-08-18T00:00:00+00:00</updated><id>/integration/ace-image-volume</id><content type="html" xml:base="/integration/ace-image-volume.html"><![CDATA[<p>You can use <a href="https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/">image volume mounts</a> in an App Connect Runtime using the App Connect operator as part of Cloud Pak for Integration.
Most Cloud Pak for Integration and IBM App Connect in containers users will have a local registry for storing their container images and will typically mirror software to ensure reliable image access.
Reusing this infrastructure for bar files makes deployment and reliability simple.</p>

<p>The image will be pulled using the same pull secrets available to the pod and can have its own pull policy specified.</p>

<h2 id="step-1-put-bar-files-into-an-image">Step 1: Put bar files into an image</h2>

<p>Use a very simple <code class="language-plaintext highlighter-rouge">FROM scratch</code> Dockerfile to copy the bar files you want onto an image.</p>

<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> scratch</span>

<span class="k">COPY</span><span class="s"> strings.bar .</span>
</code></pre></div></div>

<p>If you don’t want to install a full container client into your system, you can also use <code class="language-plaintext highlighter-rouge">crane</code> to build the image.</p>

<p>Build and push this image somewhere your cluster can pull it from:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>podman build -t quay.io/jammy/strings-bar:latest .
podman push quay.io/jammy/strings-bar:latest
</code></pre></div></div>

<h2 id="step-2-configure-your-app-connect-runtime-to-mount-the-image">Step 2: Configure your App Connect runtime to mount the image</h2>

<p>Mount the image with your bar files into the containers where the App Connect runtime can pick them up.</p>

<p>In the standard image, you can use <code class="language-plaintext highlighter-rouge">/home/aceuser/initial-config/bars/</code></p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">appconnect.ibm.com/v1beta1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">IntegrationRuntime</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">strings</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">strings</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">license</span><span class="pi">:</span>
    <span class="na">accept</span><span class="pi">:</span> <span class="no">false</span> <span class="c1"># Change this to true</span>
    <span class="na">license</span><span class="pi">:</span> <span class="s">L-RJGW-BUAA2R</span>
    <span class="na">use</span><span class="pi">:</span> <span class="s">CloudPakForIntegrationNonProductionFREE</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">runtime</span>
          <span class="na">volumeMounts</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/home/aceuser/initial-config/bars</span>
              <span class="na">name</span><span class="pi">:</span> <span class="s">strings</span>
      <span class="na">volumes</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">strings</span>
          <span class="na">image</span><span class="pi">:</span>
            <span class="na">reference</span><span class="pi">:</span> <span class="s1">'</span><span class="s">quay.io/jammy/strings-bar:latest'</span>
            <span class="na">pullPolicy</span><span class="pi">:</span> <span class="s">IfNotPresent</span>
  <span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">13.0'</span>
</code></pre></div></div>

<h2 id="test-that-your-bar-is-loaded">Test that your bar is loaded</h2>

<p>Now in the logs of the App Connect runtime pod, you’ll see the bar file being loaded:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2026-08-18 15:14:40.068456: Deploying bar files
2026-08-18 15:14:40.068567: List of bars to deploy: -a "/home/aceuser/initial-config/bars/strings.bar"
</code></pre></div></div>

<p>For most deployments you’re also likely to want some more App Connect configuration like credentials.
You can add those to the same runtime in the same way as you would using the App Connect dashboard or other gitops approaches.</p>]]></content><author><name>James Hewitt</name></author><category term="integration" /><category term="appconnect" /><category term="integration" /><category term="ace" /><category term="image" /><category term="volume" /><summary type="html"><![CDATA[You can use image volume mounts in an App Connect Runtime using the App Connect operator as part of Cloud Pak for Integration. Most Cloud Pak for Integration and IBM App Connect in containers users will have a local registry for storing their container images and will typically mirror software to ensure reliable image access. Reusing this infrastructure for bar files makes deployment and reliability simple.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="/assets/ace-image-volume.jpg" /><media:content medium="image" url="/assets/ace-image-volume.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Automatic FIPS 140-3 in Go: What Actually Works</title><link href="/kubernetes/go-automatic-fips.html" rel="alternate" type="text/html" title="Automatic FIPS 140-3 in Go: What Actually Works" /><published>2026-08-07T00:00:00+00:00</published><updated>2026-08-07T00:00:00+00:00</updated><id>/kubernetes/go-automatic-fips</id><content type="html" xml:base="/kubernetes/go-automatic-fips.html"><![CDATA[<p>Ideally, you want to build software once and run it anywhere. With FIPS and Go that
can be tricky because in upstream, FIPS is baked in at build time. Using the Red Hat
golang toolchain you can produce binaries that detect the FIPS mode of the underlying
host, and automatically run in the right mode.</p>

<p>Go 1.26 introduced <code class="language-plaintext highlighter-rouge">crypto/fips140</code> — a new standard library package giving Go
programs first-class awareness of FIPS 140-3 mode. Red Hat has been shipping a
patched Go toolchain in their UBI images for longer than that, with OpenSSL-backed
crypto that integrates with the Linux kernel’s FIPS flag.</p>

<p>But how do these two mechanisms interact? And which build combinations actually
give you automatic, kernel-aware FIPS — meaning FIPS active on a FIPS-enabled
host and transparent everywhere else, from the same binary?</p>

<p>We ran a systematic test across all meaningful build combinations on both a
FIPS-enabled RHEL 10 host and a standard non-FIPS Linux host. This post shares
what we found. The test program and Makefile are available at
<a href="https://github.com/Jamstah/fips-go">github.com/Jamstah/fips-go</a>.</p>

<h2 id="tldr">TL;DR</h2>

<p>This blog post is all based on Go 1.26. In Go 1.27, Red Hat are no longer supporting
use of openssl as the crypto for their Go toolset, see
<a href="https://github.com/golang-fips/go/blob/main/README.md#migration-to-upstream-fips-certified-cryptography">Migration to Upstream FIPS certified cryptography</a>.</p>

<p>To build a Go binary that automatically uses FIPS on FIPS-enabled hosts and runs
normally everywhere else, pick one:</p>

<p><strong>Option A — OpenSSL backend (simplest, no runtime config):</strong></p>

<p>The platform openssl is used at runtime, so your runtime OS needs to include openssl and support FIPs.</p>

<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="w"> </span><span class="s">registry.access.redhat.com/ubi10/go-toolset:1.26</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">builder</span>
<span class="k">RUN </span>go build <span class="nt">-o</span> /myapp <span class="nb">.</span>          <span class="c"># no GOFIPS140, no no_openssl</span>

<span class="k">FROM</span><span class="s"> registry.access.redhat.com/ubi10/ubi-minimal</span>
<span class="k">COPY</span><span class="s"> --from=builder /myapp /usr/local/bin/myapp</span>
<span class="k">ENTRYPOINT</span><span class="s"> ["/usr/local/bin/myapp"]</span>
</code></pre></div></div>

<p><strong>Option B — Go native FIPS module (<code class="language-plaintext highlighter-rouge">fips140.Enabled()</code> works correctly):</strong></p>

<p>You have to build on the Red Hat toolset but you can run on any OS because native Go FIPS is used. You need to set the GODEBUG option at runtime for automatic FIPS enablement.</p>

<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="w"> </span><span class="s">registry.access.redhat.com/ubi10/go-toolset:1.26</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">builder</span>
<span class="k">RUN </span>go build <span class="nt">-tags</span> no_openssl <span class="nt">-o</span> /myapp .

<span class="k">FROM</span><span class="s"> registry.access.redhat.com/ubi10/ubi-minimal</span>
<span class="k">COPY</span><span class="s"> --from=builder /myapp /usr/local/bin/myapp</span>
<span class="k">ENV</span><span class="s"> GODEBUG=fips140=auto</span>
<span class="k">ENTRYPOINT</span><span class="s"> ["/usr/local/bin/myapp"]</span>
</code></pre></div></div>

<p><strong>Option C — Go native FIPS module and config using <code class="language-plaintext highlighter-rouge">go.mod</code>:</strong></p>

<p>You need to build on the Red Hat toolset and you can run anywhere without runtime configuration because the default GODEBUG fips140 setting is linked in to the binary.</p>

<p>Add a <code class="language-plaintext highlighter-rouge">godebug</code> directive to <code class="language-plaintext highlighter-rouge">go.mod</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>godebug fips140=auto
</code></pre></div></div>

<p>Then build with <code class="language-plaintext highlighter-rouge">no_openssl</code> on the Red Hat toolset:</p>
<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="w"> </span><span class="s">registry.access.redhat.com/ubi10/go-toolset:1.26</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">builder</span>
<span class="k">RUN </span>go build <span class="nt">-tags</span> no_openssl <span class="nt">-o</span> /myapp .

<span class="k">FROM</span><span class="s"> registry.access.redhat.com/ubi10/ubi-minimal</span>
<span class="k">COPY</span><span class="s"> --from=builder /myapp /usr/local/bin/myapp</span>
<span class="k">ENTRYPOINT</span><span class="s"> ["/usr/local/bin/myapp"]</span>
</code></pre></div></div>

<hr />

<h2 id="background-two-independent-fips-mechanisms">Background: Two independent FIPS mechanisms</h2>

<p>Before looking at results, it helps to understand that there are two completely
separate ways Go can enforce FIPS on RHEL/UBI. They do not communicate with
each other.</p>

<h3 id="1-the-go-native-fips-module-gofips140">1. The Go native FIPS module (<code class="language-plaintext highlighter-rouge">GOFIPS140</code>)</h3>

<p><code class="language-plaintext highlighter-rouge">GOFIPS140</code> is a build-time environment variable consumed by <code class="language-plaintext highlighter-rouge">cmd/go</code>. When set
to anything other than <code class="language-plaintext highlighter-rouge">off</code>, it compiles the FIPS 140-3 cryptographic module
into the binary and bakes a <code class="language-plaintext highlighter-rouge">GODEBUG=fips140=on</code> default into the binary at
link time. This default is not a runtime environment variable — it is embedded
in the binary and takes effect on every host, unconditionally.</p>

<p>The <code class="language-plaintext highlighter-rouge">crypto/fips140.Enabled()</code> function introduced in Go 1.26 reflects this
mechanism.</p>

<h3 id="2-the-red-hat-openssl-backend">2. The Red Hat OpenSSL backend</h3>

<p>The Red Hat <code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code> image replaces Go’s standard crypto with an
OpenSSL-backed implementation by default (<code class="language-plaintext highlighter-rouge">GOEXPERIMENT=opensslcrypto</code>). OpenSSL
reads <code class="language-plaintext highlighter-rouge">/proc/sys/crypto/fips_enabled</code> at process startup and enforces FIPS when
the kernel flag is set — completely independently of <code class="language-plaintext highlighter-rouge">GOFIPS140</code>.</p>

<p><strong><code class="language-plaintext highlighter-rouge">crypto/fips140.Enabled()</code> is blind to this mechanism.</strong> A binary built with
<code class="language-plaintext highlighter-rouge">GOFIPS140=off</code> on a FIPS-enabled RHEL host will report <code class="language-plaintext highlighter-rouge">fips140.Enabled() = false</code>
while OpenSSL is actively enforcing FIPS.</p>

<h3 id="why-you-cant-combine-them">Why you can’t combine them</h3>

<p>Combining <code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code> (or any non-off value) with the OpenSSL backend
panics at startup:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>panic: opensslcrypto: GOLANG_FIPS and GODEBUG=fips140 are mutually exclusive
</code></pre></div></div>

<p>The Red Hat toolchain deliberately prevents both paths from being active
simultaneously. The <code class="language-plaintext highlighter-rouge">no_openssl</code> build tag disables the OpenSSL backend and
switches to the pure-Go FIPS module.</p>

<hr />

<h2 id="the-test">The test</h2>

<p>We built a small Go program that:</p>

<ol>
  <li>Prints build metadata from the binary’s embedded build info (Go version,
<code class="language-plaintext highlighter-rouge">GOFIPS140</code>, build tags) so each run is self-describing</li>
  <li>Reports <code class="language-plaintext highlighter-rouge">crypto/fips140.Enabled()</code></li>
  <li>Attempts an HTTPS connection to <code class="language-plaintext highlighter-rouge">3des.badssl.com</code>, a host that only
negotiates the non-FIPS 3DES cipher suite. The client is configured to
offer only that suite (<code class="language-plaintext highlighter-rouge">TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA</code>, TLS 1.2 max,
<code class="language-plaintext highlighter-rouge">InsecureSkipVerify</code> since cert validation is not what we’re testing). If
anything has removed 3DES from the stack, the handshake fails — no runtime
flags needed.</li>
</ol>

<p>We tested eleven combinations: eight distinct image builds, plus three of those
re-run with <code class="language-plaintext highlighter-rouge">GODEBUG=fips140=auto</code> injected at runtime.</p>

<hr />

<h2 id="results">Results</h2>

<h3 id="the-full-matrix">The full matrix</h3>

<table>
  <thead>
    <tr>
      <th>Target</th>
      <th><code class="language-plaintext highlighter-rouge">GODEBUG</code> at runtime</th>
      <th><code class="language-plaintext highlighter-rouge">fips140.Enabled()</code></th>
      <th>3DES on FIPS host</th>
      <th>3DES on non-FIPS host</th>
      <th>FIPS mechanism</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">golang:1.26</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140=off</code></td>
      <td>—</td>
      <td>false</td>
      <td>succeeded</td>
      <td>succeeded</td>
      <td>none</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">golang:1.26</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140</code> unset</td>
      <td>—</td>
      <td>false</td>
      <td>succeeded</td>
      <td>succeeded</td>
      <td>none (unset == off on upstream)</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">golang:1.26</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code></td>
      <td>—</td>
      <td><strong>true</strong></td>
      <td><strong>blocked</strong></td>
      <td><strong>blocked</strong></td>
      <td>Go native, unconditional</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140=off</code></td>
      <td>—</td>
      <td>false</td>
      <td><strong>blocked</strong></td>
      <td>succeeded</td>
      <td>OpenSSL reads kernel flag</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140</code> unset</td>
      <td>—</td>
      <td>false</td>
      <td><strong>blocked</strong></td>
      <td>succeeded</td>
      <td>same as above</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140=off</code>, <code class="language-plaintext highlighter-rouge">no_openssl</code></td>
      <td>—</td>
      <td>false</td>
      <td>succeeded</td>
      <td>succeeded</td>
      <td>none</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140</code> unset, <code class="language-plaintext highlighter-rouge">no_openssl</code></td>
      <td>—</td>
      <td>false</td>
      <td>succeeded</td>
      <td>succeeded</td>
      <td>none</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code>, <code class="language-plaintext highlighter-rouge">no_openssl</code></td>
      <td>—</td>
      <td><strong>true</strong></td>
      <td><strong>blocked</strong></td>
      <td><strong>blocked</strong></td>
      <td>Go native, unconditional</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140=off</code>, <code class="language-plaintext highlighter-rouge">no_openssl</code></td>
      <td><code class="language-plaintext highlighter-rouge">fips140=auto</code></td>
      <td>false/<strong>true</strong></td>
      <td><strong>blocked</strong></td>
      <td>succeeded</td>
      <td>Go native, kernel-aware</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140</code> unset, <code class="language-plaintext highlighter-rouge">no_openssl</code></td>
      <td><code class="language-plaintext highlighter-rouge">fips140=auto</code></td>
      <td>false/<strong>true</strong></td>
      <td><strong>blocked</strong></td>
      <td>succeeded</td>
      <td>Go native, kernel-aware</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code>, <code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code>, <code class="language-plaintext highlighter-rouge">no_openssl</code></td>
      <td><code class="language-plaintext highlighter-rouge">fips140=auto</code></td>
      <td>false/<strong>true</strong></td>
      <td><strong>blocked</strong></td>
      <td>succeeded</td>
      <td><code class="language-plaintext highlighter-rouge">auto</code> overrides link-time <code class="language-plaintext highlighter-rouge">on</code></td>
    </tr>
  </tbody>
</table>

<h3 id="key-observations">Key observations</h3>

<p><strong><code class="language-plaintext highlighter-rouge">crypto/fips140.Enabled()</code> only reflects the Go native FIPS module.</strong> When
the OpenSSL backend is enforcing FIPS, <code class="language-plaintext highlighter-rouge">fips140.Enabled()</code> returns false. You
cannot use this API to check whether FIPS is active in an OpenSSL-backed binary.</p>

<p><strong><code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code> is unconditional.</strong> The FIPS module is active on every
host, FIPS-enabled or not. There is no “activate only if the kernel has FIPS
enabled” behaviour in the upstream toolchain.</p>

<p><strong><code class="language-plaintext highlighter-rouge">GODEBUG=fips140=auto</code> overrides the link-time default in both directions.</strong>
On a <code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code> binary running on a non-FIPS host, setting
<code class="language-plaintext highlighter-rouge">GODEBUG=fips140=auto</code> at runtime disables FIPS — 3DES succeeded and
<code class="language-plaintext highlighter-rouge">fips140.Enabled()</code> returned false.</p>

<hr />

<h2 id="the-two-approaches-that-give-you-automatic-fips">The two approaches that give you automatic FIPS</h2>

<p>If you want a binary that is FIPS-compliant on a FIPS host and runs normally
elsewhere — without changing the binary between environments — there are two
approaches that work today.</p>

<h3 id="approach-1-option-a-build-with-the-red-hat-toolset-keep-the-openssl-backend">Approach 1 (Option A): Build with the Red Hat toolset, keep the OpenSSL backend</h3>

<p>This is the simplest approach and the default when using <code class="language-plaintext highlighter-rouge">ubi10/go-toolset</code>.</p>

<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="w"> </span><span class="s">registry.access.redhat.com/ubi10/go-toolset:1.26</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">builder</span>
<span class="c"># Do NOT set GOFIPS140, do NOT add no_openssl</span>
<span class="k">RUN </span>go build <span class="nt">-o</span> /myapp .

<span class="k">FROM</span><span class="s"> registry.access.redhat.com/ubi10/ubi-minimal</span>
<span class="k">COPY</span><span class="s"> --from=builder /myapp /usr/local/bin/myapp</span>
<span class="k">ENTRYPOINT</span><span class="s"> ["/usr/local/bin/myapp"]</span>
</code></pre></div></div>

<p>At runtime, OpenSSL reads <code class="language-plaintext highlighter-rouge">/proc/sys/crypto/fips_enabled</code> and automatically
enforces FIPS when the kernel flag is set. On a non-FIPS host, it runs normally.
No build flags, no environment variables, no operator action required.</p>

<p><strong>The catch:</strong> <code class="language-plaintext highlighter-rouge">crypto/fips140.Enabled()</code> returns false even when FIPS is
active, because enforcement is happening in OpenSSL, not the Go native module.
If your application code calls <code class="language-plaintext highlighter-rouge">fips140.Enabled()</code> to gate behaviour, it will
not work correctly with this approach.</p>

<p>Also note: for this to work at runtime, the container must link against OpenSSL
— meaning you need a runtime image that includes OpenSSL libraries (such as
<code class="language-plaintext highlighter-rouge">ubi-minimal</code> or <code class="language-plaintext highlighter-rouge">ubi</code>), not a scratch image.</p>

<h3 id="approach-2-build-with-the-red-hat-toolset-no_openssl-and-set-godebugfips140auto">Approach 2: Build with the Red Hat toolset, <code class="language-plaintext highlighter-rouge">no_openssl</code>, and set <code class="language-plaintext highlighter-rouge">GODEBUG=fips140=auto</code></h3>

<p>This approach uses the Go native FIPS module and activates it via the
<code class="language-plaintext highlighter-rouge">GODEBUG=fips140=auto</code> setting introduced by the Red Hat patches to the Go
toolchain. At startup, the Go runtime reads <code class="language-plaintext highlighter-rouge">/proc/sys/crypto/fips_enabled</code>.
If it is <code class="language-plaintext highlighter-rouge">1</code>, FIPS mode activates and <code class="language-plaintext highlighter-rouge">fips140.Enabled()</code> returns true. On a
non-FIPS host, <code class="language-plaintext highlighter-rouge">auto</code> resolves to off and the program runs normally.</p>

<p>There are two ways to set <code class="language-plaintext highlighter-rouge">fips140=auto</code> — pick whichever fits your workflow.</p>

<h4 id="option-b-set-it-in-the-dockerfile">Option B: Set it in the Dockerfile</h4>

<p>Bake it into the container image itself with <code class="language-plaintext highlighter-rouge">ENV</code>:</p>

<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="w"> </span><span class="s">registry.access.redhat.com/ubi10/go-toolset:1.26</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">builder</span>
<span class="k">RUN </span>go build <span class="nt">-tags</span> no_openssl <span class="nt">-o</span> /myapp .

<span class="k">FROM</span><span class="s"> registry.access.redhat.com/ubi10/ubi-minimal</span>
<span class="k">COPY</span><span class="s"> --from=builder /myapp /usr/local/bin/myapp</span>
<span class="k">ENV</span><span class="s"> GODEBUG=fips140=auto</span>
<span class="k">ENTRYPOINT</span><span class="s"> ["/usr/local/bin/myapp"]</span>
</code></pre></div></div>

<h4 id="option-c-set-it-in-gomod-go-123">Option C: Set it in <code class="language-plaintext highlighter-rouge">go.mod</code> (Go 1.23+)</h4>

<p>Add a <code class="language-plaintext highlighter-rouge">godebug</code> directive to your module’s <code class="language-plaintext highlighter-rouge">go.mod</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>godebug fips140=auto
</code></pre></div></div>

<p>The Go toolchain bakes this into the binary as a link-time default — no <code class="language-plaintext highlighter-rouge">ENV</code>
in the Dockerfile and no Kubernetes configuration needed. The binary carries
the setting with it regardless of where it is deployed.</p>

<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="w"> </span><span class="s">registry.access.redhat.com/ubi10/go-toolset:1.26</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">builder</span>
<span class="k">RUN </span>go build <span class="nt">-tags</span> no_openssl <span class="nt">-o</span> /myapp .

<span class="k">FROM</span><span class="s"> registry.access.redhat.com/ubi10/ubi-minimal</span>
<span class="k">COPY</span><span class="s"> --from=builder /myapp /usr/local/bin/myapp</span>
<span class="k">ENTRYPOINT</span><span class="s"> ["/usr/local/bin/myapp"]</span>
</code></pre></div></div>

<p><strong>The advantage over Approach 1:</strong> <code class="language-plaintext highlighter-rouge">crypto/fips140.Enabled()</code> correctly reflects
the FIPS state, so your application code can query it reliably. The binary can
also run on a scratch or non-OpenSSL runtime image since it has no OpenSSL
dependency.</p>

<p><strong>The caveat:</strong> <code class="language-plaintext highlighter-rouge">GODEBUG=fips140=auto</code> overrides the link-time default in both
directions. If it is set (via <code class="language-plaintext highlighter-rouge">ENV</code>, <code class="language-plaintext highlighter-rouge">go.mod</code>, or the host environment) and you
also deploy binaries built with <code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code>, those binaries will have
FIPS disabled on non-FIPS hosts. Only use <code class="language-plaintext highlighter-rouge">auto</code> if you want kernel-aware
activation. If you need FIPS unconditionally, use <code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code> and do not
set a runtime <code class="language-plaintext highlighter-rouge">GODEBUG</code> override.</p>

<hr />

<h2 id="a-note-on-gofips140-unset-with-the-red-hat-toolchain">A note on <code class="language-plaintext highlighter-rouge">GOFIPS140</code> unset with the Red Hat toolchain</h2>

<p>The <a href="https://github.com/golang-fips/go/blob/go1.26.5/patches/004-host-fips-auto.patch">golang-fips patch</a>
is designed to make <code class="language-plaintext highlighter-rouge">GOFIPS140</code> unset default to <code class="language-plaintext highlighter-rouge">v1.0.0</code> (a NIST-certified
snapshot) with <code class="language-plaintext highlighter-rouge">GODEBUG=fips140=auto</code> baked in at link time — giving automatic
kernel-aware activation from an unmodified build, with no explicit <code class="language-plaintext highlighter-rouge">go.mod</code>
entry and no runtime env var.</p>

<p>However, <code class="language-plaintext highlighter-rouge">ubi10/go-toolset:1.26</code> does not yet implement this default:
<code class="language-plaintext highlighter-rouge">go env GOFIPS140</code> returns <code class="language-plaintext highlighter-rouge">off</code> when unset. Until the patch ships in this image,
use Option C (<code class="language-plaintext highlighter-rouge">go.mod</code>) or Option B (<code class="language-plaintext highlighter-rouge">ENV</code>) from Approach 2 above to achieve
the same result today.</p>

<hr />

<h2 id="what-about-the-upstream-golang126-image">What about the upstream <code class="language-plaintext highlighter-rouge">golang:1.26</code> image?</h2>

<p>The upstream <code class="language-plaintext highlighter-rouge">golang:1.26</code> image does not include the Red Hat OpenSSL patches.
<code class="language-plaintext highlighter-rouge">GOFIPS140</code> unset is treated as <code class="language-plaintext highlighter-rouge">off</code>. The only way to get FIPS is
<code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code>, which activates it unconditionally on all hosts.</p>

<p>If you need a kernel-aware, automatic FIPS behaviour with the upstream image,
you would need to detect <code class="language-plaintext highlighter-rouge">/proc/sys/crypto/fips_enabled</code> yourself at startup and
set <code class="language-plaintext highlighter-rouge">GODEBUG=fips140=on</code> programmatically — but that is not standard Go
behaviour and outside the scope of this post.</p>

<p>For FIPS requirements, the Red Hat UBI toolset images are the right choice.</p>

<hr />

<h2 id="summary">Summary</h2>

<table>
  <thead>
    <tr>
      <th>Goal</th>
      <th>Approach</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>FIPS always active, all hosts</td>
      <td><code class="language-plaintext highlighter-rouge">GOFIPS140=latest</code>, <code class="language-plaintext highlighter-rouge">no_openssl</code>, no runtime override</td>
    </tr>
    <tr>
      <td>FIPS only on FIPS hosts (kernel-aware), <code class="language-plaintext highlighter-rouge">fips140.Enabled()</code> accurate</td>
      <td><code class="language-plaintext highlighter-rouge">no_openssl</code>, <code class="language-plaintext highlighter-rouge">GODEBUG=fips140=auto</code> at runtime</td>
    </tr>
    <tr>
      <td>FIPS only on FIPS hosts (kernel-aware), simpler setup</td>
      <td>Default UBI toolset build (OpenSSL backend, no <code class="language-plaintext highlighter-rouge">no_openssl</code>)</td>
    </tr>
    <tr>
      <td>No FIPS</td>
      <td><code class="language-plaintext highlighter-rouge">GOFIPS140=off</code> or upstream image</td>
    </tr>
  </tbody>
</table>

<p>For production workloads that need to run on both FIPS and non-FIPS hosts from a
single image, either the OpenSSL default or the <code class="language-plaintext highlighter-rouge">GODEBUG=fips140=auto</code> approach
will work today. The OpenSSL approach requires no runtime configuration but does
not expose FIPS state through <code class="language-plaintext highlighter-rouge">crypto/fips140.Enabled()</code>. The <code class="language-plaintext highlighter-rouge">auto</code> approach
gives you accurate <code class="language-plaintext highlighter-rouge">fips140.Enabled()</code> reporting but requires setting a runtime
environment variable until the <code class="language-plaintext highlighter-rouge">v1.0.0</code> default patch ships in the toolchain.</p>

<p>The test code, Dockerfiles, and Makefile for all eleven combinations tested here
are available at <a href="https://github.com/Jamstah/fips-go">github.com/Jamstah/fips-go</a>.</p>]]></content><author><name>James Hewitt</name></author><category term="kubernetes" /><category term="go" /><category term="fips" /><category term="redhat" /><category term="containers" /><category term="kubernetes" /><summary type="html"><![CDATA[Ideally, you want to build software once and run it anywhere. With FIPS and Go that can be tricky because in upstream, FIPS is baked in at build time. Using the Red Hat golang toolchain you can produce binaries that detect the FIPS mode of the underlying host, and automatically run in the right mode.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="/assets/go-automatic-fips.jpg" /><media:content medium="image" url="/assets/go-automatic-fips.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Fixing an IBM MQ web console redirect loop on OpenShift</title><link href="/integration/mq-web-console-http2.html" rel="alternate" type="text/html" title="Fixing an IBM MQ web console redirect loop on OpenShift" /><published>2025-06-05T00:00:00+00:00</published><updated>2025-06-05T00:00:00+00:00</updated><id>/integration/mq-web-console-http2</id><content type="html" xml:base="/integration/mq-web-console-http2.html"><![CDATA[<p>I’ve seen a number of Cloud Pak for Integration installations where the MQ web console is inaccessible because of a redirect loop attempting to log in.
Typically, this happens because HTTP/2 is pretty aggressive when it comes to reusing connections for performance reasons.
It can only happen when the MQ web console presents a certificate that is also valid for the Keycloak instance, which is more common than you might think.</p>

<!--more-->

<p>There are two fixes for this.</p>

<h2 id="fix-1-use-more-precise-certificates">Fix 1: Use more precise certificates</h2>

<p>This problem occurs when the MQ web console presents a certificate that is also valid for the Keycloak instance, which typically means it’s a wildcard certificate valid for <code class="language-plaintext highlighter-rouge">*.apps.&lt;cluster&gt;</code>.
Because it’s valid for both hostnames, and both hostnames resolve to the same load balancer, HTTP/2 will attempt to reuse the connection, sending requests meant for Keycloak to MQ.</p>

<p>To stop the browser attempting to coalesce the connections, you can use a specific certificate for just the MQ web console domain instead of reusing the wildcard certificate for the cluster apps domain.
The browser will then correctly send requests for Keycloak on a new connection to the OpenShift router.</p>

<h2 id="fix-2-turn-off-http2-on-the-mq-web-console">Fix 2: Turn off HTTP/2 on the MQ web console</h2>

<p>You can also turn off HTTP/2 (use HTTP/1.1) on the MQ web console, which means the browser will not attempt to coalesce the connections.
To do this, configure the MQ web console liberty server using the override mechanism in the MQ operator.</p>

<ol>
  <li>Create a <code class="language-plaintext highlighter-rouge">ConfigMap</code> containing the configuration:
    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">ConfigMap</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">mq-web-config-without-http2</span>
<span class="na">data</span><span class="pi">:</span>
  <span class="na">mqwebuser.xml</span><span class="pi">:</span> <span class="pi">|</span>
    <span class="s">&lt;?xml version="1.0" encoding="UTF-8"?&gt;</span>
    <span class="s">&lt;server&gt;</span>
      <span class="s">&lt;httpEndpoint id="defaultHttpEndpoint" protocolVersion="http/1.1"&gt; &lt;/httpEndpoint&gt;</span>
    <span class="s">&lt;/server&gt;</span>
</code></pre></div>    </div>
  </li>
  <li>Reference the <code class="language-plaintext highlighter-rouge">ConfigMap</code> in your <code class="language-plaintext highlighter-rouge">QueueManager</code> resources:
    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">...</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">web</span><span class="pi">:</span>
    <span class="na">manualConfig</span><span class="pi">:</span>
      <span class="na">configMap</span><span class="pi">:</span>
        <span class="na">name</span><span class="pi">:</span> <span class="s">mywebconfig</span>
<span class="nn">...</span>
</code></pre></div>    </div>
  </li>
</ol>

<h2 id="why-is-it-happening">Why is it happening?</h2>

<p>The root cause of this issue is that HTTP/2 is more aggressive in terms of connection reuse and will attempt to coalesce any hostname/IP/certificate combination that has overlap into a single connection.
Daniel Stenberg of cURL fame wrote about <a href="https://daniel.haxx.se/blog/2016/08/18/http2-connection-coalescing/">HTTP/2 Connection Coalescing</a> in his blog.</p>

<p>Typically, coalescing works on an OpenShift cluster as long as the TLS is terminated at the OpenShift router.
The browser will connect to the OpenShift router once and will reuse that connection for any other hostnames pointing at the OpenShift router that are covered by the same wildcard certificate.
The OpenShift router will correctly identify the hostname associated with each individual request over that coalesced connection and forward them on to the right backend services.
If a request for a hostname set to use Passthrough comes through to the router on that same connection, it will return a <a href="https://httpwg.org/specs/rfc7540.html#MisdirectedRequest">421 Misdirected Request</a> response, allowing the browser to try again with a new connection that can be properly terminated at the MQ web console pod instead.</p>

<p>In this case, the MQ web console is using Passthrough mode for handling TLS connections, so TLS is terminated at the MQ web console.
The OpenShift router then looks at the SNI header and tunnels the TCP connection from the browser directly to the MQ web console (i.e. Passthrough), and cannot see the individual requests.
When the browser then sees that the user is being redirected to Keycloak, it determines that it can coalesce that connection because the hostname/IP/certificate combination has overlap, and it attempts to reuse that connection.
The MQ web console doesn’t check the hostname being used (it doesn’t know the load balancer hostname, so it can’t validate it anyway), so it assumes the request is for a page in the console.
Because the user isn’t logged in, the MQ web console then redirects the user back to the login page over and over again because the coalesced connection means the browser never reaches Keycloak.</p>]]></content><author><name>James Hewitt</name></author><category term="integration" /><category term="kubernetes" /><category term="OpenShift" /><category term="MQ" /><category term="Keycloak" /><category term="http2" /><category term="redirect" /><category term="loop" /><category term="cp4i" /><category term="integration" /><summary type="html"><![CDATA[I’ve seen a number of Cloud Pak for Integration installations where the MQ web console is inaccessible because of a redirect loop attempting to log in. Typically, this happens because HTTP/2 is pretty aggressive when it comes to reusing connections for performance reasons. It can only happen when the MQ web console presents a certificate that is also valid for the Keycloak instance, which is more common than you might think.]]></summary></entry><entry><title type="html">IBM API Connect EDB Replica configuration</title><link href="/integration/api-connect-edb-replicas.html" rel="alternate" type="text/html" title="IBM API Connect EDB Replica configuration" /><published>2025-01-07T00:00:00+00:00</published><updated>2025-01-07T00:00:00+00:00</updated><id>/integration/api-connect-edb-replicas</id><content type="html" xml:base="/integration/api-connect-edb-replicas.html"><![CDATA[<p>I spin up a lot of API Connect instances for testing, which means I want to use the slimmest profile.
I’m not going to have a lot of production traffic and I want it to be quick to get going.</p>

<p>A problem I hit regularly is that the slimmest profile only includes one database replica.
To minimise disruptions, there is an automatic <a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets">pod disruption budget</a> configured to ensure the database always has at least one replica.
This stops the node the database is on from draining, preventing maintenance such as cluster upgrades.</p>

<!--more-->

<p>To fix this, you can manually configure the number of database replicas for any profile.</p>

<p>When using the top-level CR of kind <code class="language-plaintext highlighter-rouge">APIConnectCluster</code>, you can add this snippet to your CR:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">spec</span><span class="pi">:</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">mgmt-edb-cluster</span>
      <span class="na">replicaCount</span><span class="pi">:</span> <span class="m">2</span>
</code></pre></div></div>

<p>When using sub-system CRs to build your API Connect cluster, you can add this snippet to your <code class="language-plaintext highlighter-rouge">ManagementCluster</code> CR:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">spec</span><span class="pi">:</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">edb-cluster</span>
      <span class="na">replicaCount</span><span class="pi">:</span> <span class="m">2</span>
</code></pre></div></div>

<p>The operator will automatically increase the number of EDB replicas allowing your nodes to drain without violating the pod disruption budget.</p>]]></content><author><name>James Hewitt</name></author><category term="integration" /><category term="kubernetes" /><category term="OpenShift" /><category term="EDB" /><category term="PDB" /><category term="API_Connect" /><category term="cp4i" /><category term="integration" /><summary type="html"><![CDATA[I spin up a lot of API Connect instances for testing, which means I want to use the slimmest profile. I’m not going to have a lot of production traffic and I want it to be quick to get going. A problem I hit regularly is that the slimmest profile only includes one database replica. To minimise disruptions, there is an automatic pod disruption budget configured to ensure the database always has at least one replica. This stops the node the database is on from draining, preventing maintenance such as cluster upgrades.]]></summary></entry><entry><title type="html">Monitoring IBM Cloud Pak for Integration components</title><link href="/kubernetes/cp4i-monitoring.html" rel="alternate" type="text/html" title="Monitoring IBM Cloud Pak for Integration components" /><published>2022-04-13T00:00:00+00:00</published><updated>2022-04-13T00:00:00+00:00</updated><id>/kubernetes/cp4i-monitoring</id><content type="html" xml:base="/kubernetes/cp4i-monitoring.html"><![CDATA[<p>Operational visibility and the ability to understand the internals of the <strong>IBM Cloud Pak for Integration</strong> behavior is key to maintaining the health, readiness and availability of the solutions built on top of your Cloud Pak.  You can monitor components of your Cloud Pak by leveraging the out of the box monitoring stack provided by the OpenShift Container Platform 4.6+ and enhance its capabilities by deploying the monitoring services provided with the <strong>IBM Cloud Pak Foundational Services</strong>.  This configuration provides a powerful open source based solution that allows you to identify and respond to issues in a timely manner in order to guarantee the level of service and availability of your integrations.</p>

<p><strong>Author:</strong></p>

<ul>
  <li>James Hewitt: <em>Senior Technical Staff Member / Cloud Pak for Integration</em></li>
</ul>

<p><strong>Contributors:</strong></p>

<ul>
  <li>
    <p>Jorge D. Rodriguez: <em>STSM / Business Automation Solutions Architect / Automation SWAT Team</em></p>
  </li>
  <li>
    <p>Dr. Matthias Jung: <em>Enterprise Content Services / Accelerated Value Specialist / Digital Business Automation - SWAT</em></p>
  </li>
  <li>
    <p>Kevin Trinh: <em>STSM / Cloud Integration Architect / Digital Business Automation (ECM)</em></p>
  </li>
  <li>
    <p>Piotr Godowski: <em>STSM / Cloud Pak Foundational Services Architecture</em></p>
  </li>
</ul>

<p>Huge thanks to Jorge who wrote <a href="https://community.ibm.com/community/user/automation/blogs/jorge-rodriguez1/2021/10/04/how-to-monitor-ibm-cloud-pak-for-ba">How to Monitor Components of Cloud Pak for Business Automation</a>, which is the basis for this blog post.</p>

<h2 id="overview">Overview</h2>

<p><strong>IBM Cloud Pak for Integration</strong> is a hybrid integration platform with an automated, closed-loop approach that supports multiple styles of integration within a single, unified experience. Unlock business data and assets as APIs, connect cloud and on-premise applications, reliably move data with enterprise messaging, deliver real-time event interactions, transfer data across any cloud and deploy and scale with cloud-native architecture and shared foundational services — all with end-to-end enterprise-grade security and encryption.</p>

<p>The <strong>IBM Cloud Pak for Integration</strong> provides a fully integrated platform that combines IBM’s best-in-class integration software to modernise applications and provide new opportunities.  The core integration capabilities of the <strong>IBM Cloud Pak for Integration</strong> include low/no code integration development, first-class API and Event management, message streaming and queuing, and high speed data transfer.</p>

<p>While the integration platform itself provides a low code simplified experience for business and technical users alike, system administrators and operations teams still need to deploy, manage and monitor the containerized software that makes up the <strong>IBM Cloud Pak for Integration</strong> in order to keep the platform and the solutions built on top running optimally.</p>

<p>In this article I will discuss how to enable monitoring capabilities on the <strong>IBM Cloud Pak for Integration</strong> and how to extend OpenShift’s pre-configured monitoring stack to provide operational visibility across Cloud Pak components.</p>

<h2 id="pre-requisites">Pre-Requisites</h2>

<ul>
  <li>OpenShift Container Platform version 4.6 and up</li>
  <li><strong>IBM Cloud Pak for Integration</strong> version 2021.2 and up</li>
  <li>Workstation with <code class="language-plaintext highlighter-rouge">oc</code> command installed</li>
</ul>

<h2 id="brief-introduction-to-the-openshift-monitoring-stack">Brief Introduction to the OpenShift Monitoring Stack</h2>

<p>The OpenShift Container Platform monitoring stack is based on the Prometheus open source project.  At a high level the pre-configured stack includes one or more instances of the following components:</p>

<table>
  <thead>
    <tr>
      <th>Component</th>
      <th>Description</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Prometheus</td>
      <td>Used to provide a time-series data store for metrics, rule evaluation engine and alert generation.</td>
    </tr>
    <tr>
      <td>AlertManager</td>
      <td>Responsible for alerts handling and notification to external systems.</td>
    </tr>
    <tr>
      <td>Thanos</td>
      <td>Responsible for metric aggregation across Prometheus instances as well as alert generation engine.</td>
    </tr>
    <tr>
      <td>Grafana</td>
      <td>Used to provide dashboard and metric visualization capabilities. This is a read-only instance of Grafana to show platform metrics</td>
    </tr>
  </tbody>
</table>

<p>By default, the OpenShift monitoring stack provides monitoring capabilities for core platform components only.  That is, the out of the box deployment of the monitoring stack found under the <code class="language-plaintext highlighter-rouge">openshift-monitoring</code> project is fully dedicated to monitoring core OpenShift Container Platform systems and other essential kubernetes services.</p>

<p>Optionally, since OpenShift version 4.6, the default monitoring stack deployment can be extended to monitor user-defined projects and custom deployments such as the <strong>IBM Cloud Pak for Integration</strong> installations. The additional components needed to monitor user-defined projects are automatically deployed under the <code class="language-plaintext highlighter-rouge">openshift-user-workload-monitoring</code> project once the default stack is configured to support user-defined projects.  After the configuration is completed you will be able to collect, query, visualize and create alerts based on custom metrics generated by your own deployments.  This article describes how to enable these capabilities, how to expand them using additional monitoring components provided by the IBM Cloud Pak Foundational Services and how to define the necessary kubernetes resources to collect and utilize custom metrics generated by the <strong>IBM Cloud Pak for Integration</strong> platform.  For more information on how to monitor user-defined projects see the <a href="https://docs.openshift.com/container-platform/4.6/monitoring/enabling-monitoring-for-user-defined-projects.html">Enabling Monitoring for user-defined Projects</a> documentation.</p>

<h2 id="enable-user-project-monitoring-in-openshift">Enable User Project Monitoring in OpenShift</h2>

<p>The first thing that we need to do in order to monitor <strong>IBM Cloud Pak for Integration</strong> components is to turn on user project monitoring in OpenShift.  To do that we need to create an instance of a <code class="language-plaintext highlighter-rouge">ConfigMap</code> named <code class="language-plaintext highlighter-rouge">cluster-monitoring-config</code> in the <code class="language-plaintext highlighter-rouge">openshift-monitoring</code> project.  The <code class="language-plaintext highlighter-rouge">cluster-monitoring-config</code> <code class="language-plaintext highlighter-rouge">ConfigMap</code> allows you to specify the configuration details for the OpenShift monitoring stack such as enabling user project monitoring, retention policy for metrics collected, resource limits for monitoring components, node selectors for pod deployments, among other settings.  For a complete list of configurable parameters see <a href="https://docs.openshift.com/container-platform/4.6/monitoring/configuring-the-monitoring-stack.html">Configuring the OpenShift Monitoring Stack</a>.</p>

<p>For the purpose of this article we are going to deploy a simple instance of a <code class="language-plaintext highlighter-rouge">ConfigMap</code> that sets the <code class="language-plaintext highlighter-rouge">enableUserWorkload</code> attribute to true.  Setting the <code class="language-plaintext highlighter-rouge">enableUserWorkload</code> to true will enable monitoring for user-defined projects in addition to the default platform monitoring and will automatically trigger the deployment of additional monitoring components under the <code class="language-plaintext highlighter-rouge">openshift-user-workload-monitoring</code> project once the <code class="language-plaintext highlighter-rouge">ConfigMap</code> is created.  To create the <code class="language-plaintext highlighter-rouge">cluster-monitoring-config</code> <code class="language-plaintext highlighter-rouge">ConfigMap</code> complete the following steps:</p>

<ol>
  <li>Create a <code class="language-plaintext highlighter-rouge">cluster-monitoring-config.yaml</code> file.</li>
  <li>
    <p>Add the following content to the file</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
 <span class="na">kind</span><span class="pi">:</span> <span class="s">ConfigMap</span>
 <span class="na">metadata</span><span class="pi">:</span>
   <span class="na">name</span><span class="pi">:</span> <span class="s">cluster-monitoring-config</span>
   <span class="na">namespace</span><span class="pi">:</span> <span class="s">openshift-monitoring</span>
 <span class="na">data</span><span class="pi">:</span>
   <span class="na">config.yaml</span><span class="pi">:</span> <span class="pi">|</span>
     <span class="s">enableUserWorkload: true </span>
</code></pre></div>    </div>
  </li>
  <li>
    <p>Use the <code class="language-plaintext highlighter-rouge">oc</code> command to create the <code class="language-plaintext highlighter-rouge">ConfigMap</code> instance</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> oc apply <span class="nt">-f</span> cluster-monitoring-config.yaml <span class="nt">-n</span> openshift-monitoring
</code></pre></div>    </div>
  </li>
</ol>

<p>Now that the <code class="language-plaintext highlighter-rouge">cluster-monitoring-config</code> <code class="language-plaintext highlighter-rouge">ConfigMap</code> instance has been created we can verify that the additional components required for user project monitoring have been deployed under the <code class="language-plaintext highlighter-rouge">openshift-user-workload-monitoring</code> project.   You can use the following command to list the PODs under the <code class="language-plaintext highlighter-rouge">openshift-user-workload-monitoring</code> project. Wait until the pods are running and ready.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oc get pods <span class="nt">-n</span> openshift-user-workload-monitoring
</code></pre></div></div>

<p>Once the additional monitoring components have been deployed successfully the output of the command should look similar to the following listing:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-operator-5d66498b6-tgwhv         2/2     Running   0          2d4h
prometheus-user-workload-0                  4/4     Running   0          2d4h
prometheus-user-workload-1                  4/4     Running   0          2d4h
thanos-ruler-user-workload-0                3/3     Running   0          2d4h
thanos-ruler-user-workload-1                3/3     Running   0          2d4h
</code></pre></div></div>

<p>To customize the resources and the behavior specific to monitoring components deployed under the <code class="language-plaintext highlighter-rouge">openshift-user-workload-monitoring</code> project, you can create a <code class="language-plaintext highlighter-rouge">ConfigMap</code> named <code class="language-plaintext highlighter-rouge">user-workload-monitoring-config</code> under that project.  This <code class="language-plaintext highlighter-rouge">ConfigMap</code> is analogous to the <code class="language-plaintext highlighter-rouge">cluster-monitoring-config</code> created under the <code class="language-plaintext highlighter-rouge">openshift-monitoring</code> project and allows you to further customize the deployment of components specific to user project monitoring. See <a href="https://docs.openshift.com/container-platform/4.6/monitoring/configuring-the-monitoring-stack.html#creating-user-defined-workload-monitoring-configmap_configuring-the-monitoring-stack">Configuring the monitoring stack</a> for additional details.</p>

<h2 id="configure-monitoring-services-in-the-ibm-cloud-pak-foundational-services">Configure Monitoring Services in the IBM Cloud Pak Foundational Services</h2>

<p>While the OpenShift monitoring stack provides most of the capabilities required to properly address observability and operational visibility requirements for your Cloud Pak, there are some limitations around the ability to visualize metrics. Specifically, the Grafana instance included in the OpenShift core monitoring stack is read only.  This means that you can only visualize a predefined set of metrics through pre-configured dashboards provided by the platform.   This also means that the ability to create new dashboards and visualize custom metrics has been completely removed from this installation.</p>

<p>Fortunately, the <a href="https://www.ibm.com/docs/en/cloud-paks/1.0?topic=services-monitoring">IBM Cloud Pak Foundational Services</a>, installed as part of the <strong>IBM Cloud Pak for Integration</strong>, makes it easy to deploy an additional instance of Grafana that is fully functional and that is connected to the OpenShift’s Prometheus instance.  This configuration gives us the ability to use all metrics available in the platform, including custom metrics,  so that we can create additional visualizations and dashboards on top of what is provided by the OpenShift user-defined workload monitoring stack.</p>

<p>Although the <strong>IBM Cloud Pak Foundational Services</strong> are installed as part of your <strong>IBM Cloud Pak For Integration</strong> installation, the monitoring services that include Grafana might not be installed.  To verify that Grafana was installed as part of your Cloud Pak deployment you can run the following command:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oc get pods <span class="nt">-n</span> ibm-common-services <span class="nt">-l</span> app.kubernetes.io/managed-by<span class="o">=</span>ibm-monitoring-grafana-operator
</code></pre></div></div>

<p>If Grafana is installed as part of the <strong>IBM Cloud Pak Foundational Services</strong> you should see an output similar to the following listing:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NAME                                             READY   STATUS    RESTARTS   AGE
ibm-monitoring-grafana-78b9cd8688-xxq5q          4/4     Running   0          3d14h
ibm-monitoring-grafana-operator-54cbd464d-xqq82  1/1     Running   0          3d14h
</code></pre></div></div>

<p>If Grafana is not installed we can configure the monitoring services that are part of the <strong>IBM Cloud Pak Foundational Services</strong> to deploy an instance.  The steps provided below assume that the <strong>IBM Cloud Pak Foundational Services</strong> are already installed on your cluster.  To verify if the <strong>IBM Cloud Pak Foundational Services</strong> are installed you can run the following command:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oc get pods <span class="nt">-n</span> ibm-common-services <span class="nt">-l</span> app.kubernetes.io/managed-by<span class="o">=</span>ibm-common-service-operator
</code></pre></div></div>

<p>If the <strong>IBM Cloud Pak Foundational Services</strong> are installed the output of the command should be similar to the following listing:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NAME                                           READY   STATUS    RESTARTS   AGE
ibm-common-service-operator-84c9d8cc69-zsgxw   1/1     Running   0          3d15h
</code></pre></div></div>

<p>If the <strong>IBM Cloud Pak Foundational Services</strong> are not installed on your system you cannot complete the remaining steps in this section.  As already mentioned, the <strong>IBM Cloud Pak Foundational Services</strong> will be installed automatically once you install the <strong>IBM Cloud Pak for Integration</strong>, so come back to this section once your Cloud Pak is installed. See the <a href="https://www.ibm.com/docs/en/cloud-paks/1.0?topic=cloud-pak-foundational-services">IBM Cloud Pak Foundational Services</a> documentation for additional information on these services.</p>

<h3 id="deploying-grafana-using-the-platform-ui">Deploying grafana using the Platform UI</h3>

<p>The <strong>IBM Cloud Pak for Integration</strong> Platform UI can be configured to install the <strong>IBM Cloud Pak Foundational Services</strong> monitoring components for you.</p>

<ol>
  <li>
    <p>Open the OpenShift console.</p>
  </li>
  <li>
    <p>In the side bar, select <strong>Operators</strong> &gt; <strong>Installed Operators</strong>.</p>
  </li>
  <li>
    <p>Select the project for your <strong>IBM Cloud Pak for Integration</strong> Platform UI installation.</p>
  </li>
  <li>
    <p>Select <em>IBM Cloud Pak for Integration</em> from the list of operators.</p>
  </li>
  <li>
    <p>Select the <strong>Platform UI</strong> tab.</p>
  </li>
  <li>
    <p>Select your instance from the list.</p>
  </li>
  <li>
    <p>Select the <strong>YAML</strong> tab.</p>
  </li>
  <li>
    <p>Add <code class="language-plaintext highlighter-rouge">monitoring</code> to the <code class="language-plaintext highlighter-rouge">requestIbmServices</code> section.</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">spec</span><span class="pi">:</span>
  <span class="na">requestIbmServices</span><span class="pi">:</span>
    <span class="na">monitoring</span><span class="pi">:</span> <span class="no">true</span>
</code></pre></div>    </div>
  </li>
  <li>
    <p>Click <strong>Save</strong>.</p>
  </li>
  <li>
    <p>Wait for services to install.  This could take up to 30 minutes depending on how many other components are being updated or installed.   Use the following command to monitor the installation.</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oc get pods <span class="nt">-w</span> <span class="nt">-n</span> ibm-common-services <span class="nt">-l</span> <span class="nv">app</span><span class="o">=</span>grafana
</code></pre></div>    </div>

    <p>Once the installation is completed successfully you should see output similar to the following:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NAME                                      READY   STATUS           RESTARTS   AGE
ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Pending          0          0s
ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Pending          0          0s
ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Init:0/1         0          3s
ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Init:0/1         0          26s
ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Init:0/1         0          26s
ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     PodInitializing  0          29s
ibm-monitoring-grafana-797c5d4c5f-nxt8c   1/4     Running          0          54s
ibm-monitoring-grafana-797c5d4c5f-nxt8c   2/4     Running          0          69s
ibm-monitoring-grafana-797c5d4c5f-nxt8c   3/4     Running          0          71s
ibm-monitoring-grafana-797c5d4c5f-nxt8c   4/4     Running          0          78s
</code></pre></div>    </div>

    <p>Notice that the <code class="language-plaintext highlighter-rouge">ibm-monitoring-grafana-xxxxxxxxx-xxxxx</code> pod has multiple containers. All of these containers should report Ready for the installation to be considered complete.</p>
  </li>
</ol>

<h3 id="deploying-grafana-using-ibm-cloud-pak-foundational-services-directly">Deploying grafana using IBM Cloud Pak Foundational Services directly</h3>

<p>Having verified that the <strong>IBM Cloud Pak Foundational Services</strong> are deployed in your cluster, we can customize that deployment to include Grafana based services.  The Grafana based services are deployed and managed by the <strong>IBM Monitoring Grafana Operator</strong>.  This operator can be installed using the <a href="https://github.com/IBM/operand-deployment-lifecycle-manager/blob/master/docs/design/operand-deployment-lifecycle-manager.md">Operand Deployment Lifecycle Manager</a> framework that is part of the foundational services.   To install the <strong>IBM Monitoring Grafana Operator</strong>:</p>

<ol>
  <li>
    <p>Create a <code class="language-plaintext highlighter-rouge">grafana-operand-request.yaml</code> file.</p>
  </li>
  <li>
    <p>Add the following content to the file:</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">operator.ibm.com/v1alpha1</span>
 <span class="na">kind</span><span class="pi">:</span> <span class="s">OperandRequest</span>
 <span class="na">metadata</span><span class="pi">:</span>
   <span class="na">name</span><span class="pi">:</span> <span class="s">common-service</span>
   <span class="na">namespace</span><span class="pi">:</span> <span class="s">ibm-common-services</span>
 <span class="na">spec</span><span class="pi">:</span>
   <span class="na">requests</span><span class="pi">:</span>
     <span class="pi">-</span> <span class="na">operands</span><span class="pi">:</span>
         <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ibm-monitoring-grafana-operator</span>
       <span class="na">registry</span><span class="pi">:</span> <span class="s">common-service</span>
</code></pre></div>    </div>
  </li>
  <li>
    <p>Deploy the <code class="language-plaintext highlighter-rouge">OperandRequest</code> instance using the <code class="language-plaintext highlighter-rouge">oc apply</code> command.  Running this command might trigger the installation or update of additional components within the <strong>IBM Cloud Pak Foundational Services</strong>.</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> oc apply <span class="nt">-f</span> grafana-operand-request.yaml 
</code></pre></div>    </div>

    <p>The output of the command should be similar to the following listing:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> operandrequest.operator.ibm.com/common-service created
</code></pre></div>    </div>

    <p>If the approval strategy for operators has been changed to Manual, the install plan for the operator must be approved for installation to actually happen.</p>
  </li>
  <li>
    <p>Wait for services to install.  This could take up to 30 minutes depending on how many other components are being updated or installed.   Use the following command to monitor the installation.</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> oc get pods <span class="nt">-w</span> <span class="nt">-n</span> ibm-common-services <span class="nt">-l</span> <span class="nv">app</span><span class="o">=</span>grafana
</code></pre></div>    </div>

    <p>Once the installation is completed successfully you should see output similar to the following:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> NAME                                      READY   STATUS           RESTARTS   AGE
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Pending          0          0s
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Pending          0          0s
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Init:0/1         0          3s
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Init:0/1         0          26s
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     Init:0/1         0          26s
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   0/4     PodInitializing  0          29s
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   1/4     Running          0          54s
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   2/4     Running          0          69s
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   3/4     Running          0          71s
 ibm-monitoring-grafana-797c5d4c5f-nxt8c   4/4     Running          0          78s
</code></pre></div>    </div>

    <p>Notice that the <code class="language-plaintext highlighter-rouge">ibm-monitoring-grafana-xxxxxxxxx-xxxxx</code> pod has multiple containers. All of these containers should report Ready for the installation to be considered complete.</p>
  </li>
  <li>
    <p>Now that Grafana has been installed we need to find the URL that we can use to access the services.   The Grafana instance deployed through the <strong>IBM Foundational Monitoring Services</strong> is accessed via a common Cloud Pak route called <code class="language-plaintext highlighter-rouge">cp-console</code>.  To find the details of the <code class="language-plaintext highlighter-rouge">cp-console</code> route you can run the following command:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> oc get route <span class="nt">-n</span> ibm-common-services cp-console 
</code></pre></div>    </div>

    <p>The output of the command should look similar to the following listing:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>     NAME         HOST/PORT                           SERVICES                 PORT    TERMINATION          WILDCARD
     cp-console   cp-console.&lt;cluster-subdomain&gt;      icp-management-ingress   https   reencrypt/Redirect   None
</code></pre></div>    </div>

    <p>The full URL to access the cp-console route is as follows:</p>

    <p><code class="language-plaintext highlighter-rouge">https://cp-console.&lt;cluster-subdomain&gt;/</code></p>

    <p>Where<code class="language-plaintext highlighter-rouge">&lt;cluster-subdomain&gt;</code> the the ingress subdomain assigned to your cluster.</p>
  </li>
  <li>
    <p>To access the Grafana dashboard enter this URL in your browser.  You will be taken to the Cloud Pak authentication page.  Select the OpenShift as the authentication type to be taken to the Cloud Pak Administration Hub using OpenShift Credentials.</p>
  </li>
  <li>
    <p>Click Menu (top left icon) &gt; Monitoring.
 Note: Alternatively, you can open <code class="language-plaintext highlighter-rouge">https://cp-console.&lt;cluster-subdomain&gt;/grafana</code> directly to access the Grafana Dashboard.</p>
  </li>
</ol>

<h2 id="monitoring-capabilities-at-the-component-level-in-the-ibm-cloud-pak-for-integration">Monitoring Capabilities at the component level in the IBM Cloud Pak for Integration</h2>

<p>The <strong>IBM Cloud Pak for Integration</strong> components automatically enable monitoring capabilities at the component level. The components automatically advertise their metrics endpoints to the OpenShift user project monitoring stack.</p>

<h2 id="access-the-cloud-pak-for-integration-sample-dashboards">Access the Cloud Pak for Integration sample dashboards</h2>

<p>Now that we have installed <strong>IBM Cloud Pak for Integration</strong>, and deployed a fully functional instance of Grafana, we can use the sample dashboards to visualize some of the metrics generated by <strong>IBM Cloud Pak for Integration</strong> components. You can create or edit dashboards directly from the Grafana user interface, and automate the deployment of dashboards by using <code class="language-plaintext highlighter-rouge">MonitoringDashboard</code> custom resources.</p>

<h3 id="access-the-dashboards-from-the-ibm-cloud-pak-for-integration-platform-ui">Access the dashboards from the <strong>IBM Cloud Pak for Integration</strong> platform UI</h3>

<ol>
  <li>
    <p>Open the platform UI in a browser.</p>
  </li>
  <li>
    <p>Select the navigation menu, and go to the Administration section, then Integration Instances.</p>
  </li>
  <li>
    <p>Select the 3 button menu next to the integration you want to monitor, and select the Monitoring link. Only components that have dashboards available will show a Monitoring link.</p>
  </li>
</ol>

<h3 id="access-the-dashboards-within-grafana-directly">Access the dashboards within Grafana directly</h3>

<ol>
  <li>
    <p>Access your new Grafana instance using the URL off the cp-console route.  See the <a href="#configure-monitoring-services-in-the-ibm-cloud-pak-foundational-services">previous section</a> on how to access the Grafana dashboard.  As stated before the URL should have the following format:</p>

    <p><code class="language-plaintext highlighter-rouge">https://cp-console.&lt;cluster-subdomain&gt;/grafana</code></p>
  </li>
  <li>
    <p>Click on the <em>Settings</em> icon, found on the lower left-hand side of the screen (above the help “?” icon).  This will take you to the settings panel for the user that is currently authenticated in Grafana.</p>
  </li>
  <li>
    <p>Go to the <em>Organizations</em> section under the <em>Settings</em> page and look for the namespace where your <strong>IBM Cloud Pak for Integration</strong> components are deployed.  Once you find the organization, click on the <em>Select</em> button next to it.  This will switch the context used by Grafana so that it can find your Dashboard under that organization.
 Note:  You will only be able to see Grafana organizations corresponding to the OpenShift projects that your user has access to.   For more information on role-based access controls for the Grafana instance included in the <strong>IBM Cloud Pak Foundational Services</strong> see <a href="https://www.ibm.com/docs/en/cloud-paks/1.0?topic=services-monitoring">Monitoring Service</a></p>
  </li>
  <li>
    <p>Click on the <em>Dashboards</em> menu, depicted by the four squares icon, to the left of the screen and select the <em>Manage</em> option.  You should be able to see the dashboards listed for your installed <strong>IBM Cloud Pak for Integration</strong> components.</p>
  </li>
  <li>
    <p>Click on a dashboard.  You should see the selected dashboard.</p>

    <p><img src="/assets/mq-dashboard.png" alt="Sample MQ dashboard" /></p>
  </li>
</ol>

<p>You can now explore all custom metrics available via the <strong>IBM Cloud Pak for Integration</strong> components and expand the dashboard capabilities to include visualizations for additional metrics and components suited to your use case.</p>

<p>Note: There is a set of Grafana dashboards included by default in the <strong>IBM Cloud Pak Foundational Services</strong>.  You can access these dashboards by selecting the <em>Main Org.</em> organization under the <em>Settings</em> panel and accessing the <em>Dashboards</em> menu as explained above.</p>

<h2 id="configure-alerts-based-on-ibm-cloud-pak-for-integration-components">Configure Alerts Based on IBM Cloud Pak For Integration Components</h2>

<p>Arguably one of the most important features of a monitoring system is the ability to generate notifications based on conditions or events that need to be investigated in order to prevent or address application issues and potential failures.  The OpenShift Container Platform leverages the combined capabilities of Prometheus, Thanos and AlertManager to generate, correlate and route notifications to a variety of notification targets.</p>

<p>In previous sections of this article we were able to configure OpenShift’s monitoring stack to support user-defined projects and feed custom metrics generated by <strong>IBM Cloud Pak for Integration</strong> components into it.  In this section we will learn how to leverage this setup in order to create customized alerts to monitor the behavior of the Cloud Pak.  Specifically, we will create a sample alert to monitor the message throughput of an MQ queue manager.</p>

<p>To define a custom alert in the OpenShift’s monitoring stack we need to create an instance of the <a href="https://docs.openshift.com/container-platform/4.6/rest_api/monitoring_apis/prometheusrule-monitoring-coreos-com-v1.html">PrometheusRule</a> custom resource definition.  This entails creating a resource of type <code class="language-plaintext highlighter-rouge">PrometheusRule</code> and specifying the rules that would trigger the firing of notifications using the Prometheus expression language.</p>

<ol>
  <li>Create a file <code class="language-plaintext highlighter-rouge">cp4i-mq-throughput-alert.yaml</code>.</li>
  <li>
    <p>Add the following content to the file:</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">monitoring.coreos.com/v1</span>
 <span class="na">kind</span><span class="pi">:</span> <span class="s">PrometheusRule</span>
 <span class="na">metadata</span><span class="pi">:</span>
   <span class="na">name</span><span class="pi">:</span> <span class="s">cp4bi-mq-throughput</span>
   <span class="na">namespace</span><span class="pi">:</span> <span class="s">&lt;cp4i_namespace&gt;</span>
 <span class="na">spec</span><span class="pi">:</span>
   <span class="na">groups</span><span class="pi">:</span>
   <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">cp4isetup</span>
     <span class="na">rules</span><span class="pi">:</span>
       <span class="pi">-</span> <span class="na">alert</span><span class="pi">:</span> <span class="s">cp4i-mq-throughput-low</span>
         <span class="na">expr</span><span class="pi">:</span> <span class="s">rate(ibmmq_qmgr_mqput_mqput1_total{qmgr="&lt;queue manager&gt;"}[5m]) &lt; </span><span class="m">1</span>
         <span class="na">for</span><span class="pi">:</span> <span class="s">10m</span>
         <span class="na">labels</span><span class="pi">:</span>
           <span class="na">severity</span><span class="pi">:</span> <span class="s">warning</span>
         <span class="na">annotations</span><span class="pi">:</span>
           <span class="na">message</span><span class="pi">:</span> <span class="s">Potential throughput problem - queue manager is receiving no messages</span>
</code></pre></div>    </div>

    <p>Where <code class="language-plaintext highlighter-rouge">&lt;cp4i_namespace&gt;</code> should be replaced with the name of the project where the <strong>IBM Cloud Pak For Integration</strong> has been deployed and <code class="language-plaintext highlighter-rouge">&lt;queue manager&gt;</code> should be replaced with the name of the queue manager to monitor.</p>
  </li>
  <li>
    <p>Deploy the PrometheusRule instance using the <code class="language-plaintext highlighter-rouge">oc apply</code> command</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  oc apply <span class="nt">-f</span> cp4i-mq-throughput-alert.yaml
</code></pre></div>    </div>

    <p>The output of the command should be similar to the following listing:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> prometheusrule.monitoring.coreos.com/cp4i-mq-throughput created
</code></pre></div>    </div>
  </li>
  <li>
    <p>Verify the alert has been created</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> oc get PrometheusRule cp4i-mq-throughput <span class="nt">-n</span> &lt;cp4i_namespace&gt;
</code></pre></div>    </div>

    <p>Where <code class="language-plaintext highlighter-rouge">&lt;cp4i_namespace&gt;</code> should be replaced with the name of the project where the <strong>IBM Cloud Pak For Integration</strong> has been deployed.</p>

    <p>The output of the command should be similar to the following listing:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> NAME                 AGE
 cp4i-mq-throughput   18m
</code></pre></div>    </div>
  </li>
</ol>

<p>To validate if your <code class="language-plaintext highlighter-rouge">PrometheusRule</code> instance has been properly created, you can use the OpenShift Console. From the OpenShift Console you can inspect the rule created and verify if the alert is actually being fired.  For example, we can verify the <code class="language-plaintext highlighter-rouge">cp4i-mq-throughput</code> alert created above by completing the following procedure:</p>

<ol>
  <li>
    <p>Open the Administrator perspective of your OpenShit Console.</p>
  </li>
  <li>
    <p>Go to the <em>Monitoring</em> menu on the left-hand side of the console.</p>
  </li>
  <li>
    <p>Select the <em>Alerting</em> option under the <em>Monitoring</em> menu.</p>
  </li>
  <li>
    <p>By default, the <em>Alerting</em> panel shows platform based alerts (those monitoring OCP components and Kubernetes) that are being fired.  You can expand the search to include non-platform alerts by clicking on the <em>Clear all filters</em> link.  When all filters are removed, you should be able to see alerts that are part of the platform and custom alerts coming from user-defined projects.  Notice that this panel does not show the actual instance of the <code class="language-plaintext highlighter-rouge">PrometheusRule</code> created in the previous step. It shows instances of alerts that are actively being fired or about to be fired when the conditions of the <code class="language-plaintext highlighter-rouge">PrometheusRule</code> are met. If you do not see an alert in this panel it means that the conditions for the alert have not been met.</p>
  </li>
  <li>
    <p>To view the actual instance of the <code class="language-plaintext highlighter-rouge">PrometheusRule</code> previously created in this section you can click on the <em>Alerting Rules</em> tab found at the top of the <em>Alerting</em> panel.  By default, the <em>Alerting Rules</em> panel only shows <code class="language-plaintext highlighter-rouge">PrometheusRule</code> instances for the platform but you can click on the <em>Clear all filters</em> link to include alert rules for user-defined projects. Once you clear all filters you should be able to see the <code class="language-plaintext highlighter-rouge">cp4i-mq-throughput</code> alert rules.</p>
  </li>
</ol>

<p>As you can see, creating alerts for components of the <strong>IBM Cloud Pak for Integration</strong> can be done using the standard mechanism provided by Prometheus to create custom alerts.  A few things worth mentioning about these capabilities are:</p>

<ul>
  <li>Once custom metrics have been scrapped by Prometheus you can use them to create alert rules.   This gives you the flexibility to create alerts that are specific to the capabilities and inner workings of the <strong>IBM Cloud Pak for Integration</strong> components and to get real-time notifications based on potential threats to the reliability of your solutions.</li>
  <li>Custom metrics can be used in combination with functions provided by Prometheus to test for a condition.  In this example, we used the rate function to calculate the rate of change of messages being put onto an MQ queue manager.</li>
  <li>The instance of your alert resource should be created under the namespace where the resource being monitored resides.   Since we were monitoring the <strong>IBM Cloud Pak For Integration</strong> we created the alert in that namespace.</li>
</ul>

<p>For a complete explanation of how to create alerts and the Prometheus expression language see the <a href="https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/">Alerting Rules</a> documentation.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Operational visibility and the ability to understand the internals of the <strong>IBM Cloud Pak for Integration</strong> behavior is key to maintaining the health, readiness and availability of the solutions built on top of your Cloud Pak.  You can monitor components of your Cloud Pak by leveraging the out of the box monitoring stack provided by the OpenShift Container Platform 4.6+ and enhance its capabilities by deploying the monitoring services provided with the <strong>IBM Cloud Pak Foundational Services</strong>.  This configuration provides a powerful open source based solution that allows you to identify and respond to issues in a timely manner in order to guarantee the level of service and availability of your integrations.</p>

<p>Want to start your agile integration journey? See how the <a href="https://www.ibm.com/cloud/cloud-pak-for-integration/">IBM Cloud Pak for Integration</a> can help you.</p>]]></content><author><name>James Hewitt</name></author><category term="kubernetes" /><category term="kubernetes" /><category term="monitoring" /><category term="openshift" /><category term="cp4i" /><category term="integration" /><summary type="html"><![CDATA[Operational visibility and the ability to understand the internals of the IBM Cloud Pak for Integration behavior is key to maintaining the health, readiness and availability of the solutions built on top of your Cloud Pak. You can monitor components of your Cloud Pak by leveraging the out of the box monitoring stack provided by the OpenShift Container Platform 4.6+ and enhance its capabilities by deploying the monitoring services provided with the IBM Cloud Pak Foundational Services. This configuration provides a powerful open source based solution that allows you to identify and respond to issues in a timely manner in order to guarantee the level of service and availability of your integrations.]]></summary></entry><entry><title type="html">Resources in Kubernetes have no API version</title><link href="/kubernetes/kubernetes-resources-have-no-apiversion.html" rel="alternate" type="text/html" title="Resources in Kubernetes have no API version" /><published>2021-09-15T00:00:00+00:00</published><updated>2021-09-15T00:00:00+00:00</updated><id>/kubernetes/kubernetes-resources-have-no-apiversion</id><content type="html" xml:base="/kubernetes/kubernetes-resources-have-no-apiversion.html"><![CDATA[<p>OK, so its a bit of a click-bait title, and its not 100% accurate either, but its how
I have come to think about resources in Kubernetes. Once a resource has been pushed to
the API server and accepted, it no longer has an API version. Exactly how Kubernetes
stores its metadata is not my concern.</p>

<p>However, I often find people talking about checking the API version of
a resource by fetching it from the Kubernetes API server and taking a look. Because of
the nature of Kubernetes, people will often do exactly that, and believe the result
that comes back matches exactly what was pushed to the API server, but that’s not true
and can lead to missed opportunities for discovering and validating bugs.</p>

<p>In this post I’m going to try to explain how API versions are used within Kubernetes, and
what can go wrong. At the end I’ll include some examples and background if you want to dig
deeper.</p>

<h2 id="how-api-versions-are-used-within-the-api-server">How API versions are used within the API server</h2>

<p>When making calls to the Kubernetes API server, you need to include the API version that
your client understands. By enabling clients to specify their API version, the Kubernetes
API is able to change without breaking old clients by presenting the same resources in
multiple forms.</p>

<p>The API version is the schema that the Kubernetes API server uses to serialize and de-serialize
resources that you pass to it and that it passes back as responses.</p>

<p>There is no use case within Kubernetes where a user should be directly accessing
resources within the etcd database, so from an end user’s perspective, the stored object
has no API version.</p>

<p>This intentionally simple diagram skips a lot of the detail of how the API server works,
and doesn’t even include etcd, but illustrates how all API versions for a resource type
are just different ways in to the same set of resources.</p>

<p><img src="/assets/kube-api-server-apiversions.svg" alt="The client connects to either of the API versions within the Ingress API, which itself is part of the API server." class="center" /></p>

<h2 id="api-deprecation-what-could-go-wrong">API Deprecation: what could go wrong?</h2>

<p>The way that Kubernetes stores resources can trip up users who are attempting to handle
API deprecation.</p>

<p>As a concrete example, imagine you have written an operator for Kubernetes that deploys
software for your users, including <code class="language-plaintext highlighter-rouge">Ingress</code> resources, and you want to know if that’s
going to work in Kubernetes 1.22. Kubernetes 1.22 removes a number of API versions that
have been deprecated for a long time, so you need to check that you aren’t using them.</p>

<p>The immediate reaction could be to pull the resources that your operator creates from the
Kubernetes cluster and look at them, so you run <code class="language-plaintext highlighter-rouge">kubectl get ingress -o yaml</code> to check that
you’re using the right version of the ingress API. The result comes back with an apiVersion
of <code class="language-plaintext highlighter-rouge">networking.k8s.io/v1</code>, so you think great, I’m covered.</p>

<p>However, what you’ve checked here is not actually what version of the API your operator is
using, but what version of the API kubectl is using. Your application could have created its
Ingress resource using the <code class="language-plaintext highlighter-rouge">v1beta1</code> API, and the cluster would still present you with the
<code class="language-plaintext highlighter-rouge">v1</code> representation when using kubectl.</p>

<p>This test also doesn’t take into account the API version your operator uses to get resources
from the cluster.</p>

<p>How do you do it right? Follow the advice from the <a href="https://kubernetes.io/docs/reference/using-api/deprecation-guide/#what-to-do">Kubernetes deprecation guide</a>.</p>

<h2 id="example-multiple-versions-of-the-ingress-api">Example: Multiple versions of the Ingress API</h2>

<p>Lets use the example of a very simple Ingress object, straight out of the
Kubernetes documentation. The ingress API had schema changes between <code class="language-plaintext highlighter-rouge">v1beta1</code>
and <code class="language-plaintext highlighter-rouge">v1</code>, so we can use it to demonstrate the runtime conversion performed by
the API server when serializing the resource.</p>

<p>We’ll use this example straight from the Kubernetes documentation:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.k8s.io/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Ingress</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">minimal-ingress</span>
  <span class="na">annotations</span><span class="pi">:</span>
    <span class="na">nginx.ingress.kubernetes.io/rewrite-target</span><span class="pi">:</span> <span class="s">/</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">rules</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">http</span><span class="pi">:</span>
      <span class="na">paths</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/testpath</span>
        <span class="na">pathType</span><span class="pi">:</span> <span class="s">Prefix</span>
        <span class="na">backend</span><span class="pi">:</span>
          <span class="na">service</span><span class="pi">:</span>
            <span class="na">name</span><span class="pi">:</span> <span class="s">test</span>
            <span class="na">port</span><span class="pi">:</span>
              <span class="na">number</span><span class="pi">:</span> <span class="m">80</span>
</code></pre></div></div>

<p>First step, push that resource to the cluster:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[jammy@ibm007470 ~]$ kubectl create -f minimal-ingress.yaml
ingress.networking.k8s.io/minimal-ingress created
</code></pre></div></div>

<p>Notice that there is no API version in that output.</p>

<p>So, next up, get the resource back out:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[jammy@ibm007470 ~]$ kubectl get ingress minimal-ingress -o yaml
</code></pre></div></div>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.k8s.io/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Ingress</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">annotations</span><span class="pi">:</span>
    <span class="na">nginx.ingress.kubernetes.io/rewrite-target</span><span class="pi">:</span> <span class="s">/</span>
  <span class="na">creationTimestamp</span><span class="pi">:</span> <span class="s2">"</span><span class="s">2021-09-15T13:44:02Z"</span>
  <span class="na">generation</span><span class="pi">:</span> <span class="m">1</span>
  <span class="na">managedFields</span><span class="pi">:</span> <span class="s">...</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">minimal-ingress</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">jammy</span>
  <span class="na">resourceVersion</span><span class="pi">:</span> <span class="s2">"</span><span class="s">14187394"</span>
  <span class="na">uid</span><span class="pi">:</span> <span class="s">29aec213-b79a-4b96-9b05-072e9bb48541</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">rules</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">http</span><span class="pi">:</span>
      <span class="na">paths</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">backend</span><span class="pi">:</span>
          <span class="na">service</span><span class="pi">:</span>
            <span class="na">name</span><span class="pi">:</span> <span class="s">test</span>
            <span class="na">port</span><span class="pi">:</span>
              <span class="na">number</span><span class="pi">:</span> <span class="m">80</span>
        <span class="na">path</span><span class="pi">:</span> <span class="s">/testpath</span>
        <span class="na">pathType</span><span class="pi">:</span> <span class="s">Prefix</span>
<span class="na">status</span><span class="pi">:</span>
  <span class="na">loadBalancer</span><span class="pi">:</span> <span class="pi">{}</span>
</code></pre></div></div>

<p>So the default api version used by kubectl is <code class="language-plaintext highlighter-rouge">v1</code>, and the spec is exactly what we put in.</p>

<p>How about if we ask kubectl to use a specific version?</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[jammy@ibm007470 ~]$ kubectl get ingress.v1beta1.networking.k8s.io -o yaml minimal-ingress
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
</code></pre></div></div>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.k8s.io/v1beta1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Ingress</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">annotations</span><span class="pi">:</span>
    <span class="na">nginx.ingress.kubernetes.io/rewrite-target</span><span class="pi">:</span> <span class="s">/</span>
  <span class="na">creationTimestamp</span><span class="pi">:</span> <span class="s2">"</span><span class="s">2021-09-15T13:44:02Z"</span>
  <span class="na">generation</span><span class="pi">:</span> <span class="m">1</span>
  <span class="na">managedFields</span><span class="pi">:</span> <span class="s">...</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">minimal-ingress</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">jammy</span>
  <span class="na">resourceVersion</span><span class="pi">:</span> <span class="s2">"</span><span class="s">14187394"</span>
  <span class="na">uid</span><span class="pi">:</span> <span class="s">29aec213-b79a-4b96-9b05-072e9bb48541</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">rules</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">http</span><span class="pi">:</span>
      <span class="na">paths</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">backend</span><span class="pi">:</span>
          <span class="na">serviceName</span><span class="pi">:</span> <span class="s">test</span>
          <span class="na">servicePort</span><span class="pi">:</span> <span class="m">80</span>
        <span class="na">path</span><span class="pi">:</span> <span class="s">/testpath</span>
        <span class="na">pathType</span><span class="pi">:</span> <span class="s">Prefix</span>
<span class="na">status</span><span class="pi">:</span>
  <span class="na">loadBalancer</span><span class="pi">:</span> <span class="pi">{}</span>
</code></pre></div></div>

<p>Now the spec has been serialized to the <code class="language-plaintext highlighter-rouge">v1beta1</code> schema by the API server
at runtime, you can see <code class="language-plaintext highlighter-rouge">serviceName</code> instead of <code class="language-plaintext highlighter-rouge">service.name</code> in the yaml. Clients
who are using the <code class="language-plaintext highlighter-rouge">v1beta1</code> schema can continue to successfully use it to interact
with the cluster, at least until the API version is removed in Kubernetes 1.22.</p>

<h2 id="behind-the-scenes-how-does-kubernetes-actually-store-its-metadata">Behind the scenes: How does Kubernetes actually store its metadata?</h2>

<p>Kubernetes uses etcd to store cluster metadata, so its all key/value based. Reaching
in to an etcd associated with a Kubernetes cluster you can find keys like this:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/registry/deployments/kube-system/coredns
</code></pre></div></div>

<p>That’s the key for the <code class="language-plaintext highlighter-rouge">coredns</code> <code class="language-plaintext highlighter-rouge">Deployment</code> in the <code class="language-plaintext highlighter-rouge">kube-system</code> namespace. The key
itself does not have an API version, because the API version is only used for serializing
resources.</p>

<p>If you extracted the value associated with that key, you would find a protobuf serialized
representation in whatever API version the server is using for storage. When presenting
that to a user, the API server will convert it to whatever API version is requested by the
client.</p>]]></content><author><name>James Hewitt</name></author><category term="kubernetes" /><category term="kubernetes" /><category term="apiversion" /><category term="api" /><category term="deprecation" /><summary type="html"><![CDATA[OK, so its a bit of a click-bait title, and its not 100% accurate either, but its how I have come to think about resources in Kubernetes. Once a resource has been pushed to the API server and accepted, it no longer has an API version. Exactly how Kubernetes stores its metadata is not my concern.]]></summary></entry><entry><title type="html">The OpenShift anyuid SCC and its effects on storage</title><link href="/kubernetes/openshift-anyuid-scc-and-storage.html" rel="alternate" type="text/html" title="The OpenShift anyuid SCC and its effects on storage" /><published>2021-07-19T00:00:00+00:00</published><updated>2021-07-19T00:00:00+00:00</updated><id>/kubernetes/openshift-anyuid-scc-and-storage</id><content type="html" xml:base="/kubernetes/openshift-anyuid-scc-and-storage.html"><![CDATA[<p>Working for IBM on the Cloud Pak for Integration, recently I’ve been spending a lot of time
with operators, and how to make running software on OpenShift as easy as possible. I came
across a situation where when running under the <code class="language-plaintext highlighter-rouge">restricted</code> SCC, our pods behave, but under
the less restrictive <code class="language-plaintext highlighter-rouge">anyuid</code> SCC, they can fail to access storage. This post is an exploration
of why that is, and a suggestion of how to address it.</p>

<p>A lot of my understanding of how SCCs affect UIDs came from the blog post
<a href="https://www.openshift.com/blog/a-guide-to-openshift-and-uids">A Guide to OpenShift and UIDs</a>
by William Caban Babilonia, its a fantastic resource.</p>

<p>The OpenShift documentation on <a href="https://docs.openshift.com/container-platform/4.6/authentication/managing-security-context-constraints.html">Managing security context constraints</a>
is also good for understanding the different SCC settings and strategies.</p>

<h2 id="using-the-restricted-scc">Using the restricted SCC</h2>

<p>Best practice when creating an OpenShift pod is to use the <code class="language-plaintext highlighter-rouge">restricted</code> SCC. The restricted
SCC adds a number of security features to your running pods, including running as a random
user and group ID that cannot clash with other namespaces in your cluster.</p>

<p>To take advantage of those features, when creating pods you do not specify the user and group
ID so that OpenShift can assign one. This all works perfectly when the <code class="language-plaintext highlighter-rouge">restricted</code> SCC is in
use, because the assignment will be done on pod admission so the metadata is all there for
Kubernetes to use when mounting storage volumes.</p>

<p>When Kubernetes mounts storage volumes to pods, it will (caveat: not for all provisioners!)
<code class="language-plaintext highlighter-rouge">chmod</code> and <code class="language-plaintext highlighter-rouge">chown</code> the volume recursively to allow the pod to access the files. It does this
based (partly) on the <code class="language-plaintext highlighter-rouge">fsGroup</code> provided in the pod’s <code class="language-plaintext highlighter-rouge">securityContext</code>. Under the restricted
SCC, if you don’t specify the <code class="language-plaintext highlighter-rouge">fsGroup</code> yourself, one will be provided based on the allocated
range for the namespace.</p>

<h2 id="whats-different-with-anyuid">What’s different with anyuid?</h2>

<p>The <code class="language-plaintext highlighter-rouge">anyuid</code> SCC enables an important use case for OpenShift - running a pod under the user
and group defined in the image. In a plain Kubernetes distribution, you can do this by not
specifying your own values in the pod spec. In OpenShift, if you don’t specify values in the
pod spec and the <code class="language-plaintext highlighter-rouge">restricted</code> SCC is applied, it will provide defaults, so the image definition
will not be used.</p>

<p>Generally, this is a good thing, and a more secure way to run the containers, but there are
situations where you need to run using the image definition. To achieve this, you can apply
the <code class="language-plaintext highlighter-rouge">anyuid</code> SCC to the pod, which has a higher priority, and doesn’t provide defaults to the
pod when it is applied.</p>

<p>Unfortunately, if your pod is relying on defaults to be applied to successfully <code class="language-plaintext highlighter-rouge">chmod</code> and
<code class="language-plaintext highlighter-rouge">chown</code> your volumes on mount, then applying the <code class="language-plaintext highlighter-rouge">anyuid</code> SCC to a pod designed for the
<code class="language-plaintext highlighter-rouge">restricted</code> SCC can mean you can no longer access mounted storage.</p>

<h2 id="what-as-operator-developers-can-we-do-about-it">What, as operator developers, can we do about it?</h2>

<p>Ideally, we would know in advance which SCC would be applied, and could modify our resources
to specify metadata if we knew OpenShift wasn’t going to do it for us, but unfortunately, there’s
no easy way for the operator to know which SCC will be applied to a pod in advance, and it could
change over time anyway.</p>

<p>When admitting pods under the <code class="language-plaintext highlighter-rouge">restricted</code> SCC, if the <code class="language-plaintext highlighter-rouge">fsGroup</code> is not already set, the first
group in the range defined by the <code class="language-plaintext highlighter-rouge">openshift.io/sa.scc.supplemental-groups</code> annotation on the
namespace will be inserted into the pod definition. This is the part that doesn’t happen with
the <code class="language-plaintext highlighter-rouge">anyuid</code> SCC.</p>

<p>To make our pod compatible with both SCCs, what we can do is examine the annotation ourselves,
and set the <code class="language-plaintext highlighter-rouge">fsGroup</code> for our pod to the first group in the range. This will satisfy the
<code class="language-plaintext highlighter-rouge">restricted</code> SCC, because the group is in range, and will provide the metadata for Kubernetes
to prepare the mount under the <code class="language-plaintext highlighter-rouge">anyuid</code> SCC.</p>

<h2 id="can-you-back-that-up">Can you back that up?</h2>

<p>I ran some tests, and you can too, using my handy test repo:</p>

<p><a href="https://github.com/Jamstah/write-test-container">https://github.com/Jamstah/write-test-container</a></p>

<p>I ran these tests on a Red Hat OpenShift on IBM Cloud cluster using <code class="language-plaintext highlighter-rouge">ibmc-block-gold</code> storage with
OpenShift 4.6.36. The important thing to note is that the volume is always writable where the fsGroup
has been defined within the range of groups for the namespace.</p>

<p>The <code class="language-plaintext highlighter-rouge">scc-nonroot-no-context</code> job is an example of a pod designed for the <code class="language-plaintext highlighter-rouge">restricted</code> SCC only.</p>

<p>The <code class="language-plaintext highlighter-rouge">scc-anyuid-no-context</code> job shows what happens when a pod designed for the <code class="language-plaintext highlighter-rouge">restricted</code> SCC is
run using the <code class="language-plaintext highlighter-rouge">anyuid</code> SCC.</p>

<p>The <code class="language-plaintext highlighter-rouge">scc-nonroot-fsgroupproject</code> and <code class="language-plaintext highlighter-rouge">scc-anyuid-fsgroupproject</code> jobs show how adding an <code class="language-plaintext highlighter-rouge">fsGroup</code>
within the correct range can solve the problem under both SCCs.</p>

<p>Here are my results:</p>

<table>
  <thead>
    <tr>
      <th>Job</th>
      <th>Job spec fsGroup</th>
      <th>Pod Admitted</th>
      <th>Effective SCC</th>
      <th>Pod spec fsGroup</th>
      <th>UID</th>
      <th>GID</th>
      <th>Groups</th>
      <th>Volume UID</th>
      <th>Volume GID</th>
      <th>Volume perms</th>
      <th>Writable</th>
      <th>Written UID</th>
      <th>Written GID</th>
      <th>Written perms</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>scc-anyuid-fsgroup0</td>
      <td>0</td>
      <td>Yes</td>
      <td>anyuid</td>
      <td>0</td>
      <td>uid-1001(1001)</td>
      <td>gid-0(root)</td>
      <td>groups-0(root)</td>
      <td>root</td>
      <td>root</td>
      <td>drwxrwsr-x.</td>
      <td>Yes</td>
      <td>1001</td>
      <td>root</td>
      <td>-rw-r–r–.</td>
    </tr>
    <tr>
      <td>scc-anyuid-fsgroupproject</td>
      <td>1000670000</td>
      <td>Yes</td>
      <td>anyuid</td>
      <td>1000670000</td>
      <td>uid-1001(1001)</td>
      <td>gid-0(root)</td>
      <td>groups-0(root),1000670000</td>
      <td>root</td>
      <td>1000670000</td>
      <td>drwxrwsr-x.</td>
      <td>Yes</td>
      <td>1001</td>
      <td>1000670000</td>
      <td>-rw-r–r–.</td>
    </tr>
    <tr>
      <td>scc-anyuid-no-context</td>
      <td> </td>
      <td>Yes</td>
      <td>anyuid</td>
      <td> </td>
      <td>uid-1001(1001)</td>
      <td>gid-0(root)</td>
      <td>groups-0(root)</td>
      <td>root</td>
      <td>root</td>
      <td>drwxr-xr-x.</td>
      <td>No</td>
      <td> </td>
      <td> </td>
      <td> </td>
    </tr>
    <tr>
      <td>scc-default-fsgroup0</td>
      <td>0</td>
      <td>No</td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
      <td> </td>
    </tr>
    <tr>
      <td>scc-default-fsgroupproject</td>
      <td>1000670000</td>
      <td>Yes</td>
      <td>restricted</td>
      <td>1000670000</td>
      <td>uid-1000670000(1000670000)</td>
      <td>gid-0(root)</td>
      <td>groups-0(root),1000670000</td>
      <td>root</td>
      <td>1000670000</td>
      <td>drwxrwsr-x.</td>
      <td>Yes</td>
      <td>1000670000</td>
      <td>1000670000</td>
      <td>-rw-rw-r–.</td>
    </tr>
    <tr>
      <td>scc-default-no-context</td>
      <td> </td>
      <td>Yes</td>
      <td>restricted</td>
      <td>1000670000</td>
      <td>uid-1000670000(1000670000)</td>
      <td>gid-0(root)</td>
      <td>groups-0(root),1000670000</td>
      <td>root</td>
      <td>1000670000</td>
      <td>drwxrwsr-x.</td>
      <td>Yes</td>
      <td>1000670000</td>
      <td>1000670000</td>
      <td>-rw-rw-r–.</td>
    </tr>
    <tr>
      <td>scc-nonroot-fsgroup0</td>
      <td>0</td>
      <td>Yes</td>
      <td>nonroot</td>
      <td>0</td>
      <td>uid-1001(1001)</td>
      <td>gid-0(root)</td>
      <td>groups-0(root)</td>
      <td>root</td>
      <td>root</td>
      <td>drwxrwsr-x.</td>
      <td>Yes</td>
      <td>1001</td>
      <td>root</td>
      <td>-rw-r–r–.</td>
    </tr>
    <tr>
      <td>scc-nonroot-fsgroupproject</td>
      <td>1000670000</td>
      <td>Yes</td>
      <td>restricted</td>
      <td>1000670000</td>
      <td>uid-1000670000(1000670000)</td>
      <td>gid-0(root)</td>
      <td>groups-0(root),1000670000</td>
      <td>root</td>
      <td>1000670000</td>
      <td>drwxrwsr-x.</td>
      <td>Yes</td>
      <td>1000670000</td>
      <td>1000670000</td>
      <td>-rw-r–r–.</td>
    </tr>
    <tr>
      <td>scc-nonroot-no-context</td>
      <td> </td>
      <td>Yes</td>
      <td>restricted</td>
      <td>1000670000</td>
      <td>uid-1000670000(1000670000)</td>
      <td>gid-0(root)</td>
      <td>groups-0(root),1000670000</td>
      <td>root</td>
      <td>1000670000</td>
      <td>drwxrwsr-x.</td>
      <td>Yes</td>
      <td>1000670000</td>
      <td>1000670000</td>
      <td>-rw-r–r–.</td>
    </tr>
  </tbody>
</table>

<dl>
  <dt>Job</dt>
  <dd>Name of the job from the kustomize, includes the name of the SCC (above the default - restricted) that the service account has access to.</dd>
  <dt>Job spec fsGroup</dt>
  <dd>The fsGroup applied in the pod template in the job spec</dd>
  <dt>Pod Admitted</dt>
  <dd>Was the pod allowed to run on the cluster?</dd>
  <dt>Effective SCC</dt>
  <dd>The SCC that OpenShift used to admit the pod</dd>
  <dt>Pod spec fsGroup</dt>
  <dd>The fsGroup in the pod spec after admission (to see if OpenShift changed it</dd>
  <dt>UID</dt>
  <dd>The UID of the user in the container</dd>
  <dt>GID</dt>
  <dd>The GID of the user in the container</dd>
  <dt>Groups</dt>
  <dd>The additional groups of the user in the container</dd>
  <dt>Volume UID</dt>
  <dd>The UID of the root dir of the volume</dd>
  <dt>Volume GID</dt>
  <dd>The GID of the root dir of the volume</dd>
  <dt>Volume perms</dt>
  <dd>The permissions applies to the root dir of the volume</dd>
  <dt>Writable</dt>
  <dd>Could the container write to storage?</dd>
  <dt>Written UID</dt>
  <dd>What UID was the file written with?</dd>
  <dt>Written GID</dt>
  <dd>What GID was the file written with?</dd>
  <dt>Written perms</dt>
  <dd>What permissions was the file written with (controlled by umask)</dd>
</dl>]]></content><author><name>James Hewitt</name></author><category term="kubernetes" /><category term="OpenShift" /><category term="SCC" /><category term="anyuid" /><category term="Storage" /><category term="fsGroup" /><category term="SecurityContextConstraint" /><summary type="html"><![CDATA[Working for IBM on the Cloud Pak for Integration, recently I’ve been spending a lot of time with operators, and how to make running software on OpenShift as easy as possible. I came across a situation where when running under the restricted SCC, our pods behave, but under the less restrictive anyuid SCC, they can fail to access storage. This post is an exploration of why that is, and a suggestion of how to address it.]]></summary></entry><entry><title type="html">Debouncing an openHAB rule</title><link href="/openhab/debouncing-openhab-rules.html" rel="alternate" type="text/html" title="Debouncing an openHAB rule" /><published>2016-07-06T00:00:00+00:00</published><updated>2016-07-06T00:00:00+00:00</updated><id>/openhab/debouncing-openhab-rules</id><content type="html" xml:base="/openhab/debouncing-openhab-rules.html"><![CDATA[<p>I needed to debounce an openhab rule to cope with a seriously cheap 433MHz contact sensor that, on activation, would spam its ID to RFXCOM a few times. I couldn’t find much on the web so I made a debouncer using locks.</p>

<p>I thought I’d share it here, as it could be useful to others. I left in the debug message so its easy to see how it works, and verify it in the logs if you want to.</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">java.util.concurrent.locks.Lock</span>
<span class="kn">import</span> <span class="nn">java.util.concurrent.locks.ReentrantLock</span>

<span class="kt">var</span> <span class="nc">Lock</span> <span class="n">lock</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ReentrantLock</span><span class="o">()</span>

<span class="n">rule</span> <span class="s">"switch"</span>
<span class="n">when</span>
    <span class="nc">Item</span> <span class="nc">Undecoded</span> <span class="n">received</span> <span class="n">update</span> <span class="s">"107366"</span>
<span class="n">then</span>
    <span class="c1">// tryLock will acquire the lock if it is available, and fail fast otherwise.</span>
    <span class="k">if</span> <span class="o">(</span><span class="n">lock</span><span class="o">.</span><span class="na">tryLock</span><span class="o">())</span> <span class="o">{</span>
        <span class="n">logInfo</span><span class="o">(</span><span class="s">"contact"</span><span class="o">,</span> <span class="s">"Contact sensor tripped"</span><span class="o">)</span>
        <span class="nl">Thread:</span><span class="o">:</span><span class="n">sleep</span><span class="o">(</span><span class="mi">300</span><span class="o">)</span>
        <span class="n">lock</span><span class="o">.</span><span class="na">unlock</span><span class="o">()</span>
    <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
        <span class="n">logDebug</span><span class="o">(</span><span class="s">"doors"</span><span class="o">,</span> <span class="s">"Contact sensor tripped, but ignored due to lock"</span><span class="o">)</span>
    <span class="o">}</span>
<span class="n">end</span>
</code></pre></div></div>]]></content><author><name>James Hewitt</name></author><category term="openhab" /><category term="openHAB" /><summary type="html"><![CDATA[I needed to debounce an openhab rule to cope with a seriously cheap 433MHz contact sensor that, on activation, would spam its ID to RFXCOM a few times. I couldn’t find much on the web so I made a debouncer using locks.]]></summary></entry><entry><title type="html">Using Undecoded items from the RFXCom binding in openhab</title><link href="/openhab/using-undecoded-items-from-rfxcom-in-openhab.html" rel="alternate" type="text/html" title="Using Undecoded items from the RFXCom binding in openhab" /><published>2016-06-06T00:00:00+00:00</published><updated>2016-06-06T00:00:00+00:00</updated><id>/openhab/using-undecoded-items-from-rfxcom-in-openhab</id><content type="html" xml:base="/openhab/using-undecoded-items-from-rfxcom-in-openhab.html"><![CDATA[<p>In the upcoming 1.9.0 release of openhab, the RFXCom binding will support Undecoded items.</p>

<p>You can add the 1.9.0 binding to a 1.8 openhab by copying the jar to the addons folder, get a pre-release (beta 3) jar from here:
https://bintray.com/openhab/mvn/openhab/1.9.0.b3/view#files/org/openhab/binding/org.openhab.binding.rfxcom/1.9.0.b3</p>

<p>You’ll have to use <code class="language-plaintext highlighter-rouge">setMode</code> in <code class="language-plaintext highlighter-rouge">openhab.cfg</code> to turn on undecoded message. The RFXCom device won’t save the setting for undecoded as its supposed to be a debug switch for them to use when adding new devices to their list. Using <code class="language-plaintext highlighter-rouge">setMode</code> will turn it on every time. This is my <code class="language-plaintext highlighter-rouge">setMode</code> line, which basically turns on undecoded and homeeasy. You might need to turn on more, post in the comments if you need a hand.</p>

<p>/etc/openhab/configurations/openhab.cfg:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rfxcom:setMode=0d00001103531c80000f00000000
</code></pre></div></div>

<p>Once you have the new binding running, you’ll need an undecoded item to pick up the events.</p>

<p>/etc/openhab/configurations/items/undecoded.items:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>String Undecoded {rfxcom="&lt;UNDECODED:RawData"}
</code></pre></div></div>

<p>You should then start seeing items appear in your events.log.</p>

<p>/var/log/openhab/events.log:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2016-07-05 08:56:58 - Undecoded state updated to 070301271356ECC0
</code></pre></div></div>

<p>The status is made up of 4 bytes of header, then the data. From this example, the bytes are:
(07) Length, 7 bytes.
(03) Message type - 03 is Undecoded, so they should all have 03 here.
(01) Sub-type - 01 is ARC.
(27) Sequence number. This should increment for every update.
(1356ECC0) The payload, in this case, 4 bytes of data.</p>

<p>This is all documented in <a href="https://github.com/openhab/openhab/blob/master/bundles/binding/org.openhab.binding.rfxcom/src/main/java/org/openhab/binding/rfxcom/internal/messages/RFXComUndecodedRFMessage.java">RFXComUndecodedRFMessage.java</a> on github.</p>

<p>Because all the undecoded items comes through on just one item, its worth splitting them out into separate items using unbound items and a rule to process the undecoded updates. Start by making some items. My undecoded devices are contact sensors that I’ve put on doors, so the example will focus on that.</p>

<p>/etc/openhab/configurations/items/doors.items:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Switch FrontDoor "Front door [%s]"
Switch CatFlap "Cat flap [%s]"
</code></pre></div></div>

<p>For the sake of simplicity, now I’d like a rule to notify me when my front door is opened. You can obviously do anything. My sensor only triggers when the contact is broken, so we only have an Open trigger. To cope with that, I’ve told it to automatically record as closed 5 seconds later.</p>

<p>This rule uses a custom value selector of Data instead of RawData in the items file, to strip off the RFXCom headers which change every time. You’ll need to roll your own rfxcom binding to use it, or get in touch in the comments and I’ll provide one until the <a href="https://github.com/openhab/openhab/pull/4520">pull request</a> makes it.</p>

<p>The ID of the sensor was detected by looking at the events.log as above.</p>

<p>/etc/openhab/configurations/rules/frontdoor.rules:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rule "Front door sensor"
when
    Item Undecoded received update "1356ECC0"
then
    sendCommand(FrontDoor, ON)
end

rule "Front door opened"
when
    Item FrontDoor changed to ON
then
    sendBroadcastNotification("Front door opened")
    Thread::sleep(5000)
    sendCommand(FrontDoor, OFF)
end
</code></pre></div></div>

<p>Assuming you’re logged into your openhab with the app (either direct or through myopenhab), you should now be notified when your door is opened.</p>]]></content><author><name>James Hewitt</name></author><category term="openhab" /><category term="openHAB" /><summary type="html"><![CDATA[In the upcoming 1.9.0 release of openhab, the RFXCom binding will support Undecoded items.]]></summary></entry></feed>