Troubleshoot the JFrog Crossplane Providers

Triage JFrog Crossplane provider issues and fix installation, credential, reconcile, and deletion errors.

Find and fix problems with the JFrog Crossplane providers and the managed resources they reconcile.

Triage

Run the following commands to collect status, conditions, and logs:

kubectl get providers                                   # INSTALLED and HEALTHY must be True
kubectl -n crossplane-system get pods                   # one pod per provider, Running
kubectl get managed -A | grep -v 'True.*True'           # managed resources that are not Synced and Ready
kubectl describe <fq-kind>/<name> -n <ns>               # conditions and events
kubectl get <fq-kind>/<name> -n <ns> -o yaml            # annotations, status.atProvider, condition messages
kubectl -n crossplane-system logs deploy/<provider-deployment> --since=10m

Replace the following:

  • <fq-kind>: The fully-qualified Kind name of the managed resource, for example localgenericrepositories.repository.artifactory.m.crossplane.io.
  • <name>: The name of the managed resource.
  • <ns>: The namespace of the managed resource.
  • <provider-deployment>: The provider deployment in crossplane-system. To find it, see Logs and Metrics.

Check the conditions of a managed resource in this order:

  1. Synced: Shows whether the reconcile loop ran.
  2. LastAsyncOperation: Shows whether the JFrog API request succeeded. The status message displays the API response and HTTP status code.
  3. Ready: Shows whether the object exists in the target environment.

Installation Issues

The following table lists installation symptoms with their causes and fixes.

SymptomCause and fix
Provider INSTALLED=False: unauthorized, 401, deniedRegistry credentials missing or wrong. Create jfrog-registry in crossplane-system for releases-authenticated-docker.jfrog.io and list it in packagePullSecrets. Test with docker login releases-authenticated-docker.jfrog.io.
not found or manifest unknownWrong image path or tag. Use releases-authenticated-docker.jfrog.io/jfrog/crossplane-provider-<name>:v0.0.1.
secret "jfrog-registry" not foundThe Secret must be in the Crossplane namespace.
incompatible Crossplane versionUpgrade Crossplane to 2.0 or later.
x509 or TLS errors while pullingTLS-intercepting proxy. Configure Crossplane's CA bundle (Helm registryCaBundleConfig) or proxy settings.
Pod ErrImagePull or ImagePullBackOffAdd imagePullSecrets: [{name: jfrog-registry}] to the DeploymentRuntimeConfig and reference it with runtimeConfigRef.
Pod CrashLoopBackOffOn first start of provider-artifactory, no matches for kind restarts are transient. Otherwise, the cause is an unknown flag in args, OOMKilled (raise the memory limit), or RBAC.
DeploymentRuntimeConfig ignoredContainer not named package-runtime, missing selector: {}, or a typo in the runtimeConfigRef name.
no matches for kind "X", the server doesn't have a resource typeOne of these causes: the provider isn't Healthy yet, the Kind isn't activated (kubectl get managedresourcedefinitions,managedresourceactivationpolicies), the API group is wrong, or the short name is ambiguous (use the fully-qualified name).

Connection and Credentials Issues

The following table lists connection and credential error messages with their causes and fixes.

MessageCause and fix
cannot get referenced ProviderConfig: ... not foundproviderConfigRef names a missing configuration, a ProviderConfig in another namespace, or another provider's configuration group. For example, an Xray managed resource needs an xray.m.crossplane.io configuration.
unknown GVK for ProviderConfigproviderConfigRef.kind must be ClusterProviderConfig or ProviderConfig.
cannot get credentials secretSecret missing, or wrong name or namespace. Namespaced configurations read the managed resource's namespace.
cannot unmarshal <provider> credentials as JSONThe key isn't a JSON object of strings. Check the Secret content with the following command.
missing 'url' in credentials or missing authentication in credentials; 'access_token' is requiredAdd url. Use access_token (API keys aren't supported).
credential source "Environment" is not permitted on a namespaced ProviderConfigUse a ClusterProviderConfig or source: Secret.
no extraction handler registered for source: InjectedIdentityUse Secret, Environment, or Filesystem.
dial tcp ... i/o timeout, connection refused, no such hostURL not reachable from the pod. Use a routable URL or the in-cluster Service. Test the connection as described in Check the JFrog Connection.
401 Unauthorized, The access token is not validExpired or revoked token, or a token for another JFrog Platform. Update the Secret.
403 Forbidden, Insufficient permissionsThe token lacks admin or project-admin rights.
x509: certificate signed by unknown authorityPrivate CA on JFrog. Mount the CA in the provider pod (SSL_CERT_FILE or SSL_CERT_DIR plus a volume).
404 only on /xray/api or /access/api/v1/projectsXray or Projects not available on the JFrog Platform, or the URL routes only /artifactory. Use the Platform base URL.
ProviderConfig stuck TerminatingManaged resources still use it. Check with kubectl get providerconfigusages.<provider>.m.crossplane.io -A.

To check that the credentials Secret contains valid JSON, run the following command:

kubectl get secret jfrog-creds -n crossplane-system -o jsonpath='{.data.credentials}' | base64 -d | jq .

Reconcile Errors

The following table lists reconcile error messages with their causes and fixes.

MessageCause and fix
cannot resolve references: ... referenced field was emptyThe target exists but isn't created yet. Wait.
cannot resolve references: ... not found, no resources matched selectorName typo, wrong namespace, or labels don't match (matchControllerRef only inside Compositions).
spec.forProvider.<field> is a required parameterSupply the field in forProvider or initProvider. For Certificate, contentSecretRef must be set on forProvider.
spec.providerConfigRef.kind: Required valueWrite both kind and name.
`spec.managementPolicies` is set to a value(...) which is not supportedUse a supported combination. For more information, see Control What Crossplane Can Do.
unknown field "spec.forProvider.xyz", .spec.deletionPolicy: field not declared in schemaWrong field name. Or deletionPolicy is set on a namespaced managed resource: use managementPolicies instead.
metadata.name: Invalid valueUse a DNS-1123 name plus the external-name annotation.
cannot determine creation result - remove the crossplane.io/external-create-pending annotation if it is safe to proceedThe provider restarted during a create. Check JFrog. If the object exists with a server-assigned ID, set the external name to that ID first. Then run kubectl annotate <fq-kind>/<name> -n <ns> crossplane.io/external-create-pending-.
diff contains fields that require resource replacement or refuse to update the external resource because the following update requires replacing itImmutable field changed. Revert, or delete and recreate.
<Report> Update Not SupportedXray reports are one-shot. Create a new report.
Assignment change has no effect (Update not supported)Affects project Repository, ShareRepository, and ShareRepositoryWithAll. Delete and recreate.
cannot find id in tfstateArtifact or ScopedToken. For more information, see Known Issues in v0.0.1 on this page.
cannot find <field> in tfstateThe object no longer exists or the read failed. Check LastAsyncOperation.
Duplicate object createdThe external-name annotation of a server-assigned Kind was removed, or was not set when adopting. Set it to the existing ID and delete the duplicate.
object has been modified; please apply your changes to the latest versionBenign conflict, retried automatically.

Never-Ending Updates

Repeated UpdatedExternalResource events mean the provider updates the object on every reconcile. Compare spec.forProvider with status.atProvider. The causes are:

  • Two owners for one field, for example User.groups and GroupMembers.
  • A value the server normalizes: unsorted webhook eventTypes, projectEnvironments on a repository not assigned to the project, or a late-initialized proxy after you delete a Proxy.

To fix it, do one of the following:

  • Write the canonical form.
  • Move the field to initProvider.
  • Keep one owner.

The reference lists Kind-specific cases. For more information, see JFrog Crossplane Kinds Reference.

Common JFrog API Errors

The following table lists common JFrog API error messages with their causes and fixes.

MessageCause and fix
Repository <key> does not exist (virtual repository), Specified repo does not exist (policies, permissions, replication)Member not created yet (converges) or misspelled.
repository <key> doesn't exist (Watch)Set watchResource[].repoType: remote for remote repositories. The repository must exist and be Xray-indexed.
policy <name> doesn't exist (Watch)Create the policy first, or fix assignedPolicy[].name and type.
policy cannot have empty effective scope (CurationPolicy)No curated remote repositories. Set curated: true or use specific_repos.
The package type ... is not supportedThe Artifactory version is too old for that package type.
Could not find project (Environment)The project was deleted first. Delete environments before projects, or remove the finalizer if the project is gone.
Project or environment name rejectedProject key: 2-32 lowercase alphanumerics or hyphens, starting with a letter. <projectKey>-<env>: at most 32 characters.
Permission or project membership fails with not foundEvery user, group, and repository named must already exist.
Users locked out after SAML, Crowd, or HTTP SSO changesFix or delete the setting with an administrator access token.

Known Issues in v0.0.1

The following table lists the known issues in this version with their workarounds.

KindIssueWorkaround
artifactory ArtifactObserve fails with cannot find id in tfstate. The managed resource never becomes Synced and can't be deleted cleanly.Upload with jf rt upload or the REST API. To clean up, delete the file, then remove the managed resource finalizer.
artifactory ItemPropertiesCreate never completes when the item had no properties.jf rt set-props or PUT /artifactory/api/storage/<repo>/<path>?properties=...
artifactory ScopedTokenWrite-once, not importable. Observe fails.Mint tokens outside Crossplane.
artifactory SamlSettings, PermissionTarget, APIKeyDeprecated upstream (SamlSettings create fails).platform SAMLSettings, platform Permission, access tokens
Xray policiesDelete can intermittently hang in Deleting.Delete watches first. If GET /xray/api/v2/policies/<name> returns 404, remove the finalizer.
platform LifecycleDelete is a no-op.Deleting the managed resource leaves the lifecycle in place.
platform Group.membersDeprecated and ignored.Use GroupMembers.

Deletion Issues

The following table lists deletion symptoms with their fixes.

SymptomFix
Managed resource stuck TerminatingRead LastAsyncOperation. Delete dependents first (watches before policies, virtual repositories before members, environments before projects). If the JFrog object is already gone, remove the finalizer.
Managed resource stuck and the provider is uninstalledReinstall the provider, or remove finalizers (the JFrog object is kept).
Namespace stuck TerminatingManaged resources inside still have finalizers. Delete them properly first.
this resource is in useA Usage protects the resource. Delete the using resource first.

Remove a finalizer only after you confirm that the JFrog object is gone or should be kept. To remove the finalizers, run the following command:

kubectl patch <fq-kind>/<name> -n <ns> --type merge -p '{"metadata":{"finalizers":[]}}'

Verify State in JFrog

Call the JFrog REST API with the provider's URL and token (-H "Authorization: Bearer $TOKEN"). Compare the response with status.atProvider.

ObjectRequest
HealthGET /artifactory/api/system/ping, /xray/api/v1/system/ping, /access/api/v1/system/ping
RepositoriesGET /artifactory/api/repositories/<key>
Users, groups, membersGET /access/api/v2/users/<name>, /access/api/v2/groups/<name>, /access/api/v2/groups/<name>/members
Permissions, global rolesGET /access/api/v2/permissions/<name>, /access/api/v1/roles/<name>
ProjectsGET /access/api/v1/projects/<key> (plus /environments, /roles, /users, /groups)
Xray policies, watches, reportsGET /xray/api/v2/policies/<name>, /xray/api/v2/watches/<name>, /xray/api/v1/reports/<id>
WebhooksGET /event/api/v1/subscriptions/<key>
Release Bundles v2GET /lifecycle/api/v2/release_bundle/records/<name>

Expect the following timing:

  • Simple objects converge within seconds of their dependencies.
  • Each retry step adds up to 60 s.
  • Drift is corrected within one --poll interval.
  • Project deletion takes minutes.

To create a support bundle, collect the following information. Redact tokens before you share it.

  • kubectl get providers,providerrevisions,deploymentruntimeconfigs -o yaml
  • kubectl get managed -A -o yaml
  • The Crossplane and provider logs
  • kubectl get events -A --sort-by=.lastTimestamp

Logs and Metrics

Each provider revision runs as one Deployment in crossplane-system.

To find the provider deployment and read its logs:

  1. List the deployments:

    kubectl -n crossplane-system get deploy
  2. Read the logs of a provider deployment:

    kubectl -n crossplane-system logs deploy/<name>

    Replace <name> with the deployment name from the previous step.

For more detail in the logs, enable the --debug flag. For more information, see Provider Flags.

The providers serve metrics on port 8080 at the /metrics path.

Next Steps


Did this page help you?