{"schemaVersion":1,"id":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms","title":"HSMs and Cloud KMS: PKCS#11, Envelope Encryption, and Real Secrets Management Architecture","description":"A deep technical walkthrough of hardware security modules, cloud KMS services, PKCS#11 interfaces, envelope encryption patterns, and building a secrets management architecture that actually survives audit.","language":"en","datePublished":"2026-01-25","dateModified":null,"contentFormat":"text/markdown","content":"Security\n\n# HSMs and Cloud KMS: PKCS#11, Envelope Encryption, and Real Secrets Management Architecture\n\n2026-01-258 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/hardware-security-modules-hsm-cloud-kms>)\n\n## HSM Architecture and Principles\n\nA Hardware Security Module is a dedicated piece of hardware, a physical box or a PCIe card, whose sole job is to generate, store, and use cryptographic keys without ever exposing them. The key material lives inside a tamper-resistant boundary. If someone physically breaks into the HSM, the keys are automatically destroyed (this is called zeroization).\n\nThe critical property is that keys never leave the HSM in plaintext. You can ask the HSM to encrypt data using a stored key, decrypt data using a stored key, sign a message, verify a signature, but you cannot ask it to give you the key itself. This is fundamentally different from a secrets manager like [HashiCorp Vault](<https://www.vaultproject.io/>), where the secrets are stored encrypted but can be retrieved in plaintext.\n\n## PKCS#11: The Universal HSM Interface\n\nPKCS#11 (also called Cryptoki) is the standard API for interacting with cryptographic tokens, including HSMs. It's a C API from the 1990s, and it shows, but it's what everything uses.\n\n```c\n// pkcs11_example.c - Basic HSM operations via PKCS#11\n#include <pkcs11.h>\n#include <stdio.h>\n#include <string.h>\n\nint main() {\n CK_RV rv;\n CK_SESSION_HANDLE session;\n CK_SLOT_ID slot = 0;\n\n // Initialize the PKCS#11 library\n rv = C_Initialize(NULL);\n if (rv != CKR_OK) { printf(\"Init failed: %lx\\n\", rv); return 1; }\n\n // Open a session\n rv = C_OpenSession(slot, CKF_SERIAL_SESSION | CKF_RW_SESSION,\n NULL, NULL, &session);\n if (rv != CKR_OK) { printf(\"OpenSession failed: %lx\\n\", rv); return 1; }\n\n // Login as crypto user\n CK_UTF8CHAR pin[] = \"crypto-user-pin\";\n rv = C_Login(session, CKU_USER, pin, sizeof(pin) - 1);\n\n // Generate an AES-256 key inside the HSM\n CK_MECHANISM mechanism = { CKM_AES_KEY_GEN, NULL, 0 };\n CK_ULONG key_length = 32; // 256 bits\n CK_BBOOL true_val = CK_TRUE;\n CK_BBOOL false_val = CK_FALSE;\n CK_UTF8CHAR label[] = \"my-data-encryption-key\";\n\n CK_ATTRIBUTE key_template[] = {\n { CKA_TOKEN, &true_val, sizeof(true_val) }, // Persistent\n { CKA_ENCRYPT, &true_val, sizeof(true_val) }, // Can encrypt\n { CKA_DECRYPT, &true_val, sizeof(true_val) }, // Can decrypt\n { CKA_EXTRACTABLE, &false_val, sizeof(false_val) }, // CANNOT export\n { CKA_SENSITIVE, &true_val, sizeof(true_val) }, // Sensitive\n { CKA_LABEL, label, sizeof(label) - 1 },\n { CKA_VALUE_LEN, &key_length, sizeof(key_length) },\n };\n\n CK_OBJECT_HANDLE key;\n rv = C_GenerateKey(session, &mechanism, key_template,\n sizeof(key_template) / sizeof(CK_ATTRIBUTE), &key);\n\n // Encrypt data using the key (key never leaves the HSM)\n CK_MECHANISM enc_mechanism = { CKM_AES_GCM, NULL, 0 };\n CK_BYTE plaintext[] = \"sensitive database password: hunter2\";\n CK_BYTE ciphertext[256];\n CK_ULONG ciphertext_len = sizeof(ciphertext);\n\n // GCM parameters\n CK_GCM_PARAMS gcm_params = {\n .pIv = (CK_BYTE_PTR)\"random-iv-12b\",\n .ulIvLen = 12,\n .ulIvBits = 96,\n .pAAD = NULL,\n .ulAADLen = 0,\n .ulTagBits = 128,\n };\n enc_mechanism.pParameter = &gcm_params;\n enc_mechanism.ulParameterLen = sizeof(gcm_params);\n\n rv = C_EncryptInit(session, &enc_mechanism, key);\n rv = C_Encrypt(session, plaintext, sizeof(plaintext) - 1,\n ciphertext, &ciphertext_len);\n\n printf(\"Encrypted %lu bytes\\n\", ciphertext_len);\n\n // Cleanup\n C_Logout(session);\n C_CloseSession(session);\n C_Finalize(NULL);\n return 0;\n}\n```\n\nNotice ` CKA_EXTRACTABLE = false ` and ` CKA_SENSITIVE = true `. These attributes ensure the key can never be exported from the HSM. The encryption happens inside the HSM, and only the ciphertext comes back. Even a root user on the host machine cannot retrieve the key material.\n\n## Cloud KMS: HSMs as a Service\n\nCloud providers offer KMS services that are backed by HSMs but abstracted behind an API. The three major ones:\n\n### AWS KMS\n\n```python\n# aws_kms_envelope.py - Envelope encryption with AWS KMS\nimport boto3\nimport os\nfrom cryptography.hazmat.primitives.ciphers.aead import AESGCM\n\nkms = boto3.client('kms', region_name='us-east-1')\n\n# The CMK lives in KMS (backed by HSM). You never see the key material.\nCMK_KEY_ID = 'arn:aws:kms:us-east-1:123456789:key/mrk-abc123'\n\ndef encrypt_data(plaintext: bytes, context: dict) -> dict:\n \"\"\"\n Envelope encryption:\n 1. KMS generates a data key (inside the HSM)\n 2. KMS returns plaintext data key + encrypted data key\n 3. We use plaintext data key to encrypt our data locally\n 4. We discard plaintext data key\n 5. We store encrypted data key alongside the ciphertext\n \"\"\"\n # Step 1-2: Generate data key\n response = kms.generate_data_key(\n KeyId=CMK_KEY_ID,\n KeySpec='AES_256',\n EncryptionContext=context # Bound to this encryption context\n )\n\n plaintext_key = response['Plaintext'] # 32 bytes, use and discard\n encrypted_key = response['CiphertextBlob'] # Store this\n\n # Step 3: Encrypt locally with the plaintext data key\n nonce = os.urandom(12)\n aesgcm = AESGCM(plaintext_key)\n ciphertext = aesgcm.encrypt(nonce, plaintext, None)\n\n # Step 4: CRITICAL - zero out the plaintext key\n # Python doesn't guarantee this, but we try\n plaintext_key = b'\\x00' * len(plaintext_key)\n del plaintext_key\n\n # Step 5: Return everything needed for decryption\n return {\n 'ciphertext': ciphertext,\n 'nonce': nonce,\n 'encrypted_data_key': encrypted_key,\n 'encryption_context': context,\n }\n\ndef decrypt_data(envelope: dict) -> bytes:\n \"\"\"\n Reverse the envelope:\n 1. Send encrypted data key to KMS for decryption\n 2. KMS decrypts the data key (inside the HSM)\n 3. Use the plaintext data key to decrypt locally\n \"\"\"\n # Step 1-2: Decrypt the data key\n response = kms.decrypt(\n CiphertextBlob=envelope['encrypted_data_key'],\n EncryptionContext=envelope['encryption_context']\n )\n plaintext_key = response['Plaintext']\n\n # Step 3: Decrypt locally\n aesgcm = AESGCM(plaintext_key)\n plaintext = aesgcm.decrypt(\n envelope['nonce'],\n envelope['ciphertext'],\n None\n )\n\n # Zero out key\n plaintext_key = b'\\x00' * len(plaintext_key)\n del plaintext_key\n\n return plaintext\n\n# Usage\ncontext = {'service': 'payment-api', 'environment': 'production'}\nenvelope = encrypt_data(b'database_password=super_secret_123', context)\nrecovered = decrypt_data(envelope)\n```\n\nThe ` EncryptionContext ` parameter is often overlooked but it's critical for security. It binds the encrypted data key to a specific context. If someone tries to decrypt the data key with a different context, KMS refuses. This prevents an attacker from taking an encrypted data key from one service and using it in another.\n\n### Key Rotation That Doesn't Break Things\n\n```python\n# key_rotation.py - Automated key rotation with AWS KMS\nimport boto3\nfrom datetime import datetime, timedelta\n\nkms = boto3.client('kms')\n\ndef setup_automatic_rotation(key_id: str):\n \"\"\"Enable automatic annual rotation for a CMK.\"\"\"\n kms.enable_key_rotation(KeyId=key_id)\n\n # Verify rotation is enabled\n status = kms.get_key_rotation_status(KeyId=key_id)\n print(f\"Rotation enabled: {status['KeyRotationEnabled']}\")\n print(f\"Next rotation: ~{datetime.now() + timedelta(days=365)}\")\n\ndef manual_rotation_with_alias(old_key_id: str, alias_name: str):\n \"\"\"\n Manual rotation strategy using aliases.\n This is useful when you need to rotate more frequently\n than annually, or need to rotate asymmetric keys.\n \"\"\"\n # Create new key\n new_key = kms.create_key(\n Description=f'Rotated key - {datetime.now().isoformat()}',\n KeyUsage='ENCRYPT_DECRYPT',\n Origin='AWS_KMS',\n Tags=[\n {'TagKey': 'RotatedFrom', 'TagValue': old_key_id},\n {'TagKey': 'RotationDate', 'TagValue': datetime.now().isoformat()},\n ]\n )\n new_key_id = new_key['KeyMetadata']['KeyId']\n\n # Update alias to point to new key\n # All new encryptions will use the new key\n kms.update_alias(\n AliasName=alias_name,\n TargetKeyId=new_key_id\n )\n\n # OLD KEY REMAINS ACTIVE for decryption\n # This is the key insight: you don't re-encrypt existing data\n # KMS tracks which key version encrypted each data key\n print(f\"Alias {alias_name} now points to {new_key_id}\")\n print(f\"Old key {old_key_id} remains active for decryption\")\n\n # Schedule old key deletion after grace period (e.g., 90 days)\n # Only after you're certain no data encrypted with it remains\n # kms.schedule_key_deletion(KeyId=old_key_id, PendingWindowInDays=30)\n```\n\n## The Complete Secrets Management Architecture\n\nHere's the architecture I deployed for the client that started this whole journey:\n\n```\n┌─────────────────────────────────────────────────────┐\n│ Application Layer │\n│ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │\n│ │ Payment │ │ Auth │ │ Notification │ │\n│ │ Service │ │ Service │ │ Service │ │\n│ └────┬─────┘ └────┬─────┘ └────────┬─────────┘ │\n│ │ │ │ │\n│ ┌────▼──────────────▼──────────────────▼────────┐ │\n│ │ HashiCorp Vault │ │\n│ │ - Dynamic database credentials │ │\n│ │ - PKI certificate issuance │ │\n│ │ - Transit encryption (delegates to KMS) │ │\n│ │ - Lease-based secrets with auto-rotation │ │\n│ └────────────────────┬──────────────────────────┘ │\n│ │ │\n│ ┌────────────────────▼──────────────────────────┐ │\n│ │ AWS KMS │ │\n│ │ - Vault auto-unseal │ │\n│ │ - Envelope encryption for transit engine │ │\n│ │ - S3 SSE-KMS for audit logs │ │\n│ │ - RDS TDE master key │ │\n│ └────────────────────┬──────────────────────────┘ │\n│ │ │\n│ ┌────────────────────▼──────────────────────────┐ │\n│ │ AWS CloudHSM (FIPS 140-2 L3) │ │\n│ │ - Custom KMS key store │ │\n│ │ - PCI DSS key management │ │\n│ │ - Code signing keys │ │\n│ └───────────────────────────────────────────────┘ │\n└─────────────────────────────────────────────────────┘\n```\n\n```hcl\n# vault_config.hcl - Vault configuration with KMS auto-unseal\nstorage \"raft\" {\n path = \"/opt/vault/data\"\n node_id = \"vault-1\"\n\n retry_join {\n leader_api_addr = \"https://vault-2.internal:8200\"\n }\n retry_join {\n leader_api_addr = \"https://vault-3.internal:8200\"\n }\n}\n\nlistener \"tcp\" {\n address = \"0.0.0.0:8200\"\n tls_cert_file = \"/opt/vault/tls/vault.crt\"\n tls_key_file = \"/opt/vault/tls/vault.key\"\n}\n\n# Auto-unseal using AWS KMS\nseal \"awskms\" {\n region = \"us-east-1\"\n kms_key_id = \"mrk-abc123\"\n}\n\n# Audit logging to encrypted S3\naudit {\n type = \"file\"\n path = \"file\"\n options = {\n file_path = \"/var/log/vault/audit.log\"\n }\n}\n\ntelemetry {\n prometheus_retention_time = \"30s\"\n disable_hostname = true\n}\n```\n\nThe three-tier architecture is deliberate:\n\n1. **Vault** handles dynamic secrets (database credentials that rotate every hour), PKI, and application-facing secret delivery.\n2. **KMS** handles Vault's auto-unseal (so Vault operators don't need unseal keys) and provides the backing encryption for Vault's transit engine.\n3. **CloudHSM** backs the KMS custom key store for PCI-regulated encryption keys. The payment service's card data encryption key chain ultimately roots in hardware.\n\n## What I Got Wrong\n\nTwo things bit me on this project:\n\nFirst, I underestimated the cost of CloudHSM. At roughly $1.50/hr per HSM (and you need at least two for HA), that's over $2,000/month just for the HSMs. For most workloads, standard KMS at $1/key/month plus $0.03/10,000 API calls is more than sufficient. CloudHSM is for when you genuinely need FIPS 140-2 Level 3 or single-tenant hardware.\n\nSecond, the PKCS#11 tooling ecosystem is painful. The libraries are old, the documentation assumes you're running Solaris, and debugging session errors requires reading hex dumps. If you can stay in the cloud KMS API layer, do it. Drop to PKCS#11 only when you must.\n\nThe ` .env ` file is gone. The database passwords rotate hourly. The audit trail shows exactly who accessed what key, when. And the next security review? Passed without findings. That felt good.\n\n## Frequently asked questions\n\n### Do I need a dedicated HSM or is cloud KMS sufficient?\n\nCloud KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS) is sufficient for 95% of workloads. You need a dedicated HSM (CloudHSM, Azure Dedicated HSM) only when regulatory requirements mandate single-tenant hardware, when you need custom cryptographic operations not supported by KMS, or when you need PKCS#11/JCE direct access.\n\n### What's the difference between a CMK and a data key?\n\nA Customer Master Key (CMK) lives inside the KMS/HSM and never leaves. A data key is generated by KMS, used to encrypt your actual data, and then the data key itself is encrypted by the CMK. This two-tier approach is envelope encryption.\n\n### How often should I rotate encryption keys?\n\nRotate CMKs annually as a baseline. AWS KMS supports automatic annual rotation. Data keys can be rotated more frequently if needed. The important thing is that rotation is automatic and doesn't require re-encrypting existing data when using envelope encryption.\n\n## Tips\n\n1. **Tip:** Never export master keys from an HSM. If your architecture requires the master key to leave the HSM boundary, your architecture is wrong. Use ` envelope encryption ` instead.\n2. **Tip:** AWS KMS ` GenerateDataKey ` returns both plaintext and ciphertext versions of the data key. Use the plaintext to encrypt, immediately discard it from memory, store only the ciphertext alongside the encrypted data.\n3. **Tip:** Key rotation doesn't mean re-encrypting everything. With envelope encryption, you only rotate the KEK, existing data keys remain valid because they're encrypted with the old KEK version, which KMS still knows how to decrypt.\n4. **Tip:** Always set a ` DeletionWindowInDays ` of at least 14 days on KMS key deletion. Accidentally deleting a KMS key is permanent and catastrophic, those 14 days have saved more than one team I know.\n\n- hsm\n- kms\n- pkcs11\n- envelope-encryption\n- key-rotation\n- aws-cloudhsm\n- secrets-management\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"HSMs and Cloud KMS: PKCS#11, Envelope Encryption, and Real Secrets Management Architecture","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#title"},{"level":2,"text":"HSM Architecture and Principles","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#hsm-architecture-and-principles"},{"level":2,"text":"PKCS#11: The Universal HSM Interface","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#pkcs11-the-universal-hsm-interface"},{"level":2,"text":"Cloud KMS: HSMs as a Service","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#cloud-kms-hsms-as-a-service"},{"level":3,"text":"AWS KMS","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#aws-kms"},{"level":3,"text":"Key Rotation That Doesn't Break Things","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#key-rotation-that-doesnt-break-things"},{"level":2,"text":"The Complete Secrets Management Architecture","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#the-complete-secrets-management-architecture"},{"level":2,"text":"What I Got Wrong","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#what-i-got-wrong"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#questions"},{"level":3,"text":"Do I need a dedicated HSM or is cloud KMS sufficient?","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#question-1"},{"level":3,"text":"What's the difference between a CMK and a data key?","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#question-2"},{"level":3,"text":"How often should I rotate encryption keys?","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#article-tips"}],"sections":[{"heading":"HSMs and Cloud KMS: PKCS#11, Envelope Encryption, and Real Secrets Management Architecture","level":1,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms","content":"Security"},{"heading":"HSMs and Cloud KMS: PKCS#11, Envelope Encryption, and Real Secrets Management Architecture","level":1,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#title","content":"2026-01-258 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/hardware-security-modules-hsm-cloud-kms>)"},{"heading":"HSM Architecture and Principles","level":2,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#hsm-architecture-and-principles","content":"A Hardware Security Module is a dedicated piece of hardware, a physical box or a PCIe card, whose sole job is to generate, store, and use cryptographic keys without ever exposing them. The key material lives inside a tamper-resistant boundary. If someone physically breaks into the HSM, the keys are automatically destroyed (this is called zeroization).\n\nThe critical property is that keys never leave the HSM in plaintext. You can ask the HSM to encrypt data using a stored key, decrypt data using a stored key, sign a message, verify a signature, but you cannot ask it to give you the key itself. This is fundamentally different from a secrets manager like [HashiCorp Vault](<https://www.vaultproject.io/>), where the secrets are stored encrypted but can be retrieved in plaintext."},{"heading":"PKCS#11: The Universal HSM Interface","level":2,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#pkcs11-the-universal-hsm-interface","content":"PKCS#11 (also called Cryptoki) is the standard API for interacting with cryptographic tokens, including HSMs. It's a C API from the 1990s, and it shows, but it's what everything uses.\n\n```c\n// pkcs11_example.c - Basic HSM operations via PKCS#11\n#include <pkcs11.h>\n#include <stdio.h>\n#include <string.h>\n\nint main() {\n CK_RV rv;\n CK_SESSION_HANDLE session;\n CK_SLOT_ID slot = 0;\n\n // Initialize the PKCS#11 library\n rv = C_Initialize(NULL);\n if (rv != CKR_OK) { printf(\"Init failed: %lx\\n\", rv); return 1; }\n\n // Open a session\n rv = C_OpenSession(slot, CKF_SERIAL_SESSION | CKF_RW_SESSION,\n NULL, NULL, &session);\n if (rv != CKR_OK) { printf(\"OpenSession failed: %lx\\n\", rv); return 1; }\n\n // Login as crypto user\n CK_UTF8CHAR pin[] = \"crypto-user-pin\";\n rv = C_Login(session, CKU_USER, pin, sizeof(pin) - 1);\n\n // Generate an AES-256 key inside the HSM\n CK_MECHANISM mechanism = { CKM_AES_KEY_GEN, NULL, 0 };\n CK_ULONG key_length = 32; // 256 bits\n CK_BBOOL true_val = CK_TRUE;\n CK_BBOOL false_val = CK_FALSE;\n CK_UTF8CHAR label[] = \"my-data-encryption-key\";\n\n CK_ATTRIBUTE key_template[] = {\n { CKA_TOKEN, &true_val, sizeof(true_val) }, // Persistent\n { CKA_ENCRYPT, &true_val, sizeof(true_val) }, // Can encrypt\n { CKA_DECRYPT, &true_val, sizeof(true_val) }, // Can decrypt\n { CKA_EXTRACTABLE, &false_val, sizeof(false_val) }, // CANNOT export\n { CKA_SENSITIVE, &true_val, sizeof(true_val) }, // Sensitive\n { CKA_LABEL, label, sizeof(label) - 1 },\n { CKA_VALUE_LEN, &key_length, sizeof(key_length) },\n };\n\n CK_OBJECT_HANDLE key;\n rv = C_GenerateKey(session, &mechanism, key_template,\n sizeof(key_template) / sizeof(CK_ATTRIBUTE), &key);\n\n // Encrypt data using the key (key never leaves the HSM)\n CK_MECHANISM enc_mechanism = { CKM_AES_GCM, NULL, 0 };\n CK_BYTE plaintext[] = \"sensitive database password: hunter2\";\n CK_BYTE ciphertext[256];\n CK_ULONG ciphertext_len = sizeof(ciphertext);\n\n // GCM parameters\n CK_GCM_PARAMS gcm_params = {\n .pIv = (CK_BYTE_PTR)\"random-iv-12b\",\n .ulIvLen = 12,\n .ulIvBits = 96,\n .pAAD = NULL,\n .ulAADLen = 0,\n .ulTagBits = 128,\n };\n enc_mechanism.pParameter = &gcm_params;\n enc_mechanism.ulParameterLen = sizeof(gcm_params);\n\n rv = C_EncryptInit(session, &enc_mechanism, key);\n rv = C_Encrypt(session, plaintext, sizeof(plaintext) - 1,\n ciphertext, &ciphertext_len);\n\n printf(\"Encrypted %lu bytes\\n\", ciphertext_len);\n\n // Cleanup\n C_Logout(session);\n C_CloseSession(session);\n C_Finalize(NULL);\n return 0;\n}\n```\n\nNotice ` CKA_EXTRACTABLE = false ` and ` CKA_SENSITIVE = true `. These attributes ensure the key can never be exported from the HSM. The encryption happens inside the HSM, and only the ciphertext comes back. Even a root user on the host machine cannot retrieve the key material."},{"heading":"Cloud KMS: HSMs as a Service","level":2,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#cloud-kms-hsms-as-a-service","content":"Cloud providers offer KMS services that are backed by HSMs but abstracted behind an API. The three major ones:"},{"heading":"AWS KMS","level":3,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#aws-kms","content":"```python\n# aws_kms_envelope.py - Envelope encryption with AWS KMS\nimport boto3\nimport os\nfrom cryptography.hazmat.primitives.ciphers.aead import AESGCM\n\nkms = boto3.client('kms', region_name='us-east-1')\n\n# The CMK lives in KMS (backed by HSM). You never see the key material.\nCMK_KEY_ID = 'arn:aws:kms:us-east-1:123456789:key/mrk-abc123'\n\ndef encrypt_data(plaintext: bytes, context: dict) -> dict:\n \"\"\"\n Envelope encryption:\n 1. KMS generates a data key (inside the HSM)\n 2. KMS returns plaintext data key + encrypted data key\n 3. We use plaintext data key to encrypt our data locally\n 4. We discard plaintext data key\n 5. We store encrypted data key alongside the ciphertext\n \"\"\"\n # Step 1-2: Generate data key\n response = kms.generate_data_key(\n KeyId=CMK_KEY_ID,\n KeySpec='AES_256',\n EncryptionContext=context # Bound to this encryption context\n )\n\n plaintext_key = response['Plaintext'] # 32 bytes, use and discard\n encrypted_key = response['CiphertextBlob'] # Store this\n\n # Step 3: Encrypt locally with the plaintext data key\n nonce = os.urandom(12)\n aesgcm = AESGCM(plaintext_key)\n ciphertext = aesgcm.encrypt(nonce, plaintext, None)\n\n # Step 4: CRITICAL - zero out the plaintext key\n # Python doesn't guarantee this, but we try\n plaintext_key = b'\\x00' * len(plaintext_key)\n del plaintext_key\n\n # Step 5: Return everything needed for decryption\n return {\n 'ciphertext': ciphertext,\n 'nonce': nonce,\n 'encrypted_data_key': encrypted_key,\n 'encryption_context': context,\n }\n\ndef decrypt_data(envelope: dict) -> bytes:\n \"\"\"\n Reverse the envelope:\n 1. Send encrypted data key to KMS for decryption\n 2. KMS decrypts the data key (inside the HSM)\n 3. Use the plaintext data key to decrypt locally\n \"\"\"\n # Step 1-2: Decrypt the data key\n response = kms.decrypt(\n CiphertextBlob=envelope['encrypted_data_key'],\n EncryptionContext=envelope['encryption_context']\n )\n plaintext_key = response['Plaintext']\n\n # Step 3: Decrypt locally\n aesgcm = AESGCM(plaintext_key)\n plaintext = aesgcm.decrypt(\n envelope['nonce'],\n envelope['ciphertext'],\n None\n )\n\n # Zero out key\n plaintext_key = b'\\x00' * len(plaintext_key)\n del plaintext_key\n\n return plaintext\n\n# Usage\ncontext = {'service': 'payment-api', 'environment': 'production'}\nenvelope = encrypt_data(b'database_password=super_secret_123', context)\nrecovered = decrypt_data(envelope)\n```\n\nThe ` EncryptionContext ` parameter is often overlooked but it's critical for security. It binds the encrypted data key to a specific context. If someone tries to decrypt the data key with a different context, KMS refuses. This prevents an attacker from taking an encrypted data key from one service and using it in another."},{"heading":"Key Rotation That Doesn't Break Things","level":3,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#key-rotation-that-doesnt-break-things","content":"```python\n# key_rotation.py - Automated key rotation with AWS KMS\nimport boto3\nfrom datetime import datetime, timedelta\n\nkms = boto3.client('kms')\n\ndef setup_automatic_rotation(key_id: str):\n \"\"\"Enable automatic annual rotation for a CMK.\"\"\"\n kms.enable_key_rotation(KeyId=key_id)\n\n # Verify rotation is enabled\n status = kms.get_key_rotation_status(KeyId=key_id)\n print(f\"Rotation enabled: {status['KeyRotationEnabled']}\")\n print(f\"Next rotation: ~{datetime.now() + timedelta(days=365)}\")\n\ndef manual_rotation_with_alias(old_key_id: str, alias_name: str):\n \"\"\"\n Manual rotation strategy using aliases.\n This is useful when you need to rotate more frequently\n than annually, or need to rotate asymmetric keys.\n \"\"\"\n # Create new key\n new_key = kms.create_key(\n Description=f'Rotated key - {datetime.now().isoformat()}',\n KeyUsage='ENCRYPT_DECRYPT',\n Origin='AWS_KMS',\n Tags=[\n {'TagKey': 'RotatedFrom', 'TagValue': old_key_id},\n {'TagKey': 'RotationDate', 'TagValue': datetime.now().isoformat()},\n ]\n )\n new_key_id = new_key['KeyMetadata']['KeyId']\n\n # Update alias to point to new key\n # All new encryptions will use the new key\n kms.update_alias(\n AliasName=alias_name,\n TargetKeyId=new_key_id\n )\n\n # OLD KEY REMAINS ACTIVE for decryption\n # This is the key insight: you don't re-encrypt existing data\n # KMS tracks which key version encrypted each data key\n print(f\"Alias {alias_name} now points to {new_key_id}\")\n print(f\"Old key {old_key_id} remains active for decryption\")\n\n # Schedule old key deletion after grace period (e.g., 90 days)\n # Only after you're certain no data encrypted with it remains\n # kms.schedule_key_deletion(KeyId=old_key_id, PendingWindowInDays=30)\n```"},{"heading":"The Complete Secrets Management Architecture","level":2,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#the-complete-secrets-management-architecture","content":"Here's the architecture I deployed for the client that started this whole journey:\n\n```\n┌─────────────────────────────────────────────────────┐\n│ Application Layer │\n│ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │\n│ │ Payment │ │ Auth │ │ Notification │ │\n│ │ Service │ │ Service │ │ Service │ │\n│ └────┬─────┘ └────┬─────┘ └────────┬─────────┘ │\n│ │ │ │ │\n│ ┌────▼──────────────▼──────────────────▼────────┐ │\n│ │ HashiCorp Vault │ │\n│ │ - Dynamic database credentials │ │\n│ │ - PKI certificate issuance │ │\n│ │ - Transit encryption (delegates to KMS) │ │\n│ │ - Lease-based secrets with auto-rotation │ │\n│ └────────────────────┬──────────────────────────┘ │\n│ │ │\n│ ┌────────────────────▼──────────────────────────┐ │\n│ │ AWS KMS │ │\n│ │ - Vault auto-unseal │ │\n│ │ - Envelope encryption for transit engine │ │\n│ │ - S3 SSE-KMS for audit logs │ │\n│ │ - RDS TDE master key │ │\n│ └────────────────────┬──────────────────────────┘ │\n│ │ │\n│ ┌────────────────────▼──────────────────────────┐ │\n│ │ AWS CloudHSM (FIPS 140-2 L3) │ │\n│ │ - Custom KMS key store │ │\n│ │ - PCI DSS key management │ │\n│ │ - Code signing keys │ │\n│ └───────────────────────────────────────────────┘ │\n└─────────────────────────────────────────────────────┘\n```\n\n```hcl\n# vault_config.hcl - Vault configuration with KMS auto-unseal\nstorage \"raft\" {\n path = \"/opt/vault/data\"\n node_id = \"vault-1\"\n\n retry_join {\n leader_api_addr = \"https://vault-2.internal:8200\"\n }\n retry_join {\n leader_api_addr = \"https://vault-3.internal:8200\"\n }\n}\n\nlistener \"tcp\" {\n address = \"0.0.0.0:8200\"\n tls_cert_file = \"/opt/vault/tls/vault.crt\"\n tls_key_file = \"/opt/vault/tls/vault.key\"\n}\n\n# Auto-unseal using AWS KMS\nseal \"awskms\" {\n region = \"us-east-1\"\n kms_key_id = \"mrk-abc123\"\n}\n\n# Audit logging to encrypted S3\naudit {\n type = \"file\"\n path = \"file\"\n options = {\n file_path = \"/var/log/vault/audit.log\"\n }\n}\n\ntelemetry {\n prometheus_retention_time = \"30s\"\n disable_hostname = true\n}\n```\n\nThe three-tier architecture is deliberate:\n\n1. **Vault** handles dynamic secrets (database credentials that rotate every hour), PKI, and application-facing secret delivery.\n2. **KMS** handles Vault's auto-unseal (so Vault operators don't need unseal keys) and provides the backing encryption for Vault's transit engine.\n3. **CloudHSM** backs the KMS custom key store for PCI-regulated encryption keys. The payment service's card data encryption key chain ultimately roots in hardware."},{"heading":"What I Got Wrong","level":2,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#what-i-got-wrong","content":"Two things bit me on this project:\n\nFirst, I underestimated the cost of CloudHSM. At roughly $1.50/hr per HSM (and you need at least two for HA), that's over $2,000/month just for the HSMs. For most workloads, standard KMS at $1/key/month plus $0.03/10,000 API calls is more than sufficient. CloudHSM is for when you genuinely need FIPS 140-2 Level 3 or single-tenant hardware.\n\nSecond, the PKCS#11 tooling ecosystem is painful. The libraries are old, the documentation assumes you're running Solaris, and debugging session errors requires reading hex dumps. If you can stay in the cloud KMS API layer, do it. Drop to PKCS#11 only when you must.\n\nThe ` .env ` file is gone. The database passwords rotate hourly. The audit trail shows exactly who accessed what key, when. And the next security review? Passed without findings. That felt good."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#questions","content":""},{"heading":"Do I need a dedicated HSM or is cloud KMS sufficient?","level":3,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#question-1","content":"Cloud KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS) is sufficient for 95% of workloads. You need a dedicated HSM (CloudHSM, Azure Dedicated HSM) only when regulatory requirements mandate single-tenant hardware, when you need custom cryptographic operations not supported by KMS, or when you need PKCS#11/JCE direct access."},{"heading":"What's the difference between a CMK and a data key?","level":3,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#question-2","content":"A Customer Master Key (CMK) lives inside the KMS/HSM and never leaves. A data key is generated by KMS, used to encrypt your actual data, and then the data key itself is encrypted by the CMK. This two-tier approach is envelope encryption."},{"heading":"How often should I rotate encryption keys?","level":3,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#question-3","content":"Rotate CMKs annually as a baseline. AWS KMS supports automatic annual rotation. Data keys can be rotated more frequently if needed. The important thing is that rotation is automatic and doesn't require re-encrypting existing data when using envelope encryption."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#article-tips","content":"1. **Tip:** Never export master keys from an HSM. If your architecture requires the master key to leave the HSM boundary, your architecture is wrong. Use ` envelope encryption ` instead.\n2. **Tip:** AWS KMS ` GenerateDataKey ` returns both plaintext and ciphertext versions of the data key. Use the plaintext to encrypt, immediately discard it from memory, store only the ciphertext alongside the encrypted data.\n3. **Tip:** Key rotation doesn't mean re-encrypting everything. With envelope encryption, you only rotate the KEK, existing data keys remain valid because they're encrypted with the old KEK version, which KMS still knows how to decrypt.\n4. **Tip:** Always set a ` DeletionWindowInDays ` of at least 14 days on KMS key deletion. Accidentally deleting a KMS key is permanent and catastrophic, those 14 days have saved more than one team I know.\n\n- hsm\n- kms\n- pkcs11\n- envelope-encryption\n- key-rotation\n- aws-cloudhsm\n- secrets-management\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/hardware-security-modules-hsm-cloud-kms"},{"text":"HashiCorp Vault","url":"https://www.vaultproject.io/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms"},"headline":"HSMs and Cloud KMS: PKCS#11, Envelope Encryption, and Real Secrets Management Architecture","description":"A deep technical walkthrough of hardware security modules, cloud KMS services, PKCS#11 interfaces, envelope encryption patterns, and building a secrets management architecture that actually survives audit.","datePublished":"2026-01-25","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://lucioduran.com/favicons/og-default.png","url":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Security","keywords":"hsm, kms, pkcs11, envelope-encryption, key-rotation, aws-cloudhsm, secrets-management","wordCount":1551,"timeRequired":"PT8M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/hardware-security-modules-hsm-cloud-kms"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"HSMs and Cloud KMS: PKCS#11, Envelope Encryption, and Real Secrets Management Architecture","item":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"Do I need a dedicated HSM or is cloud KMS sufficient?","acceptedAnswer":{"@type":"Answer","text":"Cloud KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS) is sufficient for 95% of workloads. You need a dedicated HSM (CloudHSM, Azure Dedicated HSM) only when regulatory requirements mandate single-tenant hardware, when you need custom cryptographic operations not supported by KMS, or when you need PKCS#11/JCE direct access."}},{"@type":"Question","name":"What's the difference between a CMK and a data key?","acceptedAnswer":{"@type":"Answer","text":"A Customer Master Key (CMK) lives inside the KMS/HSM and never leaves. A data key is generated by KMS, used to encrypt your actual data, and then the data key itself is encrypted by the CMK. This two-tier approach is envelope encryption."}},{"@type":"Question","name":"How often should I rotate encryption keys?","acceptedAnswer":{"@type":"Answer","text":"Rotate CMKs annually as a baseline. AWS KMS supports automatic annual rotation. Data keys can be rotated more frequently if needed. The important thing is that rotation is automatic and doesn't require re-encrypting existing data when using envelope encryption."}}]}]}],"contentHash":"fa874d2b8ddc2f30993f36a7dbb8e40001da815b0d886e6cc8149f8233fc95d9","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/hardware-security-modules-hsm-cloud-kms","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":65367,"contentBytes":14989}}