starred/proxmoxer

Fork 0

mirror of https://github.com/proxmoxer/proxmoxer.git synced 2026-04-24 22:56:01 +03:00

[GH-ISSUE #193] Unable to use cicustom with yaml input #109

New issue

Open

opened 2026-02-27 15:46:29 +03:00 by kerem · 3 comments

kerem commented

2026-02-27 15:46:29 +03:00

Owner

Originally created by @harshguptaserver on GitHub (Dec 14, 2024).
Original GitHub issue: https://github.com/proxmoxer/proxmoxer/issues/193

Hello proxmoxer team,

I am a long time user of your library in my homelab. I find it great and really love the simplicity.

aim : I am trying to create vm in proxmox using cloud-init from a .yml file ( https mode of communication to proxmox )

source: :ref: https://proxmoxer.github.io/docs/latest/examples/cloud-init/

suggests : 
proxmox.nodes("<node_name>").qemu(clone_vm_id).config.set(
    ....
    cicustom="<explicit YAML cloud-config>",
    ....
)

Here is my sample code:


proxmox = ProxmoxAPI(
            cluster_config['host'],
            user=cluster_config['username'],
            password=cluster_config['password'],
            verify_ssl=cluster_config.get('verify_ssl', False),
            timeout=30
        )

def prepare_cloud_init_config(vmid, cloud_config_dir):
    cloud_init_file = os.path.join(cloud_config_dir, f'vm-{vmid}.yml')

    try:
        with open(cloud_init_file, 'r') as file:
            cloud_init_config = yaml.safe_load(file)
        return cloud_init_config
    except FileNotFoundError:
        logging.warning(f"No cloud-init configuration found for VM {vmid}")
        return None
    except Exception as e:
        logging.error(f"Error reading cloud-init config: {e}")
        return None

cloud_init_config = prepare_cloud_init_config(vmid, cloud_config_dir)
cloud_init_str = "#cloud-config\n" + yaml.dump(cloud_init_config)

# below code gives me error
update_cloud_init_config = proxmox.nodes(node).qemu(vmid).config.set(
                cicustom= cloud_init_str
            ) 
# my requirement is to read the cloud-config from a yml file and upload to server/set it for vm somehow

Error:

2024-12-14 22:47:58,199 - ERROR: Failed to create or configure VM: 400 Bad Request: Parameter verification failed. - {'cicustom': 'invalid format - value without key, but schema does not define a default key\n'}

desired solution : read the yml file and create cloud-init of the vm ( the yml file has multiple keys and config )

Note: the yml file works fine if i use shell command by copying yml manually :

this_ci_custom = "qm set {0} --cicustom \"{1}\"".format(vm_id, vm_ci_location)

Sample yml file

#cloud-config
hostname: test-faas-01
manage_etc_hosts: true
user: pi
chpasswd:
  expire: False
users:
  - default

ssh_pwauth: true

package_upgrade: true
package_update: true


packages:
 - wget
 - nload
 - htop
 - curl
 - net-tools
 - python3-dev
 - python3-pip
 - qemu-guest-agent

apt:
  primary:
    - arches: [default]
      uri: http://us.archive.ubuntu.com/ubuntu/



runcmd:
 - date >> /etc/birthday_vm
 - systemctl enable --now qemu-guest-agent
final_message: "The system is finally up, after $UPTIME seconds"
power_state:
  delay: now
  mode: reboot
  message: Bye Bye
  timeout: 30
  condition: True
ssh_authorized_keys:
  - ssh-rsa AAAAB......8V harsh@gd03

manage_resolv_conf: true
resolv_conf:
  nameservers: [10.0.0.20, 10.0.0.1]
locale: un_US

Originally created by @harshguptaserver on GitHub (Dec 14, 2024). Original GitHub issue: https://github.com/proxmoxer/proxmoxer/issues/193 Hello proxmoxer team, I am a long time user of your library in my homelab. I find it great and really love the simplicity. >> ##### aim : I am trying to create vm in proxmox using cloud-init from a .yml file ( https mode of communication to proxmox ) ##### source: :ref: https://proxmoxer.github.io/docs/latest/examples/cloud-init/ ``` suggests : proxmox.nodes("<node_name>").qemu(clone_vm_id).config.set( .... cicustom="<explicit YAML cloud-config>", .... ) ``` Here is my sample code: ``` proxmox = ProxmoxAPI( cluster_config['host'], user=cluster_config['username'], password=cluster_config['password'], verify_ssl=cluster_config.get('verify_ssl', False), timeout=30 ) def prepare_cloud_init_config(vmid, cloud_config_dir): cloud_init_file = os.path.join(cloud_config_dir, f'vm-{vmid}.yml') try: with open(cloud_init_file, 'r') as file: cloud_init_config = yaml.safe_load(file) return cloud_init_config except FileNotFoundError: logging.warning(f"No cloud-init configuration found for VM {vmid}") return None except Exception as e: logging.error(f"Error reading cloud-init config: {e}") return None cloud_init_config = prepare_cloud_init_config(vmid, cloud_config_dir) cloud_init_str = "#cloud-config\n" + yaml.dump(cloud_init_config) # below code gives me error update_cloud_init_config = proxmox.nodes(node).qemu(vmid).config.set( cicustom= cloud_init_str ) # my requirement is to read the cloud-config from a yml file and upload to server/set it for vm somehow ``` Error: ``` 2024-12-14 22:47:58,199 - ERROR: Failed to create or configure VM: 400 Bad Request: Parameter verification failed. - {'cicustom': 'invalid format - value without key, but schema does not define a default key\n'} ``` ##### desired solution : read the yml file and create cloud-init of the vm ( the yml file has multiple keys and config ) Note: the yml file works fine if i use shell command by copying yml manually : ``` this_ci_custom = "qm set {0} --cicustom \"{1}\"".format(vm_id, vm_ci_location) ``` Sample yml file ``` #cloud-config hostname: test-faas-01 manage_etc_hosts: true user: pi chpasswd: expire: False users: - default ssh_pwauth: true package_upgrade: true package_update: true packages: - wget - nload - htop - curl - net-tools - python3-dev - python3-pip - qemu-guest-agent apt: primary: - arches: [default] uri: http://us.archive.ubuntu.com/ubuntu/ runcmd: - date >> /etc/birthday_vm - systemctl enable --now qemu-guest-agent final_message: "The system is finally up, after $UPTIME seconds" power_state: delay: now mode: reboot message: Bye Bye timeout: 30 condition: True ssh_authorized_keys: - ssh-rsa AAAAB......8V harsh@gd03 manage_resolv_conf: true resolv_conf: nameservers: [10.0.0.20, 10.0.0.1] locale: un_US ```

kerem commented

2026-02-27 15:46:30 +03:00

Author

Owner

@morph027 commented on GitHub (Dec 14, 2024):

Unfortunately, you can't pass the YAML content, it must be a file located on the proxmox node itself. So you need to transfer the YAML file from the client to a server location accessible for PVE and then the VM can use it.

There's a pending bug report to add an endpoint for the snippets storage type, which would be very feasible for this task: https://bugzilla.proxmox.com/show_bug.cgi?id=2208

As a workaround, I'm using a third party server for uploading the files to a location also being used as snippet store (Example).

You can find the quite complex project here: Source + Docs

@morph027 commented on GitHub (Dec 14, 2024): Unfortunately, you can't pass the YAML content, it must be a file located on the proxmox node itself. So you need to transfer the YAML file from the client to a server location accessible for PVE and then the VM can use it. There's a pending bug report to add an endpoint for the snippets storage type, which would be very feasible for this task: https://bugzilla.proxmox.com/show_bug.cgi?id=2208 As a workaround, I'm using a third party server for uploading the files to a location also being used as snippet store ([Example](https://gitlab.com/morph027/pve-cloud-init-creator/-/blob/master/pve_cloud_init_creator.py?ref_type=heads#L244-L275)). You can find the quite complex project here: [Source](https://gitlab.com/morph027/pve-cloud-init-creator/) + [Docs](https://morph027.gitlab.io/pve-cloud-init-creator/)

kerem commented

2026-02-27 15:46:30 +03:00

Author

Owner

@harshguptaserver commented on GitHub (Dec 15, 2024):

Thank you for the quick reply @morph027 . I checked the code shared by you.
For a quick workaround, I have decided to SSH to the node and upload the yml file.

For anyone who stumbles here see my code below ( this copies the yml based in str format to the snippets for 'local' storage), don't forget to set the cicustom afterwards:


def upload_cloud_init_to_proxmox(
        hostname,
        username,
        password,
        cloud_init_content,
        vm_id,
        remote_path='/var/lib/vz/snippets',
        port=22
):
    """
    Upload cloud-init content to Proxmox server and set permissions.

    :param hostname: Proxmox server hostname or IP address
    :param username: SSH username
    :param password: SSH password
    :param cloud_init_content: String containing cloud-init configuration
    :param vm_id: VM ID to be used in the snippet filename
    :param remote_path: Destination directory on the server (default: /var/lib/vz/snippets)
    :param port: SSH port (default: 22)
    :return: Full path of the uploaded snippet on the server
    """
    # Validate content is not empty
    if not cloud_init_content or not cloud_init_content.strip():
        raise ValueError("Cloud-init content cannot be empty")

    # Construct remote filename
    remote_filename = f'user-{vm_id}.yml'
    full_remote_path = f'{remote_path}/{remote_filename}'

    try:
        print(f"Going to create ssh connection to {username}@{hostname}:{port}")
        # Establish SSH connection
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect(hostname, port=port, username=username, password=password)

        # Create SFTP client
        sftp = ssh.open_sftp()

        try:
            # Upload file-like object directly
            with sftp.file(full_remote_path, 'w') as remote_file:
                remote_file.write(cloud_init_content)
        finally:
            # Always close SFTP session
            sftp.close()

        # Set permissions and ownership
        # Uses chmod to set 0644 permissions (rw-r--r--)
        # Uses chown to set owner and group to root
        ssh.exec_command(f'chmod 0644 {full_remote_path}')
        ssh.exec_command(f'chown root:root {full_remote_path}')

        print(f"Successfully uploaded cloud-init file to {full_remote_path}")
        return full_remote_path

    except Exception as e:
        print(f"Error uploading cloud-init file: {e}")
        raise
    finally:
        # Ensure SSH connection is closed
        if 'ssh' in locals():
            ssh.close()

resp_copy = upload_cloud_init_to_proxmox(
                hostname=cluster_config['host'],
                username=cluster_config['ssh_username'],
                password=cluster_config['password'],
                cloud_init_content=cloud_init_str,
                vm_id=vmid,
            )
print("Going to set cloud-init config")
proxmox.nodes(node).qemu(vmid).config.set(
    cicustom='user=local:snippets/user-{}.yml'.format(vmid)
)

@harshguptaserver commented on GitHub (Dec 15, 2024): Thank you for the quick reply @morph027 . I checked the code shared by you. For a quick workaround, I have decided to SSH to the node and upload the yml file. For anyone who stumbles here see my code below ( this copies the yml based in str format to the snippets for 'local' storage), don't forget to set the cicustom afterwards: ``` def upload_cloud_init_to_proxmox( hostname, username, password, cloud_init_content, vm_id, remote_path='/var/lib/vz/snippets', port=22 ): """ Upload cloud-init content to Proxmox server and set permissions. :param hostname: Proxmox server hostname or IP address :param username: SSH username :param password: SSH password :param cloud_init_content: String containing cloud-init configuration :param vm_id: VM ID to be used in the snippet filename :param remote_path: Destination directory on the server (default: /var/lib/vz/snippets) :param port: SSH port (default: 22) :return: Full path of the uploaded snippet on the server """ # Validate content is not empty if not cloud_init_content or not cloud_init_content.strip(): raise ValueError("Cloud-init content cannot be empty") # Construct remote filename remote_filename = f'user-{vm_id}.yml' full_remote_path = f'{remote_path}/{remote_filename}' try: print(f"Going to create ssh connection to {username}@{hostname}:{port}") # Establish SSH connection ssh = paramiko.SSHClient() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ssh.connect(hostname, port=port, username=username, password=password) # Create SFTP client sftp = ssh.open_sftp() try: # Upload file-like object directly with sftp.file(full_remote_path, 'w') as remote_file: remote_file.write(cloud_init_content) finally: # Always close SFTP session sftp.close() # Set permissions and ownership # Uses chmod to set 0644 permissions (rw-r--r--) # Uses chown to set owner and group to root ssh.exec_command(f'chmod 0644 {full_remote_path}') ssh.exec_command(f'chown root:root {full_remote_path}') print(f"Successfully uploaded cloud-init file to {full_remote_path}") return full_remote_path except Exception as e: print(f"Error uploading cloud-init file: {e}") raise finally: # Ensure SSH connection is closed if 'ssh' in locals(): ssh.close() resp_copy = upload_cloud_init_to_proxmox( hostname=cluster_config['host'], username=cluster_config['ssh_username'], password=cluster_config['password'], cloud_init_content=cloud_init_str, vm_id=vmid, ) print("Going to set cloud-init config") proxmox.nodes(node).qemu(vmid).config.set( cicustom='user=local:snippets/user-{}.yml'.format(vmid) ) ```

kerem commented

2026-02-27 15:46:30 +03:00

Author

Owner

@morph027 commented on GitHub (Dec 15, 2024):

Nice one 👍 SSH is such a obvious way of uploading that i just missed it somehow. Probably going to update my project to using SSH as it's much less hassle setting up 😆

@morph027 commented on GitHub (Dec 15, 2024): Nice one :+1: SSH is such a obvious way of uploading that i just missed it somehow. Probably going to update my project to using SSH as it's much less hassle setting up :laughing:

kerem referenced this issue

2026-02-27 15:46:47 +03:00

[PR #109] [MERGED] Allow specifying resource_id as 0 for firewall rule compatibility #178

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/proxmoxer#109

No description provided.

Rows
Columns