[GH-ISSUE #139] Two extra bytes in ISO file when uploading via API / Proxmoxer

kerem commented

2026-02-27 15:46:17 +03:00

Owner

Originally created by @rudolfbyker on GitHub (Apr 5, 2023).
Original GitHub issue: https://github.com/proxmoxer/proxmoxer/issues/139

Originally assigned to: @jhollowe on GitHub.

When I upload the same ISO file manually through the GUI and through the code below, I get two slightly different results:

Manual upload: The ISO works.
Upload using code below: The file has two extra leading bytes, specifically 0x0d0a (\r\n), and it's seen as a corrupt ISO file.

My uploading code:

def upload_iso(
    *,
    prox: ProxmoxAPI,
    node_name: str,
    storage_name: str,
    iso_path: Path,
) -> Any:
    with iso_path.open("rb") as f:
        return prox(f"/nodes/{node_name}/storage/{storage_name}/upload").post(
            filename=f,
            content="iso",
        )

Hex dump of the first kilobyte:

root@hv1:/var/lib/vz/template/iso# hexyl -n 1024 foo.iso
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 0d 0a 00 00 00 00 00 00 ┊ 00 00 00 00 00 00 00 00 │__000000┊00000000│
│00000010│ 00 00 00 00 00 00 00 00 ┊ 00 00 00 00 00 00 00 00 │00000000┊00000000│
│*       │                         ┊                         │        ┊        │
│00000400│                         ┊                         │        ┊        │
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘
root@hv1:/var/lib/vz/template/iso# hexyl -n 1024 foo-manual.iso
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 00 00 00 00 00 00 00 00 ┊ 00 00 00 00 00 00 00 00 │00000000┊00000000│
│*       │                         ┊                         │        ┊        │
│00000400│                         ┊                         │        ┊        │
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘

Note the two byte difference:

root@hv1:/var/lib/vz/template/iso# ls -l foo*.iso
-rw-r--r-- 1 root root 550914 Apr  5 10:56 foo.iso
-rw-r--r-- 1 root root 550912 Apr  5 10:59 foo-manual.iso

The one on my local machine is also 550912 bytes, as expected.

Originally created by @rudolfbyker on GitHub (Apr 5, 2023). Original GitHub issue: https://github.com/proxmoxer/proxmoxer/issues/139 Originally assigned to: @jhollowe on GitHub. When I upload the same ISO file manually through the GUI and through the code below, I get two slightly different results: - Manual upload: The ISO works. - Upload using code below: The file has two extra leading bytes, specifically `0x0d0a` (`\r\n`), and it's seen as a corrupt ISO file. My uploading code: ``` def upload_iso( *, prox: ProxmoxAPI, node_name: str, storage_name: str, iso_path: Path, ) -> Any: with iso_path.open("rb") as f: return prox(f"/nodes/{node_name}/storage/{storage_name}/upload").post( filename=f, content="iso", ) ``` Hex dump of the first kilobyte: ``` root@hv1:/var/lib/vz/template/iso# hexyl -n 1024 foo.iso ┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐ │00000000│ 0d 0a 00 00 00 00 00 00 ┊ 00 00 00 00 00 00 00 00 │__000000┊00000000│ │00000010│ 00 00 00 00 00 00 00 00 ┊ 00 00 00 00 00 00 00 00 │00000000┊00000000│ │* │ ┊ │ ┊ │ │00000400│ ┊ │ ┊ │ └────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘ root@hv1:/var/lib/vz/template/iso# hexyl -n 1024 foo-manual.iso ┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐ │00000000│ 00 00 00 00 00 00 00 00 ┊ 00 00 00 00 00 00 00 00 │00000000┊00000000│ │* │ ┊ │ ┊ │ │00000400│ ┊ │ ┊ │ └────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘ ``` Note the two byte difference: ``` root@hv1:/var/lib/vz/template/iso# ls -l foo*.iso -rw-r--r-- 1 root root 550914 Apr 5 10:56 foo.iso -rw-r--r-- 1 root root 550912 Apr 5 10:59 foo-manual.iso ``` The one on my local machine is also 550912 bytes, as expected.

kerem

2026-02-27 15:46:17 +03:00

closed this issue
added the
type:bug 🐞

backend:https
labels

kerem commented

2026-02-27 15:46:18 +03:00

Author

Owner

@rudolfbyker commented on GitHub (Apr 6, 2023):

A few comments after I did some debugging:

I followed the instructions at https://proxmoxer.github.io/docs/1.2/examples/files/#uploading-files-to-pve to write the code above. I find it very strange that a file object should be passed to an argument called filename. I also find it very strange that the API docs https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/storage/{storage}/upload have filename but nothing like file or data. Maybe these two peculiarities are related?
With my code sample, the Request object is being created with files={'filename': ('foo.iso', <_io.BufferedReader name='foo.iso'>)} at github.com/proxmoxer/proxmoxer@cdcb40da0d/proxmoxer/backends/https.py (L229) . Is this correct? The API docs of the request package seem to indicate that this should look like files={'foo.iso': <_io.BufferedReader name='foo.iso'>} instead.
I found this smoking gun:
I would normally expect \r\n to be a Windows-only thing, but I'm getting the same results when running this in Windows and in a python-alpine Docker container on a Linux host. I have heard that the HTTP spec requires \r\n but have not confirmed this for myself yet. I usually don't bother with low-level stuff like that.

@rudolfbyker commented on GitHub (Apr 6, 2023): A few comments after I did some debugging: 1. I followed the instructions at https://proxmoxer.github.io/docs/1.2/examples/files/#uploading-files-to-pve to write the code above. I find it very strange that a file object should be passed to an argument called `filename`. I also find it very strange that the API docs https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/storage/{storage}/upload have `filename` but nothing like `file` or `data`. Maybe these two peculiarities are related? 2. With my code sample, the [Request object](https://requests.readthedocs.io/en/latest/api/#requests.Request) is being created with `files={'filename': ('foo.iso', <_io.BufferedReader name='foo.iso'>)}` at https://github.com/proxmoxer/proxmoxer/blob/cdcb40da0d3cd58e2f0588a6eb10278379f7dd34/proxmoxer/backends/https.py#L229 . Is this correct? The API docs of the `request` package seem to indicate that this should look like `files={'foo.iso': <_io.BufferedReader name='foo.iso'>}` instead. 3. I found this smoking gun: ![image](https://user-images.githubusercontent.com/10025342/230318152-bd1ef255-a3e0-4b47-9162-506fe9b86b05.png) I would normally expect `\r\n` to be a Windows-only thing, but I'm getting the same results when running this in Windows and in a python-alpine Docker container on a Linux host. I have heard that the HTTP spec requires `\r\n` but have not confirmed this for myself yet. I usually don't bother with low-level stuff like that.

kerem commented

2026-02-27 15:46:18 +03:00

Author

Owner

@rudolfbyker commented on GitHub (Apr 6, 2023):

I'm having the same issue when uploading from the command line. So maybe this isn't an issue with proxmoxer after all.

https --verify=no --form POST 192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload 'Authorization:PVEAPIToken=user@pve!mytoken=secret' content=iso filename@foo.iso

I would love to hear if anyone else has had this issue, but I'm guessing that the appropriate place for discussion is at https://forum.proxmox.com/threads/two-extra-bytes-in-iso-file-when-uploading-via-api-proxmoxer.125411 rather than here.

@rudolfbyker commented on GitHub (Apr 6, 2023): I'm having the same issue when uploading from the command line. So maybe this isn't an issue with `proxmoxer` after all. ```shell https --verify=no --form POST 192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload 'Authorization:PVEAPIToken=user@pve!mytoken=secret' content=iso filename@foo.iso ``` I would love to hear if anyone else has had this issue, but I'm guessing that the appropriate place for discussion is at https://forum.proxmox.com/threads/two-extra-bytes-in-iso-file-when-uploading-via-api-proxmoxer.125411 rather than here.

kerem commented

2026-02-27 15:46:18 +03:00

Author

Owner

@jhollowe commented on GitHub (Apr 6, 2023):

What version of PVE, proxmoxer, and Python are you using?

There have been some issues with file upload that I recently found and got merged into PVE's API server. I would suggest updating to the latest PVE and installing the develop branch of proxmoxer and try using the proxmoxer.tools.files's .

from proxmoxer.tools import Files
f = Files(your_proxmoxapi_object, "node_name", "storage_name")
f.upload_local_file_to_storage("/path/to/file.iso")

This will calculate a checksum from the local file and then get PVE to validate that.

@jhollowe commented on GitHub (Apr 6, 2023): What version of PVE, proxmoxer, and Python are you using? There have been some issues with file upload that I recently found and got merged into PVE's API server. I would suggest updating to the latest PVE and [installing the `develop` branch of proxmoxer](https://proxmoxer.github.io/docs/2.0/development/#installing-from-source) and try using the `proxmoxer.tools.files`'s . ``` python from proxmoxer.tools import Files f = Files(your_proxmoxapi_object, "node_name", "storage_name") f.upload_local_file_to_storage("/path/to/file.iso") ``` This will calculate a checksum from the local file and then get PVE to validate that.

kerem commented

2026-02-27 15:46:18 +03:00

Author

Owner

@jhollowe commented on GitHub (Apr 6, 2023):

I would imagine that the issue is related to the Content-Type on the file's multipart. The develop branch now adds this to its uploads as it seems PVE isn't following RFC 2388 which specifies the Content-Type as optional.

See #116 for previous errors concerning this regression.

@jhollowe commented on GitHub (Apr 6, 2023): I would imagine that the issue is related to the `Content-Type` on the file's multipart. The `develop` branch now adds this to its uploads as it seems PVE isn't following [RFC 2388](https://www.ietf.org/rfc/rfc2388.txt) which specifies the `Content-Type` as optional. See #116 for previous errors concerning this regression.

kerem commented

2026-02-27 15:46:18 +03:00

Author

Owner

@rudolfbyker commented on GitHub (Apr 6, 2023):

My versions:

proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.3-3
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.6.4
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-1
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.11-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

I'll try the latest proxmoxer in a moment…

@rudolfbyker commented on GitHub (Apr 6, 2023): My versions: ``` proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve) pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a) pve-kernel-5.15: 7.3-3 pve-kernel-5.15.102-1-pve: 5.15.102-1 ceph-fuse: 15.2.17-pve1 corosync: 3.1.7-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.24-pve2 libproxmox-acme-perl: 1.4.4 libproxmox-backup-qemu0: 1.3.1-1 libproxmox-rs-perl: 0.2.1 libpve-access-control: 7.4-2 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.3-4 libpve-guest-common-perl: 4.2-4 libpve-http-server-perl: 4.2-1 libpve-rs-perl: 0.7.5 libpve-storage-perl: 7.4-2 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 5.0.2-2 lxcfs: 5.0.3-pve1 novnc-pve: 1.4.0-1 proxmox-backup-client: 2.3.3-1 proxmox-backup-file-restore: 2.3.3-1 proxmox-kernel-helper: 7.4-1 proxmox-mail-forward: 0.1.1-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.6.4 pve-cluster: 7.3-3 pve-container: 4.4-3 pve-docs: 7.4-2 pve-edk2-firmware: 3.20230228-1 pve-firewall: 4.3-1 pve-firmware: 3.6-4 pve-ha-manager: 3.6.0 pve-i18n: 2.11-1 pve-qemu-kvm: 7.2.0-8 pve-xtermjs: 4.16.0-1 qemu-server: 7.4-3 smartmontools: 7.2-pve3 spiceterm: 3.2-2 swtpm: 0.8.0~bpo11+3 vncterm: 1.7-1 zfsutils-linux: 2.1.9-pve1 ``` I'll try the latest proxmoxer in a moment…

kerem commented

2026-02-27 15:46:18 +03:00

Author

Owner

@rudolfbyker commented on GitHub (Apr 17, 2023):

This has been resolved in Proxmox itself: Just use apt to update libpve-http-server-perl to anything >= 4.2-2. Details: https://forum.proxmox.com/threads/two-extra-bytes-in-iso-file-when-uploading-via-api-proxmoxer.125411/post-548156

@rudolfbyker commented on GitHub (Apr 17, 2023): This has been resolved in Proxmox itself: Just use `apt` to update `libpve-http-server-perl` to anything `>= 4.2-2`. Details: https://forum.proxmox.com/threads/two-extra-bytes-in-iso-file-when-uploading-via-api-proxmoxer.125411/post-548156

Rows
Columns

[GH-ISSUE #139] Two extra bytes in ISO file when uploading via API / Proxmoxer #73