mirror of
https://github.com/proxmoxer/proxmoxer.git
synced 2026-04-25 07:06:00 +03:00
[GH-ISSUE #116] Files upload not working #60
Labels
No labels
backend:https
backend:local
backend:openssh
backend:ssh_paramiko
pull-request
status:ansible-issue
status:help-wanted
status:info-needed
status:proxmox-issue
status:review-needed
type:bug 🐞
type:dependency ⛓️
type:docs 📝
type:enhancement ⏫
type:maintenance 🛠️
type:meta
type:question ❓
type:request ✋
type:testing 🧪
version:1.x
version:latest
version:py2
version:py3
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/proxmoxer#60
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @cclecle on GitHub (Oct 31, 2022).
Original GitHub issue: https://github.com/proxmoxer/proxmoxer/issues/116
Originally assigned to: @jhollowe on GitHub.
Not sure since when but I can not get files uploading to work using proxmoxer.
See my thread on proxmox forum:
Proxmox Forum
Copy of the thread:
Hi,
Using proxmoxer I am trying to upload lxc template to my hypervisor.
It used to work just fine but since a few weeks (or month) it fails.
On proxmoxer side it say:
Remote end closed connection without response
Then I checked on hypervisor side:
In /var/log/syslog :
Oct 30 22:01:55 hypervisor pveproxy[2126784]: problem with client ::ffff:XXX.XXX.XXX.XXX; No space left on device
In /var/log/pveproxy/ :
::ffff:XXX.XXX.XXX.XXX- dabfactory-bot@pam!XXXXXXXXXXXX[30/10/2022:22:04:34 +0100] "POST /api2/json/nodes/hypervisor/storage/WORK/upload HTTP/1.1" 500 -
There is plenty of space:
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
udev 16G 0 16G 0% /dev
tmpfs 3,2G 13M 3,2G 1% /run
/dev/mapper/pve-root 28G 13G 14G 50% /
tmpfs 16G 49M 16G 1% /dev/shm
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
/dev/mapper/pve-data 65G 17G 49G 26% /var/lib/vz
/dev/sdc1 4,6T 4,2T 374G 92% /mnt/BRICK2
/dev/sdb1 4,6T 4,3T 345G 93% /mnt/BRICK1
/dev/fuse 128M 24K 128M 1% /etc/pve
tmpfs 3,2G 32K 3,2G 1% /run/user/0
localhost:TMP_FAST 100G 0 100G 0% /media/TMP_FAST
localhost:ARRAY 11T 11T 752G 94% /media/ARRAY
localhost:WORK 3,7T 3,4T 252G 94% /media/WORK
What I have try:
{'shared': 0, 'total': 3998318010368, 'active': 1, 'type': 'dir', 'content': 'vztmpl,rootdir', 'used': 3727743836160, 'enabled': 1, 'avail': 270574174208}
Other info:
proxmox VE version: 7.2-11
proxmoxer version: 1.3.1
EDIT 1:
test using pveproxy start --debug=1 :
worker[2323338]: PVE::APIServer::AnyEvent +1714: (eval): ACCEPT FH10 CONN0
worker[2323338]: PVE::APIServer::AnyEvent +1325: (eval): start upload /api2/json/nodes/hypervisor/storage/local/upload multipart/form-data e4e26ace364a4f9488cf9a9f06dbc860
worker[2323338]: PVE::APIServer::AnyEvent +1764: client_do_disconnect: close connection AnyEvent::Handle=HASH(0x559afd76bfd8)
worker[2323338]: PVE::APIServer::AnyEvent +1764: client_do_disconnect: CLOSE FH10 CONN0
Compared to through webGUI:
(...)
worker[2326306]: PVE::APIServer::AnyEvent +1325: (eval): start upload /api2/json/nodes/hypervisor/storage/local/upload multipart/form-data ---------------------------221127871937183401032155461004
(...)
EDIT2:
Seems to work using low-level API (requests + requests_toolbelt)....
So not a Proxmox issue, will make a ticket on proxmoxer-side.. !
Working sample:
@jhollowe commented on GitHub (Oct 31, 2022):
What version of PVE, python, and proxmoxer are you using? What file are you trying to upload?
@cclecle commented on GitHub (Oct 31, 2022):
PVE 7.2-11
Python 3.9.2 (Linux) + Python 3.9.5 (windows)
Proxmoxer 1.3.1
Trying to upload a .tar.gz lxc appliance template
Was able to fix it modifying proxmoxer:
github.com/cclecle/proxmoxer@07924b7edfWould not lie, I found this fix by comparing with working solution, not sure why 'text/plain' content type is needed now (and was not). But it is doing the trick !
Tested and works with both user/password and token auth style.
EDIT:
The working code (with 'text/plain') comes from the requests-toolbelt reference code :
https://toolbelt.readthedocs.io/en/latest/user.html
@jhollowe commented on GitHub (Nov 1, 2022):
What is the python code you are running when it fails. It does seem strange that the type needs to be "text/plain"
@cclecle commented on GitHub (Nov 11, 2022):
Sorry for delay.
I already sent you template but I reworked it, just change values according your need.
Reproduce defect in both plain-auth and api token.
This script tested only on windows, but should also work on Linux..
And, my proxmoxer based code was:
@jhollowe commented on GitHub (Nov 11, 2022):
is
self.nodean instance ofself.proxmox.nodes("your_node_name")?I'll try to reproduce the issue using your promoxer example code
@cclecle commented on GitHub (Nov 11, 2022):
Yes excatly, it is !
@jhollowe commented on GitHub (Nov 12, 2022):
What version of proxmoxer was this working for you previously?
And can you give me some info on
self.CfgMng.template_storage? What type of storage is is? where are the bytes stored (on the PVE node or somewhere over the network)?Can you try again and paste in the status and log from the Task?
@cclecle commented on GitHub (Nov 12, 2022):
Unfortunatly I can not remember the context when it was working ...
Not even sure it is actually caused by a proxmoxer update, it can also be a dependency or a proxmox update that break it :-/
self.CfgMng.template_storageis just a string containing storage name.Problem occured both using Directory storage or LVM (thin), both are on the local proxmox node.
Don't forget this line:
STREAMING_SIZE_THRESHOLD = 100 * 1024 * 1024 # 10 MiBYou must upload a file > 100MiB (another fix in comment :) ) to raise the issue using proxmoxer.
My file was 1GB+
@jhollowe commented on GitHub (Nov 12, 2022):
Hrm, I reproduced with the same proxmoxer (1.3.1) and python (3.7.13) against
7.2-7(d0dd0e85) and7.2-11(b76d3178) using the alpine 3.16 template.The issue is only present on my
7.2-11node. If you are able, please try on a previous version of PVE. Also if you are able, try using one of the ssh backends to try the upload.I'm going to keep looking at this and try to figure if we can make a change to proxmoxer to support
7.2-11or if this is a PVE bug that should be fixed on the Proxmox side of things.@jhollowe commented on GitHub (Nov 12, 2022):
I've done a packet capture against 7.1-7 (the latest ISO I had on hand) and 7.2-11 (same PVE node updated to latest). From what I can tell there is nothing wrong on the client side (the upload is the exact same request with different auth and multipart boundary). The request goes through fine and then hits the
client_do_disconnect: close connection AnyEvent::Handlein the pveproxy which then sends an RST to the client.It looks like this is an issue with PVE, so I'm going to label it as such but leave this issue open until it gets resolved.
captures.zip
@cclecle commented on GitHub (Nov 12, 2022):
I do not have proxmox test environnent sorry :(.
For the moment I can live with my patched version...
I made a quick check on proxmox web UI, the encoding it also set !
See:

Value is not the same but there is one... !
Maybe we can just consider this is the way to go ?
And fix my patch to use application/x-gzip like PVE UI is doing... ?
@jhollowe commented on GitHub (Nov 13, 2022):
It looks like a pve-http-server commit from May (inadvertently?) requires a Content-Type for the file's multipart part but it never uses the value.
I've raised bug 4344 to remove this requirement and have submitted a fix to the mailing list.
@cclecle commented on GitHub (Nov 15, 2022):
Right, but if you look the full situation:
On the other hand, adding content-type to proxmoxer will solve the current issue and will not cause problem with older proxmox...
So for proxmoxer users I guess a proxmoxer-side fix would be prefered, even if I agree with the fact that the initial break comes from proxmox side..
@jhollowe commented on GitHub (Nov 22, 2022):
Since PVE has regular updates and there is not a big reason to not update to the point releases, I'm hesitant to make a proxmoxer code change to add extra stuff which just increases the likelihood of breakage in the future. The error was only in a few point (or dash?) releases and so I don't see a reason to add a change to proxmoxer which is only relevant for a few versions which will be upgraded from in a few weeks/months.
@cclecle commented on GitHub (Nov 22, 2022):
Checked RFC: RFC 2388
=>§3 says: As with all multipart MIME types, each part has an optional "Content-Type", which defaults to text/plain
So you are right, it is a regression as the standard do not state requirement for this field..
@elelayan commented on GitHub (Mar 22, 2023):
In case one is stuck at a specific PVE version,
patch https://github.com/proxmoxer/proxmoxer/blob/develop/proxmoxer/backends/https.py#L204