mirror of
https://github.com/GameServerManagers/LinuxGSM.git
synced 2026-04-25 22:25:59 +03:00
[GH-ISSUE #2597] Older versions of curl are unable to download large files using newer SSL/HTTP technologies #1887
Labels
No labels
Atomic
Epic
cannot reproduce
command: backup
command: console
command: debug
command: details
command: fast-dl
command: install
command: mods
command: monitor
command: post-details
command: restart
command: send
command: start
command: stop
command: update
command: update-lgsm
command: validate
command: wipe
distro: AlmaLinux
distro: Arch Linux
distro: CentOS
distro: Debian
distro: Fedora
distro: RedHat
distro: Rocky Linux
distro: Ubuntu
distro: openSUSE
engine: goldsrc
engine: source
game: 7 Days to Die
game: ARMA 3
game: Ark: Survival Evolved
game: Assetto Corsa
game: Avorion
game: BATTALION: Legacy
game: Barotrauma
game: Battalion 1944
game: Battlefield 1942
game: Black Mesa: Deathmatch
game: Blade Symphony
game: Call of Duty 2
game: Call of Duty 4
game: Call of Duty: United Offensive
game: Counter-Strike 1.6
game: Counter-Strike 2
game: Counter-Strike: Global Offensive
game: Counter-Strike: Source
game: Day of Infamy
game: Dayz
game: Death Match Classic
game: Don't Starve Together
game: ET: Legacy
game: Eco
game: Factorio
game: Factorio
game: Garry's Mod
game: Half-Life
game: Hurtword
game: Insurgecy
game: Insurgecy
game: Insurgency: Sandstorm
game: Just Cause 3
game: Killing Floor
game: Killing Floor 2
game: Left 4 Dead 2
game: Minecraft
game: Minecraft Bedrock
game: Mordhau
game: Multi Theft Auto
game: Mumble
game: Natural Selection 2
game: No More Room in Hell
game: Pavlov VR
game: Post Scriptum
game: Project Zomboid
game: Quake 3
game: QuakeWorld
game: Red Orchestra: Ostfront 41-45
game: Return to Castle Wolfenstein
game: Rising World
game: Rust
game: San Andreas Multiplayer
game: Satisfactory
game: Soldat
game: Soldier of Fortune 2
game: Squad
game: Squad 44
game: Starbound
game: Stationeers
game: Sven Co-op
game: Team Fortress 2
game: Teamspeak 3
game: Teeworlds
game: Terraria
game: The Front
game: Unreal Tournament 2004
game: Unreal Tournament 3
game: Unreal Tournament 99
game: Unturned
game: Valheim
game: Wurm Unlimited
game: Zombie Master Reborn
game: label missing
good first issue
help wanted
info: alerts
info: dependency
info: docker
info: docs
info: email
info: query
info: steamcmd
info: systemd
info: tmux
info: website
info: website
needs more info
outcome: duplicate
outcome: issue resolved
outcome: issue resolved
outcome: issue unresolved
outcome: pr accepted
outcome: pr rejected
outcome: unconfirmed
outcome: wontfix
outcome: wrong forum
potential-duplicate
priority
pull-request
type: bug
type: feature
type: feature
type: feature request
type: game server request
type: refactor
waiting response
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/LinuxGSM#1887
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @lorentedford on GitHub (Nov 17, 2019).
Original GitHub issue: https://github.com/GameServerManagers/LinuxGSM/issues/2597
User Story
As a [user description], I want [desired action] so that [desired outcome].
https://linuxgsm.com/lgsm/cod4server/
Basic info
Further Information
A clear description of what the bug is and any ideas on how to resolve it. Plus any further context that might be relevant to the issue.
To Reproduce
add user follow directions on website and type ./cod4server install
This is where the error occurs.
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
downloading cod4x18_1772_dedrun.tar.bz2...OK
verifying cod4x18_1772_dedrun.tar.bz2 with MD5...FAIL
cod4x18_1772_dedrun.tar.bz2 returned MD5 checksum: 809b16e34396d6ef588d28748962dc3f
expected MD5 checksum: de29f29d79f9cc24574b838daa501e46
fetching core_exit.sh...OK
Here is logs:
End of Logs
Steps to reproduce the behaviour:
Expected behaviour
Installation of cod4.
@issue-label-bot[bot] commented on GitHub (Nov 17, 2019):
Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.
@dgibbs64 commented on GitHub (Nov 17, 2019):
Looks like the download didnt complete so you have a corrupted download. you will need to delete the download and try again
@lorentedford commented on GitHub (Nov 17, 2019):
I have done this twice.. Looks like its an installation script issue.
@dgibbs64 commented on GitHub (Nov 17, 2019):
Just tested this myself with no issues. The download is large at 4GB in size. The curl error shows the download failed at 19%. The download simply didn't complete.
Here are my results.
@lorentedford commented on GitHub (Nov 17, 2019):
Trust me this isn't adding up.
@dgibbs64 commented on GitHub (Nov 17, 2019):
Your error clearly states the download failed at 19.4%. The md5 sum WILL fail with a file that's only 19% downloaded. For whatever reason, the connection dropped and the download failed
Here is your original error see where it stated 19.4%
@lorentedford commented on GitHub (Nov 17, 2019):
Clearly it has to be the server where its being downloaded possibly?
https://youtu.be/lNE2Hbw-9l8
@dgibbs64 commented on GitHub (Nov 17, 2019):
My monitoring software has reported no outages or downtime for over a week. I host all the files on my dedicated server. I have tested with my own test servers with no issues. It appears that your server connection is dropping the download for some reason. Is the server hosted on a local network or the internet? As it might be a poor connection. Here is the url if you want to download the using a browser https://files.linuxgsm.com/
@lorentedford commented on GitHub (Nov 17, 2019):
The server is hosted in a data center environment even I have to connect to it from the outside.. We are doing a full fresh re-installation of this box. At this time I will keep you posted.
@lorentedford commented on GitHub (Nov 17, 2019):
Even with a fresh installation of Ubuntu 18.04 LTS this fails
Oddest thing ever.
@lorentedford commented on GitHub (Nov 17, 2019):
So I have tried installing this on my server at home at my home network which i know is up and working correctly I got this...
I have now verified that this is an issue with either your connection where the files are located or an issue with some thing posted on on your cod 4 installation page maybe its downloading an outdated version of linuxgsm when i run the commands idk..
Here is the directions i have followed.
https://linuxgsm.com/lgsm/cod4server/
Here is the video of the dedicated box re installation from a fresh install Its at 2x the speed for you.
https://youtu.be/AsftEIomanU
@UltimateByte commented on GitHub (Nov 18, 2019):
@lorentedford The issue here is rather a network issue than anything else.
What about trying with iptables off?
Also, what server provider are you using and which country is your server located in? Maybe runing some "ping" commands to the file hosting server would provide us some useful info.
@dgibbs64 commented on GitHub (Nov 18, 2019):
Potential issue related to fail2ban blocking Cloudflare IP addresses. I have made a change. Please test again.
@lorentedford commented on GitHub (Nov 18, 2019):
Well since this happened with my home data center servers I this is not the issue. Besides iptables wouldn't affect downloads by default even on a fresh install.. You would have to program that into iptables..
Now..
No this didn't work on a fresh install again. I have dedicated business class fiber with five static ip's at my data center here in my basement where I actually host web and email servers along with many other gamer servers that run linuxgsm with out any issues. I also play with asterisk phone systems and many other projects in the amateur radio community.. I can assure you this is a download issue from your servers.. Especially since I have verified this on an outside data center that is not at or near my network.
Here are my static ip ranges. 157.185.79.34 to 157.185.79.38. According to pf-sense its pulling from 157.185.79.38 non of my other projects have any issues and my download stream has been solid over the past 24 hrs to our gateway.
Here is what Pfsense is showing on traffic information over the past 8 hours.
From what i can see on the router side is that some thing from the download server side disconnects us.
@lorentedford commented on GitHub (Nov 18, 2019):
Please read my above posts and i am off to bed will be up anywhere between 6 to 8 hours from now I will check my email through out my sleep to see If i get another notice from github of reply on this.. Chat in the morning
Best Wishes
@dgibbs64 commented on GitHub (Nov 18, 2019):
Just to be clear. I have identified a potential issue on my server. It appears fail2ban is blocking Cloudflare IP addresses (instead of the offending bad IP address) This might cause connections to be dropped. I have disabled the fail2ban jail that is causing this so the issue might be resolved. Please test again to see if this has helped.
A potential solution that I will investigate later to allow log files to see original visitor IPs addresses that will allow fail2ban to work correctly and not block Cloudflare IP ranges
https://support.cloudflare.com/hc/en-us/articles/200170786-Restoring-original-visitor-IPs-Logging-visitor-IP-addresses-with-mod-cloudflare-
@lorentedford commented on GitHub (Nov 18, 2019):
Nope Its still denying it..
I do have to wipe the directory and start fresh each time when I go to install..
Here is a screen shot..
@h3o66 commented on GitHub (Nov 18, 2019):
some time ago i tried to do a "hack" to do a auto resume of the download.
github.com/h3o66/LinuxGSM@ff48b8dee6Do you think this could help as addition for lgsm ?
@dgibbs64 commented on GitHub (Nov 18, 2019):
@h3o66 yeah something ensure that the download retries would be good. A potential solution will need to be investigated. I will be doing maintenance and testing to make sure that the potential solution resolves the problem.
@lorentedford commented on GitHub (Nov 18, 2019):
A linode server is like 5 dollars a month to test it on.. Just saying.. I wouldn't use it to necessarily host the server on but installation wise it should work and tell you if you found out why the server doesn't like to download and install correctly.
@dgibbs64 commented on GitHub (Nov 18, 2019):
@lorentedford Linode already sponsor me with a free account ;-).
@lorentedford commented on GitHub (Nov 18, 2019):
Well there ya go :) So what's wrong with the downloading from your server? Do you have any ideas yet?
@lorentedford commented on GitHub (Nov 18, 2019):
Keep in mind i have tried on several systems here in our network and at the data center network and still no go on downloading and installation its so odd...
@lorentedford commented on GitHub (Nov 18, 2019):
Are my ip's blacklisted?
@dgibbs64 commented on GitHub (Nov 18, 2019):
Yes I'm working on a fix as we speak. I think some Cloudflare's IP are being blocked by my server which users proxy though
@lorentedford commented on GitHub (Nov 18, 2019):
Yeah personally I haven't been a fan of cloud flare services yet.. Just my thoughts.
@dgibbs64 commented on GitHub (Nov 18, 2019):
Right, I think things might have settled down. I have reconfigured fail2ban and it now correctly sends bans to the cloudflare firewall. This will have to be monitored.
@lorentedford commented on GitHub (Nov 18, 2019):
Well this server on my network failed..
Here are my static ip ranges. 157.185.79.34 to 157.185.79.38.
And the one at the data center eventually failed as well at 29%
@dgibbs64 commented on GitHub (Nov 18, 2019):
Found the request in the logs and it's not being blocked. Even shows me how far the download got. I no longer think the issue is with fail2ban blocking anything. I'm stumped with this one currently, especially since I can't replicate this.
Whats possibly non-standard with your server is you are using link aggregation across 4 NIC's. Maybe the file transfer has an issue with that...but I'm just guessing as it has been a while since I have used such a configuration. Possibly doesn't like the particular bonding mode as there are 7 different modes. 🤷♂️
https://www.interserver.net/tips/kb/network-bonding-types-network-bonding/
@lorentedford commented on GitHub (Nov 18, 2019):
Let m boot up a linode really quick and try one from linode and see what happens.. Again if its not working on my personal data center network or the dedicated box and you can see where both ips pulled for the data example my server from my personal data center should have been pulling from 157.185.79.38 and not sure on the data center side.. Can you tell me if you see the 157.185.79.38? Why does the server drop this connection?
@lorentedford commented on GitHub (Nov 18, 2019):
I created a linode vps really quickly to try out your theory that it might be on both of my networks this was spun up literally like 10 minutes ago as stated and it failed.. Here is the ip of my linode server.
198.58.123.110 and 2600:3c00::f03c:92ff:feb0:3d0e
This re affirms my theory that its on your network or your server connections.
See if there is any logs of this ip trying to connect up to your server.
@dgibbs64 commented on GitHub (Nov 18, 2019):
Possible lead. Tested on 7 different distros and the download only worked on newer versions of curl, researching found a possible issue with curl using certain types https encryption. Bed for me now but I will continue investigating this. I think I will have to tweak https settings for compatibility.
@markie83 commented on GitHub (Nov 19, 2019):
probobly has outdated Certificate Authorities in the older versions of cURL......try it with a newer version of cURL from backport repos.
@markie83 commented on GitHub (Nov 19, 2019):
ok turns out there is not a backport for cURL on 18.04 so I manually added the ppa for 18.10
deb http://ppa.launchpad.net/xapienz/curl34/ubuntu cosmic main
deb-src http://ppa.launchpad.net/xapienz/curl34/ubuntu cosmic main
and then installed curl from there and then it worked.........not something I really wanted to do in a production environment. Ubuntu REALLY needs to do a backport for 18.04 LTS
@lorentedford commented on GitHub (Nov 19, 2019):
Yes this actually worked
Waiting for @dgibbs64 to respond before i do this i don't want to mess this up on my friends server that I am installing for him..
@dgibbs64 commented on GitHub (Nov 19, 2019):
Yup updating curl would be a good fix for this. I will, however, look at making changes to my webserver to ensure compatibility with older versions of curl.
@lorentedford commented on GitHub (Nov 19, 2019):
Me and Markie talked about this last night this should have been back ported because this is going to cause issues down the line.. Markie is my boyfriend I brought him over to my case because I wanted to make sure I wasn't missing anything..
@dgibbs64 commented on GitHub (Nov 19, 2019):
I agree curl should be backported. Especially with the speed that new web technologies and certificate authorities are changing. Looking through the settings I think the easiest and most reliable option is to revert to HTTP. I would prefer to use https but since there is no sensitive data being transferred I think it makes sense in this case.
So, in summary, the quick fix is to upgrade curl to a newer version. On my end, I will make changes to move to HTTP, which I will make ready for the next release.
@lorentedford commented on GitHub (Nov 19, 2019):
Well now we are back to this again...
@lorentedford commented on GitHub (Nov 19, 2019):
@dgibbs64 What did you do from yesterday?? Because now I can't download the files either way..
Just trying to figure out what happened...
@lorentedford commented on GitHub (Nov 19, 2019):
Now this is the same server that installed everything just fine yesterday after we modified with the latest curl.
Here is what is happening now after creating a new user and running the commands.
wget -O linuxgsm.sh https://linuxgsm.sh && chmod +x linuxgsm.sh && bash linuxgsm.sh cod4server./cod4server install@lorentedford commented on GitHub (Nov 21, 2019):
Ok so where does that leave us?
@Frisasky commented on GitHub (Nov 22, 2019):
i think its related to cloudflare's compatibility with older software, try turn off http2 and tls 1.3 in CF's website config
@lorentedford commented on GitHub (Dec 1, 2019):
Hey this problem still exist 12-1-2019
Why is this closed????
@Frisasky commented on GitHub (Dec 1, 2019):
did u updated to latest version of lgsm?
@lorentedford commented on GitHub (Dec 1, 2019):
Yes
@lorentedford commented on GitHub (Dec 1, 2019):
Did you update your website??
Here is my reinstall script since i got tired of making user accounts..
I don't appreciate being treated like i don't know shit.. I get it working one time give you guys the information then you all break it.. Oh then close the ticket after I have been waiting patiently for over a week..
Here is what i am going off of... https://linuxgsm.com/lgsm/cod4server/
It would have been nice if you guys tagged me in my own ticket and asked me if the issue was fully resolved instead of just assuming that the problem is fixed and moving on..
@Frisasky commented on GitHub (Dec 1, 2019):
kindly asking can you join our discord server so we could perform better diagnostic
https://linuxgsm.com/discord
@lorentedford commented on GitHub (Dec 1, 2019):
I am in the general chat and waiting on a video to finish rendering with my fresh install
https://youtu.be/re3enYFImjc
@dgibbs64 commented on GitHub (Dec 1, 2019):
Further testing is ongoing with this issue. Using http has helped improve reliability but is not currently working all the time. I have found a way for curl to continue if the connection drops for any reason so I will be adding that feature to the curl command. Testing is ongoing to confirm if cloud flare is causing a connection issue or not but is very hard to confirm either way.
@lorentedford commented on GitHub (Dec 2, 2019):
This problem was resolved as of Mon Dec 2 19:33:03 UTC 2019 Thank you so much for all your hard work! I do still notice a performance issue with the download but at least it works. Thanks again! However I do notice that it disconnects and reconnects to finish downloading but its a slow process. I still think there is some thing underlining that is going on maybe even server side..
@dgibbs64 commented on GitHub (Dec 6, 2019):
Removal of cloudflare caching and reverting to http has resolved the issue. Methods to improve downloads for non-steam games will need to be investigated moving forward
@Frisasky commented on GitHub (Dec 6, 2019):
theres some other CDN providers around, maybe have a try on them?
@dgibbs64 commented on GitHub (Dec 6, 2019):
@Frisasky KeyCDN offer a free account for open source. They might be what we are after.
@github-actions[bot] commented on GitHub (Jan 5, 2021):
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.