[GH-ISSUE #1255] netboot.xyz not loading in Equinix Metal #367

Closed
opened 2026-02-27 14:51:31 +03:00 by kerem · 13 comments
Owner

Originally created by @enkelprifti98 on GitHub (Jun 7, 2023).
Original GitHub issue: https://github.com/netbootxyz/netboot.xyz/issues/1255

Originally assigned to: @antonym on GitHub.

Describe the bug
Deploying Equinix Metal instances with custom ipxe and https://boot.netboot.xyz does not work. It gets stuck chainloading the netboot.xyz image.

To Reproduce
Steps to reproduce the behavior:

  1. Deploy Equinix Metal server instance with Custom iPXE operating system. Under the URL field use https://boot.netboot.xyz

Expected behavior
The netboot.xyz image should load.

Originally created by @enkelprifti98 on GitHub (Jun 7, 2023). Original GitHub issue: https://github.com/netbootxyz/netboot.xyz/issues/1255 Originally assigned to: @antonym on GitHub. **Describe the bug** Deploying Equinix Metal instances with custom ipxe and `https://boot.netboot.xyz` does not work. It gets stuck chainloading the netboot.xyz image. **To Reproduce** Steps to reproduce the behavior: 1. Deploy Equinix Metal server instance with Custom iPXE operating system. Under the URL field use `https://boot.netboot.xyz` **Expected behavior** The netboot.xyz image should load.
kerem 2026-02-27 14:51:31 +03:00
Author
Owner

@vielmetti commented on GitHub (Jun 7, 2023):

Happy to provide support for anyone interested in looking into this.

<!-- gh-comment-id:1580875238 --> @vielmetti commented on GitHub (Jun 7, 2023): Happy to provide support for anyone interested in looking into this.
Author
Owner

@antonym commented on GitHub (Jun 7, 2023):

Any particular machine type this is occurring on or specific architecture? Have you tried loading older releases instead of rolling? Was it working before?

<!-- gh-comment-id:1581087169 --> @antonym commented on GitHub (Jun 7, 2023): Any particular machine type this is occurring on or specific architecture? Have you tried loading older releases instead of rolling? Was it working before?
Author
Owner

@enkelprifti98 commented on GitHub (Jun 7, 2023):

@antonym Seems to be mainly the m3.small.x86 instance type affected. Others like the c3.small seem to work fine.

<!-- gh-comment-id:1581138515 --> @enkelprifti98 commented on GitHub (Jun 7, 2023): @antonym Seems to be mainly the m3.small.x86 instance type affected. Others like the c3.small seem to work fine.
Author
Owner

@antonym commented on GitHub (Jun 13, 2023):

if the implementation is working on other instance types of the same arch, I don't see why it wouldn't work unless there is something new with the hardware. If it was working previously, it could be an iPXE change upstream that rolled in, and then we could try booting older releases to see if anything works on that type.

<!-- gh-comment-id:1588543879 --> @antonym commented on GitHub (Jun 13, 2023): if the implementation is working on other instance types of the same arch, I don't see why it wouldn't work unless there is something new with the hardware. If it was working previously, it could be an iPXE change upstream that rolled in, and then we could try booting older releases to see if anything works on that type.
Author
Owner

@jackson-tim commented on GitHub (Jun 13, 2023):

This also seems to be happening on the n3.xlarge instances, maybe it's a common NIC type? e810s maybe?

<!-- gh-comment-id:1589784606 --> @jackson-tim commented on GitHub (Jun 13, 2023): This also seems to be happening on the n3.xlarge instances, maybe it's a common NIC type? e810s maybe?
Author
Owner

@vielmetti commented on GitHub (Jun 29, 2023):

@antonym I have added some credits to your Equinix account, to assist in any testing. Unclear to me at this point what the scope of this problem is right now.

<!-- gh-comment-id:1613287949 --> @vielmetti commented on GitHub (Jun 29, 2023): @antonym I have added some credits to your Equinix account, to assist in any testing. Unclear to me at this point what the scope of this problem is right now.
Author
Owner

@enkelprifti98 commented on GitHub (Jun 29, 2023):

Hi @antonym @vielmetti ! I did some more testing and I believe this is an issue with nodes that have the Intel E810 NIC.

I deployed 5x m3.small.x86 instances. The ones with Mellanox NICs get to the netboot.xyz menu just fine. The ones with the Intel E810 NIC get stuck on the following screen seen through the Out-of-Band serial console.

Updated version of netboot.xyz is available:

Running version.....1.0.0+
Updated version.....2.x

Please download the latest version from netboot.xyz.

Attempting to chain to latest version...
http://boot.netboot.xyz/ipxe/netboot.xyz-metal.efi... ok
iPXE initialising devices...
<!-- gh-comment-id:1613526390 --> @enkelprifti98 commented on GitHub (Jun 29, 2023): Hi @antonym @vielmetti ! I did some more testing and I believe this is an issue with nodes that have the Intel E810 NIC. I deployed 5x m3.small.x86 instances. The ones with Mellanox NICs get to the netboot.xyz menu just fine. The ones with the Intel E810 NIC get stuck on the following screen seen through the Out-of-Band serial console. ``` Updated version of netboot.xyz is available: Running version.....1.0.0+ Updated version.....2.x Please download the latest version from netboot.xyz. Attempting to chain to latest version... http://boot.netboot.xyz/ipxe/netboot.xyz-metal.efi... ok iPXE initialising devices... ```
Author
Owner

@antonym commented on GitHub (Jun 29, 2023):

I saw this link https://github.com/ipxe/ipxe/issues/498 but not sure if there was any resolution. Any idea what the firmware of the NIC might be on and if there is a newer rev available from Intel?

<!-- gh-comment-id:1613730204 --> @antonym commented on GitHub (Jun 29, 2023): I saw this link https://github.com/ipxe/ipxe/issues/498 but not sure if there was any resolution. Any idea what the firmware of the NIC might be on and if there is a newer rev available from Intel?
Author
Owner

@antonym commented on GitHub (Jun 30, 2023):

Also found this thread:

https://github.com/ipxe/ipxe/issues/937

I'll try and create some snp efi images this weekend and we can see if it provides a better result.

<!-- gh-comment-id:1614993012 --> @antonym commented on GitHub (Jun 30, 2023): Also found this thread: https://github.com/ipxe/ipxe/issues/937 I'll try and create some snp efi images this weekend and we can see if it provides a better result.
Author
Owner

@antonym commented on GitHub (Jul 1, 2023):

Try this url for custom_url to see if you have any issues:

https://s3.amazonaws.com/dev.boot.netboot.xyz/854df13acc3969f659f2fd3aa973c60d736a82e7/menu.ipxe

I've added building of the snp EFI kernels and also have set the default to use snp kernels as the default on everything as it should avoid using the built in iPXE drivers that are not always kept up to date as newer hardware is deployed.

I've tested on one machine that was having the issue and it seemed to start loading after that. I'm not sure of the wider impact on other machine types yet, but it should be a more reliable path to go.

github.com/netbootxyz/netboot.xyz@854df13acc
github.com/netbootxyz/netboot.xyz@da3e1d9243

This is mainly in development now and not pushed to production yet, will give it a bit of time to bake so others can test with.

<!-- gh-comment-id:1615997855 --> @antonym commented on GitHub (Jul 1, 2023): Try this url for custom_url to see if you have any issues: ``` https://s3.amazonaws.com/dev.boot.netboot.xyz/854df13acc3969f659f2fd3aa973c60d736a82e7/menu.ipxe ``` I've added building of the snp EFI kernels and also have set the default to use snp kernels as the default on everything as it should avoid using the built in iPXE drivers that are not always kept up to date as newer hardware is deployed. I've tested on one machine that was having the issue and it seemed to start loading after that. I'm not sure of the wider impact on other machine types yet, but it should be a more reliable path to go. https://github.com/netbootxyz/netboot.xyz/commit/854df13acc3969f659f2fd3aa973c60d736a82e7 https://github.com/netbootxyz/netboot.xyz/commit/da3e1d9243a42999b5a022d50fecbb1c1479ecb1 This is mainly in development now and not pushed to production yet, will give it a bit of time to bake so others can test with.
Author
Owner

@antonym commented on GitHub (Jul 6, 2023):

Closing this out as I think this should be resolved and work better going forward, please ping me if you notice any oddness. Thanks to @vielmetti for the testing credits!

<!-- gh-comment-id:1622794998 --> @antonym commented on GitHub (Jul 6, 2023): Closing this out as I think this should be resolved and work better going forward, please ping me if you notice any oddness. Thanks to @vielmetti for the testing credits!
Author
Owner

@vielmetti commented on GitHub (Jul 6, 2023):

Thanks @antonym for checking this out - the credits should be ample for any reasonable test setup that is necessary now or in the future.

<!-- gh-comment-id:1622820318 --> @vielmetti commented on GitHub (Jul 6, 2023): Thanks @antonym for checking this out - the credits should be ample for any reasonable test setup that is necessary now or in the future.
Author
Owner

@enkelprifti98 commented on GitHub (Jul 11, 2023):

Tested an E810 node with https://boot.netboot.xyz and it works fine now. I guess the change is live in production at this point. Thanks for addressing the issue!

<!-- gh-comment-id:1631184250 --> @enkelprifti98 commented on GitHub (Jul 11, 2023): Tested an E810 node with `https://boot.netboot.xyz` and it works fine now. I guess the change is live in production at this point. Thanks for addressing the issue!
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/netboot.xyz#367
No description provided.