Adam Strickfaden

Category: Troubleshooting

  • Reset or Change NUTANIX NXS2U1NL06G610 BMC Password

    Reset or Change NUTANIX NXS2U1NL06G610 BMC Password

    I came across what appears to be a rebranded Supermicro server. This is a NUTANIX NXS2U1NL06G610. Unfortunately despite many attempts and various user/pass combos I was unable to get into the BMC. This post will explain how to manually set the BMC password using Ubuntu and IPMITOOL. This is not something you can do in the BIOS like Dell or HP.

    For starters get a copy of Ubuntu and make a bootable USB drive.

    Once booted you can open a terminal and install IPMITOOL. The following command will update the package repositories and install the utility automatically:

    sudo apt update && sudo apt install -y ipmitool

    My recommendation is to perform a complete factory reset of the BMC. This will ensure a clean slate. It’s possible you have a server with a non-default password. You can reset and try the defaults once more. This did not work for me but maybe it will for you.

     

    sudo ipmitool raw 0x3c 0x40

    There won’t be any output for this command but you will hear the fans spin up, indicating that the server is resetting.

    Once reset you can attempt the default credentials again, otherwise proceed to set a manual password of your own.

    List the user accounts on the system with the following command. You should see a user called ADMIN:

    sudo ipmitool user list 1
    sudo ipmitool user list 1

    The most important column is the ID field. In my case the ADMIN account corresponds to ID field 2.

    Now we can set a password of our own:

    sudo ipmitool user set password 2 NewPassword!

    Here we instruct ipmitool to set a new password by giving it the ID of the admin user, in our case 2, and then setting the new password.

    This command will tell you if it was successful or not. Sometimes there’s rules on the length and complexity of the password you can use. If you get errors just try other passwords until it lets you through.

    Now you can login in to the BMC interface without problems.

  • R740xd – Loud Fans With Rear Flex Bay

    R740xd – Loud Fans With Rear Flex Bay

    This post will be short and sweet. If you have an R740xd with a rear flex bay you need high performance fans installed otherwise fan speed will remain at 100%. I’ve seen this behavior time and time again.

    Here’s a screenshot of the fans with standard cooling fans installed:

    Fan speed with standard fans

    And here is a screenshot right after the installation of high performance fans:

    fan speed after high performance fan installation

    It may be necessary to reset iDRAC after installation of high performance fans.

     

  • No memory found – Dell PowerEdge Server and Intel Xeon 2nd Gen CPU

    No memory found – Dell PowerEdge Server and Intel Xeon 2nd Gen CPU

    If you upgrade your 14 gen DELL server to an Intel Xeon Scalable 2nd Gen CPU, you may receive this error.

    Common second gen CPUs:

    • Xeon Gold 5218 (16 cores)
    • Xeon Gold 5220 (18 cores)
    • Xeon Gold 6240 (18 cores)
    • Xeon Gold 6252 (24 cores)
    • Xeon Platinum 8253 (16 cores)
    • Xeon Platinum 8276 (28 cores)

    Chances are your BIOS is outdated and doesn’t support second gen processors. DELL did not introduce support for these processors until BIOS version 2.2.10. If you have a version below that you will receive this error without a doubt.

    What you’ll have to do is install a first GEN CPU, update the BIOS, and then reinstall the second GEN processors.

    Examples of first gen processors you can use:

    • Xeon Bronze 3104
    • Xeon Bronze 3106
    • Xeon Silver 4108
    • Xeon Gold 6126
    • Xeon Gold 6148 (20 cores)
    • Xeon Platinum 8176 (28 cores)

    Quick way to identify the generation

    • 31xx, 41xx, 51xx, 61xx, 81xx → 1st Generation (Skylake)
    • 32xx, 42xx, 52xx, 62xx, 82xx → 2nd Generation (Cascade Lake)

    Just update the BIOS and you’ll be good to go.

  • DELL R640 freezes loading BIOS drivers

    DELL R640 freezes loading BIOS drivers

    I’ve probably had hundreds of servers freeze at this particular screen over my time dealing with servers. It gets to the loading bios drivers screen. You count the dots one by one and watch closely. One dot. two dots. 5 dots. And then nothing. The dots stop coming.

    In almost every case I’ve found bad memory to be the culprit. In fact, I am documenting this while dealing with the exact problem. An R640 with 16 sticks of memory. It’s freezing. I know there’s a bad stick somewhere.

    The only question is what’s the fastest way to locate the bad stick so you can move on with your life?

    With a lot of memory I like to remove all the DIMMs from the black slots first. It’s up to you. The whole idea is to keep removing DIMMs until the darn thing boots up. If you remove sticks and it still freezes, that means the sticks you pulled out were good. In a 2 CPU configuration remove them evenly from both sides to prevent unbalanced memory errors.

    Now, if you remove a bunch of sticks and it boots up, that means the bad stick is in the group of DIMMs you just pulled out. To find the bad one you now put them back in the server one by one. Once it freezes up you have found the bad DIMM. Replace it.

    Sometimes you’ll do this process only to find the DIMMs just needed to be reseated. That’s fine, in the end it’s just about fixing the problem.

  • HP DL380 G9 Not Booting Into Smart Storage Administrator

    HP DL380 G9 Not Booting Into Smart Storage Administrator

    You may run into a case where a G9 server fails to boot into the HP Smart Storage Administrator. In my case the server would freeze here after hitting enter.

    It’s not actually frozen but hitting enter does nothing and your only recourse is to just reboot the server.

    Beyond that you may also notice Intelligent Provisioning appears to be completely borked. And most of the time this is the case.

    The resolution is to simply reinstall Intelligent Provisioning. Start by downloading a copy here.

    The file will come in an ISO format. Turn this into a bootable USB or alternatively just boot from it using the remote console in iLO. If you have difficulties making the USB I find booting it from the virtual KVM almost always works.

    This is the screen you want to see. You’ll get a progress bar and the whole process takes about 10 – 15 minutes. The UID light will flash blue indicating a firmware update is taking place.

    Once the system restarts you can attempt to boot into the Smart Storage Administrator once again.

    In my case I was now able to successfully boot into the software and configure my drives.