Sometimes, it can be very useful to manage your IPMI settings from the command line… Managing a Nutanix environment, daily administration jobs will not cost you a lot of time.
However, even the automation you are counting on, is not always a guarantee of success. Lately I performed an update for the BIOS and BMC, leveraging the LCM module of Nutanix.
After firing off the update, I just waited for two hours and the update was failing on the first node. The node was running Phoenix, as seen more often after a failed update. But now, the classic ‘reboot-to-host’ script did not work. What made things worse, the IPMI web interface was not available anymore. The IPMI seems to respond on pings, but no other communication was possible. I had to drive to the DC and power off the node by hand, and start the node again.
This time the node also booted in Phoenix, but I managed to reboot the host to AHV after issuing the ‘reboot to host’ script. Still the IPMI did not respond, not even to ping right now. Then you are lucky, you can logon to AHV and use the ipmitool command.
Review network settings
First command I always issue is one to review the network settings.
[root@NTNX-TEST04 ~]# ipmitool lan print 1
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback : MD2 MD5 PASSWORD
: User : MD2 MD5 PASSWORD
: Operator : MD2 MD5 PASSWORD
: Admin : MD2 MD5 PASSWORD
: OEM : MD2 MD5 PASSWORD
IP Address Source : Static Address
IP Address : 0.0.0.0
Subnet Mask : 255.255.255.0
MAC Address : da:23:45:ef:e5:86
SNMP Community String : public
IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP : 0.0.0.0
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12
Cipher Suite Priv Max : XaaaXXaaaXXaaXX
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
Bad Password Threshold : 3
Invalid password disable: yes
Attempt Count Reset Int.: 300
User Lockout Interval : 300
The output of the command shows explicitly that the IP settings of the IPMI interface are not correct anymore… No worries yet, you can easily set these with the IPMI tool.
Change network settings
IP settings can be set using the ipmitool command.
[root@NTNX-TEST04 ~]# ipmitool lan set 1 ipaddr 10.10.5.101 [root@NTNX_TEST04 ~]# ipmitool lan set 1 netmask 255.255.255.0 [root@NTNX-TEST04 ~]# ipmitool lan set 1 defgw ipaddr 10.10.5.254
After issuing the correct IP settings, I was not able to ping the interface. This time I tried to reset the IPMI interface.
[root@NTNX-TEST04 ~]# ipmitool mc reset cold
Even the reset did not work at all. I did another check on the chassis of the node and checked the mac address of the earlier output with the mac address on the sticker on the chassis. It was not the same.
[root@NTNX-TEST04 ~]# ipmitool lan set 1 macaddr ac:1f:6b:35:f1:6c
After changing the mac address to the correct one, the ping request was successful.
LCM update continues…
After all these changes, the LCM update for this node was successful at the second try. However, the second node had the exact behavior as the first one. I could perform all steps another time. I decided to perform all other BMC updates manually. The idea was that with up-to-date firmware versions, all other LCM updates are going to run smoothly.
While performing these updates manually, I ran into a new issue on (just) one node. The ADMIN password was unknown, or at least not working anymore. So again, I used the ipmitool to save my day.
Modifying user accounts
You can change the password of an IPMI user as follows.
[root@NTNX-TEST04 ~]# ipmitool user list ID Name Callin Link Auth IPMI Msg Channel Priv Limit 1 true false false Unknown (0x00) 2 ADMIN true false false Unknown (0x00) 3 true false false Unknown (0x00) 4 true false false Unknown (0x00) 5 true false false Unknown (0x00) 6 true false false Unknown (0x00) 7 true false false Unknown (0x00) 8 true false false Unknown (0x00) 9 true false false Unknown (0x00) 10 true false false Unknown (0x00) [root@NTNX-TEST04 ~]# ipmitool user set password 2 myadminPassword
Normally I always create a second administrative user for the IPMI.
[root@NTNX-TEST04 ~]# ipmitool user set name 3 bart [root@NTNX-TEST04 ~]# ipmitool user set password 3 my1stpassword [root@NTNX-TEST04 ~]# ipmitool user priv 3 0x4 1 [root@NTNX-TEST04 ~]# ipmitool user enable 3 [root@NTNX-TEST04 ~]# ipmitool user list ID Name Callin Link Auth IPMI Msg Channel Priv Limit 1 true false false Unknown (0x00) 2 ADMIN true false false Unknown (0x00) 3 bart true false false Unknown (0x00) 4 true false false Unknown (0x00) 5 true false false Unknown (0x00) 6 true false false Unknown (0x00) 7 true false false Unknown (0x00) 8 true false false Unknown (0x00) 9 true false false Unknown (0x00) 10 true false false Unknown (0x00)
Conclusion
So, after a lot of work using the ipmitool, and the IPMI webinterface for performing the update manually, the LCM updates were running smoothly at last.
Leave a Reply