Quick Start

This document will outline the basics of how to perform hardware maintenance on this ESI cluster, and how to provision new hardware into the ESI cluster.

OpenStack CLI

During troubleshooting you'll need to interact with OpenStack often, which requires setting up your client to connect to the MOC ESI cluster. Click Here to install OpenStack. 

• Login through the MOC ESI dashboard and create an application credential.
• Navigate to Identity > Application Credential > Create Application Credential to do that
• Create the file ~/.config/openstack/clouds.yaml and populate it with this content (Replace <> items

    cache:        auth: true    clouds:        esi:            auth:                auth_url: https://esi.massopen.cloud:13000                application_credential_id: "<PROJECT ID>"                application_credential_secret: "<SECRET>"            interface: "public"            identity_api_version: 3            auth_type: "v3applicationcredential"

You should now be able to run the command

openstack --os-cloud=esi esi node list

and you should see all the hwbroken nodes.

IPMI Network Bastion VM

Often times access to the BMC interface of a node is required to troubleshoot. For this we have a VM acting as a bastion host which you can use sshuttle or similar tools to access any BMC on your local machine.

Provide your public key to ESI admins via Slack, they will add it to this VM

After that is done, connect using these details: 

ssh techsquare@techsquare.massopen.cloud

The BMC address of each node can be found using 

openstack --os-cloud=esi baremetal node show <NODE LABEL> --fields=driver_info

 Which will be accessible via the bastion VM.

If you need to login to a chassis controller, if it exists, it will be the first IP in that block of 10. For example, the CMC address for the blade

What Happens when Server Breaks

• Remove any leases/offers on the node

List and delete any active offersopenstack esi offer list --resource <node\>openstack esi offer delete <offer-uuid\>  

List and delete any active leasesopenstack esi lease list --resource <node\>openstack esi lease delete <lease-uuid\>

• Set the node owner to the hwbroken project

openstack baremetal node set --owner hwbroken <node>

• Lease the node to the hwbroken project indefinitely (makes it easy for admins to see if a node is broken at a glance)

openstack esi lease create --resource-type dummy_node --resource <node> --project hwbroken

• Put the node in maintenance mode

openstack baremetal node maintenance set <node> --reason "Hardware issue - see GitHub issue #<issue-number>"

• Create a GitHub issue for tracking the maintenance of the issue

gh issue create \  --repo CCI-MOC/<repo-name> \  --title "Node <node>: hardware maintenance" \  --body "Node <node> has been placed in maintenance mode and leased to hwbroken. Tracking hardware issue." \  --label "hardware"

Server Diagnosing/Fixing

1. To run inspection on a node it must be managed:

openstack --os-cloud=esi baremetal node manage <NODE LABEL>

2. To run inspection use the command

openstack --os-cloud=esi baremetal node inspect <NODE LABEL>

The node state will either go back to managed if successful, or inspect failed if not

1. To clean a node it must be managed 

openstack --os-cloud=esi baremetal node manage <NODE LABEL>

2. To clean a node use the command

openstack --os-cloud=esi baremetal node provide <NODE LABEL>

The node will be available if successful (no longer managed), or clean failed if not

1. The node must be able to be deployed with an image.
2. For this you can use metalsmith:

metalsmith --os-cloud=esi --resource-class <NODE RESOURCE CLASS> --image ubuntu-image --network hwbroken --candidate <NODE LABEL> --ssh-public-key <PATH TO LOCAL SSH PUBLIC KEY>

3. To undeploy a node (either in failed or active state), use the command

metalsmith --os-cloud=esi undeploy <NODE LABEL>

The node will be active if successful, or deploy failed if not

After all of this is successful, let ESI admins know and they will take the node out of the hwbroken project.

Hardware Issues

If the issue is related to a hardware component, you can use any spare hardware from the spare hardware storage rack R4-PA-C22. If the issue is with an individual blade in a chassis you can replace without notice. If the chassis requires a hardware swap and not all the blades in the chassis are hwbroken, then that would need to be coordinated with ESI admins.

Physical Hardware

Node Labeling

Every node in ESI is labeled in the convention <OWNER>-<CABINET><U>-<Slot>. For example, MOC-R4PAC10U29-S3 is the server that is in R4-PA-C10 in U29 and in slot 3 on the server. Servers that are standalone (not blades) omit the slot. ie. MOC-R4PAC10U29.

Dell FX2 Chassis

A lot of MOC ESI hardware are Dell blades in Dell FX2 chassis. The user manual for this chassis can be found here. The FX2 can be configured in 8-node (Dell FC430 blades), 4-node (Dell FC630 blades), or 2-node (Dell FC830 blades) layouts. To determine what slot is which you can either flash in the iDrac, or see the user manual linked above on page 23. That document also has the layouts for which PCIe slots map to which blades, and which network interfaces match to which blade, should that be relevant.

Factory Resetting a CMC

If there is a need to factory reset the chassis management controller in a CMC, the process is somewhat convoluted unfortunately.

If you don't know the password or it doesn't work for some reason then

1. Remove the CMC and apply a jumper on the password reset pin, slot CMC back in
2. Serial into the CMC, login with root/calvin

Use These Commands Below

racadm racresetcfg

To factory reset the CFG

racadm setniccfg -d

To enable dhcp on the CMC

Discussion/Collaboration

You can reach all the ESI admins in the #ESI slack channel in the MOCA slack.