Tracked servers
49
Current hardware inspection, BMC connectivity, and power-state issues tracked by the operations team.
49
46
3
Most recent admin update: Jul 7, 2026, 4:14 PM
During sync_power_state, max retries exceeded for node 03137208-5e80-4aee-9be1-4301a096e078, node state None does not match expected state 'power off'. Updating DB state to 'None' Switching node to maintenance mode. Error: IPMI call failed: power status.
Failed to change power state to 'power off'. Error: IPMI call failed: power status.
During sync_power_state, max retries exceeded for node 210716ec-faab-479f-af59-24b1cc1f4f22, node state None does not match expected state 'power off'. Updating DB state to 'None' Switching node to maintenance mode. Error: IPMI call failed: power status
Failed to inspect hardware. Reason: unable to start inspection: Failed to create neutron ports for node's 499423c9-c2a9-45f6-ac12-5ba3e50844f2 ports [Port(address=e0:d8:48:e3:0d:03,created_at=2024-02-02T15:46:36Z,extra={},id=4116,internal_info={},is_smartnic=False,local_link_connection={'switch_info': 'MOC-R4PAC08-SW-TORS-A', 'port_id': 'tengigabitethernet 1/28/3', 'switch_id': 'd8:9e:f3:ae:e5:a2'},name=None,node_id=2274,physical_network='datacentre',portgroup_id=None,pxe_enabled=True,updated_at=2025-12-17T17:45:23Z,uuid=96a621bb-7ce8-47fe-8b9f-c9cb905f2707), Port(address=e0:d8:48:e3:0d:06,created_at=2024-02-02T15:46:36Z,extra={},id=4119,internal_info={},is_smartnic=False,local_link_connection={'switch_info': 'MOC-R4PAC08-SW-TORS-B', 'port_id': 'tengigabitethernet 1/28/3', 'switch_id': '4c:76:25:e7:e3:42'},name=None,node_id=2274,physical_network='datacentre',portgroup_id=None,pxe_enabled=True,updated_at=2025-12-17T17:45:23Z,uuid=3d7225e5-0dc9-4fdc-9d9f-4f82f372ecf7)].
Agent returned error for clean step {'step': 'erase_devices_metadata', 'priority': 10, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True, 'requires_ramdisk': True} on node 4720d355-7eba-4cfc-9ccb-4ce4b77c96f1: Error performing clean_step erase_devices_metadata: Error erasing block device: Failed to erase the metadata on the device(s): "/dev/sda": Unexpected error while running command.
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/cb63d0cb-2c8e-4d15-9b1c-7ec0db7dd30d?manage_boot=False, Internal server error
enroll
enroll
error
INSPECT FAILED
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/8a0e3d17-7fcf-44ea-ace3-c5f4b9ce2172?manage_boot=False, Internal server error
ironic-inspector inspection failed: Introspection timeout
ironic-inspector inspection failed: Introspection timeout
Timeout reached while inspecting the node
DEPLOY FAILED - escalate to ESI admin
INSPECT FAILED
maintenance_reason: During sync_power_state, max retries exceeded for node 91130dd5-bd54-4485-b31e-92de3cd62bb9, node state None does not match expected state 'power off'. Updating DB state to 'None' Switching node to maintenance mode. Error: IPMI call failed: power status.
maintenance_reason: During sync_power_state, max retries exceeded for node 91130dd5-bd54-4485-b31e-92de3cd62bb9, node state None does not match expected state 'power off'. Updating DB state to 'None' Switching node to maintenance mode. Error: IPMI call failed: power status.
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/d67849c2-e6f2-422b-a677-4c4bc51bad6c?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/7b606779-6740-44a2-96b3-484c0beaf1dd?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/af568ba6-c9d6-4991-a9ff-18f60097b414?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status.
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status - Cannot connect to BMC
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status
Failed to change power state to 'power off'. Error: IPMI call failed: power status
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/aff0d858-57b6-4ba1-930c-283c606012d7?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/201b25a2-17dc-40fd-b36c-f8576fa0b727?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/3b6bd707-0f78-41f2-a2a1-435ea52946fd?manage_boot=False, Internal server error
During sync_power_state, max retries exceeded for node d909ab63-1e09-4b5e-850e-842d75245941, node state None does not match expected state 'power on'. Updating DB state to 'None' Switching node to maintenance mode. Error: IPMI call failed: power status
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/8cbd7863-23b8-432e-89dd-bcb6581b29dd?manage_boot=False, Internal server error
During sync_power_state, max retries exceeded for node 747a3072-cb5e-4658-84f3-d6f6d3f64515, node state None does not match expected state 'power on'. Updating DB state to 'None' Switching node to maintenance mode. Error: IPMI call failed: power status.
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status.
Failed to inspect hardware. Reason: unable to start inspection: Failed to set DHCP BOOT options for any port on node 5c5af27d-aaa4-4464-a7ab-b55d8447fe
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/2c2f2098-2a31-4e4f-9464-161d38ddb403?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: IPMI call failed: power status.
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/152190b7-a761-4d5c-89f9-758986718975?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/0e9648cc-298a-4e1c-a4e4-7b4eb05b7f41?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/300cc62f-d51d-4e1d-acb1-b20cc766df1b?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/b385e960-810d-457f-96bf-deffaa12f49a?manage_boot=False, Internal server error
Failed to inspect hardware. Reason: unable to start inspection: HttpException: 500: Server Error for url: http://192.168.24.17:5050/v1/introspection/2d32ca75-04d6-47f4-97c7-fa61bb4ab91f?manage_boot=False, Internal server error
inspect wait
ACTIVE
AVAILABLE
ACTIVE