Fabric Interconnect booting to bash?

So, I thought I’d share an experience I had yesterday where my Cisco UCS Fabric Interconnect (FI) wasn’t feeling well and in my attempt to resurrect it, I seemed to break it even more. I’m sure that never happens to you…The FI was now booting to a bash prompt instead of the normal UCS console interface. It would get to the point where it would say “System is coming up….Please wait” and it would say this about 12 times and then display the bash prompt. I won’t bore you with what I actually did in my attempt to get beyond this, but lets just say I spent about 2 hours debugging it when the fix should have only taken about 5 minutes (hindsight is 20/20). It goes without saying that this situation should not happen under normal circumstances, but I’ve heard rumblings of people seeing this here and there after upgrading to 2.2.x. So if Google brought you here looking for a solution, you’re in luck.

All you need to do is:

bash# shutdown -r now

As the FI boots, press Ctrl + r to interrupt the boot process


Get the UCS kickstart file name – it would be preferable for you to boot the actual file name that the FI has in the /installables/switch folder which is the name of the kickstart image that you last installed. This can be found by looking at the working FI and running “scope firmware” and then “show image” (that’s from memory but I think that’s it). However, you can use the kickstart in the root if you can’t figure it out.

loader> boot ucs-6100-k9-kickstart.5.0.3.N2.<xyzabc>.bin

When the system comes up and sits are the boot prompt, run “erase configuration”

switch(boot)# erase configuration

That should do it. The FI needs to reboot and come back up as if it were brand new and ask to create/join the cluster.

Hope this saves someone some time.


7 thoughts on “Fabric Interconnect booting to bash?

  1. Hi Jeff,

    We’ve encountered this issue several times now, but I just found your post today. And sadly it did not work for us.

    bash-2.05b# erase configuration
    bash: erase: command not found

    How could you perform this action in the bash prompt?

  2. Pingback: Cool New UCS Feature–Firmware Sync | Jeff Said So

  3. After doing the erase config and giving up all the required commands to start up new !!

    It agains goes back to bash shell ???

    is there a solution for this !

  4. Hi Jeff,

    Is there any good way of landing into bash once the system has booted?
    I want to clear up some space on the altflash partition that the FI mounts from the IOM.
    So far I managed to dump into bash into a very unofficial way but I wasn’t able to locate how to mount the altflash part (I also don’t have the required time).
    If you want to experiment or anyone is interested as a hobby (this won’t be available in any official manual or doc too soon probably):
    Connect to the Fabric Interconnect with a Console Cable and reboot.
    Press CTRL+R and end up in the bootloader.
    From there load the kickstart image.
    Then load the normal system image.
    While the system image is loading there is a “programming fault” with not stopping all the Linux signals from interfering with the process and by running a shell script to trigger the actual Cisco console.
    This shell script can be broken out of and one can land into bash as root.
    After you see the INIT prompt for Linux just press CTRL + Backslash and keep pressing it to kill the pseudo shell script until it gives up and dies (it has to be killed about 10-15 times).
    CTRL+C won’t work as I think SIG INT is being blocked from doing its things.
    SIG QUIT on the other hand (I am too lazy to google now and see if this is what CTRL+\ is sending) is allowed.
    Eventually you land into bash as root.
    Then you have to:
    – use sed and replace in /etc/passwd the nobash shell with /bin/bash
    – use echo to add a new line for root in /etc/shadow with a null password (this you will change upon login for the first time)
    – run the binary you killed with CTRL+\ so that the system continues its normal boot
    I ran it by hand and added a & at the end so that it runs in the background (I was afraid it might kill my newly activated shell console).
    It’s something with /isanboot/bin…
    If you look a bit throughout the system you’ll also find some Indian programmer names (a guy called Chandra that did parts of the linux binaries:) (same if you play with strings on some binaries).
    Also some interesting loopbacks used for communication and some internal 127 IPs to go with them.
    Same for all the block devices or flash in /dev.
    Some can be mounted..I had no patience to take it logically so I mounted all that was mountable in /mnt with a script.
    Now if you ssh as root or admin into the UCS you will land in a linux shell automatically.
    Just look in /etc/passwd and you will see for the user admin the shell that you have to start so that you get the classic Cisco command prompt.
    Starting it manually works perfect and you have access again to the nice frontend Cisco CLI instead of the backend Linux one (backend is though useful for those that are tech friendly enough to be curious what’s going on behind the curtains).
    My detective time has run out but if anyone discovers how the altflash can be mounted from the FI Linux, I would greatly appreciated (it has to be something either from /dev or via some protocol accessible via those 127 IPs that interconnect the various components).
    arp -an shows some IPs on the internal communication plane.

  5. Hi Jeff,

    I’m looking through your web page and am wondering if you have seem something like this… I posted this on the Cisco Community, but would definitely like to hear from you.

    My Fabric Interconnects 6248 hangs with the following single line message ”

    N5000 BIOS v.3.5.0, Thu 02/03/2011, 05:12 PM

    I’ve tried going into the Golden Bios and showing debug message and the following was displayed before it hanged.

    N5000 BIOS v.3.5.0, Thu 02/03/2011, 05:12 PM

    Booting Golden BIOS

    Enable BIOS debug messages (y/n) :

    N5000 BIOS v.3.5.0, Thu 02/03/2011, 05:12 PM

    — Turning ON Amber LED —
    — Turning ON Blue LED —
    ME Platform PEI Init start.
    Register PPI Notify: 64c96700-6b4c-480c-a3e1-b8bde8f602b2
    ME Platform PEI Init end.
    HECI PEI Init start.
    HECI PEI Init end.
    ========== Entering UNCOREINIT ==========
    RtidProfile before: 0
    Setup variable is valid – QPI BS (BIOS STUFF).
    MemCeil : 0xD0000000
    Pci Resource Mem32 Range: 0xD0000000 – 0xFBFFFFFF
    PciResourceMem64Base: 0x100000000
    RB[0] Mem64 Range : 0x100000000 – 0x7fffffffff
    RB[1] Mem64 Range : 0x0 – 0x0
    RtidProfile after: 0
    Tohm : 0x0
    HighMmioBase : 0x100000000
    HighMmioLimit : 0x8000000000
    ========== Enter CSI RC code ==========
    ========== Exit CSI RC code ==========
    LowGap: 12
    Notify: PPI Guid: 71a8917b-0891-4e27-8a73-a9b334840393, Peim notify entry point: fff9d798


    ========== Enter MRC code ==========

    MRC rev: 00900000
    Memory behind processor 0 running at DDR3-1066
    RDIMM population
    Channel Early Config
    DDR training
    Command/Clock TrainingRead DQ/DQS
    Receive Enable
    Write Leveling
    Write Leveling Fix-up
    Write DQ/DQS
    Command phase 0 Re-center RdDqs Re-center WrDq Re-run Rd Vref RTL
    Checking margins for all ranks with loop count = 10…

    RxDqLeft RxDqRight RxVLow RxVCenter RxVHigh TxDqLo TxDqHi TxVLow TxVCenter TxVHigh
    N0.C0.D0.R0: 18 18 27 -1 25 25 25 31 0 31
    N0.C0.D0.R1: 17 17 28 -1 26 24 25 31 0 31
    N0.C1.D0.R0: 18 17 27 -1 23 24 25 31 0 31
    N0.C1.D0.R1: 18 17 27 -1 24 24 26 31 0 31
    Independant channel mode enabled on socket 0
    ECC is enabled
    Start Hardware Memory Init

    MemInit latency 1815 ms
    Initialize Memory Map
    Clear Errors
    Set RAS Config

    Total MRC latency = 7074 ms

    MRC latency – MemTest and MemInit = 5259 ms
    Software memory test Passed!

    DIMM location | dimmPresent | Ranks | Size | mapOut | Mfg. ID | Mfg. Date | DRAM Id | Part #
    N0.C0.D0 | 1 | 2 | 8192 MB | 0 | Samsung | 2017 WW35 | Samsung |M393B1K70CH0-YH9
    N0.C0.D1 | 0
    N0.C1.D0 | 1 | 2 | 8192 MB | 0 | Samsung | 2017 WW35 | Samsung |M393B1K70CH0-YH9
    N0.C1.D1 | 0
    N0.C2.D0 | 0
    N0.C2.D1 | 0

    MRC is done!
    ========== Exit MRC code ==========
    Notify: PPI Guid: 64c96700-6b4c-480c-a3e1-b8bde8f602b2, Peim notify entry point: fffa7938
    Install PEI Memory.
    Memory Installed: Address=BF800000; Length=10000000
    PEI_STACK: Address=BF800000; Length=100000
    HOBLIST address before memory init = 0xffae0000
    HOBLIST address after memory init = 0xbf900000

    Data in MemoryInit
    sizeof(struct hostNvram): 3300
    sizeof(MRC_HOST_HOB): 3336
    sizeof(MEMORY_INFO): 3344
    MrcHostHob->SizeOf_NvRam: 3300
    MrcHostHob->SizeOf_MRCHOSTHOB: 3336
    MrcHostHob->SizeOf_MEMORYINFO: 3344

    MemoryInit Completion!!!!
    After PeiCpuWorkaround (UncoreInitLateInit)
    After PeiIohWorkaround (UncoreInitLateInit)
    Tohm : 0x42C000000
    HighMmioBase : 0x430000000
    HighMmioLimit : 0x8000000000
    Updated PciResourceMem64Base: 0x430000000
    Updated RB[0] Mem64 Range : 0x430000000 – 0x7FFFFFFFFF
    After OemUncoreInitHook (UncoreInitLateIohChipsetInit)
    — Turning OFF Blue Led —
    After OemUncoreInitHook (UncoreInitLateInit)
    ========== Exiting UNCOREINIT ==========
    PEI core reallocated to memory
    Total Cache as RAM: 131072 bytes.
    CAR stack ever used: 65532 bytes.
    CAR heap used: 6904 bytes.
    Notify: PPI Guid: f894643d-c449-42d1-8ea8-85bdd8c65bde, Peim notify entry point: fff9d908
    Starting all APs.
    Load AP Microcode. BSP and maybe NBSPs Microcode loaded in SEC.
    Enable BSP and AP Cache.
    APs waiting.
    NumCpus = 2.
    CPU initialization.
    BSP APIC ID = 0.
    Register PPI Notify: 605ea650-c65c-42e1-ba80-91a52ab618c6
    Exit PEI CPU Driver.
    CK505Pei_Init Entry.
    Program CK505 start.
    Select the SMBUS mux. Assuming to be on SMBUS Segment 0
    Program CK505 end.
    PEIM 8401a046-6f70-4505-8471-7015b40355e3 was not started!!
    PEIM e008b434-0e73-440c-8612-a143f6a07bcb was not started!!
    DXE IPL Entry(CF7F67D0)
    Notify: PPI Guid: 605ea650-c65c-42e1-ba80-91a52ab618c6, Peim notify entry point: fff946bc
    DXE Status Code Available
    GenericSIO: Variable Write Available!
    GenericSIO: Get ISA_IRQ_MASK Status=EFI_NOT_FOUND Updating with DEFAULT E305
    GenericSIO: Get ISA_DMA_MASK Status=EFI_NOT_FOUND Updating with DEFAULT 10

    Intended Cpu Freq = 1729
    Actual Cpu Freq = 1745
    AcpiResLib: LibGetDsdt(): LocateProtocol(ACPISupport) returned EFI_NOT_FOUND
    IN ACPI Start: 0
    IN ACPI 1: 0
    DSDT21 addres 0x15FAB50; -> EFI_SUCCESS
    AcpiResLib: LibGetDsdt(): Found v1.0b RSDT->DSDT @ 0x15FAB50; -> EFI_SUCCESS
    AcpiResLib: LibGetDsdt(): Found v2.0&UP XSDT->DSDT @ 0x15FAB50; -> EFI_SUCCESS
    SIO[0]: Aml=> Collected 5 DepFunc Items of UAR1 Object
    SIO: Updating ISA_IRQ_MASK = 0xE305 with 0x10 for IRQ# 4
    SIO[1]: Aml=> Failed to Locate ASL Object UAR2 in DSDT at 015FAB50
    SIO[1]: Tbl=> Entry is Empty! Check ‘YourSioName_DevLst’ table.
    GenericSIO: FAIL to Enumerate SIO Chip # 1 EFI_NOT_FOUND
    AcpiResLib: LibGetDsdt(): Found v1.0b RSDT->DSDT @ 0x15FAB50; -> EFI_SUCCESS
    AcpiResLib: LibGetDsdt(): Found v2.0&UP XSDT->DSDT @ 0x15FAB50; -> EFI_SUCCESS
    ACPI: SetAcpiTable() Table=0x1006318; Handle=0xBF8FF520; *Handle=0x0
    ACPI: SetAcpiTable() Exiting… Status = EFI_SUCCESS
    PciHostCSHooks: LocateProtocol(ACPISupport)=EFI_SUCCESS
    ACPI: SetAcpiTable() Table=0x161B298; Handle=0x16F7FB0; *Handle=0x0
    ACPI: SetAcpiTable() Exiting… Status = EFI_SUCCESS
    PciHostCSHooks: ACPISupport->SetAcpiTable(MCFG) = EFI_SUCCESS
    TSEG Base: cf800000
    GCD: AddMemSpace B=CF800000, L=800000, i=0, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=CF800000, L=800000, i=0, S=EFI_SUCCESS
    GCD: AddMemSpace B=FEE00000, L=100000, i=1, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FEE00000, L=100000, i=1, S=EFI_SUCCESS
    GCD: AddMemSpace B=FEB00000, L=100000, i=2, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FEB00000, L=100000, i=2, S=EFI_SUCCESS
    GCD: AddMemSpace B=FEA00000, L=20, i=3, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FEA00000, L=20, i=3, S=EFI_SUCCESS
    GCD: AddMemSpace B=FE800000, L=200000, i=4, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FE800000, L=200000, i=4, S=EFI_SUCCESS
    GCD: AddMemSpace B=FE000000, L=800000, i=5, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FE000000, L=800000, i=5, S=EFI_SUCCESS
    GCD: AddMemSpace B=FD000000, L=1000000, i=6, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FD000000, L=1000000, i=6, S=EFI_SUCCESS
    GCD: AddMemSpace B=FC000000, L=1000000, i=7, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FC000000, L=1000000, i=7, S=EFI_SUCCESS
    GCD: AddMemSpace B=FBFFE000, L=2000, i=8, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FBFFE000, L=2000, i=8, S=EFI_SUCCESS
    GCD: AddMemSpace B=D0000000, L=10000000, i=9, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=D0000000, L=10000000, i=9, S=EFI_SUCCESS
    GCD: AddI/OSpace B=CF8, L=8, i=10, S=EFI_ACCESS_DENIED
    GCD: AllocI/OSpace B=CF8, L=8, i=10, S=EFI_SUCCESS
    GCD: SpaceAttr A=8000000000000000 B=CF800000, L=800000, i=0, S=EFI_SUCCESS
    GCD: SpaceAttr A=1 B=FEE00000, L=100000, i=1, S=EFI_SUCCESS
    GCD: SpaceAttr A=1 B=FEB00000, L=100000, i=2, S=EFI_SUCCESS
    GCD: SpaceAttr A=1 B=FEA00000, L=20, i=3, S=EFI_UNSUPPORTED
    GCD: SpaceAttr A=1 B=FE800000, L=200000, i=4, S=EFI_SUCCESS
    GCD: SpaceAttr A=1 B=FE000000, L=800000, i=5, S=EFI_SUCCESS
    GCD: SpaceAttr A=1 B=FD000000, L=1000000, i=6, S=EFI_SUCCESS
    GCD: SpaceAttr A=1 B=FC000000, L=1000000, i=7, S=EFI_SUCCESS
    GCD: SpaceAttr A=1 B=FBFFE000, L=2000, i=8, S=EFI_SUCCESS
    GCD: SpaceAttr A=8000000000000001 B=D0000000, L=10000000, i=9, S=EFI_SUCCESS
    Initializing NTB for IIO 0
    Port 0:3 Active. Max Link Width = 4
    Port 0:4 Active. Max Link Width = 4
    Port 0:5 Active. Max Link Width = 4
    Port 0:6 Active. Max Link Width = 4
    TSEG Address cf800000.
    TSEG size 800000.
    Creating Memory Data for SMBIOS.
    –DIMM Speed = 1067 MHz
    –Array[0] Device[0] Size = 8192 MB
    –Array[0] Device[2] Size = 8192 MB
    GCD: AddMemSpace B=FF000000, L=1000000, i=0, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FF000000, L=1000000, i=0, S=EFI_SUCCESS
    GCD: AddMemSpace B=FED1C000, L=4000, i=1, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FED1C000, L=4000, i=1, S=EFI_SUCCESS
    GCD: AddMemSpace B=FED00000, L=4000, i=2, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FED00000, L=4000, i=2, S=EFI_SUCCESS
    GCD: AddMemSpace B=FEC00000, L=100000, i=3, S=EFI_ACCESS_DENIED
    GCD: AllocMemSpace B=FEC00000, L=100000, i=3, S=EFI_SUCCESS
    GCD: AddI/OSpace B=400, L=80, i=4, S=EFI_ACCESS_DENIED
    GCD: AllocI/OSpace B=400, L=80, i=4, S=EFI_SUCCESS
    GCD: AddI/OSpace B=500, L=80, i=5, S=EFI_ACCESS_DENIED
    GCD: AllocI/OSpace B=500, L=80, i=5, S=EFI_SUCCESS
    GCD: AddI/OSpace B=1180, L=20, i=6, S=EFI_ACCESS_DENIED
    GCD: AllocI/OSpace B=1180, L=20, i=6, S=EFI_SUCCESS
    GCD: SpaceAttr (UC ALL) B=FEC00000; L=1400000;
    GCD:(UC ALL) setting B=FEC00000, L=104000, A=1; S=EFI_SUCCESS
    GCD:(UC ALL) setting B=FED04000, L=18000, A=1; S=EFI_SUCCESS
    GCD:(UC ALL) setting B=FED1C000, L=4000, A=1; S=EFI_SUCCESS
    GCD:(UC ALL) setting B=FED20000, L=E0000, A=1; S=EFI_SUCCESS
    GCD:(UC ALL) skipping B=FEE00000, L=100000, A=1; S=EFI_SUCCESS
    GCD:(UC ALL) setting B=FEF00000, L=100000, A=1; S=EFI_SUCCESS
    GCD:(UC ALL) setting B=FF000000, L=1000000, A=1; S=EFI_SUCCESS
    GCD: SpaceAttr A=8000000000000001 B=FF000000, L=1000000, i=0, S=EFI_SUCCESS
    GCD: SpaceAttr A=8000000000000001 B=FED1C000, L=4000, i=1, S=EFI_SUCCESS
    GCD: SpaceAttr A=1 B=FED00000, L=4000, i=2, S=EFI_SUCCESS
    GCD: SpaceAttr A=1 B=FEC00000, L=100000, i=3, S=EFI_SUCCESS
    HPET: FED1F404 = 80
    HPET LocateProtocol(ACPISupport)- EFI_SUCCESS Success
    ACPI: SetAcpiTable() Table=0x16C3798; Handle=0x171F2B8; *Handle=0x0
    ACPI: SetAcpiTable() Exiting… Status = EFI_SUCCESS
    ACPISupport.SetAcpiTable() = EFI_SUCCESS
    ERROR: AhciSmm.Entry(CF83E6B0)=EFI_NOT_FOUND
    ERROR: AhciSmm.Entry(18AA6B0)=EFI_NOT_FOUND
    SmiFlash: Flash Protocol CF831000
    InitializeSupNuovaTcoHandler – Not in Smm
    InitializeSupNuovaTcoHandler – In SMM.
    InitializeSupNuovaTcoHandler – Registering TCO handler
    InitializeSupNuovaTcoHandler – Successfully registered
    InitializeSupNuovaTcoHandler – Not in Smm
    Driver 4a37320b-3fb3-4365-9730-9e89c600395d was discovered but not loaded!!
    Driver 899407d7-99fe-43d8-9a21-79ec328cac21 was discovered but not loaded!!
    AMI PCI Bus Driver.Start(17C6C30)=<<<>>>
    ASSERT in C:\Projects\AMI\O296\argon_o2_src_work\Core\EM\SMM\SmmDispatcher.c on 462: gUseSbsp

    – See more at: https://supportforums.cisco.com/discussion/12488456/ucs-fabric-interconnect-hangs#comment-10460856

Leave a Reply

Your email address will not be published. Required fields are marked *