Oracle 6140 Copyback doesnt start automatically

Again this week we had a disk fail in one of our ageing 6140 arrays. no big deal, but once I’d pulled the failed disk, waited around for a few mins, and replaced the disk with a new one copybook didn’t start of its own accord. I’ve seen this a number of times before, but not for a year or so now.
It pretty much does it if the disk fails with:

Event Message: Drive by-passed
Component type: Drive

It can sometimes also declare the disk Missing. Here’s how to breath some life into the process. Ok, before we start, this will only work when the array is on 07.xx firmware.

You will have a disk looking something like this:

Replaced Drive - No copyback

Replaced Drive – No copyback

First click on the volume group(VG) in the logical tab that the disk belongs to.

Now right click the volume group name and click on replace drives.

Replace drives

Replace drives….

It will now bring up a new window and in here you will see the missing disk mentioned in the top panel and in the bottom panel it should list the hot spare(HS) that is in-use (if you had any) and any unassigned disk that might be in the array.

Select drives to replace

Select drives to replace

Your disk you replaced should show here as an unassigned disk so click on it in the bottom panel and ‘replace drive’ will now become available, click it and the copyback will now happen.

If you did not want to have the copyback happen and would like to keep the HS as part of the VG you would click on that, it would then flip the disk from a ‘in-use hot spare’ to a member of the VG but you would now be one HS less unless you made the other disk the HS.

Once you make your disk selection, copyback should begin.

Copyback Begins

Copyback Begins

Pretty simple, but perhaps not obvious, I certainly can’t remember it first time every time, so in some ways, this post is for me.

Storage market continuous change

The storage market has to be the most unsettled it’s been in years. The wide variety of storage vendors, with good independents is a shrinking landscape. The power plays and buy outs makes for some very careful decisions if you’re about to buy storage. My short advice is be careful and if you can..wait it out.

I guess the start for me was Sun buying Storagetek, that was back in 2005, but I mark it as one of the very early indicators of change. However after that, things appeared settled, but as a Storagetek customer, there was a shift from what felt like a more intimate organisation in Storagetek, to a very rigid feeling environment within Sun. The irony of that was even further bureaucracy and rigidity was to come in 2009 when Oracle purchased Sun.
I guess I was pretty close to both these acquisitions as I was a Storagetek customer throughout. As a customer I felt the changes quite severely. When using Storagetek for support, I felt like I could reach into the heart of the organisation to get advice or help, with Sun, those portals closed, things became very regimented, almost certainly better for a company with scale, but as a customer it was sometimes frustrating.

With Dell continuing to resell EMC, they didn’t have that storage piece they could call their own. Dell had long since been in the server space, if you didn’t buy HP/Compaq, you bought Dell tin. They went shopping and Dell bought Equallogic.
This of course soured their relationship with EMC for a while, but it seemed to more or less heal. My thoughts are here on that. Equallogic was a move away from the traditional frame based FC type SAN, with it’s iSCSI technology. It was an expensive purchase, (to put it mildly) the largest ever cash purchase of a private venture backed technology company! That said they’ve made it work for them, a very established and profitable product line now.

HP, not to be outdone, realising the changing and growing storage space, go out and acquire Lefthand Networks, who are pretty much an Equallogic competitor. Interesting in that it’s a like for like purchase, and really only touching the Enterprise level market. Lefthand a heavy iSCSI pioneer.

Oracle purchasing Sun posed many questions, with a lot of speculation that they would drop the storage, they only purchased MySQL. At the time I maintained the storage piece had value as Oracle DB turn key solutions, and low and behold that’s exactly what they’re offering now, tuned storage to create a harmony between software and hardware.

Dell, had hoped to sew up their Entry/Mid/Enterprise solution with the Equallogic and clearly with designs on 3Par products in mind went shopping and made a very interesting purchase by purchasing a storage technology company called Ocarina. They’re a dedupe/storage optimisation technology. This signalled their intention to invest further in their storage offering and continue to innovate.

In the same month IBM announced the acquisition of Storwize a data compression specialist. One can only assume they have the intention of offering it as part of their product suite.

Along comes HP, predominantly a server, printer and desktop tin vendor, they had the EVA storage solution,  it had issues and was becoming a bit of a dinosaur in a rapidly changing storage technology market. The LeftHand acquisition had put them in a good place, and for a while were the all round vendor. If you went to HP, it was pretty much a 1 stop shop. However the EVA was ageing, Lefthand wasn’t making inroads into the heavy enterprise user markets, they lacked mid to enterprise punch, a storage piece that would move them on to next generation technology. A bidding war ensued between Dell and HP for the independent 3Par. 3Par had great technology, an excellent reputation and was perceived as a fantastic purchase by both HP and Dell. HP won out. Some say they paid way over the odds, probably, but they were ahead in the race!
Ironically I don’t feel HP are pushing the 3Par sales very hard, I suspect it’s because it just doesn’t integrate into their Storageworks management platform, and that’s a piece of development that is ongoing. Just take a walk around their website, it’s all Lefthand P-series and EVA, 3Par seems to exist as a separate entity. Try and engage HP about 3Par products, you’ll see what I mean.

Step into 2011 and events occur at pace.

Dell expand their empire again. While Equallogic is a great product, it sits pretty much in the entry to mid level space, whereas they were filling Enterprise heavy orders with the EMC offerings. To take on that market themselves they needed that heavy hitter, they lost 3Par, so they went to their closest rival Compellent. The recession had meant that R&D for the smaller storage houses was slightly under funded, and of course made them ripe for purchase. Compellent has fantastic technology, a well respected support operation and has potential for scale and improvements. Compellent also makes a nice model for the Ocarina innovation. Dedupe in-line anyone?

Then the pace steps up!
Western Digital announces the acquisition of Hitachi disk arm. This is a very interesting purchase on many levels, clearly WD are making a play here to up the ante on Seagate, their closest rival. Hitachi parting with the disk arm is curious, they’re a storage array vendor themselves and part of their sales pitch is that they make the array end to end.. well, not anymore! Odd. That puts question marks over Hitachi HDS, are they ripe for acquisition?

Confused by that, NetApp announces their acquisition of the the LSI storage arm, Engenio. This is HUGE news. Underneath a plethora of arrays, sits LSI tech. Oracle’s 6000 series, IBM DS and FastT to me are the big fish. NetApp and Oracle compete for the same business in that market, Oracle aren’t going to want to buy from NetApp – NetApp aren’t going to want to sell to Oracle or indeed IBM, they want that business for themselves! The PR on this is that all is well, OEM relationships are safe… yeah, ok. This is business and big money. At present I can only see the end of the line for those big ticket OEM ranges. This one change puts huge question marks over further investment in those technologies, and without committed re-assurance from all parties, to my mind it’s time for change!

Next thing we know Seagate upsize to catch WD, and acquire Samsungs disk arm!

To emphasise the pace at which events occur, I’ve put together this timeline.

Storage market evolution

This isn’t exhaustive by any means, little twists and turns have come and gone, but for me these were the ones that had knock forward effect and therefor the most impact.


What next?! Rumour and prediction collide…

Cisco buy NetApp: Well, Cisco are desperate to get a slice of the storage/server market, they’re largely regarded as a 1 trick pony, king of networking. They do an AWFUL lot of collaboration with NetApp… feels like only a matter of time.

Oracle dump Hitachi and LSI/Engenio: they aren’t going to want to buy from the competition, they either buy them or drop them.

Oracle buy NetApp: They have a choice to make, dig deep and buy NetApp, which makes a lot of sense on many levels, but would be costly.

IBM buy NetApp: Similar reasons for Oracle to buy them, get back control or lose a chunk of their range

IBM buy EMC: This would also make sense for IBM, EMC has the enterprise (banking) marketplace very well sewn up and IBM has that sort of support operation to match.

Sometimes you HAVE to invest, and if you take the hard line that it’s strategic or an upgrade with an existing vendor relationship, you can’t go far wrong. Otherwise, it’s becoming a small storage vendor list, my advice, watch now, buy much later.

Sun Storagetek/Oracle 6000 series firmware Preparation

I recently had some good feedback from a couple of really nice folk asking for advice/help/comment on some Sun/STK/Oracle 6xxx series issues they were having. I’ll start to try and put more stuff up about the arrays.

One very decent document I read was the upgrade guide. It’s pretty simple to read doc and contains good information about the upgrade process.

Sun Storage 6000 Series Array Firmware Upgrade Guide

To that end, it’s not exactly obvious, since the move away from Sunsolve, how to find the software in the MOS portal. To locate the 6xxx series software for the firmware upgrade process, I’ve put together this ‘where is it’ step by step.

1. Login to My Oracle Support at
2. Along the top of the window that opens as your first page, click on the ‘Patches & Updates’ tab.
3. In the Patch Search pane, click on “Product or Family(Advanced Search).”
4. Tick or check the box for “Include all products in a family.”
5. In the Product field, clock the drop down and select “Sun StorageTek 6000 Series Software”
6. In the Release field, select “Sun StorageTek 6000 Series software 1.0”. It should already be selected, but just check.
7. Select the platform to install the tool and click search OR as i wanted to see all I just clicked Search.

My Oracle Support - Sun storagetek 6000 series firmware

8. This will take you to a new window with your search results. -Patch 10265930: “Sun StorageTek 6000 Series Array Firmware Upgrade Utility”
9. Download the zip file and extract the executables.

Once you have the file extracted and installed, we proceed to the firmware updates themselves.

vSphere4 causes AVT on Oracle (Sun/STK) 6140

Ever since we started our vSphere4 upgrade, our 6140 arrays are behaving well, but differently. Each new vSphere4 server we bring online results in an AVT (automated volume transfer) of a random lun. This process ,is ordinarily used for link failures or maintenance when the array needs to serve a host via an alternate path, which is of course the beauty of multiple-path I/O. Recovery Guru throws out a message like this:

Volume not on preferred path

As soon as we bring a new vSphere4 server onto the bus, it ‘walks’ the bus, and this results in a random lun or more moving between controllers. The first time it did this I was very dubious and went ploughing through logs before reaching a conclusion. However it’s repeatable and in reality fairly harmless, but if you see it, don’t panic, just move it back again and make sure your luns are balanced.

Why is it doing it? Well vSphere4 does a discovery on any bus, so it knows about all disks etc being offered or present on any bus. This is meant to be helpful in that you don’t have to use the virtual centre to do storage adaptor refreshes, it’s done for you. vSphere4 also tries to encapsulate any volume or lun it finds – all meant to happen, but on the sun/oracle hardware, just be aware of this side effect.

This is also the reason why with vSphere4 and crystal firmware, you MUST delete loon ID 31 to avoid any issues going forward with vmware having encapsulated your inbound santricity lun (lun 31)

Modifying or clearing Sun STK DACstore on foreign disks

This procedure only works for 06.xx code running on the controllers, and is intended for Santricity users.

You are not clearing the DACstore so much as you are removing the foreign one to add the correct one of the receiving subsystem. The effect is the same, you can use the foreign disk.

1.       In the source subsystem, locate the non-data (unassigned) disks you wish to transfer and remove them one at a time with approx. two minutes between each. If the array is off, just remove them.
2.       In the target subsystem, locate a hot spare of equivalent spec/size etc
3.       In SANtricity, fail the hot spare.
4.       Remove it from the tray, wait a minute or so.
5.       Place the ‘foreign’ disk into the hot spare slot, wait for the system to settle and mark it as hot spare. (Santricity can lag behind actual events)
6.       Unassign it as a hot spare.
7.       Reassign it as a hot spare.
8.       Install the previous disk from this slot in a desired unassigned slot within this subsystem
9.       Repeat steps 2 through 8 for all disk to be installed in this target subsystem
10.       When the last disk is done, leave it as the hot spare.

For 7.x firmware subsystems it is different for adding ‘foreign’ disks as they could be from 6.x or 7.x firmware subsystems. (to follow)