Oracle 6140 Copyback doesnt start automatically

Again this week we had a disk fail in one of our ageing 6140 arrays. no big deal, but once I’d pulled the failed disk, waited around for a few mins, and replaced the disk with a new one copybook didn’t start of its own accord. I’ve seen this a number of times before, but not for a year or so now.
It pretty much does it if the disk fails with:

Event Message: Drive by-passed
Component type: Drive

It can sometimes also declare the disk Missing. Here’s how to breath some life into the process. Ok, before we start, this will only work when the array is on 07.xx firmware.

You will have a disk looking something like this:

Replaced Drive - No copyback

Replaced Drive – No copyback

First click on the volume group(VG) in the logical tab that the disk belongs to.

Now right click the volume group name and click on replace drives.

Replace drives

Replace drives….

It will now bring up a new window and in here you will see the missing disk mentioned in the top panel and in the bottom panel it should list the hot spare(HS) that is in-use (if you had any) and any unassigned disk that might be in the array.

Select drives to replace

Select drives to replace

Your disk you replaced should show here as an unassigned disk so click on it in the bottom panel and ‘replace drive’ will now become available, click it and the copyback will now happen.

If you did not want to have the copyback happen and would like to keep the HS as part of the VG you would click on that, it would then flip the disk from a ‘in-use hot spare’ to a member of the VG but you would now be one HS less unless you made the other disk the HS.

Once you make your disk selection, copyback should begin.

Copyback Begins

Copyback Begins

Pretty simple, but perhaps not obvious, I certainly can’t remember it first time every time, so in some ways, this post is for me.

Storage market continuous change

The storage market has to be the most unsettled it’s been in years. The wide variety of storage vendors, with good independents is a shrinking landscape. The power plays and buy outs makes for some very careful decisions if you’re about to buy storage. My short advice is be careful and if you can..wait it out.

I guess the start for me was Sun buying Storagetek, that was back in 2005, but I mark it as one of the very early indicators of change. However after that, things appeared settled, but as a Storagetek customer, there was a shift from what felt like a more intimate organisation in Storagetek, to a very rigid feeling environment within Sun. The irony of that was even further bureaucracy and rigidity was to come in 2009 when Oracle purchased Sun.
I guess I was pretty close to both these acquisitions as I was a Storagetek customer throughout. As a customer I felt the changes quite severely. When using Storagetek for support, I felt like I could reach into the heart of the organisation to get advice or help, with Sun, those portals closed, things became very regimented, almost certainly better for a company with scale, but as a customer it was sometimes frustrating.

With Dell continuing to resell EMC, they didn’t have that storage piece they could call their own. Dell had long since been in the server space, if you didn’t buy HP/Compaq, you bought Dell tin. They went shopping and Dell bought Equallogic.
This of course soured their relationship with EMC for a while, but it seemed to more or less heal. My thoughts are here on that. Equallogic was a move away from the traditional frame based FC type SAN, with it’s iSCSI technology. It was an expensive purchase, (to put it mildly) the largest ever cash purchase of a private venture backed technology company! That said they’ve made it work for them, a very established and profitable product line now.

HP, not to be outdone, realising the changing and growing storage space, go out and acquire Lefthand Networks, who are pretty much an Equallogic competitor. Interesting in that it’s a like for like purchase, and really only touching the Enterprise level market. Lefthand a heavy iSCSI pioneer.

Oracle purchasing Sun posed many questions, with a lot of speculation that they would drop the storage, they only purchased MySQL. At the time I maintained the storage piece had value as Oracle DB turn key solutions, and low and behold that’s exactly what they’re offering now, tuned storage to create a harmony between software and hardware.

Dell, had hoped to sew up their Entry/Mid/Enterprise solution with the Equallogic and clearly with designs on 3Par products in mind went shopping and made a very interesting purchase by purchasing a storage technology company called Ocarina. They’re a dedupe/storage optimisation technology. This signalled their intention to invest further in their storage offering and continue to innovate.

In the same month IBM announced the acquisition of Storwize a data compression specialist. One can only assume they have the intention of offering it as part of their product suite.

Along comes HP, predominantly a server, printer and desktop tin vendor, they had the EVA storage solution,  it had issues and was becoming a bit of a dinosaur in a rapidly changing storage technology market. The LeftHand acquisition had put them in a good place, and for a while were the all round vendor. If you went to HP, it was pretty much a 1 stop shop. However the EVA was ageing, Lefthand wasn’t making inroads into the heavy enterprise user markets, they lacked mid to enterprise punch, a storage piece that would move them on to next generation technology. A bidding war ensued between Dell and HP for the independent 3Par. 3Par had great technology, an excellent reputation and was perceived as a fantastic purchase by both HP and Dell. HP won out. Some say they paid way over the odds, probably, but they were ahead in the race!
Ironically I don’t feel HP are pushing the 3Par sales very hard, I suspect it’s because it just doesn’t integrate into their Storageworks management platform, and that’s a piece of development that is ongoing. Just take a walk around their website, it’s all Lefthand P-series and EVA, 3Par seems to exist as a separate entity. Try and engage HP about 3Par products, you’ll see what I mean.

Step into 2011 and events occur at pace.

Dell expand their empire again. While Equallogic is a great product, it sits pretty much in the entry to mid level space, whereas they were filling Enterprise heavy orders with the EMC offerings. To take on that market themselves they needed that heavy hitter, they lost 3Par, so they went to their closest rival Compellent. The recession had meant that R&D for the smaller storage houses was slightly under funded, and of course made them ripe for purchase. Compellent has fantastic technology, a well respected support operation and has potential for scale and improvements. Compellent also makes a nice model for the Ocarina innovation. Dedupe in-line anyone?

Then the pace steps up!
Western Digital announces the acquisition of Hitachi disk arm. This is a very interesting purchase on many levels, clearly WD are making a play here to up the ante on Seagate, their closest rival. Hitachi parting with the disk arm is curious, they’re a storage array vendor themselves and part of their sales pitch is that they make the array end to end.. well, not anymore! Odd. That puts question marks over Hitachi HDS, are they ripe for acquisition?

Confused by that, NetApp announces their acquisition of the the LSI storage arm, Engenio. This is HUGE news. Underneath a plethora of arrays, sits LSI tech. Oracle’s 6000 series, IBM DS and FastT to me are the big fish. NetApp and Oracle compete for the same business in that market, Oracle aren’t going to want to buy from NetApp – NetApp aren’t going to want to sell to Oracle or indeed IBM, they want that business for themselves! The PR on this is that all is well, OEM relationships are safe… yeah, ok. This is business and big money. At present I can only see the end of the line for those big ticket OEM ranges. This one change puts huge question marks over further investment in those technologies, and without committed re-assurance from all parties, to my mind it’s time for change!

Next thing we know Seagate upsize to catch WD, and acquire Samsungs disk arm!

To emphasise the pace at which events occur, I’ve put together this timeline.

Storage market evolution

This isn’t exhaustive by any means, little twists and turns have come and gone, but for me these were the ones that had knock forward effect and therefor the most impact.


What next?! Rumour and prediction collide…

Cisco buy NetApp: Well, Cisco are desperate to get a slice of the storage/server market, they’re largely regarded as a 1 trick pony, king of networking. They do an AWFUL lot of collaboration with NetApp… feels like only a matter of time.

Oracle dump Hitachi and LSI/Engenio: they aren’t going to want to buy from the competition, they either buy them or drop them.

Oracle buy NetApp: They have a choice to make, dig deep and buy NetApp, which makes a lot of sense on many levels, but would be costly.

IBM buy NetApp: Similar reasons for Oracle to buy them, get back control or lose a chunk of their range

IBM buy EMC: This would also make sense for IBM, EMC has the enterprise (banking) marketplace very well sewn up and IBM has that sort of support operation to match.

Sometimes you HAVE to invest, and if you take the hard line that it’s strategic or an upgrade with an existing vendor relationship, you can’t go far wrong. Otherwise, it’s becoming a small storage vendor list, my advice, watch now, buy much later.

Dell buying Compellent – Virtualising Storage

Over the past couple of weeks I have been doing various seminars and meetings with storage vendors, more specifically, the storage vendors I regard as offering modern thinking, and up to date technology.
I’ve made no bones about my liking for Compellent, I really think they are forward thinking and innovative, whereas the traditional storage vendors are clinging on to repeat business and people doing whatever they have done before.
I went to Dell’s Accelerate the Virtual Era 2011 era session last week at the Mandeville hotel in central London. I wanted to get more confidence in where they are heading with storage. I left feeling that not even Dell know for sure, well, nothing they’re able to give away to the guys on the ground at any rate.

Then i gave it serious thought, they want to stand alone with their storage offering, they’ve more or less said as much. It’s been made pretty clear Dell want to stand alone as a storage partner, and that be based on the emerging FCoE technologies, data management– NEW technology.
Now, whilst I like the Equallogic product it’s what I would consider an entry to mid level product. It has a vast hole in it’s disk by not offering fibre channel support, (no, i dont mean FC connectivity to the array per say, but reliable fast data delivery). They jump from SAS to SSD, which is just…daft. They cover that up with all sorts of flannel saying the box will sort it all out with blocks moving about… Yes, ok, but we live in the real world. That said, I accept that 6Gbps SAS is on the Market, but it is still serial architecture, it’s the latency that worries me. It has MASSIVE potential however, four X 4Gbps pipes with a 6Gbps SAS drive represents a sizeable 24Gbps throughput.
Their recent acquisition of Compellent adds the fibre piece and introduces a mid to enterprise level array, after all Compellent go toe to toe with the likes of EMC in the market right now.

However…. Dell don’t only sell Equallogic and Compellent right now (there’s already a reseller deal in place even though the buyout will only complete end of Feb), there is the Celerra(VNX) and Clarion arrays, BUT these are both rebadged EMC products. This is a reseller arrangment in essence, yes they sell it as Dell badged, but we all know it’s EMC when purple badges and Dell-EMC when black badged. Neither of these arrays are ‘modern’ thinking, good as they are, they’re older frame based technology which is being dragged into the FCoE era, with ‘bolt on’ data management solutions.

Now if you remember the 3Par bun fight Dell had with HP, even though HP won the bid, Dell somewhat soured their relationship with EMC in the process. EMC weren’t happy that Dell were effectively undermining them. It was all ‘smoothed’ over in one way or another with both sides making noise about exploring new avenues of collaborating etc, but now they’ve shown their cards with the Compellent deal. I seriously think the EMC deal won’t be renewed, apparently that has about 18 months to run before renewal. Whether Dell walk away from that or EMC give them the shove I don’t know, but my guess is it will die a death. What they MIGHT keep is the EMC software (Data Domain is an EMC product) for the dedupe, but again, anyones guess. Software reseller?

So….my opinion, given 12 – 18 months we will see Dell finally sort their storage offerings with Equallogic occupying the entry to mid range Market and Compellent pushing on with their mid to Enterprise Market. It kind of makes sense to me now.

With Compellent going, they are pretty much the last big independent vendor, it’s all come back to the big boys, again. In this economic climate nobody is in a position to pour big money into R&D, so this technology is here to stay for a while.

What benefits does the deal bring for Compellent as such? Well a nice simple one is buying power, Compellent buy in relatively small numbers compared to the Dell buying more power, it’s a simple economic and should see the disk price come down for Compellent. Also, currently Compellent use the Supermicro base servers to run as their controllers, Dell servers would instantly replace these. There is huge scope for bringing the big boy Dell into the Compellent business, support/partners/oem etc.
Negatives? You might argue that the clearly excellent support structure that Compellent have built a lot of business on, could be ‘diluted’ by Dell, but 1. they will tread careful, and 2. I’ve actually had good service through Dell support.

The only solid conclusion I came to was this is the worst possible time to be buying new storage, and the best decision is to sit and watch and wait for it to all play out.

vSphere4 causes AVT on Oracle (Sun/STK) 6140

Ever since we started our vSphere4 upgrade, our 6140 arrays are behaving well, but differently. Each new vSphere4 server we bring online results in an AVT (automated volume transfer) of a random lun. This process ,is ordinarily used for link failures or maintenance when the array needs to serve a host via an alternate path, which is of course the beauty of multiple-path I/O. Recovery Guru throws out a message like this:

Volume not on preferred path

As soon as we bring a new vSphere4 server onto the bus, it ‘walks’ the bus, and this results in a random lun or more moving between controllers. The first time it did this I was very dubious and went ploughing through logs before reaching a conclusion. However it’s repeatable and in reality fairly harmless, but if you see it, don’t panic, just move it back again and make sure your luns are balanced.

Why is it doing it? Well vSphere4 does a discovery on any bus, so it knows about all disks etc being offered or present on any bus. This is meant to be helpful in that you don’t have to use the virtual centre to do storage adaptor refreshes, it’s done for you. vSphere4 also tries to encapsulate any volume or lun it finds – all meant to happen, but on the sun/oracle hardware, just be aware of this side effect.

This is also the reason why with vSphere4 and crystal firmware, you MUST delete loon ID 31 to avoid any issues going forward with vmware having encapsulated your inbound santricity lun (lun 31)

Modifying or clearing Sun STK DACstore on foreign disks

This procedure only works for 06.xx code running on the controllers, and is intended for Santricity users.

You are not clearing the DACstore so much as you are removing the foreign one to add the correct one of the receiving subsystem. The effect is the same, you can use the foreign disk.

1.       In the source subsystem, locate the non-data (unassigned) disks you wish to transfer and remove them one at a time with approx. two minutes between each. If the array is off, just remove them.
2.       In the target subsystem, locate a hot spare of equivalent spec/size etc
3.       In SANtricity, fail the hot spare.
4.       Remove it from the tray, wait a minute or so.
5.       Place the ‘foreign’ disk into the hot spare slot, wait for the system to settle and mark it as hot spare. (Santricity can lag behind actual events)
6.       Unassign it as a hot spare.
7.       Reassign it as a hot spare.
8.       Install the previous disk from this slot in a desired unassigned slot within this subsystem
9.       Repeat steps 2 through 8 for all disk to be installed in this target subsystem
10.       When the last disk is done, leave it as the hot spare.

For 7.x firmware subsystems it is different for adding ‘foreign’ disks as they could be from 6.x or 7.x firmware subsystems. (to follow)