In Malaysia, it won't. Malaysia doesn't have this kind of culture.

A single hardisk problem can cause whole core trading system jam up, no redundancy or backup plan? 

Better yet, saw this in the Star biz section print version (not found in the online version)

I'll try to recap as much as I can the chrnology of events (newspaper in me boss office at the moment but you can pick up the Star print version and check it out yourself)

6.00 am the hard disk died, hard disk replaced
6.30 am replacement hard disk causing problems, and triggering failure in another hard disk and the CPU
8.30 am activate the backup system
1.00 pm backup system start up took too long, switch back to production site to reattempt.
from then onwards, connection problems bla bla bla

Question time. Hard disk problematic causing failure in another hard disk and the CPU??

Second question. Only one failover system? That's it?

The other interesting thing to keep in mind here is that the explanation given focuses so much on the failure of a single harddisk. So what happened to the RAID array? Unless the attempt by the RAID controller to rebuild the data on the new disk went haywire as well.

The other thing of note also is the need to activate the backup systems manually, especially considering that such real time systems would have automated failovers.

this is not a entry level Dell Server, the trading system are running on mainframe class hardware, i find it really hard to believe a single failed harddisk cause this problem, even if some of the disk and CPU is down it can still keep on running, also BNM have requirement all financial company have DR site that can take over within a few minute, this is not available here, the DR site is not working properly, does BURSA have the same requirement as bank and financial institution to perform regular DR trial and report the result to BNM, if so there maybe someone that lie in the report. If there is no requirement for Bursa to have have hot standby DR system i will be shock, such a important part of our country economy is at risk.

here is official timeline



http://biz.thestar.com.my/news/story.asp?f...38&sec=business

CIO said they are using HP NonStop server, this is fault tolerate server, all the part in the system is have redundancy there is no single point of failure.

http://en.wikipedia.org/wiki/NonStop

robertngo,

A NonStop aka Tandem computer has NO SINGLE point of failure.  So, a single hard disk failure cannot bring the system down.  In summary, there is something WRONG here.

Dreamer

this is not a entry level Dell Server, the trading system are running on mainframe class hardware, i find it really hard to believe a single failed harddisk cause this problem, even if some of the disk and CPU is down it can still keep on running, also BNM have requirement all financial company have DR site that can take over within a few minute, this is not available here, the DR site is not working properly, does BURSA have the same requirement as bank and financial institution to perform regular DR trial and report the result to BNM, if so there maybe someone that lie in the report. If there is no requirement for Bursa to have have hot standby DR system i will be shock, such a important part of our country economy is at risk.

here is official timeline



http://biz.thestar.com.my/news/story.asp?f...38&sec=business

CIO said they are using HP NonStop server, this is fault tolerate server, all the part in the system is have redundancy there is no single point of failure.

http://en.wikipedia.org/wiki/NonStop

well, that's the first thing that came to mind too, that the reasonings that they've provided for failure's more appropriate for the downtime of entry level servers.

To ensure this, Bursa is mulling over the idea of an automated start-up of its back-up system in a situation when a partial or single point at the primary site fails.

Chief information officer Yew Kim Keong said trials on its recovery system were previously done for the scenario of a total system breakdown and not when a single point failed.

This happened on Thursday when the derivatives and bond trading were still operating despite the failure in the equities trading system. As such, the exchange decided to only switch on that part to the back-up system.

However, the synchronisation of data between the primary site and back-up system was longer than the anticipated three hours, stopping Bursa from resuming trading in the afternoon, Yew said.

Hewlett-Packard (M) Sdn Bhd managing director T.F. Chong said the design of the computer system depended on the business environment and the requirements of the respective stock exchange.

Hewlett-Packard is the vendor of the HP Non-Stop Hardware, which is the existing architecture being used by Bursa.

On whether cost was a constraint for Bursa to have a variety of situational recovery trials, Yusli said the exchange had to be practical.

“We could spend all our time testing on business continuity process (BCP) or draw a line somewhere. Our BCP is in line with international practice,” he said.

Bursa spends the most on technology after manpower, which stood at 30% and 50% respectively of operating cost. Total operating cost amounts to RM200mil.

the BCP plan fail to prepare for the situation of single trading system and recovery was not done immediatly, also they dont have live data replication setup between production and DR site and require more than three hour to sync their production data. This is a big problem for a trading system, if you lose more than three hour time you will be losing entire trading session. Banks have been require by BNM to have live data replication to DR for instance recovery during disaster, why does a vital financial institution like Bursa is not required to do the same??

fyire,

In conclusion, some Malaysian IT people are NOT capable of using the the Nonstop server properly.

"First world infrastructure, third world process and mentality"

Dreamer

One of the better possibility answer for the root cause. 

To be fair, I would change from some Malaysian IT people to Bursa IT people or its vendor that responsible for that as don't know whether they sub-con out or manage their own in term of the computer server issue.

cherroy,

1)  In this case, it is BURSA IT people.   But, I had seen ENOUGH examples that to know this problem is MORE wide spread than the bursa.

2) Recently, some Alliance Bank branches are out for one or 2 days too.

As per Godfather movie, "Fish rot from the head first".

Dreamer

The SHORT answer is to manage and run ANY kind of complex system, you NEED GOOD EXPERIENCED PEOPLE to support IT.  So, essentially, you have 2 choices:

A) Train your internal people and pay them well enough so that they stay technical

Or

B) Outsource to someone else but you still NEED GOOD people to monitor them

Malaysian culture is WE spent a lot of MONEY on the hardware.  But, we neither train, pay and keep REAL TECHNICAL people to support and maintain those hardware.  There is NO career path in Malaysia for senior technical people with a lot of experience.  Some took the short cut and hire ex-pats.  But, they REFUSE to hire oversea Malaysians with similar or better experience.  The REASON is they think it is EASIER to control the ex-pats.  But, that is harmful for the future of the country.  In the end, we are in this catch 22 situation that we NEVER get out

A) There are NO GOOD EXPERIENCE TECHNICAL people that deserve very high pay

B) Technical people do not stay technical because they make more money as manager.

In the end, nothing changes and Malaysia cannot go to the next level of IT capabilities.

There are just really basic things from an IT point of view that doesn't make sense in their timeline:

>> Faulty disk replaced

It is now getting standard for servers to have redundancy built-in, especially in the disk storage sub-systems. With RAID arrays, a single or even multiple disk failures don't cause system outages. You can replace those faulty items at some later convenient time, and with hot-swappable/pluggable systems, you don't even need to shut the system down.

>> Backup site start-up process takes longer than expected.

They obviously didn't test this properly and/or regularly enough, or whatever they have implemented is just not appropriate.

Sounds likes there's a few incompetent people in their IT.

Hardware is there, just how people manage it. This is more on management issue on IT side rather than hardware issue.

Bursa's server that crashed was a HP  NonStop K-Series which is already obsolete and 2 generations old.  The K-Series has already been replaced by the S-Series back in the late 1990s.  The S-Series was replaced this year by HP's NonStop Integrity line which runs on Intel Itanium processors.  The K-Series and S-Series runs on MIPS processors.

The Edge had an article that implies that Bursa stopped investing on the old server because it was replacing the system with a new trading system from Atos Euronext... but this project was delayed for 2 years...so the old machine had to keep running and finally died

Bursa's server that crashed was a HP  NonStop K-Series which is already obsolete and 2 generations old. 

HP released a new line of NonStop S-Series to replace the K-Series back in the late 1990s. In 2008, HP phased out the the S-Series with its new HP's NonStop Integrity line which runs on Intel Itanium processors.  (The K-Series and S-Series runs on MIPS processors)

The Edge had an article this week that implies that Bursa stopped investing on the old server because it was replacing the system with a new trading system from Atos Euronext... but this project was delayed for 2 years...so the old K-Series machine had to keep running and finally died