Message from Bursa Malaysia

* High Priority **

Equity trading for today (Thursday 3 July 2008) is suspended for the first session. Only Equity Trading is suspended for the first session.

Please be informed that any orders entered or trades executed in the first session will be cancelled by Bursa.

We will keep you updated on the progress and when trading can resume.

Regards

Bursa Helpdesk

What happened is it another Bursa low limit breaker breach?

So far no update from bursa, pending investigation.
But every broker unable to online their Broker Front End (BFE) server ....
I guess is due to Bursa Centre Host Server failure (this is the server where all the broker connect to everyday during trading hours)

hmmmmm... what a goooood system of "host server".... yesterday drop 20+point ..... today stop fuctioning....

i think this whole week will keep dropping.

From Star.....

KUALA LUMPUR: Bursa Malaysia faced connectivity issue in early trade on Thursday after opening slightly lower and suspension was expected to be for the entire morning session.

The KLCI opened down 0.37 point to 1,153.33 on some selling. However, it had not updated the data since then.

In an immediate response, Bursa Malaysia Securities said trading would be suspended for at least the morning trading session from 9am to 12.30pm following a multi hardware failure in its core trading system.

“The derivatives market will continue to trade as normal, however, there will not be any underlying KLCI feed for the FKLI and OKLI trading,” it said.

Bursa Securities said all market participants have been duly notified and will be updated periodically of the progress.

“Bursa Malaysia is investigating the problem and will notify the market of any updates,” it added.

damn.....i thought it was a time to buy in.....then i need to wait until afternoon..

here comes all kind of rumours and hearsay. sigh.

fellow remisiers can go minum teh for the meantime?

we been dropped 5 days and all red (my eye also strained), is time to take a break 'jom minum'

down down afternoon....i guess

Do not know when is the technical rebound? Already five days.

afternoon will drop sharply.... rumor said ????

No technical rebound in sight. Nikkei down for 10 straight days. I don't think KLCI will put a miracle.

treders should sue bursa for loss of profit

any news from bursa yet?


Thanks

It's time to open... Gogogo.. Why it's still down

Haih, Malaysia Boleh

Seems Bursa close for the whole day

This post has been edited by Digital_Life: Jul 3 2008, 03:11 PM

mmmm fishy indeed...maybe someone attacK kl market now lols!..
1997 repeat..ahahaha

LOL, computer system being infected by virus or trojan?

Malaysia tentu boleh... so many sodomy scandal.. what's new to the KLSE down.. this nation has no future.

someone torrent in main server..

8am - said down
11am - said switch to DR site
1.30 said - switch back to production.
2pm said - trading start at 3.30
3.25pm said - whole day will be suspended....

Let's go propose a new HA IT system for KLSE

..........whole day gone...no trading at all.
anybody here know wat was happening?

the core trading system already have HA feature, but they said is multiple hardware failure, then must be so unlucky the DR system also having hardware problem. There must be some system admin there under tremendous pressure to fix this issue, do anyone here know how much revenue bursa get in a typical day?

but i wonder why it take so long to replace the hardware, such critical system should have support contract to get the parts within hours, maybe there is some program update to mess up the data so they take so long time to recover.

This post has been edited by robertngo: Jul 3 2008, 06:23 PM

Mayb Khairy wanna press sell all AA shares @ 90cents but key in wrongly @9cents, so they have to shut it down and then propose to cancel all transaction for today just for that.

maybe to prevent massive selling activities caused by the Najib news?

the future market still open, it is down 27 point, so tomorrow still will be dropping

I think should be maintenance wise issue..

You know lah..

Malaysia..

The suspension of the Bursa is probably due to Anwar's press conference. I am very sure tomorrow we will be triggering the circuit breaker.

circuit breaker will automatically take effect if market drop 10%, it need to drop more than 100 point to hit it.

and for the 1st time ever, it happened on 10th March 2008.

Tomorrow is Friday and my guess is that the hardware still gonna fail and Bursa chief is going to secure his job by following orders from BN that KLSE still above 1100 after the weekend. Hooray for the CEO and moolah for BN.

Tommorow sure big sell down. If hardware failure or glitches sure easily solve. I think KLSE got hacked.

This post has been edited by LittleBro: Jul 3 2008, 11:14 PM

like typical lar..if not attacked wat else..

you never know the real deal behind till you are in it.

as mentioned before...it is iether internally attcked or externally...

cannot said for sure what happen unless you work in BURSA, i am working on enterprise server system and have experience with financial system before. The core transaction system are the crown jewel of the business, it is likely a mainframe or a fault tolerant server like HP NonStop server. These server have very very high uptime measure in years, it is also very hard to hack because this is not Windows or UNIX/LINUX with available exploit code and script can be download online, you need to write the exploit code and be a expert in these system to success.

Although the system are tough something sh*t happen and maybe due some weird combination of hardware and software issue they cannot bring the server up, for bursa the impact will be huge the CEO and CIO may lost its job. or you can believe this is a conspiracy by the gov


http://themalaysianinsider.com/index.php/h...st-ceo-his-job-

the official story is the harddisk problem, i find it hard to believe this, the DR site should be bring up resume operation but looks like their DR is not working, something is not right about this. SC and BNM should investigate if they have fake the DR test report, as all financial institution need to DR working and able to swing transaction over within minute, they need to have regular DR test and submit report to BNM, not sure if BURSA system have the same requirement

This post has been edited by robertngo: Jul 4 2008, 08:40 AM

What's the news now guys?

any news or shut down for today?

hehe no shutdown today ler, been running fine since morning.

In Malaysia, it won't. Malaysia doesn't have this kind of culture.

A single hardisk problem can cause whole core trading system jam up, no redundancy or backup plan?

Better yet, saw this in the Star biz section print version (not found in the online version)

I'll try to recap as much as I can the chrnology of events (newspaper in me boss office at the moment but you can pick up the Star print version and check it out yourself)

6.00 am the hard disk died, hard disk replaced
6.30 am replacement hard disk causing problems, and triggering failure in another hard disk and the CPU
8.30 am activate the backup system
1.00 pm backup system start up took too long, switch back to production site to reattempt.
from then onwards, connection problems bla bla bla

Question time. Hard disk problematic causing failure in another hard disk and the CPU??

Second question. Only one failover system? That's it?

The other interesting thing to keep in mind here is that the explanation given focuses so much on the failure of a single harddisk. So what happened to the RAID array? Unless the attempt by the RAID controller to rebuild the data on the new disk went haywire as well.

The other thing of note also is the need to activate the backup systems manually, especially considering that such real time systems would have automated failovers.

For Christ sake we're talking about stock exchange system here, either they're really dumb which we should be proud of ourselves as the IT guys at SE not even half of our knowledge on how to implement a proper real time online system or they're stupid enough to believe that they would get away with this kind of excuse... Malaysia really whatever also boleh...

This post has been edited by hpteh: Jul 5 2008, 09:22 AM

While the market is dropping, let's discuss which stock is worth buying :-D

I SUGGEST RESORT & BJTOTO.....

this is not a entry level Dell Server, the trading system are running on mainframe class hardware, i find it really hard to believe a single failed harddisk cause this problem, even if some of the disk and CPU is down it can still keep on running, also BNM have requirement all financial company have DR site that can take over within a few minute, this is not available here, the DR site is not working properly, does BURSA have the same requirement as bank and financial institution to perform regular DR trial and report the result to BNM, if so there maybe someone that lie in the report. If there is no requirement for Bursa to have have hot standby DR system i will be shock, such a important part of our country economy is at risk.

here is official timeline



http://biz.thestar.com.my/news/story.asp?f...38&sec=business

CIO said they are using HP NonStop server, this is fault tolerate server, all the part in the system is have redundancy there is no single point of failure.

http://en.wikipedia.org/wiki/NonStop

This post has been edited by robertngo: Jul 4 2008, 05:39 PM

robertngo,

A NonStop aka Tandem computer has NO SINGLE point of failure. So, a single hard disk failure cannot bring the system down. In summary, there is something WRONG here.

Dreamer

yah i work with Nonstop server before, that is why i find it very hard to believe the story it said the replacement disk cause other hard disk and CPU to fail which will be really strange.

This post has been edited by robertngo: Jul 5 2008, 09:26 AM

well, that's the first thing that came to mind too, that the reasonings that they've provided for failure's more appropriate for the downtime of entry level servers.

fyire,

In conclusion, some Malaysian IT people are NOT capable of using the the Nonstop server properly.

"First world infrastructure, third world process and mentality"

Dreamer

This post has been edited by dreamer101: Jul 6 2008, 01:43 AM

the BCP plan fail to prepare for the situation of single trading system and recovery was not done immediatly, also they dont have live data replication setup between production and DR site and require more than three hour to sync their production data. This is a big problem for a trading system, if you lose more than three hour time you will be losing entire trading session. Banks have been require by BNM to have live data replication to DR for instance recovery during disaster, why does a vital financial institution like Bursa is not required to do the same??

robertngo,

1) Why do we have whole radar station burn down because nobody check the fuse of the fire alarm??

2) Why do KLIA out of power for 4 hours even though we supposed to have redundant power feed from 2 power suppliers and 2 power networks??

Understand this and you would understand ALMOST everything WRONG about Malaysia. Why Malaysia is NOT going forward in spite of the tremendous amount of resources that we have.

Dreamer

One of the better possibility answer for the root cause.

To be fair, I would change from some Malaysian IT people to Bursa IT people or its vendor that responsible for that as don't know whether they sub-con out or manage their own in term of the computer server issue.

cherroy,

1) In this case, it is BURSA IT people. But, I had seen ENOUGH examples that to know this problem is MORE wide spread than the bursa.

2) Recently, some Alliance Bank branches are out for one or 2 days too.

As per Godfather movie, "Fish rot from the head first".

Dreamer

The SHORT answer is to manage and run ANY kind of complex system, you NEED GOOD EXPERIENCED PEOPLE to support IT. So, essentially, you have 2 choices:

A) Train your internal people and pay them well enough so that they stay technical

Or

B) Outsource to someone else but you still NEED GOOD people to monitor them

Malaysian culture is WE spent a lot of MONEY on the hardware. But, we neither train, pay and keep REAL TECHNICAL people to support and maintain those hardware. There is NO career path in Malaysia for senior technical people with a lot of experience. Some took the short cut and hire ex-pats. But, they REFUSE to hire oversea Malaysians with similar or better experience. The REASON is they think it is EASIER to control the ex-pats. But, that is harmful for the future of the country. In the end, we are in this catch 22 situation that we NEVER get out

A) There are NO GOOD EXPERIENCE TECHNICAL people that deserve very high pay

B) Technical people do not stay technical because they make more money as manager.

In the end, nothing changes and Malaysia cannot go to the next level of IT capabilities.


This post has been edited by dreamer101: Jul 6 2008, 10:41 AM

There are just really basic things from an IT point of view that doesn't make sense in their timeline:

>> Faulty disk replaced

It is now getting standard for servers to have redundancy built-in, especially in the disk storage sub-systems. With RAID arrays, a single or even multiple disk failures don't cause system outages. You can replace those faulty items at some later convenient time, and with hot-swappable/pluggable systems, you don't even need to shut the system down.

>> Backup site start-up process takes longer than expected.

They obviously didn't test this properly and/or regularly enough, or whatever they have implemented is just not appropriate.

Sounds likes there's a few incompetent people in their IT.

I don't know Alliance Bank has such serious issue, never deal with them before either. Very poor indeed.

I think this problem in Malaysia (I don't know elsewhere)

Technical people doesn't being paid well in general as most Malaysia company don't view technical people as a valuable asset. Most of the time we saw experienced technician or engineer up to 10 years of expertise and experience, their wages also can't match the a newly hired manager with not much experience. Experience and expertise is an intangible asset for a company especially in dealing IT stuff which is much more difficult to be replaced easily compared to ordinary manager. (There are some very good manager which is also valuable to the company, what I mentioned is those ordinary manager).

Also, one of most important issue, is that some company culture, promotion is done based on preferential, not because of merit point.

Hardware is there, just how people manage it. This is more on management issue on IT side rather than hardware issue.

This post has been edited by cherroy: Jul 6 2008, 11:24 AM

Ultimately it comes down to people/management. But those high-end, expensive, hardware are not supposed to behave in the way they described.

Bursa's server that crashed was a HP NonStop K-Series which is already obsolete and 2 generations old.

HP released a new line of NonStop S-Series to replace the K-Series back in the late 1990s. In 2008, HP phased out the the S-Series with its new HP's NonStop Integrity line which runs on Intel Itanium processors. (The K-Series and S-Series runs on MIPS processors)

The Edge had an article this week that implies that Bursa stopped investing on the old server because it was replacing the system with a new trading system from Atos Euronext... but this project was delayed for 2 years...so the old K-Series machine had to keep running and finally died



This post has been edited by gilabola: Jul 6 2008, 11:45 AM

Ah, that could explain a lot then...

Use some server that obsolete 10 years ago?

Bursa make hefty profit every year even giving special dividend and capital repayment to the shareholders, in term of financial wise should be no problem for them to invest up to date hardware.

What can the IT people do if the management refuses to invest ? Or how could they test the new softwares effectively with the very old hardwares. ( I am a nobody in Tech )

Initially the CEO blames it on IT people, as to cover himself.

This post has been edited by SKY 1809: Jul 7 2008, 08:43 AM

my god the K-series, i see this system only once in one of my previous company DR site, even the S-series is consider a old machine.


They have quite a large budget for IT about 60 million a year, why is this not done earlier?

This post has been edited by robertngo: Jul 6 2008, 06:54 PM

If I'm not mistaken, each K-Series processor has only a max of 128 MB memory... even my PC has more processing power and memory

this server is not about speed, it is about ensuring the integrity of every transaction, a process is executed on two CPU on the same time, if one CPU is down, the process can still complete on the other CPU. This design make nonstop server very suitable to financial transaction where the integrity of the processing result is more important that the speed. That said, they really should have upgraded the server years ago, who is the one that approved the plan to keep running their critical trading system on a obsolete platform.

I'll give you an example for you to compare for you to see why your reasoning here isn't very accurate.

At one of my client's place, their old billing system runs on a Digital Alpha machine, and is right now, 12 years old. It still runs perfectly for the purpose that it needs to have. And they're still making use of it right now for their secondary market, as they do not trust their new system to be able to handle multiple markets.

Now, the new system runs off a HP SuperDome, with all the high spec stuff and so forth, but the stability of the system is something that really constantly gives the support ppl sleepless nights. There's nothing wrong with the hardware, but its the software part that they do not trust.

This is a rather good example that such hardware despite being old can still very much run properly without any issues. Don't forget that these are not PC quality hardware which you can expect stuff to start breaking down after 2-3 years.

To put it in layman terms in regards to the hardware specs: I had visited my aunt's office once a few years back, and there's this PC there that's still a 486. But surprise surprise, Microsoft Word 6.0 on Windows 3.1 running on that 486 is still faster than Microsoft Word 2003 running on Windows XP, on a Pentium IV. It is for that reason why they've not have the need to upgrade or replace that PC as yet, unless it breaks down on them.

---

Added on July 7, 2008, 12:13 amWell, such systems tends to be rather customized, as well as certified only to certain ranges of hardware. To get new hardware will require a re-certification of the old system for the new hardware, of which the cost will be difficult for them to justify should they be planning an upgrade of the entire system. In a way, the delays in the rollouts of the new system can also be partly to blame for this.

This post has been edited by fyire: Jul 7 2008, 01:34 AM

They are changing to a new trading system together with new hardware soon, the project cost like 60+ million, let hope they will have disaster recovery plan work out properly this time.

robertngo,

If I am NOT mistaken, each process is executed on 3 CPUs. The result is voted on the egress where the majority wins. If one CPU is different than the other 2, it is voted out and detected to be faulty.

<<it is about ensuring the integrity of every transaction>>

You CANNOT ensure the integrity of every transaction with ONLY 2 CPUs. You need at least 3 CPUs to do that. With ONLY 2 CPUs, if the results are different, you cannot tell which CPU has the correct result.

<<That said, they really should have upgraded the server years ago, who is the one that approved the plan to keep running their critical trading system on a obsolete platform.>>

What does that tell you about the Bursa IT system?? It is an OLD system. It should have VERY STABLE process and procedure to handle disaster recovery. People have more than 10 years to work out the problems. But, they cannot do it. So, what makes you think the NEW system with LESS TIME to work out the process and procedure will be BETTER??

It is the PEOPLE that make the system succeed or fail.

Dreamer

Insanity: doing the same thing over and over again and expecting different results.
Albert Einstein

This post has been edited by dreamer101: Jul 7 2008, 05:02 AM

this is true for the new NonStops that run on Itanium. The old NonStops have only 2 cpus that run in lock step.