Re: 4-way Opteron vs. Xeon-IBM X3 architecture

George Macdonald <fammacd=!SPAM^nothanks@xxxxxxxxxxxxx> wrote:
> On Thu, 29 Dec 2005 19:19:38 +0000 (UTC), David Wang <foo@xxxxxxxxxxx>
> wrote:

> >George Macdonald <fammacd=!SPAM^nothanks@xxxxxxxxxxxxx> wrote:
> >
> >> Maybe but IF YOU"D ONLY READ, there's more than THG - this stuff is also
> >> common knowledge in discussion forums. OTOH Micron/Crucial, among other
> >> mfrs, for what is now a small premium, is producing PC4200/DDR500 DIMMS for
> >> the enthusiast market; so the server market is asleep?... err, not
> >> enthusiastic enough to pursue this?:-) If this DDR500 needs 2.8V... so
> >> what?... specs can be changed to adapt to evolution of silicon.
> >
> >1. I did not see any concrete reference other than to THG. If you have
> >anything other than citing "common knowledge from the enthusiast market"

> While it may suit your agendum, that is not a quote of what I said.

My "agenda" is only to report facts and data. The facts on the ground
is that I have not seen any references that has shown that you can
reliably add more memory to the Opteron system @ 400 Mb/s.

The point here is that the issue concerns both speed AND capacity.
The references given suggest that higher speeds are possible, but
none shows that higher speeds are possible with full capacity.

> >then automatically applying it to the server market, please reference
> >that.

> I did not give any "concrete reference" URL... not even to THG - that was a
> casual mention of one I remembered among several I've seen and I do *not*
> keep bookmarks for such things... as common as they are. You need to get
> out more - the "common knowledge" is in discussion groups all over the
> place - I visit several Web Forums to gather info and troubleshoot and I've
> seen such discussion of memory configs in almost all.

As you may suspect, I read plenty about memory systems, and I would
vigorously challenge your memory in regards to the "common knowledge"
here. I do not believe it's common knowledge for anyone to have
demonstrated running 64 DDR(1) SDRAM devices reliabliably @ 400 Mb/s.
If you can, please do cite a concrete reference URL. What you remember
to have seen may not actually be what it is.

> >2. I don't read THG, so I am not familiar with details of their test.
> >However, there is a difference between "memory targeted for the enthusasists"
> >and "memory targeted for servers". For one thing, servers use RDIMMS
> >while desktop folks use UDIMMS. The difference is that servers tend
> >to go for capacity, so they'd want to load up on the ranks, and you'll
> >usually see 16 x4 devices crammed into a single rank, then the DIMM
> >provides the buffering to electrically isolate the load.

> With the right devices and with registering the server market should be
> able to do better - that's all I'm saying. There's nothing about an x4
> device which would prohibit making the higher speed versions in x4 form.

The issue that prohibits making the higher speed version with x4 devices
is that you have to hang 16 of them (18 with ECC) on the same address
and command busses per rank. That's a rather heavy electrical load to
run @ 200 MHz. So, no, you can't just look at "enthusiast memory" built
with x8 parts and automatically assume that x4 parts will work just the
same at the same data rate, because you're going to need even faster
parts to meet the same timing.

> > The "enthusiast
> >memory" have to go for speed, so AFAICT, they don't use 16 x4 devices
> >in any single rank, and there's no buffering there. So even if THG or
> >whatever website may have succeeded in running 4 ranks of DDR(1) devices
> >@ 400 Mb/s, we have to ask the next question, "What is the configuration
> >of each rank?" If those are not fully loaded 16 device ranks, then the
> >capacity per channel is still limited to 32 devices @ 400 Mb/s.

> Ah so the servers don't "go for speed"? What I'm saying is that with the
> devices available now I don't see why they could do better. You know
> damned well that UDIMMs are 8 devices per rank so why do you have "to ask
> the next question"?

Because you cited some rather nebulous references in regards to memory
from the enthusiast market and assumed that it would work in the server
world. I was simply pointing out that's not going to work here because
of configuration differences and electrical loading considerations.

> > Even if you show that it is
> >capable of running DDR(1) with 64 devices @ 400 Mb/s on Opteron/A64,
> >the next question to ask is how many simulation and test hours do you
> >have behind that configuration. The point here is that no one is doing
> >that sort of study to get DDR(1) to work @ 400 Mb/s in the big servers,
> >because DDR2 is here, and all of the energy is going into qualifying
> >those parts and getting the 64 device configurations to run @ 400 Mb/s.

> Ah now we have it: "conservative":-)... that's my complaint.;-)

Which is what is shipping in HP's Opteron server, and guarenteed to
work by HP. That guarentee provides the effective upper limit to
the maximum memory capacity of an Opteron server as of today. That
limit cannot be exceeded or changed arbitrarily. This, I believe was
the crux of the contentious point. . .

> >> >Getting 4 ranks of DDR(1) to work @ 400 Mb/s probably doesn't take
> >> >rocket science, as I am reasonably certain that I am using it in the
> >> >Intel based box that I sit in front of right now. The difference is
> >> >in guarenteeing that it'll work with sufficient margins and reliability
> >> >that HP or anyone would feel comfortable in putting their name on it...
> >> >in a server.
> >
> >> Let's hope someone else might get more umm, enthusiastic.:-)
> >
> >That's the enthusiast market, not the server market. I believe the
> >topic of discussion is the 4P Opteron box and its memory capacity.
> >If gaming ethusiasts start shelling out the dough to cram 16 GB of
> >memory into those boxes, then some vendor will certainly step up, do
> >the required engineering support and get it to work, but until then,
> >the Opteron is limited to running DDR(1) with only 32 devices @ 400 Mb/s,
> >because that's the maximum amount of memory that HP is willing to
> >stand behind at that datarate.

> No, the topic was *not* limited to 4P Opteron box, not that it's make much
> difference with Opteron anyway. I'm not sure how you're counting your "32
> devices" but HP only makes the rules for its boxes... not Opteron in
> general.

32 devices is not counting ECC. 36 counting ECC.

The limit is 2 ranks of 18 x4 DDR(1) devices running @ 400 Mb/s, and
that's the same number I am seeing over and over again. Not just HP,
but Tyan as well. So the limitation isn't just HP Opterons, or even
Opterons of any brand of servers, but DDR(1) SDRAM memory controllers
@ 400 Mb/s. The Opteron just happen to have a DDR(1) SDRAM memory
controller that has to follow the same constraints as everyone else.
If you claim that the limit can be exceeded, please show me where
you're getting your impression from, because I'd certainly like to
see where someone is getting a fully loaded DDR(1) memory system to
run @ 400 Mb/s. **

** By fully loaded, I mean 4 ranks of 16 x4 devices, for a total of
64 devices (not counting ECC) per channel. With 1 Gbit devices, you'll
get 8 GB of memory per channel, and 16 GB of memory per Opteron
processor. In a 4P config, that would push your total memory
capacity to 64 GB instead of the current limit of 32 GB @ 400 Mb/s.

> >> >HP was being conservative and drops the capacity down to 2 ranks when
> >> >running DDR(1) @ 400 Mb/s. That shouldn't be a surprise. IMO, seeing
> >> >DDR(1) @ 400 Mb/s in a server is itself a bit of a surprise,
> >> >since DDR(1) @ 400 Mb/s is really "overclocked" memory with the 2.6 V
> >> >spec.
> >
> >> DDR(1)400 is no more of an overclock than PC100 was over PC66 and PC133
> >> after that - it's just progress and enhancement of the processes... happens
> >> all the time.
> >
> >There are sets of baseline spec's for SDRAM, DDR SDRAM, DDR2 SDRAM,
> >respectively, and DDR(1) 400 is a push product that broke the
> >baseline spec for DDR SDRAM. PC133 (and even PC166) uses the same
> >voltage specs as PC100. DDR(1) 400 does not use the same set of
> >spec's as DDR 266 or DDR 333.

> So a recommended .1V difference for DDR400 makes it overclocked and a
> different "set of specs"?<boggle> As I've already tried to point out the
> "baseline spec's" for DDR are >3 years old - if you're going to deny the
> ability to push as the silicon opportunity presents, I have to ask why? We
> "allow" that CPUs, GPUs, etc. increase voltage and speed over a silicon
> design lifecycle... why not memory?

The difference is that CPU's and GPU's are single vendor to single customer
parts. That is, Intel can change the specs of these devices to whatever
it wants, whenever it wants, as long as its customers doesn't mind
following the new spec. Same with AMD, NVIDIA, ATI, etc. DRAM doesn't
work that way. The dynamics of the commodity market means that the parts
are supposed to be completely interchangeable. So the "interchangeable"
aspect of things greatly limits the standards definition process.

For example, Samung can probably crank out much faster DDR parts because
it has excellent process tech, but some of the less-well funded fabs
can barely meet the spec, and they would be hard pressed to meet these
push spec parts, so they would be resistant to changes in the JEDEC
standards definition.

The limitation of the JEDEC standard means that the faster guys can't
really run ahead of the slow guys, although they're finding some ways
around that with the push spec parts designed for the "enthusiast
market". So, no, you can't just take advantage of opportunities
made available with faster process technology to make your own faster
DRAM parts. You have to wait until sufficient number of DRAM manufacturers
can agree with you on the new addendum to the spec, and a suffient
number of design houses (Intel, IBM, Sun, AMD, etc etc) agree to the
same set of proposed addendum to the spec, before the standard can be
created, and you can sell you part as (JEDEC) DDRx xyz MHz, and customers
can be reasonably certain that your DDRx xyz MHz parts can operate
interchangeably with parts from Infineon, Samsung, Micron, Elpida,
etc.

> >> > The 400 Mb/s bin was supposed to have been reserved for DDR2,
> >> >but market demand from the desktop segment pushed 400 Mb/s down into
> >> >DDR(1) territory and made it happen. So no other server that I know of
> >> >use DDR(1) @ 400 Mb/s, and they're all waiting for DDR2 to run 4 ranks
> >> >@ 400 Mb/s.
> >
> >> Not expecting some evolution and progress in functional specs is a bit
> >> myopic IMO. No need to slavishly follow specs which are >3 years old.
> >
> >If you're selling servers and your reputation depends on the reliability
> >of said servers, you too would gladly follow specs regardless of their
> >age. If you're pushing out a product that has better spec than others
> >in the market, that's great, but I'd want to know how many test hours
> >you had behind it. I wouldn't want to hear that you just came up with
> >the product with Micron's help last week.

> Oh you mean like FBDIMMs... with AMBs which "will brown a burger better
> than a George Foreman Grill can" [quote from this NG]?:-)

FBD's have been in developement for more than 2 years, and they're still in
development/testing/tweaking.

They'll enable servers with incredible amount of memory, and the power
headache that comes with it. The AMB is just part of the problem. With
16 device per FBD, you can get the ratio of AMB device power to DRAM
device power down to 15~20%.

As we've discussed, the practical limit for DDR(1) @ 400 Mb/s is 32 devices
per channel, and for DDR2 devices it's 64 devices @ 400 Mb/s per
channel. Each channel will cost you about 100 control/data pins.

FBD's can get you 256 devices per channel with much fewer pins. You can
basically hang 10x more DRAM bits per pin. Now imagine a memory system with
10x more devices, and the amount of power that memory system can consume.
AMB is a (relatively) small problem.

--
davewang202(at)yahoo(dot)com
.

Relevant Pages

  • Re: 4-way Opteron vs. Xeon-IBM X3 architecture
    ... I did not see any concrete reference other than to THG. ... >reliably add more memory to the Opteron system @ 400 Mb/s. ... >> With the right devices and with registering the server market should be ... >following the new spec. ...
    (comp.sys.ibm.pc.hardware.chips)
  • RE: server problems
    ... This newsgroup only focuses on SBS technical issues. ... >Thread-Topic: server problems ... >> web proxy service or SQL Server will normally use large memory. ...
    (microsoft.public.windows.server.sbs)
  • Re: Allocated Memory alerts
    ... If you do not encounter any real server performance issue, ... the high memory usage is expected. ... The Health monitor is running on the SBS server. ... A memory allocation threshold is configured on the particular monitored ...
    (microsoft.public.windows.server.sbs)
  • Re: FTS Performance in SQL 2005
    ... I don't think you have left enough memory for the OS and MSSearch. ... Looking for a SQL Server replication book? ... Looking for a FAQ on Indexing Services/SQL FTS ...
    (microsoft.public.sqlserver.fulltext)
  • Re: Allocated memory alerts and changed threshold still doesnt si
    ... multiple alerts on a daily basis. ... week or so of server operation without a restart. ... have to check which process used high CPU and Memory utilization. ... please refer to the following Microsoft Knowledge Base ...
    (microsoft.public.windows.server.sbs)
Loading