insufficient memory allocation

[SAS User]

A colleague is trying to run the following proc iml in SAS v6.12
on UNIX/Solaris:

Proc iml;
...
A=diag(Pi)*(I(&n)-diag(Pi)); **Here Pi is a 1 times 1440 vector.


And getting the following error:

Worksize = 262128
Symbol size = 262128
NOTE: IML Ready
ERROR: (execution) Unable to allocate sufficient memory. At least 16588832
more ^MERROR: (execution) Unable t$
bytes required.^M bytes required.^M bytes required.


It has been so long since I have had to deal with large datasets,
that I've forgotten all the tricks o' the trade. We do not have
any user-restraints here, everyone just sharing the system as they
need (wrt space/cpu/etc). The SAS website says that there are no
fixes available and that the ERROR is actually wrong. Any suggestions?

Thanks in advance,

casey

[Real SAS User]

Post by SAS User
A colleague is trying to run the following proc iml in SAS v6.12
Proc iml;
...
A=diag(Pi)*(I(&n)-diag(Pi)); **Here Pi is a 1 times 1440 vector.
Worksize = 262128
Symbol size = 262128
NOTE: IML Ready
ERROR: (execution) Unable to allocate sufficient memory. At least 16588832
more ^MERROR: (execution) Unable t$
bytes required.^M bytes required.^M bytes required.
It has been so long since I have had to deal with large datasets,
that I've forgotten all the tricks o' the trade. We do not have
any user-restraints here, everyone just sharing the system as they
need (wrt space/cpu/etc). The SAS website says that there are no
fixes available and that the ERROR is actually wrong. Any suggestions?

In SAS: OPTIONS MEMSIZE; I'd also suggest NOOVP.

In Unix: man ulimit.

--
Charming man. I wish I had a daughter so I could forbid her to marry one...

[David L. Cassell]

Post by SAS User
Proc iml;
...
A=diag(Pi)*(I(&n)-diag(Pi)); **Here Pi is a 1 times 1440 vector.

But what your colleague is doing in this line requires working with
(at least one) 1440x1440 matrix! At 8 bytes per number, that's
16,588,800 bytes. Per matrix. Which is really wasteful here, since
this can be done in a less computational manner. You're only working
with the diagonal, so you cna do this as a vector calc, instead of a
full matrix calc.

Post by SAS User
Worksize = 262128
Symbol size = 262128
NOTE: IML Ready
ERROR: (execution) Unable to allocate sufficient memory. At least

16588832 more

Post by SAS User
. . .

So now you can see where that '16588832' could be coming from.

You may want to try starting PROC IML with a big WORKSIZE= value:
proc iml worksize=100000;
would request 100,000 K available for the WORKSIZE option. But IML
is supposed to allocate this stuff dynamically, so I have to wonder if
your colleague hasn't already clogged up all possible RAM before trying
to compute 'A'. So here are a few PROC IML tips you might suggest to
your IML'er:

Use FREE to free matrices you're no longer using.
Use STORE if you have a large matrix you won't need for a bit, and then
FREE
up its storage space.
Re-design your matrix calcs so you don't need such massive matrices.

I personally recommend #3 there.

HTH,
David
--
David Cassell, CSC
***@epa.gov
Senior computing specialist
mathematical statistician

[Dale McLerran]

Well, I can't inform you as to whether the error message is
wrong or not. However, I would note that the code below is
quite inefficient, both in terms of memory utilization and in
terms of performing a lot of unnecessary addition and
multiplication operations. A better approach would be to write:

A = diag( Pi # (1 - Pi) );


For Pi={.3, .6}, the above would construct the temporary matrix
temp1 = 1 - Pi = {.7, .4} and then compute a second temporary
matrix temp2 = {.3*.7, .6*.4} = {.21, .24}. Finally, it expands
temp2 into a 2x2 matrix with zeros off the diagonal. Note that
there are two subtraction operations and two multiplication
operations performed.

Contrast this with the code below which constructs temporary
2x2 matrices for diag(Pi), I(2), diag(Pi), and I(2) - diag(Pi).
The I(2) - diag(Pi) performs n^2 subtraction operations.
Temp1*temp2 performs n^3 multiplication operations and (n-1)*(n^2)
addition operations in addition to constructing three diagonal
matrices instead of one.

Note that if A is a diagonal matrix, then it may not even be
necessary to keep more than just the diagonal portion. Down
the road where A is referenced, there may need to be some
changes in coding much like I have demonstrated for the
construction of A. These further changes may provide additional
savings in terms of memory requirements as well as in terms
of arithmetic operations. Thus, your colleague may be best
off just leaving A as Pi#(1-Pi).

With n of length 2, there is not a great savings in the code
I offer, but for n=1400 there is tremendous savings. As there
are fewer matrices of size n^2 hanging around, your memory
requirement is less, too. Perhaps this change in code will
save the day. Still, it seems that this is a problem of rather
trivial size. Each square matrix requires about 16mb. With
todays computers, that is not all that much. Maybe in a
multiuser environment, there are constraints on how much
memory can be allocated to any one user?


Dale

Post by SAS User
A colleague is trying to run the following proc iml in SAS v6.12
Proc iml;
...
A=diag(Pi)*(I(&n)-diag(Pi)); **Here Pi is a 1 times 1440 vector.
Worksize = 262128
Symbol size = 262128
NOTE: IML Ready
ERROR: (execution) Unable to allocate sufficient memory. At least 16588832
more ^MERROR: (execution) Unable t$
bytes required.^M bytes required.^M bytes
required.
It has been so long since I have had to deal with large datasets,
that I've forgotten all the tricks o' the trade. We do not have
any user-restraints here, everyone just sharing the system as they
need (wrt space/cpu/etc). The SAS website says that there are no
fixes available and that the ERROR is actually wrong. Any
suggestions?
Thanks in advance,
casey

=====
---------------------------------------
Dale McLerran
Fred Hutchinson Cancer Research Center
mailto: ***@fhcrc.org
Ph: [phone]
Fax: [phone]
---------------------------------------

__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com