topic Question about Arrays in GDL? in GDL

Question about Arrays in GDL?

Nader Belal — Mon, 26 Sep 2022 20:49:10 GMT

Hi there,

I would like to ask if there was a maximum size for arrays (either parameter or variable) for 1D and 2D arrays ?

And if there was any, how big it can get? And what is the safe limit that I should not break for safe usage ?

And if Array type (integer, real, string, etc) have an impact on its size ?

Thank you in advance

Re: Question about Arrays in GDL?

Peter Baksa — Tue, 14 Jul 2020 10:44:36 GMT

The upper limit for an array index is currently 2147483647.
The size in memory depends on the data type, but with huge amount of data you will probably have speed issues sooner than being out of memory.
A safe limit is hard to declare, it also depends on how complex the model generated from the data is.

Re: Question about Arrays in GDL?

Nader Belal — Tue, 14 Jul 2020 11:38:30 GMT

@Peter Baksa

Thank you for your quick reply.

by Peter Baksa
The upper limit for an array index is currently 2147483647.

Is the 2147483647 indices are for single column or total ?

by Peter Baksa
The size in memory depends on the data type, but with huge amount of data you will probably have speed issues sooner than being out of memory.
A safe limit is hard to declare, it also depends on how complex the model generated from the data is.

I was expecting something in that direction, and I know that it's difficult to give an accurate answer for such a complex question, I just need something as a point of reference.

By the way, how much would impact each data type for arrays on its size in memory ?? In simple multipler number, also as a reference

Thank you in advance

Re: Question about Arrays in GDL?

Peter Baksa — Tue, 14 Jul 2020 16:59:33 GMT

2147483647 for a single column, but an array[2147483647][2147483647] will be surely out of memory.
Consider all numeric types the same size, not much optimization can be done on GDL side.

Re: Question about Arrays in GDL?

Nader Belal — Tue, 14 Jul 2020 23:03:27 GMT

Thank you Peter

Please check my calculations,

So an array with 2,147,483,647 items, for numeric type supposing (integer or array) usually will occupy 4 bytes per item, which will make almost 8 GB ...

That is impressive ....

And if it was string, i would say it would be variable per item with an average 64 byte per item (total speculation from my part)

So if my numbers were true,
The safe bet for a median spec. pc, (let say a Ryzen 1700 + 16GB Ram) with 1 instance of the special GSM file in an almost empty file, with no operation associated with it would be 214,750,000 items assuming they are numeric (total), outside of these specs, and light work in GDL would be 21,475,000 items.

But if we have to consider all conditions, PC specs, project size, array type, and other criteria, the array size should not go beyond the 215,000 items.

Am I correct !!??

Re: Question about Arrays in GDL?

Kristian Bursell — Tue, 14 Jul 2020 23:41:13 GMT

I am curios about that number as I am getting errors once the value of my 2D arrays goes beyond 32,627; not sure if that is the exact number as it was given to me 4 years ago when i hit the limit.

I'm populating arrays with data from XML files to generate catalogues within an object. When the catalogues get larger and go over 32,627 resulting array cells, we get a "stack error" when importing it into the array.