topic Re: Point cloud - Too many points in Project data & BIM

Point cloud - Too many points

Anonymous — Tue, 30 May 2023 10:39:16 GMT

I may have pushed the limits of how many cloud files to import in a blank ARCHICAD file in attempts to find a workflow on how to use them.

There was a total of 8 files e57 with roughly 10GB of Data. Are xyz files better?

I made an independent layer for each scans so I can turn it on and off easily, however I wanted to test to see what would happen when all are on and open the 3D view. In case I didn't know ARCHICAD and turned everything on.

The file just became unusable as it was in a permanent frozen state, or may take 20 minutes or more to respond, this was expected....

Are there better ways to manipulate the information besides layer management?

Is there a way to disable Turning on all layers, or specific layers?

I was thinking I could make a renovation status for each scans and use the pin option for each scan. See attachment.

Other thoughts on what others are doing would be greatly appreciated...

Re: Point cloud - Too many points

Laszlo Nagy — Mon, 25 Jan 2016 14:25:18 GMT

Well, it seems like you are really pushing the limit here.
10 GB is 10 000 000 000 bytes, which I think is probably in the range of 100-200 million individual points in those Point Clouds (if we assume that 1 point takes 50-100 bytes of data).
I don't think it matters whether it is an E57 or XYZ file, both are converted to a Point Cloud Object file when imported into ARCHICAD.

I was thinking about this thing for a while since it is a thing that can definitely bring ARCHICAD, or any BIM application for that matter, to its knees.

What I think would be useful is for GRAPHISOFT to implement some kind of technology using which it would be possible to make the point cloud less dense. This would mean that, for example, only every second point in the point cloud file is imported. Or every 3rd or 4th. This could be set by the user during the import process. This way a very large point cloud file could be made much lighter and more manageable.

Handling XYZ files in this regard is easier than handling E57 files since XYZ is a textual file format it which each line represents one Point of the Point Cloud.
If you had some utility program which could take an XYZ file and delete every 2nd line in it, then same it under a new name, then using this method you could manually reduce the size of those Point Cloud files, then import those.
Or you could delete 3 lines, keep every 4th line and then save it as a new XYZ file, that would decrease the number of points in each file by a factor of 4.
I think this method should/could work, probably you would still get the geometry, there would only be fewer points everywhere. So you wouldn't have to worry about losing parts of your point cloud geometry by reason of deleting so many points.

It would be interesting to know which component of the computer is choking when you try to display them in 3D. During the generation of 3D while it seems frozen, you could check how much memory is used, because if the 16 GB is not enough ARCHICAD swaps data to virtual memory on the hard drive, which will slow it down to a crawl. This can be remedied by putting in another 16 GBs of RAM.
Or it is the CPU? Do you see the CPU running at 100%. If you do it is probably working as hard as it can but this is how fast it can process all that data. This can be remedied only by a new CPU but even that will not increase the speed as much (and may require buying a new computer, which is not a really good option).

Re: Point cloud - Too many points

Erwin Edel — Mon, 25 Jan 2016 14:35:07 GMT

I've got zero experience with pointclouds, but I know there is software that can 'vectorise' the data, resulting in much smaller models. www.pointfuse.com for example.

Sadly, another piece of software to buy. Hopefully in the future ArchiCAD will be able to reconize simple geometry like planes and tubes, rather than having to work with millions of points.

For now we are sticking to the old way of single point laser measurement, along with ye olde tape measure in our office.

Re: Point cloud - Too many points

Gorazd — Tue, 26 Jan 2016 07:32:57 GMT

I had a good result with free open source application Cloud Compare (http://www.danielgm.net/cc/). You can open your pointcloud and reduce the number of points. I had a 1 by 1 milimeter dense pointcolud, that was too much for ArchiCAD. I reduced it to 1/10th of the size. One of pointclouds was also without RGB information, just intensity, so all points in ArchiCAD turned black. With Cloudcompare I managed to get at least grayscale information.
Hope it helps.

Re: Point cloud - Too many points

Erwin Edel — Tue, 26 Jan 2016 12:32:49 GMT

That sounds very interesting, Gorazd.

Re: Point cloud - Too many points

Anonymous — Fri, 29 Jan 2016 13:19:00 GMT

Thank you for the information Laszlo, I will try and get smaller files and see how the xyz format works.

I also will try Goraz suggestion.

To make things a lot worst the cloud files I have have a crazy origin point because we started off the model with Civil coordinates which puts our model x=64572' and y= 252831' from the origin point.

Placing point clouds can be challenging as one has to set all the correct parameters under the object tool first (layers, rotation, and height maybe too?). So that the object doesn't rotate or move up or down.

To respond to the software freezing: Need to retry it and document it.

I agree that I need to find a way of filtering the cloud information of what is needed for the import into ARCHICAD for best results.

Gorazd wrote:
I had a good result with free open source application Cloud Compare (http://www.danielgm.net/cc/). You can open your pointcloud and reduce the number of points. I had a 1 by 1 milimeter dense pointcolud, that was too much for ArchiCAD. I reduced it to 1/10th of the size. One of pointclouds was also without RGB information, just intensity, so all points in ArchiCAD turned black. With Cloudcompare I managed to get at least grayscale information.
Hope it helps.

laszlonagy wrote:
Well, it seems like you are really pushing the limit here.
10 GB is 10 000 000 000 bytes, which I think is probably in the range of 100-200 million individual points in those Point Clouds (if we assume that 1 point takes 50-100 bytes of data).
I don't think it matters whether it is an E57 or XYZ file, both are converted to a Point Cloud Object file when imported into ARCHICAD.

I was thinking about this thing for a while since it is a thing that can definitely bring ARCHICAD, or any BIM application for that matter, to its knees.

What I think would be useful is for GRAPHISOFT to implement some kind of technology using which it would be possible to make the point cloud less dense. This would mean that, for example, only every second point in the point cloud file is imported. Or every 3rd or 4th. This could be set by the user during the import process. This way a very large point cloud file could be made much lighter and more manageable.

Handling XYZ files in this regard is easier than handling E57 files since XYZ is a textual file format it which each line represents one Point of the Point Cloud.
If you had some utility program which could take an XYZ file and delete every 2nd line in it, then same it under a new name, then using this method you could manually reduce the size of those Point Cloud files, then import those.
Or you could delete 3 lines, keep every 4th line and then save it as a new XYZ file, that would decrease the number of points in each file by a factor of 4.
I think this method should/could work, probably you would still get the geometry, there would only be fewer points everywhere. So you wouldn't have to worry about losing parts of your point cloud geometry by reason of deleting so many points.

It would be interesting to know which component of the computer is choking when you try to display them in 3D. During the generation of 3D while it seems frozen, you could check how much memory is used, because if the 16 GB is not enough ARCHICAD swaps data to virtual memory on the hard drive, which will slow it down to a crawl. This can be remedied by putting in another 16 GBs of RAM.
Or it is the CPU? Do you see the CPU running at 100%. If you do it is probably working as hard as it can but this is how fast it can process all that data. This can be remedied only by a new CPU but even that will not increase the speed as much (and may require buying a new computer, which is not a really good option).

Re: Point cloud - Too many points

Anonymous — Fri, 29 Jan 2016 13:25:44 GMT

Yes, we do not scans building either and use the laser measurer and tape as well.

For this particular project the CM does scans of the existing conditions as part of their coordination process and I happen to be given the scans in order to coordinate some existing conditions.

Erwin wrote:
I've got zero experience with pointclouds, but I know there is software that can 'vectorise' the data, resulting in much smaller models. www.pointfuse.com for example.

Sadly, another piece of software to buy. Hopefully in the future ArchiCAD will be able to reconize simple geometry like planes and tubes, rather than having to work with millions of points.

For now we are sticking to the old way of single point laser measurement, along with ye olde tape measure in our office.

Re: Point cloud - Too many points

Frank Beister — Thu, 06 Oct 2016 12:57:09 GMT

Are there experiences about #of nodes/clouds and clouds/project to keep performance alive?

-> Wishlist: Imported clouds are objects. Objects can be intelligent. Optional cropping of data extension and density inside of the object. Or deviding import-cloud into several Objects with low density, managed by a main object, which switches individual density higher.

Re: Point cloud - Too many points

Laszlo Nagy — Sun, 30 Oct 2016 22:09:10 GMT

I just learned about a free Point Cloud processing software called CloudCompare which may help you with this:

http://archicad-talk.graphisoft.com/viewtopic.php?t=54196

Re: Point cloud - Too many points

Anonymous — Tue, 01 Nov 2016 20:22:03 GMT

Cool, thanks a lot for that info

laszlonagy wrote:
I just learned about a free Point Cloud processing software called CloudCompare which may help you with this:

http://archicad-talk.graphisoft.com/viewtopic.php?t=54196

Re: Point cloud - Too many points

Frank Beister — Wed, 02 Nov 2016 07:40:40 GMT

Thanks for the reply. I will have a look on it. Sounds very helpful.

Re: Point cloud - Too many points

Da3dalus — Wed, 16 Aug 2017 02:40:01 GMT

Using ArchiCAD 21, I'm just finally getting into some Point Clouds. Unfortunately, the first attempts have been riddled with problems. Too many points is the big one, and I'm going to try CloudCompare.

Besides that, the first XYZ files we received had no RGB data, just XYZ. ArchiCAD is pick and won't accept that. The next file had no .XYZ extension, and in the Mac, there's a less-than-obvious trick to adding useful file extensions. Then the Surveyor added the color, but the file was comma-delimited. ArchiCAD only recognizes space or tab-delimited. After all that, still too big.

By the way, I have opened these files in a text editor. I use TextWrangler on the Mac. FYI, a 600 MB XYZ file has about 18 million points. That might be a useful ratio: about 30,000 points per MB.

I have no idea what the optimum size for ArchiCAD is, especially with the 16 GB ceiling on most Macs. It would be useful for GS to publish.

Re: Point cloud - Too many points

Da3dalus — Wed, 16 Aug 2017 15:59:28 GMT

CloudCompare is they key to making point clouds work!

I was able to easily open the file, and after fiddling for a bit, I resampled the resolution and previewed several settings, settled on 1,000,000 points, saved back into XYZ, and changed it to space-delimited, just like ArchiCAD wants it. Works like a dream!

I think I can do more than a million, but that will need some experimentation.

Re: Point cloud - Too many points

ruivenda — Sun, 28 Oct 2018 12:28:03 GMT

Hi all.

I'm using point clouds in AC since AC20, with some nice results and a good workflow to follow. I will explain how I use them.

First I do the site job - laser scanning and then point cloud registering - normally the surveyor does that.

Then I ask for an e57/xyz file with all the information available. In these case I have the full resolution of the point cloud in order to mesure it or evaluate it further. Here there are several pieces of free software to evaluate these things - Recap, Leica, Faro, or even Cloud Compare (the one I personally use). Recap files are also good, because they have those 360 photos so we can work and evaluate things from the photo and not from the point cloud itself.

Normally I get very big files, with 20GB, 50GB or even 85GB. These files contains all the information, unfiltered so we can evaluate them. Cloud Compare can struggle with files that big, but there are alternatives, and in last resort I ask for a file that will be a little bit smaller, so I can open it in CloudCompare and work with it.

In CloudCompare I'll do 4 things: fuse all the individual scans, that come with the xyz/e57 file, in order to get rid of the comum points and get a file a little bit smaller just by that.
~~[list=2]~~
then I delete all the information that is not needed, like far surroundings and other information that is irrelevant to the job to be done.
~~[list=3]~~
then I try to level the point cloud by a known level ground plane - sometimes it's tricky but is very easy to get it done, by selecting 3 points defining a horizontal plane.
~~[list=4]~~
last I prepare the file to import to AC, subsampling it. I first subsample the file to 1mm density, that for a 5mm density, then to 10mm density and so on until I get a file around 1GB of file size
With that done, I manage to import the point cloud into AC, with a reasonable file size and responsiveness that allow me to develop the 3d modeling process into AC.

In AC I use plan region settings in the point cloud to make several heights in floor plan - normally small bands of 100mm thick is all we need at a good height. And for sections and elevations the trick is the same: small depths, so we can see "lines" instead of a lot of noise that do not help modeling the building.

Yes is a process that is time consuming but at least we manage to use point clouds in AC in a way that is helping and not disturbing. If the subsampled file lost valuable information, I get back to CloudCompare, section a little bit of a higher density point cloud and export just that little bit of the cloud, and then import just that fragment to AC, on top of the previous one, or on top of the already modeled part of the building, and so on.

My main complain is that AC22 somehow is worst that AC21 handling point clouds in section and elevation. I don't find out why, but in AC21 I get all the cross section straight and in AC22 the very same section only displays half of the point cloud, so I do not get the proper section to work. For that reason, and because time do not stop I'm using AC21 to get the job with the clouds done, and then shift to AC22 to get the best AC yet.

I had tested other workflows, like PointCAB and other tools, but I think that they are even more time consuming, and all lacks the "Architect evaluation eyes" to get the things that really matter in these solutions.

Hope it help. I'de spoken about these workflow in AC Summer School in Lisbon... 😉

Re: Point cloud - Too many points

Daniel Kovacs — Tue, 30 Oct 2018 17:04:38 GMT

ruivenda wrote:
...
My main complain is that AC22 somehow is worst that AC21 handling point clouds in section and elevation. I don't find out why, but in AC21 I get all the cross section straight and in AC22 the very same section only displays half of the point cloud, so I do not get the proper section to work. For that reason, and because time do not stop I'm using AC21 to get the job with the clouds done, and then shift to AC22 to get the best AC yet.
...

Hi,

First of all, neat workflow tips, I like it!

The issue you mentioned about ARCHICAD 22 is something we are aware of, and we want to fix it in our next update (AC22 50xx). You will find it as fix #250366 in our Release Note.

As far as we know this only happen on macOS. If you turn off the 2D Drawing Hardware Acceleration in the Work Environment (or sometimes it is enough to just set it lower), the Point Cloud should appear properly - though that should make things slower.

Regards,