Huge Memory Usage

[tyliggity]

Hi there. First off, thanks for this great library.

When trying to scan a 700Mb *osm.pbf file, the mem usage spikes to over 145Gb with just a single go routine. I know this because I am trying this task in an AWS CodeBuild project with max memory allowed (145 Gb) and it still gets killed.

This is surprising considering that the Osmosis CLI tool can accomplish the same task (scanning the 700Mb pbf file and filtering by a bounding box) without breaching 7Gb of memory.

Why does this tool take over 20 times more memory than Osmosis, and is there anything we can do to improve this? I'm willing to help if it's a big task.

Thanks.

[paulmach]

This was a problem if you had a lot of cores, it was fixed in #22 which is in v0.2.1

If that's not it, can you share the pbf you're using, or something similar?

[tyliggity]

Hi Paul. Sure, the file is here: https://download.geofabrik.de/north-america/canada/ontario-latest.osm.pbf

The code is basically the same as the example:

file, err := os.Open(fileNameIn)
if err != nil {
  panic(err)
}
defer file.Close()

fullOSM := &osm.OSM{
  Nodes:     make([]*osm.Node, 0),
  Ways:      make([]*osm.Way, 0),
  Relations: make([]*osm.Relation, 0),
}

scanner := osmpbf.New(context.Background(), file, runtime.GOMAXPROCS(1))
defer scanner.Close()
for scanner.Scan() {
  switch obj := scanner.Object().(type) {
  case *osm.Node:
    if isCoordWithinBounds(obj.Lat, obj.Lon) {
      fullOSM.Nodes = append(fullOSM.Nodes, obj)
    }
  case *osm.Way:
    if obj.Bounds != nil && isBoundsWithinBounds(obj.Bounds) {
      fullOSM.Ways = append(fullOSM.Ways, obj)
    }
  case *osm.Relation:
    if obj.Bounds != nil && isBoundsWithinBounds(obj.Bounds) {
      fullOSM.Relations = append(fullOSM.Relations, obj)
    }
  }
}
if err := scanner.Err(); err != nil {
  panic(err)
}

But maybe there's something I'm doing wrong. I'm ultimately trying to go from an *.osm.pbf to a geojson, filtering by a bounding box.

[tyliggity]

Oh, the CodeBuild environment has 4 CPUs and 7Gb RAM but I've also tried the higher options of 15 GB memory, 8 vCPUs and 145 GB memory, 72 vCPUs

[paulmach]

@oflebbe This may be due to #13. If you preallocate 8000 nodes, keep a pointer to 1 of those, the whole 8000 node slice can't be GC'd.

I was not able to duplicate 145Gb with the above file. But I was able to verify the above is true.

[paulmach]

I created a PR fo the issue I described above #26

[oflebbe]

I would rather propose to copy over the needed objects, otherwise we will have a huge GC overhead.
And btw, it is much more performant (memory overhead wise) to have a array of objects than having an array of pointers to objects, since a pointer is not that lightweight

fullOSM := &osm.OSM{
Nodes: []osm.Node{},
Ways: []osm.Way{},
Relations: []osm.Relation{},
}

if isCoordWithinBounds(obj.Lat, obj.Lon) {
node := osm.Node{}
copy( node, obj)
fullOSM.Nodes = append( fullOSM.Nodes, node)
}

[paulmach]

merged #26 and published v0.2.2