Perf: reduce memory usage when reloading with large configurations.

[tkan145]

What

Small perf to reduce memory bloat for big configuration.

For a setup with 1000 services, each has 2 policy and 100 mapping rules.

Version	Boot init	After reload
2.15	360MiB	684MiB
2.16	275MiB	443MiB
master	282Mib	436MiB
this PR	248MiB	334MiB

Verification steps

• Build a new runtime-image for master

make runtime-image IMAGE_NAME=apicast-test-master

• Checkout this branch
• Build another runtime-image

make runtime-image IMAGE_NAME=apicast-test

• Get into dev-environements

cd dev-environments/plain-http-upstream

• Create a new file with the following content

#!/usr/bin/env python3
"""Generate a large APIcast configuration file with N services for reload testing."""
import json
import sys

N = int(sys.argv[1]) if len(sys.argv) > 1 else 1000
OUT = sys.argv[2] if len(sys.argv) > 2 else "apicast-config-large.json"
CONFIG_VERSION = sys.argv[3] if len(sys.argv) > 3 else "1"


def make_service(i):
    sid = str(1000 + i)
    return {
        "backend_version": "1",
        "id": sid,
        "config_version": CONFIG_VERSION,
        "proxy": {
            "hosts": [f"service-{i}.example.com"],
            "api_backend": "https://echo-api.3scale.net:443",
            "backend": {
                "endpoint": "http://127.0.0.1:8081",
                "host": "backend",
            },
            "policy_chain": [
                {"name": "apicast", "version": "builtin", "configuration": {}},
                {"name": "apicast.policy.3scale_batcher", "configuration": {}},
            ],
            "proxy_rules": [
                {
                    "http_method": "GET",
                    "pattern": f"/path-{i}-{j}",
                    "metric_system_name": "hits",
                    "delta": 1,
                    "parameters": [],
                    "querystring_parameters": {},
                }
                for j in range(100)
            ],
        },
    }


config = {"services": [make_service(i) for i in range(N)]}

with open(OUT, "w") as f:
    json.dump(config, f, indent=2)

print(f"Wrote {N} services to {OUT}")

• Generate big config file

python ./generate_large_config.py

• Modify docker-compose.yaml file as follow

diff --git a/dev-environments/plain-http-upstream/docker-compose.yml b/dev-environments/plain-http-upstream/docker-compose.yml
index ebf84ebc..fcca3c78 100644
--- a/dev-environments/plain-http-upstream/docker-compose.yml
+++ b/dev-environments/plain-http-upstream/docker-compose.yml
@@ -11,10 +11,10 @@ services:
     environment:
       THREESCALE_CONFIG_FILE: /tmp/config.json
       THREESCALE_DEPLOYMENT_ENV: staging
-      APICAST_CONFIGURATION_LOADER: lazy
+      APICAST_CONFIGURATION_LOADER: boot
       APICAST_WORKERS: 1
       APICAST_LOG_LEVEL: debug
-      APICAST_CONFIGURATION_CACHE: "0"
+      APICAST_CONFIGURATION_CACHE: "30"
     expose:
       - "8080"
       - "8090"
@@ -22,7 +22,7 @@ services:
       - "8080:8080"
       - "8090:8090"
     volumes:
-      - ./apicast-config.json:/tmp/config.json
+      - ./apicast-config-large.json:/tmp/config.json
   example.com:
     image: quay.io/openshift-logging/alpine-socat:1.8.0.0
     container_name: example.com
(END)

• Start the gateway with master image

make gateway IMAGE_NAME=apicast-test-master

• In another terminate run

docker stats

• Check the memory usage of APIcast container and wait for the reload
• Stop the containers

CTRL-C

• Start the gateway with this branch image

make gateway IMAGE_NAME=apicast-test

Check the memory stats again, you should see it lower memory usage