< Back to blog home PostgreSQL Zone

Fujitsu Enterprise Postgres high availability - Network latency results

Rajesh Kandasamy | September 19, 2024

Preparing to benchmark

One of the customers is implementing Fujitsu Enterprise Postgres with high availability using Mirroring Controller (MC) and Server Assistant in the AWS cloud.

To leverage the zone redundancy provided by AWS using Availability Zones (AZ) and place each of the three Fujitsu Enterprise Postgres nodes in various AZ in a region, we tested how the network latency will affect the placement of Fujitsu Enterprise Postgres nodes.

AWS claims single-digit millisecond roundtrip latency between AZs in the same Region. So, would it have any detrimental effect on the functioning of a HA cluster which would lead to unwanted failover due to network latency?
As you may be aware, one of the pre-conditions for Mirroring Controller is streaming replication configured in synchronous mode, where two-phase commit must happen to complete any DML operations.

We have used

• Operating system: Red Hat Enterprise Linux release 9.4 (Plow)
• Database: Fujitsu Enterprise Postgres 15 SP2 Advanced Edition
• vCPUs: 4
• Benchmarking tool: pgbench
• Distribution: AWS EC2 instances housed in different AZ of a region

For the initial data load, using pgbench

$ pgbench -i -s 150 testdb
-i, --initialize    invokes initialization mode
-s, --scale=NUM    scaling factor

Load test with (-N) to perform simple updates. The sample looks like the below:

pgbench -c 5 -j 5 -T 120 -n -N testdb
-c, --client=NUM        number of concurrent database clients (default:  1)
-j, --jobs=NUM          number of threads (default: 1)
-T, --time=NUM          duration of benchmark test in seconds
-n, --no-vacuum         do not run VACUUM before tests
-N, --skip-some-updates skip updates of pgbench_tellers and pgbench_branches (same as "-b simple-update")

To induce latency to eth0 network card, we ran ‘tc’ (Traffic Control) command, as root user, as below:

• To add latency of 50ms

  tc qdisc add dev eth0 root netem delay 50ms

• To change latency to 100ms

  tc qdisc change dev eth0 root netem delay 100ms

• To remove latency

  tc qdisc del dev eth0 root netem

Note: You may need the operating system packages iproute and kernel-modules-extra for executing the above tc commands.

Once you altered the latency, you can verify the round-trip time (rtt) in the ping command response.

Testing

First, we ran test with five clients for pgbench — pgbench -c 5

We tested further, then

Next, we increased the number of clients to 50.

Note: The data mentioned in the table above is for reference only and should be used as benchmarks. These results were achieved in a controlled testing infrastructure and hence results could vary from test environment to environment.

Observations

We observed that in the cases with up to 100ms of latency in the network, the Mirroring Controller status remains
switchable, which indicates the transactions are in sync, and in case of any abnormality, the Mirroring Controller can trigger automatic failover, as expected.

Another observation to note is that as network latency increases, both the transaction processed per second (TPS) and transaction processed decrease, while write latency spikes as anticipated.

Wrapping up: key takeaways

If you consider implementing a Fujitsu Enterprise Postgres High Availability cluster across availability zones in a cloud environment, it is highly advised to conduct failover tests.

In this article, we have shown how to simulate a network latencies using the pgbench benchmarking tool and the tc qdisc netem command.