Health Samurai recently published an open-source FHIR server performance benchmark that compares Aidbox, HAPI FHIR, Medplum, and the Microsoft FHIR Server on identical hardware. The dataset they used is 1,000 synthetic patient records generated by Synthea, totaling around 2 million resources. The repository is open and the harness reruns daily, which is good benchmark hygiene. The honest question for the reader is what a Synthea-driven test actually shows, and where it stops.
What Synthea Is Good At
Synthea generates patient records that are structurally valid FHIR. The resources reference each other correctly, the demographic distributions are plausible, and the volume scales linearly with the patient count. For a comparison harness, that is exactly what you want at the data layer: every server under test sees the same resources in the same order, and the variable being measured is server behavior, not data quality.
In practice, Synthea is the right shape for measuring three kinds of things. First, raw resource ingestion throughput: the bundle-import RPS numbers in the Health Samurai report come out cleanly because every server processes the same payload. Second, single-resource CRUD throughput: create, read, update, delete cycles against generated resources produce consistent per-server numbers. Third, search across simple parameters: queries against Patient by identifier, Observation by code, Encounter by date all behave predictably on Synthea data.
What Synthea Cannot Show
The first thing it cannot show is the long tail of profile extensions that real US clinical data carries. Production data is messy in specific ways: USCDI extensions that bind to local code systems, vendor-specific extensions that survive from older HL7 v2 feeds, narrative text with embedded HTML that some servers tokenize differently, partial values where a real EHR submitted a Patient resource with no birthDate. Synthea data is clean in ways production data never is.
The second thing it cannot show is the burst pattern that clinical workloads actually produce. Hospital traffic is not steady. It spikes around shift changes, when discharge summaries get filed in batches, when scheduled lab pulls run at 2 AM. Honestly, the only way to test a server's behavior under that shape is to replay actual traffic patterns from a system in production, and Synthea cannot generate those.
The third thing is search behavior under chained queries that mirror real applications. A real care-management app does not query "all Observations" or "all Patients" in isolation. It chains: Patient with active Care Plan, with Observations of a specific code in a value set, within a date range, sorted by recency. The chained pattern under production cardinality is where servers diverge most dramatically, and Synthea-shaped data tends to produce too-clean cardinality at the chain points.
Where the Benchmark Numbers Are Still Useful
That said, the numbers in the Health Samurai report are still useful as a baseline. The 5,212 CRUD RPS Aidbox posted and the 440 CRUD RPS the Microsoft FHIR Server posted are not equivalent under production data, but the relative ordering across servers tends to hold even when the absolute numbers shift. A server that posts strong Synthea numbers is unlikely to perform terribly on production data; a server that struggles on Synthea is unlikely to recover under heavier load.
The bundle import numbers and the storage footprint comparison are the cleanest. Both are largely shape-independent: a bundle of 100 resources takes roughly the same time to ingest whether the resources are Synthea-clean or production-messy, and on-disk size after import scales with the resource count more than with the resource content. For the FHIR resource library and the broader question of how server selection should account for these limits, the data parity is the part that holds up.
How to Read Vendor-Run Synthetic Benchmarks
A few practical filters help. The benchmark is published by Health Samurai, which develops Aidbox, so the obvious caveat applies. The repository is open source, which means you can re-run it against your own data if you want a number that reflects your workload. The next post in the Health Samurai series is supposed to test at scale, which will surface the in-memory limitations that a 1K-patient dataset hides.
The honest move is to treat Synthea numbers as a sanity check on the underlying server engineering, not as a forecast of your production behavior. For more on what the broader FHIR terminology and server ecosystem looks like under real US workloads, the FHIR terminology servers in 2026 buyer's guide covers adjacent ground, and the FHIR R4 vs FHIR R5 for US EHR projects comparison covers the version dimension that benchmark choices implicitly assume.
