Apache Pulsar.

Apache Pulsar is a distributed messaging and event-streaming system for teams that need to run producers, consumers, and data flows across several applications or regions. Its important boundary is architectural: Pulsar separates brokers from durable storage, organises access through tenants and namespaces, and brings backlog handling, replication, and multiple subscription modes into one platform. That is valuable when those operating requirements are real; for one small internal queue, the cluster is usually more complexity than the workload deserves.

Who should use Pulsar?

Pulsar fits platform, data, and backend teams that do not want to reinvent event contracts, ownership, and recovery in every application. Typical candidates include product catalogues, order and payment events, telemetry, CDC pipelines, and multi-tenant SaaS systems. It becomes interesting when several consumers need independent views of the same event stream, when consumers must replay retained data, or when events need to move between clusters.

It is not automatically a good fit for a small project with no appetite for distributed operations. Pulsar does not replace a schema and ownership model, nor does it create observability by itself. If the actual need is simply to feed one worker reliably, compare a smaller broker first.

The components in a real workflow

A producer writes to a topic. A broker accepts the connection, looks up topic ownership, and dispatches messages to consumers. Persistent data is handled by Apache BookKeeper, while a metadata store keeps cluster, topic, and namespace information. This separation allows brokers and storage to be planned independently, but it also creates more operational surfaces than a single-process queue.

Tenants and namespaces provide organisational boundaries for permissions, quotas, and policies. The subscription type and cursor determine whether consumers share work or keep independent views of the stream. Schemas, retention, and backlog rules therefore belong in the event contract, not only in client configuration.

A practical adoption workflow

1. Choose one bounded event, such as order.created, and document its owner, schema, key, ordering expectation, and retention.
2. Run a test cluster with a real producer and consumer. Check acknowledgements, retries, backlog growth, and what happens when a consumer restarts.
3. Set up the tenant, namespace, roles, and quotas before more teams create topics. Naming rules prevent an event estate that nobody can navigate.
4. Simulate broker, bookie, and consumer failures. Measure recovery time, duplicates, data loss, and the cost of replaying a growing backlog, not only throughput.
5. Promote the design to production with runbooks for schema changes, backlog alarms, certificate rotation, partitioning, and rollback.

Integration and everyday operations

Pulsar provides official client libraries including Java, C++, Go, Python, Node.js, and C#. Clients support reconnection, acknowledgements, and transactions; transactions are useful when producing and acknowledging messages across multiple topics must be atomic. Pulsar Functions and Pulsar IO can run computation or connectors close to the messaging system, but they do not remove the need to test the surrounding data pipeline.

Operations teams need visibility into BookKeeper capacity, metadata health, broker load, storage growth, replication lag, and consumer backlog. Geo-replication is not a complete business-continuity plan: regions, failover, conflicts, RPO/RTO, permissions, and the route back must be tested separately.

Quality and decision criteria

A credible pilot uses production-like failure modes instead of a synthetic success story. Test schema compatibility, per-key ordering, duplicate delivery, traffic spikes, consumer failure, and replay from a growing backlog. Record measurements and failure messages so the decision does not depend on one impressive demo.

Pulsar is the stronger choice when independent subscription views, namespace governance, durable storage, and cluster-to-cluster replication are needed together. Without that combination, a simpler broker may be easier to operate and cheaper in team time and risk.

Security, privacy, and governance

The official documentation treats encryption, authentication, and authorization as separate controls that must be configured deliberately. A production design should include TLS, centrally managed roles, least-privilege access to tenants and namespaces, and credentials that can be rotated. Proxy and broker roles need their own review rather than one broad administrator identity.

Before customer data enters a topic, identify the payloads, logs, schemas, and technical metadata that will be stored or replicated. Retention, backlog, exports, and replication targets can keep data available for longer or in more regions than an application owner expects. Durable events are not exempt from privacy obligations: deletion, access, encryption, and minimisation need an explicit security and privacy decision.

Pricing and real operating costs

Apache Pulsar is open-source software. Running it still costs compute for brokers, storage and I/O for BookKeeper, metadata and network resources, monitoring, backups, replication, and on-call coverage. A managed service adds provider pricing and may charge for traffic or support. Compare more than a per-message figure: include cluster size, retention, backlog, regions, and the team effort required to operate the platform.

Editorial Assessment

We recommend Pulsar to platform and data teams that genuinely need several consumer views, clear tenant boundaries, and durable events under active operations. Its value appears when there is a documented event contract, an accountable operations owner, and recovery targets that have been tested.

For one queue, a small team without streaming-operations capacity, or a short-lived integration, we would first compare RabbitMQ or NATS. Pulsar earns its extra infrastructure when the pilot proves that backlog, replication, and governance requirements outweigh the additional operational surface.