BigPipe

Sends pages using the BigPipe technique that allows browsers to show them much faster by streaming content in chunks.

big_pipe
2 sites
9
drupal.org
api utility
performance scalability BigPipe streaming page load rendering placeholders caching

Overview

BigPipe is a page rendering technique originally developed by Facebook that dramatically improves perceived page load performance. Instead of waiting for the entire page to be rendered before sending anything to the browser, BigPipe streams the page content in chunks.

The module works by first sending the static 'skeleton' of the page (headers, navigation, footer, etc.) immediately, while dynamic content (user-specific blocks, personalized data, CSRF tokens) is rendered asynchronously and streamed as it becomes ready. This allows browsers to start rendering and displaying static content immediately while dynamic placeholders are replaced progressively.

BigPipe automatically integrates with Drupal's render system and placeholder strategy. It identifies content that should be placeholdered based on cacheability metadata, then replaces those placeholders using either JavaScript (for browsers with JS enabled) or server-side streaming (no-JS fallback). The module requires no manual configuration - it automatically activates for authenticated users and anonymous users with sessions.

Features

  • Streams HTML responses in chunks using Transfer-Encoding: chunked, allowing browsers to start rendering immediately
  • Automatic JavaScript detection with intelligent fallback to no-JS streaming mode
  • Integrates seamlessly with Drupal's auto-placeholdering system to identify dynamic content
  • Supports two placeholder strategies: JS placeholders (replaced via AJAX at end of page) and no-JS placeholders (replaced inline during streaming)
  • Handles both HTML placeholders and HTML attribute value placeholders (like CSRF tokens in URLs)
  • Provides session.exists cache context for session-aware caching
  • Sets appropriate HTTP headers for CDN/proxy compatibility (Surrogate-Control, X-Accel-Buffering)
  • Automatic exclusion for routes incompatible with BigPipe (batch processing, module installation)
  • XSS protection via placeholder ID whitelisting in drupalSettings

Use Cases

Improving perceived performance for authenticated users

When authenticated users load pages, BigPipe sends the page skeleton immediately while user-specific content (personalized blocks, status messages, etc.) is rendered and streamed progressively. Users see the page layout instantly and watch dynamic content appear as it becomes ready.

E-commerce sites with shopping carts

Anonymous users with shopping carts have sessions and benefit from BigPipe. The product listings and page structure load immediately, while the cart widget and personalized recommendations are streamed as placeholders are resolved.

Pages with expensive dynamic content

When pages contain content that is expensive to generate (complex database queries, external API calls), BigPipe allows the cacheable parts of the page to display immediately while the slow dynamic content is computed and streamed.

CSRF token handling

Forms and links with CSRF tokens benefit from BigPipe's no-JS placeholder strategy. The page structure is sent immediately, and CSRF tokens are streamed inline as they are generated, without requiring JavaScript.

Progressive enhancement for no-JS users

BigPipe automatically detects when JavaScript is disabled and falls back to no-JS streaming mode, where placeholders are replaced inline during page delivery. This ensures all users benefit from streaming, not just those with JavaScript enabled.

Tips

  • BigPipe works best when content is properly annotated with cacheability metadata. Ensure your render arrays include appropriate #cache contexts and tags.
  • Use #lazy_builder for dynamic content that should be placeholdered. This integrates seamlessly with BigPipe's streaming mechanism.
  • For NGINX with php-fpm, BigPipe automatically sets 'X-Accel-Buffering: no' header. For other proxies, you may need additional configuration.
  • Test your site with JavaScript disabled to ensure the no-JS fallback works correctly.
  • BigPipe automatically handles asset (CSS/JS) dependencies for streamed content, loading new libraries as needed via AJAX commands.
  • The module sets 'Surrogate-Control: no-store, content="BigPipe/1.0"' header to inform edge caches not to cache BigPipe responses.

Technical Details

Hooks 2
hook_help

Implements hook_help() to provide help text on the module's help page explaining what BigPipe does and that it requires no configuration.

hook_page_attachments

Implements hook_page_attachments() to add no-JS detection mechanism. When a session exists and the no-JS cookie is not set, adds a <noscript> meta refresh that redirects to /big_pipe/no-js to set the cookie. When the cookie exists, adds JavaScript to delete it (indicating JS is available).

Troubleshooting 5
BigPipe doesn't seem to be working - pages load all at once

Verify: 1) The user has an active session (BigPipe only works for authenticated users or anonymous users with sessions like shopping carts). 2) The hosting environment supports chunked transfer encoding and doesn't buffer the entire response. 3) Check if a reverse proxy or CDN is buffering responses. 4) Ensure the route doesn't have '_no_big_pipe' option set.

JavaScript errors occur when placeholders are replaced

This was addressed in a recent patch. Ensure you're using the latest version of the module. The issue occurred when the same placeholder appeared multiple times on a page and was being rendered/sent multiple times.

BigPipe causes redirect loops

The no-JS detection mechanism uses cookies and meta refresh redirects. If you're seeing redirect loops, check that cookies are being set properly. The big_pipe_nojs cookie should be set on the first no-JS detection redirect and prevent subsequent redirects.

Content appears briefly then disappears or shows wrong content

Check browser console for JavaScript errors. BigPipe's JavaScript validates placeholder IDs against a whitelist in drupalSettings.bigPipePlaceholderIds. Unknown placeholders are ignored for security reasons.

Placeholders not being replaced on specific routes

Some routes are automatically excluded from BigPipe (batch pages, module installation). Check if your route has '_no_big_pipe' option set. You can add this option to custom routes in your routing.yml if needed.

Security Notes 4
  • BigPipe responses are always marked as private and should never be cached by shared caches, as they contain session-specific content.
  • Placeholder IDs are whitelisted in drupalSettings to prevent XSS attacks via the placeholder replacement mechanism.
  • CSRF tokens are handled via no-JS placeholders to ensure they're properly generated per-session without exposing them to timing attacks.
  • The big_pipe_nojs cookie is set without httpOnly flag intentionally, allowing JavaScript to delete it when JS is available.