❗️ This guide is intended only for development of PostHog itself. If you're looking to deploy PostHog for your product analytics needs, go to Self-host PostHog.
What does PostHog look like on the inside?
Before jumping into setup, let's dissect a PostHog.
The app itself is made up of 4 components that run simultaneously:
- Celery worker (handles execution of background tasks)
- Django server
- Node.js plugin server (handles event ingestion and apps/plugins)
- React frontend built with Node.js
These components rely on a few external services:
- ClickHouse – for storing big data (events, persons – analytics queries)
- Kafka – for queuing events for ingestion
- MinIO – for storing files (session recordings, file exports)
- PostgreSQL – for storing ordinary data (users, projects, saved insights)
- Redis – for caching and inter-service communication
- Zookeeper – for coordinating Kafka and ClickHouse clusters
When spinning up an instance of PostHog for development, we recommend the following configuration:
- the external services run in Docker over
docker compose
- PostHog itself runs on the host (your system)
This is what we'll be using in the guide below.
It is also technically possible to run PostHog in Docker completely, but syncing changes is then much slower, and for development you need PostHog dependencies installed on the host anyway (such as formatting or typechecking tools). The other way around – everything on the host, is not practical due to significant complexities involved in instantiating Kafka or ClickHouse from scratch.
The instructions here assume you're running macOS or the current Ubuntu Linux LTS (24.04).
For other Linux distros, adjust the steps as needed (e.g. use dnf
or pacman
in place of apt
).
Windows isn't supported natively. But, Windows users can run a Linux virtual machine. The latest Ubuntu LTS Desktop is recommended. (Ubuntu Server is not recommended as debugging the frontend will require a browser that can access localhost.)
In case some steps here have fallen out of date, please tell us about it – feel free to submit a patch!
Option 1: Developing with Codespaces
This is a faster option to get up and running. If you don't want to or can't use Codespaces, continue from the next section.
- Create your codespace.
- Update it to 8-core machine type (the smallest is probably too small to get PostHog running properly).
- Open the codespace, using one of the "Open in" options from the list.
- In the codespace, open a terminal window and run
docker compose -f docker-compose.dev.yml up
. - In another terminal, run
pnpm i
(and use the same terminal for the following commands) - Then run
pip install -r requirements.txt -r requirements-dev.txt
- Now run
./bin/migrate
and then./bin/start
. - Open browser to http://localhost:8000/.
- To get some practical test data into your brand-new instance of PostHog, run
DEBUG=1 ./manage.py generate_demo_data
.
Option 2: Developing locally
Prerequisites
macOS
Install Xcode Command Line Tools if you haven't already:
xcode-select --install
.Install the package manager Homebrew by following the instructions here.
After installation, make sure to follow the instructions printed in your terminal to add Homebrew to your$PATH
. Otherwise the command line will not know about packages installed withbrew
.
Install OrbStack – a more performant Docker Desktop alternative – with
brew install orbstack
. Go to OrbStack settings and set the memory usage limit to at least 4 GB (or 8 GB if you can afford it) + the CPU usage limit to at least 4 cores (i.e. 400%). You'll want to use Brex for the license if you work at PostHog.Continue with the common prerequisites for both macOS and Linux.
Ubuntu
Install Docker following the official instructions here.
Install the
build-essential
package:Terminalsudo apt install -y build-essentialContinue with the common prerequisites for both macOS and Linux.
Common prerequisites for both macOS & Linux
Append line
127.0.0.1 kafka clickhouse
to/etc/hosts
. You can do it in one line with:Terminalecho '127.0.0.1 kafka clickhouse' | sudo tee -a /etc/hostsClickHouse and Kafka won't be able to talk to each other without these mapped hosts.
If you are using a newer (>=4.1) version of Podman instead of Docker, the host machine's
/etc/hosts
is used as the base hosts file for containers by default, instead of container's/etc/hosts
like in Docker. This can make hostname resolution fail in the ClickHouse container, and can be mended by settingbase_hosts_file="none"
incontainers.conf
.Clone the PostHog repository. All future commands assume you're inside the
posthog/
folder.Terminalgit clone https://github.com/PostHog/posthog && cd posthog/
Get things up and running
1. Spin up external services
In this step we will start all the external services needed by PostHog to work.
docker compose -f docker-compose.dev.yml up
Friendly tip 1: If you see
Error while fetching server API version: 500 Server Error for http+docker://localhost/version:
, it's likely that Docker Engine isn't running.
Friendly tip 2: If you see "Exit Code 137" anywhere, it means that the container has run out of memory. In this case you need to allocate more RAM in OrbStack settings.
Friendly tip 3: On Linux, you might need
sudo
– see Docker docs on managing Docker as a non-root user. Or look into Podman as an alternative that supports rootless containers.
Friendly tip 4: If you see
Error: (HTTP code 500) server error - Ports are not available: exposing port TCP 0.0.0.0:5432 -> 0.0.0.0:0: listen tcp 0.0.0.0:5432: bind: address already in use
, you have Postgres already running somewhere. Trydocker compose -f docker-compose.dev.yml
first, alternatively runlsof -i :5432
to see what process is using this port.Terminal
sudo service postgresql stop
Second, verify via docker ps
and docker logs
(or via the OrbStack dashboard) that all these services are up and running. They should display something like this in their logs:
# docker ps NAMESCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES5a38d4e55447 temporalio/ui:2.10.3 "./start-ui-server.sh" 51 seconds ago Up 44 seconds 0.0.0.0:8081->8080/tcp posthog-temporal-ui-189b969801426 temporalio/admin-tools:1.20.0 "tail -f /dev/null" 51 seconds ago Up 44 seconds posthog-temporal-admin-tools-181fd1b6d7b1b clickhouse/clickhouse-server:23.6.1.1524 "/entrypoint.sh" 51 seconds ago Up 50 seconds 0.0.0.0:8123->8123/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:9009->9009/tcp, 0.0.0.0:9440->9440/tcp posthog-clickhouse-1f876f8bff35f bitnami/kafka:2.8.1-debian-10-r99 "/opt/bitnami/script…" 51 seconds ago Up 50 seconds 0.0.0.0:9092->9092/tcp posthog-kafka-1d22559261575 temporalio/auto-setup:1.20.0 "/etc/temporal/entry…" 51 seconds ago Up 45 seconds 6933-6935/tcp, 6939/tcp, 7234-7235/tcp, 7239/tcp, 0.0.0.0:7233->7233/tcp posthog-temporal-15313fc278a70 postgres:12-alpine "docker-entrypoint.s…" 51 seconds ago Up 50 seconds (healthy) 0.0.0.0:5432->5432/tcp posthog-db-1c04358d8309f zookeeper:3.7.0 "/docker-entrypoint.…" 51 seconds ago Up 50 seconds 2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp posthog-zookeeper-109add699866e maildev/maildev:2.0.5 "bin/maildev" 51 seconds ago Up 50 seconds (healthy) 0.0.0.0:1025->1025/tcp, 0.0.0.0:1080->1080/tcp posthog-maildev-161a44c094753 elasticsearch:7.16.2 "/bin/tini -- /usr/l…" 51 seconds ago Up 50 seconds 9200/tcp, 9300/tcp posthog-elasticsearch-1a478cadf6911 minio/minio:RELEASE.2022-06-25T15-50-16Z "sh -c 'mkdir -p /da…" 51 seconds ago Up 50 seconds 9000/tcp, 0.0.0.0:19000-19001->19000-19001/tcp posthog-object_storage-191f838afe40e redis:6.2.7-alpine "docker-entrypoint.s…" 51 seconds ago Up 50 seconds 0.0.0.0:6379->6379/tcp posthog-redis-1# docker logs posthog-db-1 -n 12021-12-06 13:47:08.325 UTC [1] LOG: database system is ready to accept connections# docker logs posthog-redis-1 -n 11:M 06 Dec 2021 13:47:08.435 * Ready to accept connections# docker logs posthog-clickhouse-1 -n 1Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.# ClickHouse writes logs to `/var/log/clickhouse-server/clickhouse-server.log` and error logs to `/var/log/clickhouse-server/clickhouse-server.err.log` instead of stdout/stsderr. It can be useful to `cat` these files if there are any issues:# docker exec posthog-clickhouse-1 cat /var/log/clickhouse-server/clickhouse-server.log# docker exec posthog-clickhouse-1 cat /var/log/clickhouse-server/clickhouse-server.err.log# docker logs posthog-kafka-1[2021-12-06 13:47:23,814] INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)# docker logs posthog-zookeeper-1# Because ClickHouse and Kafka connect to Zookeeper, there will be a lot of noise here. That's good.
Friendly tip: Kafka is currently the only x86 container used, and might segfault randomly when running on ARM. Restart it when that happens.
Finally, install Postgres locally. Even if you are planning to run Postgres inside Docker, we need a local copy of Postgres (version 11+) for its CLI tools and development libraries/headers. These are required by pip
to install psycopg2
.
- On macOS:Terminalbrew install postgresql
This installs both the Postgres server and its tools. DO NOT start the server after running this.
- On Debian-based Linux:Terminalsudo apt install -y postgresql-client postgresql-contrib libpq-dev
This intentionally only installs the Postgres client and drivers, and not the server. If you wish to install the server, or have it installed already, you will want to stop it, because the TCP port it uses conflicts with the one used by the Postgres Docker container. On Linux, this can be done with sudo systemctl disable postgresql.service
.
On Linux you often have separate packages: postgres
for the tools, postgres-server
for the server, and libpostgres-dev
for the psycopg2
dependencies. Consult your distro's list for an up-to-date list of packages.
2. Prepare the frontend
- Install nvm, with
brew install nvm
or by following the instructions at https://github.com/nvm-sh/nvm. If using fish, you may instead prefer https://github.com/jorgebucaran/nvm.fish.
After installation, make sure to follow the instructions printed in your terminal to add NVM to your $PATH
. Otherwise the command line will use your system Node.js version instead.
Install the latest Node.js 18 (the version used by PostHog in production) with
nvm install 18
. You can start using it in the current shell withnvm use 18
.Install pnpm with
corepack enable
and check it withpnpm --version
.Install Node packages by running
pnpm i
.Run
pnpm typegen:write
to generate types for Kea state management logics used all over the frontend.
The first time you run typegen, it may get stuck in a loop. If so, cancel the process (
Ctrl+C
), discard all changes in the working directory (git reset --hard
), and runpnpm typegen:write
again. You may need to discard all changes once more when the second round of type generation completes.
3. Prepare plugin server
- Install the
brotli
compression library andrust
stable viarustup
:
- On macOS:Terminalbrew install brotli rustuprustup default stablerustup-init# Select 1 to proceed with default installation
- On Debian-based Linux:Terminalsudo apt install -y brotlicurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh# Select 1 to proceed with default installation
- Run
pnpm i --dir plugin-server
to install all required packages. We'll actually run the plugin server in a later step.
Note: If you face an error like
ld: symbol(s) not found for architecture arm64
, most probably your openssl build flags are coming from the wrong place. To fix this, run:Terminal
export CPPFLAGS=-I/opt/homebrew/opt/openssl/includeexport LDFLAGS=-L/opt/homebrew/opt/openssl/libpnpm i --dir plugin-server
Note: If you face an error like
import gyp # noqa: E402
, most probably need to installpython-setuptools
. To fix this, run:Terminal
brew install python-setuptools
4. Prepare the Django server
Install a few dependencies for SAML to work. If you're on macOS, run the command below, otherwise check the official xmlsec repo for more details.
On macOS:
Terminalbrew install libxml2 libxmlsec1 pkg-configIf installing
xmlsec
doesn't work, try updating macOS to the latest version (Sonoma).On Debian-based Linux:
Terminalsudo apt install -y libxml2 libxmlsec1-dev libffi-dev pkg-config
Install Python 3.11.
On macOS, you can do so with Homebrew:
brew install python@3.11
.On Debian-based Linux:
Terminalsudo add-apt-repository ppa:deadsnakes/ppa -ysudo apt updatesudo apt install python3.11 python3.11-venv python3.11-dev -y
Make sure when outside of venv
to always use python3
instead of python
, as the latter may point to Python 2.x on some systems. If installing multiple versions of Python 3, such as by using the deadsnakes
PPA, use python3.11
instead of python3
.
You can also use pyenv if you wish to manage multiple versions of Python 3 on the same machine.
- Install
uv
uv
is a very fast tool you can use for python virtual env and dependency management. See https://docs.astral.sh/uv/. Once installed you can prefix any pip
command with uv
to get the speed boost.
Create the virtual environment in current directory called 'env':
Terminaluv venv env --python 3.11Activate the virtual environment:
Terminal# For bash/zsh/etc.source env/bin/activate# For fishsource env/bin/activate.fishUpgrade pip to the latest version:
Terminaluv pip install -U pipInstall requirements with pip
If your workstation is an Apple Silicon Mac, the first time your run
pip install
you must set custom OpenSSL headers:Terminalbrew install opensslCFLAGS="-I /opt/homebrew/opt/openssl/include $(python3.11-config --includes)" LDFLAGS="-L /opt/homebrew/opt/openssl/lib" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 uv pip install -r requirements.txtFriendly tip: If you see
ERROR: Could not build wheels for xmlsec
, refer to this issue.These will be used when installing
grpcio
andpsycopg2
. After doing this once, and assuming nothing changed with these two packages, next time simply run:Terminaluv pip install -r requirements.txt -r requirements-dev.txt
5. Prepare databases
We now have the backend ready, and Postgres and ClickHouse running – these databases are blank slates at the moment however, so we need to run migrations to e.g. create all the tables:
cargo install sqlx-cli # If you haven't alreadyDEBUG=1 ./bin/migrate
Friendly tip: The error
fe_sendauth: no password supplied
connecting to Postgres happens when the database is set up with a password and the user:pass isn't specified inDATABASE_URL
. Tryexport DATABASE_URL=postgres://posthog:posthog@localhost:5432/posthog
.
Another friendly tip: You may run into
psycopg2
errors while migrating on an ARM machine. Try out the steps in this comment to resolve this.
6. Start PostHog
Now start all of PostHog (backend, worker, plugin server, and frontend – simultaneously) with:
./bin/start
Friendly tip: If you get the error
Configuration property "enable.ssl.certificate.verification" not supported in this build: OpenSSL not available at build time
, make sure your environment is using the rightopenssl
version by setting those environment variables, and then run./bin/start
again.
Open http://localhost:8000 to see the app.
Note: The first time you run this command you might get an error that says "layout.html is not defined". Make sure you wait until the frontend is finished compiling and try again.
To get some practical test data into your brand-new instance of PostHog, run DEBUG=1 ./manage.py generate_demo_data
. For a list of useful arguments of the command, run DEBUG=1 ./manage.py generate_demo_data --help
.
You can also use mprocs to run all development processes in a single terminal window. mprocs provides a clean interface for starting, stopping, and monitoring each service individually, with separate log views for easier debugging. This includes docker compose, so make sure to stop it if it's already running. Once you have mprocs installed, run
./bin/start-mprocs
7. Develop
This is it! You can now change PostHog in any way you want. See Project Structure for an intro to the repository's contents.
To commit changes, create a new branch based on master
for your intended change, and develop away. Just make sure not use to use release-*
patterns in your branches unless putting out a new version of PostHog, as such branches have special handling related to releases.
Testing
For a PostHog PR to be merged, all tests must be green, and ideally you should be introducing new ones as well – that's why you must be able to run tests with ease.
Frontend
For frontend unit tests, run:
pnpm test:unit
You can narrow the run down to only files under matching paths:
pnpm jest --testPathPattern=frontend/src/lib/components/IntervalFilter/intervalFilterLogic.test.ts
To update all visual regression test snapshots, make sure Storybook is running on your machine (you can start it with pnpm storybook
in a separate Terminal tab). You may also need to install Playwright with pnpm exec playwright install
. And then run:
pnpm test:visual
To only update snapshots for stories under a specific path, run:
pnpm test:visual:update frontend/src/lib/Example.stories.tsx
Backend
For backend tests, run:
pytest
You can narrow the run down to only files under matching paths:
pytest posthog/test/test_example.py
Or to only test cases with matching function names:
pytest posthog/test/test_example.py -k test_something
To see debug logs (such as ClickHouse queries), add argument --log-cli-level=DEBUG
.
End-to-end
For Cypress end-to-end tests, run bin/e2e-test-runner
. This will spin up a test instance of PostHog and show you the Cypress interface, from which you'll manually choose tests to run. You'll need uv
installed (the Python package manager), which you can do so with brew install uv
. Once you're done, terminate the command with Cmd + C.
Extra: Working with feature flags
When developing locally with environment variable DEBUG=1
(which enables a setting called SELF_CAPTURE
),
all analytics inside your local PostHog instance is based on that instance itself – more specifically, the currently selected project.
This means that your activity is immediately reflected in the current project, which is potentially useful for testing features
– for example, which feature flags are currently enabled for your development instance is decided by the project you have open at the very same time.
So, when working with a feature based on feature flag foo-bar
, add a feature flag with this key to your local instance and release it there.
If you'd like to have ALL feature flags that exist in PostHog at your disposal right away, run DEBUG=1 python3 manage.py sync_feature_flags
– they will be added to each project in the instance, fully rolled out by default.
This command automatically turns any feature flag ending in _EXPERIMENT
as a multivariate flag with control
and test
variants.
Backend side flags are only evaluated locally, which requires the POSTHOG_PERSONAL_API_KEY
env var to be set. Generate the key in your user settings.
Extra: Debugging with VS Code
The PostHog repository includes VS Code launch options for debugging. Simply go to the Run and Debug
tab in VS Code, select the desired service you want to debug, and run it. Once it starts up, you can set breakpoints and step through code to see exactly what is happening. There are also debug launch options for frontend and backend tests if you're dealing with a tricky test failure.
Note: You can debug all services using the main "PostHog" launch option. Otherwise, if you are running most of the PostHog services locally with
./bin/start
, for example if you only want to debug the backend, make sure to comment out that service from the start script temporarily.
Extra: Debugging the backend in PyCharm
With PyCharm's built in support for Django, it's fairly easy to setup debugging in the backend. This is especially useful when you want to trace and debug a network request made from the client all the way back to the server. You can set breakpoints and step through code to see exactly what the backend is doing with your request.
Setup PyCharm
- Open the repository folder.
- Setup the python interpreter (Settings… > Project: posthog > Python interpreter > Add interpreter): Select "Existing" and set it to
path_to_repo/posthog/env/bin/python
. - Setup Django support (Settings… > Languages & Frameworks > Django):
- Django project root:
path_to_repo
- Settings:
posthog/settings/__init__py
- Django project root:
Start the debugging environment
- Instead of manually running
docker compose
you can open thedocker-compose.dev.yml
file and click on the double play icon next toservices
- From the run configurations select:
- "PostHog" and click on debug
- "Celery" and click on debug (optional)
- "Frontend" and click on run
- "Plugin server" and click on run
Extra: Developing paid features (PostHog employees only)
If you're a PostHog employee, you can get access to paid features on your local instance to make development easier. Learn how to do so in our internal guide.
Extra: Working with the data warehouse and a MySQL source
If you want to set up a local MySQL database as a source for the data warehouse, there are a few extra set up steps you'll need to complete:
- Setting up a local MySQL database to connect to.
- Installing MS SQL drivers on your machine.
- Defining additional environment variables for the Temporal task runner.
First, install MySQL:
brew install mysqlbrew services start mysql
Once MySQL is installed, create a database and table, insert a row, and create a user who can connect to it:
mysql -u root
CREATE DATABASE posthog_dw_test;CREATE TABLE IF NOT EXISTS payments (id INT AUTO_INCREMENT PRIMARY KEY, timestamp DATETIME, distinct_id VARCHAR(255), amount DECIMAL(10,2));INSERT INTO payments (timestamp, distinct_id, amount) VALUES (NOW(), 'testuser@example.com', 99.99);CREATE USER 'posthog'@'%' IDENTIFIED BY 'posthog';GRANT ALL PRIVILEGES ON posthog_dw_test.* TO 'posthog'@'%';FLUSH PRIVILEGES;
Next, you'll need to install some MS SQL drivers for PostHog the application to connect to the MySQL database. Learn the entire process in posthog/warehouse/README.md. Without the drivers, you'll get the following error when connecting a SQL database to data warehouse:
symbol not found in flat namespace '_bcp_batch'
Lastly, you'll need to define these environment variables in order for the Temporal task runner monitor the correct queue and work as expected:
# Ask for the values in #team-data-warehouseexport PYTHONUNBUFFERED=export DJANGO_SETTINGS_MODULE=export DEBUG=export CLICKHOUSE_SECURE=export KAFKA_HOSTS=export DATABASE_URL=export SKIP_SERVICE_VERSION_REQUIREMENTS=export PRINT_SQL=export BUCKET_URL=export AIRBYTE_BUCKET_KEY=export AIRBYTE_BUCKET_SECRET=export AIRBYTE_BUCKET_REGION=export AIRBYTE_BUCKET_DOMAIN=export TEMPORAL_TASK_QUEUE=export AWS_S3_ALLOW_UNSAFE_RENAME=export HUBSPOT_APP_CLIENT_ID=export HUBSPOT_APP_CLIENT_SECRET=
If you put them in a .temporal-worker-settings
file, you can run source .temporal-worker-settings
before you call DEBUG=1 ./bin/start
.
To verify everything is working as expected:
- Navigate to "Data pipeline" in the PostHog application.
- Create a new MySQL source using the settings above.
- Once the source is created, click on the "MySQL" item. In the schemas table, click on the triple dot menu and select the "Reload" option.
After the job runs, clicking on the synced table name should take you to your data.