Skip to main content

Codebase tour

This page answers "I want to add or change X, where do I touch?". It is a directory map, not a behavior reference. For what each piece does, see the Simulator and Profiler sections.

Repository layout​

Selected directories and entry points:

LLMServingSim/
├── serving/ Simulator Python, the core loop
├── profiler/ Profiler Python, vLLM-based latency capture
├── bench/ Bench Python, real vLLM run + sim validation
├── workloads/ Workloads JSONL traces + generators
├── configs/ Configs Cluster/model JSON and PIM INI
├── scripts/ Env scripts Docker launchers + builders
└── astra-sim/ Backend C++ analytical network simulator

Each domain has a clear boundary. A typical PR touches one or two of these, not all of them. If you find yourself editing four domains for a single change, stop and reconsider the scope.

Simulator (serving/)​

Where most contributor work happens.

serving/
├── __main__.py CLI + main loop
└── core/
├── scheduler.py vLLM-style continuous batching
├── trace_generator.py Profile lookup -> trace rows
├── memory_model.py KV / weight / CXL byte accounting
├── component_sizes.py Contextual component loading and byte accounting
├── graph_generator.py Trace rows -> Chakra protobuf
├── controller.py ASTRA-Sim subprocess IPC
├── router.py Request routing across instances
├── gate_function.py MoE expert routing (incl. group-limited)
├── spec_decode.py Speculative-decoding acceptance model
├── cudagraph.py Target graph padding and DP shape synchronization
├── config_builder.py Cluster config -> ASTRA-Sim inputs
├── hardware_defaults.py Measured hardware defaults and override precedence
├── communication.py Collective tensor sizes and vocabulary shards
├── moe_components.py Native DP+EP component table lookup
├── moe_execution.py Native MoE compute/collective ordering
├── power_model.py Power / energy estimation
├── pim_model.py PIM device model
├── request.py Request / Batch dataclasses
├── block_pool.py Per-tier KV block pool + prefix-cache index
├── kv_cache_manager.py Tiered KV cache manager (block hashing, allocation)
├── logger.py Rich-based logging + stdio capture
└── utils.py Model config loading, formatters

Where to touch by intent:

IntentEdit
Change scheduling policyscheduler.py
Change how latency is looked uptrace_generator.py (_lookup_*)
Change byte accounting (KV, weights, prefix cache)memory_model.py
Change inter-instance routingrouter.py
Add a new CLI flag__main__.py (argparse), then thread through
Change MoE expert distributiongate_function.py
Change speculative acceptancespec_decode.py, and configs/spec_decode.json for the published defaults
Change ASTRA-Sim input generationconfig_builder.py
Add a new power componentpower_model.py

Profiler (profiler/)​

profiler/
├── __main__.py Model, coverage, skew and hardware CLI dispatch
├── core/ internals (runner, engine, categories, skew_calibration)
│ ├── catalog_path.py model_type -> yaml resolution (shared with serving/)
│ ├── component_layouts.py Runtime-selected canonical component layouts
│ ├── context_tables.py Contextual category publication and metadata
│ ├── context_lookup.py Strict contextual category loading and lookup
│ ├── callable_profile.py Declared callable acquisition and lookup
│ └── stack.py per-layer block resolution (shared with serving/)
├── models/<model_type>.yaml Architecture catalogs with declared aliases
├── perf/<hw>/<model>/... Output bundles (CSV per category)
└── profile.sh Editable user template

Several core/ modules are shared with the simulator, including catalog resolution, stack composition, attention geometry, skew calibration and MoE deployment contracts. Keep these importable in the CPU simulator environment without vLLM or GPU acquisition dependencies. Changes can alter simulation results; audit imports and run regression validation.

Where to touch by intent:

IntentEdit
Add a new hardware targetRun python -m profiler profile <model> --hardware <hw>, or edit MODEL and HARDWARE inside profiler/profile.sh; output lands in profiler/perf/<hw>/. See Profiler / Adding hardware
Add a new model architectureDrop a YAML in profiler/models/<model_type>.yaml, then python -m profiler coverage <model> --hardware <hw> until it binds every kernel. See Profiler / Adding model architecture
Teach the layer resolver a new per-layer ruleprofiler/core/stack.py — read the rule out of vLLM's source, never guess; the vendors' conventions genuinely disagree
Change the default skew calibrationprofiler/core/skew_calibration.py
Change what categories get profiledprofiler/core/categories.py + profiler/core/runner.py
Change runtime-selected component boundariescore/component_layouts.py, core/callable_profile.py, core/hooks/runtime_admission.py
Change contextual category publication or lookupcore/context_tables.py, core/context_lookup.py, serving/core/component_sizes.py
Change output CSV columnscore/writer.py (and _load_perf_db() in serving/core/trace_generator.py to consume them)

Bench (bench/)​

bench/
├── __main__.py CLI (run / validate)
├── core/ AsyncLLM driver, recorder, validator
├── examples/<hw>/<model>/ Committed end-to-end runs
└── results/<run_id>/ Output for ad-hoc runs

You'll touch this only if you change the validation methodology itself (how vLLM is driven, what metrics are compared, what plots are emitted). For day-to-day "did my change regress?" use, see Validating your changes.

Configs (configs/)​

configs/
├── cluster/<name>.json Cluster topology (the main thing)
├── model/<org>/<name>.json Model configs consumed by profiling and simulation
└── pim/<name>.ini PIM device specs (DRAMSim3 format)

Cluster configs are the most edited file outside serving/. Adding a new scenario almost always means dropping a new configs/cluster/<scenario>.json and not touching simulator code at all. Field-by-field schema lives in Reference / Cluster config.

Workloads (workloads/)​

workloads/
├── *.jsonl Datasets (one request or session per line)
├── generators/ JSONL builders. One subcommand today: `sharegpt`
└── README.md JSONL format reference

Adding a new workload generator is a contained change: a new module under generators/, registered in generators/__main__.py and runnable as python -m workloads.generators <name>. See Workloads / ShareGPT generators for the existing pattern.

ASTRA-Sim (astra-sim/)​

C++ network simulator, lives as a submodule. Don't edit unless the change targets simulator integration. Most simulator-side changes never touch this.

The handful of files you might edit:

FileWhy
astra-sim/extern/graph_frontend/chakra/src/converter/llm_converter.pyNew trace comm_type syntax, new trace header field, new memory location enum
astra-sim/astra-sim/workload/Workload.ccCustom collective issuance, involved_dim handling
astra-sim/astra-sim/system/AstraMemoryAPI.hhNew memory tier enum (paired with llm_converter.py)
astra-sim/inputs/...Don't edit. Generated by config_builder.py on every run

If you do edit ASTRA-Sim, rerun ./scripts/compile.sh before testing. Chakra is installed into the container's site-packages, so editing llm_converter.py alone has no effect until you reinstall it (cd astra-sim/extern/graph_frontend/chakra && pip3 install ., which compile.sh does for you).

Scripts (scripts/)​

scripts/
├── docker-sim.sh Sim container launcher
├── docker-vllm.sh vLLM container launcher (profiler / bench)
├── install-vllm.sh Bare-metal vLLM install (uv venv)
├── compile.sh ASTRA-Sim + Chakra build
└── monitor_run.py Bounded process execution and resource telemetry

You'll touch these rarely. If you add a new entry point, prefer python -m <module> (handled inside the existing containers) over adding more shell scripts.

Tests and fixtures​

There is no unit-test suite. The simulator is deterministic instead, so validation is exact equality against recorded results:

./serving/validate.sh

Every scenario compared against its recorded Total clocks (ns), then each bench/examples entry's sim.csv and validation/summary.txt checked by md5. See Validating your changes for what to do when something differs, and for the case where no scenario covers what you changed.

When adding a feature that has clean inputs and outputs (a new _lookup_* function, a new memory accounting helper), a scenario in serving/validate.sh is usually the cheapest way to pin it down. A formal unit-test framework is still an open contribution opportunity.

Where docs live​

AudienceLocation
User-facing docs (this site)docs/
Per-module developer notes<module>/README.md (each top-level Python module has one)
Top-level project READMEREADME.md
Project context for AI agentsCLAUDE.md (mirrors AGENTS.md)

When you change behavior, update the relevant page under docs/. When you add a feature, also update the module's own README.md if it covers something the website doesn't.

What's next​