Running V-JEPA 2.1 on an iPhone

V-JEPA 2.1 encoder and predictor with 1.9 billion parameters

iPhone 15 Pro | Core ML | Local inference

The main result

We ran the complete V-JEPA 2.1 ViT-G encoder and predictor on an iPhone 15 Pro, with no server involved in inference. Together, these models contain about 1.9 billion parameters. With four-frame clips at 128 × 128 pixels, the loaded model stack took about 108 ms per fixed-input run. In a separate live camera test, the app produced about five updates per second for more than five minutes.

Three design choices made this deployment possible: four-bit lookup-table storage for the encoder weights, four encoder stages that stay loaded between calls, and a separate FP16 predictor. The model files occupy about 1.05 GB. Core ML runs the stages in sequence, without reloading them from storage for each update.

These measurements describe an interactive phone demonstration. The live test reached high thermal states, so sustained operation is the next target. We explain how the model was prepared for the phone, what compression changed, and which parts of the approach other engineers can reuse.

1. Why JEPA

We want to build an assistant that follows a person’s activity and prepares useful help for what comes next. It could bring up the next instruction during a task, or leave the interface unchanged when help is not needed. This goal led us to JEPA, short for Joint Embedding Predictive Architecture. Its encoder converts observed video into learned numerical features, called latent representations. Its predictor uses those features to estimate the representation of a hidden part of the video. When that hidden part comes later in time, we can study possible future states without generating the future video itself. [1, 2]

These features do not directly tell an interface what to display. We need an action head, a small model that learns to turn features into labels for human actions. A separate policy would then decide whether assistance is useful and what to show. Our phone demonstration already includes an action head. The next part we want to develop is the policy that connects these outputs to useful assistance, including when to leave the interface unchanged.

We chose to run the encoder and predictor on the phone. This keeps video processing local and removes the need for a network connection during inference. We first made the encoder run, then added the predictor and tested repeated updates from the camera. This gave us a working system for testing action heads. Longer operation on wearable devices will also require tests of battery use and heat.

2. What runs on the phone

For each update, the app collects four recent camera frames into one clip. Each frame is resized to 128 × 128 pixels before the clip enters the encoder.

The encoder does not process the clip as one large image. It first divides it into tubelets: small image patches that cover the same location across two consecutive frames. Each tubelet is converted into a list of numbers called a token. The encoder processes these tokens together to learn relationships across the image and over time.

The encoder produces numerical features that describe the video. The predictor uses observed features to estimate features for a target part of the video. This target can be a later video segment. Both models run through Core ML, Apple’s software framework for running machine-learning models on the device.

The app displays the action head’s highest-scoring labels. This gives us a readable output to inspect as the camera view changes. Our next step is to connect these labels to a UI policy. That policy would decide whether to offer help and what the interface should show.

3. How the model fits on the phone

The phone needs space both to store the models and to run them. Storage keeps the model files. Working memory, or RAM, holds data used while the models run. Model weights are the numbers learned during training. With FP16, each weight uses two bytes. Our encoder and predictor would need about 3.8 GB for weights alone. The models do not have all the phone’s RAM to themselves. iOS, the app, and the temporary results produced during inference also need memory. We compressed the encoder weights while keeping all 48 encoder blocks and the predictor.

We compressed the encoder weights with LUT4 palettization, shown in Figure 3a. Each group of weights shares a table of 16 values. Each compressed weight uses a four-bit index that selects an approximate weight value from this table. The predictor stays in FP16. Together, the model files occupy about 1.05 GB. This is file storage, not the total RAM needed to run the models. Temporary results, called activations, remain in floating point. LUT4 therefore describes how we store weights, not the precision of every calculation. [5]

Compressing the encoder provided enough storage reduction for the phone demonstration. The smaller predictor occupies about 120 MB in FP16 and fits alongside it. We kept activations in floating point because this was faster in our matched phone test. With the LUT4 weights and FP16 predictor unchanged, adding INT8 activation quantization increased total model time from about 115 ms to 316 ms, as shown in Figure 3c. The demonstration therefore keeps mainly FP16 activations, with selected attention operations in FP32. Further predictor compression remains a separate optimization opportunity.

A small model file does not guarantee that the phone can load it. Core ML also needs working memory to prepare the model for execution. When we packaged the complete encoder as one file, iOS stopped the app during loading, before it could process a clip. This was consistent with memory pressure during preparation, although we did not confirm the exact cause. We therefore divided the encoder into four smaller model files, which loaded successfully. The boundaries follow four points where the original encoder provides features to the predictor, so all 48 blocks remain in use. The app loads these stages and the predictor at startup and keeps them available. Each new clip then passes through the stages in order, without loading the model files again.

Using fewer bits does not guarantee a faster model. Converting values or changing how operations run can add extra work. In our test, INT8 slowed the encoder, while the predictor took the same time. Profiling is the next step to find out why.

4. Preparing the model for local inference

The released model uses PyTorch, the framework used to build and train it. Our iPhone app uses Core ML to run the model, so we needed to convert it. We first traced each encoder stage and the predictor with example inputs to record their calculations. We then used Core ML Tools to convert these calculations and the learned weights into model packages for the app. This step did not retrain the model. We fixed the input clip size at four frames, each 128 × 128 pixels. The video content can change; only the number and size of the frames stay fixed. This gives the conversion tools a known workload to prepare for execution. [3, 4]

Before measuring speed, we checked the converted model’s outputs against the original PyTorch model on the same two video clips. We first checked for invalid numerical values, such as infinities. We then used cosine similarity to compare the output features. This measure compares the direction of two feature vectors; a value near one means their directions are close. It helps us measure changes caused by conversion and compression. Feature similarity does not tell us whether an action label is correct.

For live use, we prioritize recent activity over processing every camera frame. The app runs one clip at a time while it continues to collect new frames. When that run finishes, it starts the next clip from the latest available frames. It does not keep a growing list of clips waiting for processing. This prevents the results from falling further behind the camera view when frames arrive faster than the model can process them.

5. Measured phone performance

We measured the complete encoder and predictor on an iPhone 15 Pro through Core ML. Each input contained four frames at 128 × 128 pixels. The encoder used LUT4 weight storage, and the predictor used FP16.

5.1. Model speed and live updates

We measured speed in two ways. First, we ran the loaded encoder and predictor ten times with the same prepared input. The median run time was about 108 ms. This measures model execution, without camera capture or screen updates. We then tested the complete app with the live camera. It completed 1,605 updates in about 5.3 minutes, averaging five updates per second. This second measurement includes the work needed to collect frames, prepare clips, run the models, and update the screen.

The phone was already warm from earlier tests when the live run started. During the run, iOS reported serious and critical thermal states. [6] The app continued to produce results without a logged error or crash. The five-updates-per-second result therefore describes operation under these test conditions. Our next step is to repeat the test from a cool start and track update speed alongside thermal state over a longer period. This will help us choose an update rate for longer use.

Before processing clips, the app must load and prepare the four encoder stages and the predictor. This took about 49 seconds in the recorded startup test. The models then stayed loaded, so subsequent runs did not repeat this preparation. The 108 ms result measures execution after loading, not the wait before the first result. Reducing this startup delay is another target for future work.

5.2. Comparing model outputs

For this comparison, we used the Core ML version with LUT4 encoder weights and an FP16 predictor. On the two test clips, its observed-activity features stayed closer to the original PyTorch outputs than its predicted-future features did. This result comes from the centred cosine comparison with the original PyTorch outputs. It shows that the two types of output changed by different amounts in this test. It does not tell us how often an action label would be correct. To guide further compression, we will extend the comparison to more clips and measure action recognition accuracy alongside feature similarity.

Figure 6 shows a separate encoder test on an iPhone 15 Pro using a dog video. To display the model’s features, we use principal component analysis (PCA). This reduces each set of feature values to three values, which we display as red, green, and blue. We calculate this mapping from the reference model’s features, then use the same mapping and color scale for the phone outputs. This makes the visible patterns directly comparable. The colors show a simplified view of the features, not reconstructed video or an image of the future. [1]

6. From model features to useful assistance

To turn the model’s numerical features into readable action labels, we train a small model called an action head. For the four-frame phone demonstration, we use Something-Something V2, a video dataset with 174 action categories. Each training example pairs video features with an action label. The head learns to score these categories, and the app displays the labels with the highest scores. During this training, the encoder’s learned weights do not change. We can therefore train the action head without retraining the large encoder. [7]

6.1. From action labels to interface responses

Recognizing an activity is only the first step toward useful assistance. If someone is repairing a bicycle tire, the interface could offer instructions for the next step. The next challenge is learning when that help is useful and when the interface should stay quiet.

We did not train a UI policy: the component that decides what the interface should show and when. Recognizing an action does not tell us which response would help. Two people assembling the same object may need different guidance, and neither may want an interruption. Training this component would need examples that pair a person’s activity with a useful response, including cases where the interface should do nothing. We would then test those responses with users. The phone demo produces action labels; learning how to use them to offer help remains future work.

7. Toward sustained wearable assistance

The phone demo gives us a starting point. The next question is whether this system can stay useful throughout the day. On a wearable device, speed is only part of the problem: battery life and heat also matter. We may not need to run the model as often when little is changing. For example, someone sitting at a desk may need fewer updates than someone moving through an assembly task. Longer tests would help us find an update rate that keeps the system responsive without doing unnecessary work.

The model must also provide useful information at that operating rate. Broader comparisons with the reference model and action-level tests can guide further compression. A comparison with the original V-JEPA results requires matched inputs and evaluation methods. [1] For future-action prediction, the next tests should measure whether predicted features improve decisions about what happens next. User-reviewed examples can then connect those decisions to useful interface responses.

For someone wearing smart glasses, the useful result is not an action label. It is getting the right help at the right time: the next assembly instruction, a relevant diagram, or information they would otherwise need to search for. Just as important is knowing when to stay quiet. Our phone demo gives us a way to develop and test these ideas with local video processing. The longer-term goal is an assistant that helps people continue their task, while leaving them in control of when and how it responds.

References

[1] L. Mur-Labadia et al. (2026). V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning. arXiv:2603.14482v3. Model background, hierarchical training, evaluation protocols, and PCA visualization. V-JEPA 2.1 paper

[2] M. Assran et al. (2025). V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv:2506.09985. Latent prediction and the separate action-conditioned model. V-JEPA 2 paper

[3] Apple. PyTorch Conversion Workflow. Guide to Core ML Tools. Graph capture and conversion. Apple conversion guide

[4] Apple. Model Prediction. Guide to Core ML Tools, section Using Compiled Python Models for Prediction. Package compilation and device specialization. Apple compilation guide

[5] Apple. Palettization Overview. Guide to Core ML Tools. Lookup-table weight storage and grouped-channel compression. Apple palettization guide

[6] Apple. ProcessInfo.ThermalState.serious. Foundation documentation. Thermal limits and reduced system performance. Apple thermal-state documentation

[7] Qualcomm Technologies. Something-Something v. 2 Dataset. Official dataset description and 174 action labels. Something-Something V2 dataset

[8] Apple. iPhone 15 Pro specifications. Apple hardware

[9] Apple. Choosing a resource storage mode for Apple GPUs. shared memory

[10] Apple. MLComputeUnits.cpuAndNeuralEngine. Core ML settings

[11] Ego-Exo4D. Official dataset website. Source of the bicycle-repair frame in Figure 7 (cmu_bike01_4, 00:22). Ego-Exo4D dataset