The Apple Vision Pro: Is Spatial Computing the Future of MedTech?
Tim Cook's vision for technology without limits
From Microsoft’s HoloLens, to Google’s Google Glass, tech companies have been trying to crack the virtual reality market, and have struggled big time. It seems like Apple is the first to make a dent.
Whether it’s pioneering the smartphone revolution with the iPhone, setting new sales records with the AirPods. Apple has continuously been at the forefront of pushing the boundaries of possibility in consumer tech.
As of last month, they’ve decided to add one more thing to their list of groundbreaking technologies with the inception of the Apple Vision Pro, a device that marks Apple's ambitious journey into the world of mixed reality.
Put simply, the Vision Pro is an augmented reality headset, coined by Apple as a ‘spatial computer’. Aside from looking like a pair of ski goggles, it allows the user to visualise their personal digital environment onto the physical world.
The headset gives rise to an array of immersive experiences, from enhancing productivity with the projection of multiple work screens in the 3D world, to total zen with Environments, a feature that allows the user to virtually transport their self to whichever natural landscape they wish.
The most impressive part of the tech is that the device doesn’t require any controllers. With the simultaneous use of 12 cameras (6 world-facing, 4 eye-tracking and 2 main cameras), it allows the user to control the device using only their eyes and simple finger gestures, namely the subtle tapping of the thumb and index finger together. Many have said this makes it feel like the device is simply reading their mind, similar to Elon Musk’s vision with Neuralink.
It uses 2 CPU chips (M2 and R1) for processing all the data it captures, making it as powerful as Apple’s flagship M2-powered Mac. The device allows for a seamless integration of hardware and software, moving technology forward in reducing the barrier between humans and digital environments.
But how will the technology affect the world of medicine and healthcare?
Surgical Simulations - MEDIVIS
One of the largest use cases for XR (extended reality) in medicine is the simulation of surgical situations, allowing for the visualisation of scenarios that would otherwise be difficult to access in training, such as the embolisation of a brain tumour, or resection of a pancreatic mass. American company MEDIVIS is already on top of this.
Their holographic surgical navigation technology allows surgeons to visualise a 3D holographic render of a patient’s anatomy, mapped directly onto them in real time. This opens a world of possibilities, with easier pre-surgical planning, reducing complications & increasing safety.
The technology expands to 3D holograms of otherwise 2D radiology scans, such as CTs and MRIs. Aside from helping radiologists better map out sometimes complex pathologies, the app looks like it’d be genuinely fun to use by the average AVP user.
Drug Design and Discovery - Nanome
The field of biochemistry also benefits. Nanome is a molecular design company, focused on projecting otherwise minuscule chemical compounds and structures into large 3D models, through virtual and mixed reality. In the end, this helps structural biologists better analyse their study protein, and chemists more easily discover and design novel drugs.
Their app has already proven to be very effective for existing pharmaceutical companies such as Nimbus Therapeutics, aiming to help them save tens of thousands of dollars a year per drug developed.
We create an immersive environment where chemists can build small molecules in the context of a protein's binding site. It's like being able to build a key from inside a lock.
Sam Hessenauer, Nanome CTO
Nanome was originally made for Meta’s Quest Pro, and is currently being tested on Apple’s Vision Pro.
Mental Health Therapy - Xaia
Xaia, an app created by company Cedars-Sinai, aims to deliver mental health therapy through the use of a virtual avatar, modelled to imitate the advice and practices of trained, highly experienced therapists.
A large barrier to mental health therapy is limitations in geographical access to therapists. This is particularly true for those heavily afflicted with conditions like anxiety and bipolar disorder, who may otherwise struggle to make it to a psychologist.
Xaia overcomes this, delivering therapy whilst completely immersed in serene, relaxing environments, one’s that would otherwise only be available during a retreat to the mountains of Peru or Mayan villages of Mexico.
These examples are currently being actively worked on and implemented onto the Apple Vision Pro. However, they don’t cover all the possible use cases; here are some more I’ve come up with off the top of my head:
Blindness - most people with blindness don’t have complete lack of sight. The device could help by highlighting environmental surroundings for the partially blind using high contrast colours, something similar to below. This would aid the visually impaired in navigating their environments and recognising particular objects.
Bedside examinations - using a similar technology to that of MEDIVIS, physicians could use the device as a diagnostic companion when examining ward patients, leading to easier identification and analysis of important clinical signs. For example, the device could recognise a patient who is tachycardic (using variations in light reflection from the fingertips), jaundiced and ascitic - helping a gastroenterologist diagnose decompensated liver disease in an alcoholic patient.
Phobia desensitisation - most mental health use-cases for the Vision Pro would take advantage of its immersive environments. A person with agoraphobia could be placed in the busy streets of central London, virtually, to help them overcome their fear of public spaces (this is already a form of therapy, named ‘Virtual Reality Exposure Therapy (VRET)’.
My thoughts in a nutshell:
It may be early, but in my opinion Apple’s Vision Pro is the first step into the future of computing. Imagine a device that can do everything an iPhone, iPad and most computers can and more, all without a physical screen. It’s a huge engineering feat - the device uses 2 super powerful processing chips (M2 and R1), and can fit all its necessary components into such a small space.
However, at this stage, there are a few limitations to the device:
Typing is a chore:
Typing on the device is quite cumbersome. It uses a virtual keyboard, forcing you to type using one index finger at a time. This makes something so pivotal to work and productivity so much more time-consuming.
Right now, the solution is connecting a wireless keyboard to the device, giving the tactile feedback you need when typing.
It’s hard to use for more than a few hours:
The AVP is pretty heavy compared to other XR devices, with most of its weight distributed to the front of the head. During long use, such as during a neurosurgery or an educational simulation session, this could become very uncomfortable.
It’s also pretty bulky, which could make it awkward to use in intimate environments like a hospital ward or patient consultation.
Its battery life is capped at ~2 hours; to use it for longer, you’d need to plug its external battery pack into a wall, limiting its portability.
$3500 is a big ask:
Its price point is on the higher end of the spectrum, compared to its competitive counterparts. An outlier to this is Microsoft’s HoloLens 2 ($5599), which isn’t positioned for general or household use, but targeted towards enterprise users for specialised work scenarios, particularly in manufacturing factories, construction sites and engineering firms.
This makes the device (at its current version) more of a developer and tech enthusiasts’ toy, rather than something you’d see the average person using.
It also makes it a hard investment for healthcare systems across the world, who can often be reluctant to adopt new innovative technology. Apple would really need to sell the benefits of the device to hospitals, focusing on cost savings and improved patient outcomes.
In my opinion, each of these limitations will be addressed in future iterations of the Vision Pro. As the technology develops, Apple will undoubtedly make the device more lightweight, portable and intuitive.
In regards to widespread adoption, the naming of the device tells us all we need to know. In a few years, Apple will most likely release a non-pro ‘Apple Vision’, a cheaper version with less features than the Pro of that time, for use by the less tech-savvy general public. I believe it’s all part of Apple’s vision for the future of computing.
All in all, could I see it being adopted in the medical environment?
I think this is a possibility further down the line, once the device resembles a pair of sunglasses rather than ski goggles. It’s lack of portability and bulkiness right now are some of its biggest drawbacks.
The nature of the device’s hardware (a bunch of cameras reflecting their recordings onto its internal screen) opens up a can of worms in regards to patient privacy. The device must permanently capture video input to work, leading to the potential for recording in private environments without consent (intentionally or unintentionally).
Apple has addressed this, claiming to limit the amount of data they store in their servers by processing as much as they can on the device itself
I can see the device being accepted eventually in controlled environments, such as the operating room and during seminars. However, for the device to be used on wards and at the patient bedside, there would need to be a big shift in societal norms - to the point where the tech is as widely used as our mobile phones, and trust has been built with the device to overcome any potential privacy concerns.
What do you think? Will the AVP have an impact on medicine, and the world as a whole? Feel free to reply to this email with your thoughts, or leave them in the comments.
If you liked this article, check out other Medtechish articles like it. And make sure to subscribe for more, direct to your inbox.
Great read! Especially enjoyed the original ideas on how to utilise the AVP’s further!
Hadn't spent much energy considering the applications of AVP in other industries, glad to see you've done that for me. Really interesting on the blindness point. Google Gemini 1.5 is apparently really good at processing videos. Even pre-gemini we could do stuff like this.
https://x.com/charliebholtz/status/1724815159590293764?s=20
That blindness scenario may not even require VR or it could be very intelligent with the intersection of XR/AI.