Embarking on a journey with Meta to shape the future of smart glasses, I've found myself immersed in a thrilling intersection of mobile development, graphics programming, and computational photography.
My Day-to-Day at Meta
When our team comes up with a new photo technique, I get to figure out how to make it work on iOS or the glasses. Sometimes, this means tweaking things so they run faster or use less battery. It's a bit like translating a recipe so it tastes just as good but cooks quicker.
What's Next?
Right now, I'm diving deep into getting our video tricks running on the smart glasses. It's full of challenges, but that's the fun part! This job feels more like a cool project than work because it's all about blending the latest research with real-world tech.
My XR Whirlwind with Meta: More Than Just a Hackathon
Ever tried to fit 2 million pieces of data into one virtual headset? Well, that was my wild challenge for Meta's XR hackathon, which I dived into just before officially joining Meta.
The task? Bring my entire apartment to virtual life. Not just some 3D model, but a point-by-point reconstruction. Sounds breezy? Not quite. With those 2 million data points, the headset was like, "Whoa, slow down!"
But every problem's got a solution, right? Mine came wrapped in a brilliant bachelor thesis by Simon Fraiss. He'd conjured a neat method of breaking down that massive cloud of points using something called an octree and storing them neatly in a folder structure.
However, there was a twist. Unity, the platform I was using, wasn't too keen on fetching these files in the background. But sidelining the main rendering thread for file retrieval? Big no-no in the VR world.
Solution time! I wore my coding hat and whipped up a custom Java plugin. Why Java? Well, the Quest headset's pretty much an Android system backstage. This nifty plugin bypassed Unity's restrictions, doing all the file magic in the background.
Pairing Simon's genius with my Java magic, users could stroll through my virtual apartment as if they were grabbing a cup of coffee from my kitchen. And let me tell you, diving deep into this project, I had an 'aha' moment. It was clear as day: I was meant to be at Meta, making XR wonders for years to come.
LIV’s Mixed Reality App: Bringing VR to Life!
Ever thought about jumping inside your favorite VR game? With LIV’s Mixed Reality app, you can! It's a mobile magic trick that lets you record yourself right in the heart of games like Beat Saber and Super Hot.
The Cool Bits & Challenges
Here's the magic: If you've got an Oculus Quest and you're playing a game that works with Mixed Reality Capture, our app combines the real you with the virtual game world. Sounds simple, but trust me, it's a bit like putting together a jigsaw puzzle blindfolded!
The behind-the-scenes stuff? We connect to your headset using low level posix sockets, grab the video streaming from it, and then compose it together. Speed's the name of the game here, so we gave the app a bit of a turbo boost using hardware-accelerated encoding.
Initially, I thought, "Hey, why not use AVFoundation?" But, needing more control, we dove deeper and got our hands dirty with Apple’s own VideoToolBox.
The fun didn't stop there! We had to tell the headset where the phone was floating in space and make sure both were speaking the same "coordinate" language. Enter Unity 3D and its awesome math tools, bridging the gap between ARKit and the headset.
And guess what? All this tech talk meant we had to build a special plugin just for Unity! Then, we tapped into Unity’s shader capabilities to mix the videos together. Doing all this meant we were juggling memory like a circus act, keeping a close eye so nothing crashed or burned. We kept memory so tight we had to turn off ARC in iOS because it was getting in the way.
Speaking of memory, XCode Instruments and I? We're like two peas in a pod. I used it to make sure our app didn't forget any unused data lying around.
And oh, our app looks slick too! We built a super-fast UI tool right in Unity, making our designers' lives a breeze. It's all coded up in C# and reacts faster than popcorn in a microwave.
We did hit a snag with ARKit when it came to background segmentation. It wasn’t cutting the mustard in some places, and Android's ARCore didn't have the tools we needed. So, we teamed up with Segmentive.io to supercharge our background segmentation game.
Languages, you ask? This app required speaking C# for Unity, Objective-C for the iOS bits, and C++ for the heavy-duty streaming.
When we announced our app on Reddit, the world took notice!, it got featured in 'Road to VR', and guess what? Over 6,000 folks raced to download our app in just 20 days!
3D Magic from Above: The Aerialytic Adventure
Imagine taking satellite snaps and elevation info and crafting them into lifelike 3D models. Sound exciting? It was. And challenging! It's easily one of the top projects that's kept me on my toes.
A bit of backstory: I co-founded Aerialytic with a big goal. We wanted to sell solar panels in the smartest way possible. How do you figure out the perfect rooftop for a solar panel? You need to know how much sunshine it bathes in. And for that, a crystal-clear 3D model of the house is gold.
Now, here's the kicker: the tech we were using was the hottest and newest in town. We trained machines to pick out areas of interest from pictures, used smart algorithms to make sense of the elevation data, and then craftily pieced together a 3D mesh.
Ever heard of the U-net architecture for image segmentation? It was the new kid on the block, initially made for medical imaging. We were among the first daredevils to use it for satellite images.
And when it came to dissecting a roof into its bits and pieces, I cooked up a special image processing recipe. It smartly calculated the roof's angles and organized them into neat clusters.
The real twist? A lot of the cool tricks I used were so new they were mostly just scribbles in research papers. So, this journey turned me into a sort of tech detective, decoding academic mysteries and bringing them to life.
Oh, how I wish for another whirlwind project like this one!
After hanging up my Aerialytic boots, I thought, "Why not dive into a new digital playground?" And that's how I plunged into the world of augmented reality, with ARKit leading the way. The deeper I delved, the more I became engrossed in all things AR.
Now, my background in computer vision meant I had an insider's guide on the who's who of the AR world. Sure, ARKit was cool, but my radar caught onto this nifty thing called 6d.ai, a gem from Oxford’s vision lab.
Here's a fun tidbit: I played around with 6d.ai before it got swooped up by Niantic. Yep, the same folks behind the worldwide frenzy, Pokemon Go.
But what made 6d.ai stand out in the crowded AR space? These wizards figured out how to make objects in AR hide behind real-world objects WITHOUT those fancy depth cameras. They whipped up this snazzy SLAM engine that could map out a room on-the-go and let digital creations mingle with our world.
Stepping into my Technical Lead shoes at Guestlogix, I rolled up my sleeves and conjured up a nifty chat bot service for travelers. Whether you’re using our mobile SDKs or browsing the web, this chatty friend is just a click away.
Now, what does our chat bot ace at? It's got dual superpowers:
Serving up tailored content for our users like a personal concierge.
Giving our backend a crystal-clear picture of what the user really wants.
And guess what powers this digital genius? A slick AWS Lex instance, snugly integrated into our SDKs.
But here's the icing on the cake: We've coupled it with a plug-and-play UI SDK. So, our clients can sprinkle our chat bot magic onto Android, iOS, and web in, well, just 10 minutes. And trust me, I'm not just throwing numbers around – I took on the challenge and clocked myself setting it up on all three platforms. Nifty, right?
Ever wondered what happens when you blend third-party suppliers into a snazzy, single platform? Well, at Guestlogix, we did just that, and I got to build the magic wand: the Traveler SDK.
Imagine a toolkit so smooth that it packs a full-blown UI Kit, streamlined app flows, and a payment gateway. The kicker? You can have your app jazzed up with it in a mere 30 minutes. Yep, that's right!
Now, here's where things got fun (and by fun, I mean a brain-teasing puzzle): We had a bouquet of suppliers, each with its own set of rules and buy flows. The challenge? Funnel all that into one cohesive SDK. And guess who got to architect this masterpiece? Yours truly!
I channeled my inner architect to craft an SDK that’s not just user-friendly but also agile enough to welcome new suppliers without throwing a wrench in the existing setup.
A peek under the hood: We’re talking type-safe design, Facade pattern flair, and a dash of Protocol Oriented Programming. And because we love efficiency, it multitasks with an operations-based backbone, juggling network calls and data processing like a pro.
Handling the myriad suppliers? We dipped into some cutting-edge polymorphism for JSON parsing, using chic techniques like type erasing.
And if you’re wondering about the tech ingredients: It's baked in Swift, almost entirely free from third-party add-ons, save for our trusty payment ally - Stripe.
