Posts

Stuart Bradley is the founder and CEO of not one, but two companies – Carolina Speech Pathology LLC and Altaravision. We caught up with him on a busy Monday afternoon in between meetings, and he was gracious enough to take some time to talk with us about his experience as founder of Altaravision and the interesting journey of their flagship product, NDŌʜᴅ.

Put simply, NDŌʜᴅ is the most portable, high-definition endoscopic imaging system on the market today and an invaluable tool for speech pathologists. It has been extremely well received by the medical community, but its path from concept to market was not without its obstacles.

Where did the idea for NDŌʜᴅ come from? Because it is a very specific product.

It came from a need. Specifically, the need to be able to do machine vision on a Macintosh. Surprisingly, there really wasn’t any software that addressed it anywhere in the marketplace.

Would you mind just briefly explaining what machine vision is?

Sure. Machine vision is the ability for a computer to view imagery or an object, take that information and then display it. Essentially, it is a computer’s ability to see.

And the capacity to do that wasn’t on a Mac? That’s interesting.

Well, no. There was plenty of software out there, but it was all secondary purpose. The bigger issue was that nothing had the capabilities you would need in a medical setting.

It all comes down to video capture. All of the off-the-shelf software could capture images, but they suffered from significant lag. What you saw on the screen might be a full second behind what was happening in real time. That might not seem like much, but when you are dealing with medical procedures that kind of lag isn’t going to cut it.

I played around with off-the-shelf software for a number of years and finally found something I thought might work, but there were a ton of features that I didn’t want or need. I reached out to the developer to make me a one-off, but he was ultimately unable to deliver a final product. That’s what led me to Oak City Labs.

Once you had your software developer in Oak City Labs, what was the hardest part about going from this idea you had to an actual finished product?

By far, the biggest hurdle was doing it in a way that maintains compliance with FDA regulations. Jay Lyerly, the one who was doing the coding, knew that from the start and was able to work with my FDA consultant in a way that we could survive an FDA audit.

The thing is, FDA audits are worse than IRS audits and you’re guaranteed to get one, whereas IRS audits are random. As a medical device company, we are audited every two years by the FDA. Thanks to Jay and Carol at OCL, we’ve been able to pass every single audit with zero deficiencies, which is nearly unheard of.

Was there a moment when you got NDŌʜᴅ out into the world and thought “ok, we did it.”

Yea, there was. With FDA-regulated software you actually do have to draw that line in the sand. Unlike other software development cycles, where updates are always being pushed out, you can’t do that with medical devices. It has to be the finished product from the day it comes out. If you add features, it has to go back through the FDA approval process, which, as you can imagine, is pretty lengthy.

If you could do it all over again, is there anything that you’d do differently?

We bootstrapped the entire thing, with CNP essentially acting like an angel investor for the product. That was really tough, especially when there are people out there actively looking for good products to invest in. If I had to do it again, I would have taken the time to seek out some outside investment instead of just jumping in and doing it all myself.

When you think about where you are today as a business owner, is there anything that sticks out to you as the thing you are most proud of?

Honestly, being able to take on, create, sell and make an actual viable business out of a medical device when I had no prior experience in that industry. I had owned Carolina Speech Pathology for years, but the journey with Altaravision and NDŌʜᴅ was an entirely new one.

What’s your favorite part about doing what you do?

It has to be the satisfaction I get from solving hard problems, and the fact that it’s never boring.

Finally, whenever you have clients or colleagues that are talking about Altaravision or the NDŌʜᴅ product, what do you want them to say or know about it?

I want them to know two things. First, I want them to know it works, and always works. Second, that it is designed to be incredibly easy to use. If you can use Facebook, you can use NDŌʜᴅ.

For more on Oak City Lab’s work with Stuart Bradley and Altavision, check out this article Jay wrote on Computer Vision for Medical Devices via Core Image. If you have an idea and need a software development partner, or if you just have some questions about the development process, we’d love to talk to you. Follow the link below to contact us!

At the end of next month, I’ll be hanging up my project manager hat and embarking on a new journey. Over the past year, I’ve been balancing my role here at Oak City Labs with graduate school as I’m pursuing my Master’s of Arts degree in Teaching from Meredith College. Though I’ve spent the better part of the past decade working in the technology industry, I felt a shift in my goals and interests and, with the amazing support of the Oak City Labs leadership team, I decided to make a career change.

Why tell you this?

Because in some ways, I’m not really leaving the technology industry at all. I’ll just be applying my skills in a different way to a different set of “clients” (read: elementary school students).

In my time spent in graduate school and in field placement positions this past year, it has become increasingly clear to me that there is more of a need for globally-minded, technologically-equipped educators than ever before. The reality is that educators need to be preparing students for jobs that don’t even exist yet. Yes, you read that correctly. According to the World Economic Forum, 65% students entering elementary school now (aka my future “clients” if you will) will hold jobs that don’t even exist yet. And that data is two years old. The numbers have certainly increased since then.

I think about some of my recent blog posts on artificial intelligence, machine learning, computer vision and machine vision. As cutting edge as these technologies are, odds are they will have significantly evolved by the time current primary and younger secondary school students graduate high school in 8-10 years. Therefore, instead of preparing students for specific jobs, we are charged with preparing students with skill sets that will grow with them as this world also grows.

Figuratively, that preparation is a multi-layered, interdisciplinary approach to learning beginning with the earliest grades through high school graduation. It looks different for every student and every teacher. Practically that preparation begins with integrating meaningful technology in the classroom, expanding student learning through social studies and science, as well as enhancing student understanding through the arts.  

The hope is through all of our efforts, we’ll prepare students not for the jobs that artificial intelligence will certainly replace, but for new jobs that work alongside artificial intelligence. While machine vision may eliminate the need for a factory worker to inspect products, machine vision will certainly create the need for software engineers to manage the inspection system. And desirable software engineers will need to possess specific skills in technology, along with soft skills like critical thinking/problem solving, collaboration, communication and creativity/innovation.

So I leave this role, company and industry with a lot of change ahead, but I’m hopeful that my efforts will foster students that are better prepared for those jobs that don’t even exist yet. And maybe even some future employees of Oak City Labs.

PS – Did you hear? We’re currently on the hunt for a project manager and software developer. Check out our Careers page for more information and job details.

Today’s blog post will demonstrate some basic neural network code that uses the fast.ai library to train a model to categorize images into one of n categories and get predictions on single images. You can take this code and change it to train the model on any image categories you would like. I won’t go into too much detail on what is going on behind the scenes and will focus on showing you how to get a working model.

The code is based off of lesson 1 (and partially lesson 3) of the fast.ai course, which I strongly recommend taking a look at if you are interested in machine learning. To run the code, you will need the fast.ai library, which is built on top of the PyTorch deep learning framework. Additionally, to run the code in any reasonable amount of time you will likely want a GPU. Take a look at fast.ai lesson 1 to learn how to set up a GPU computer in the cloud using Paperspace.

The code above constructs a neural network using the ResNet-34 model, which is pre-trained on the ImageNet dataset. Our code simply adds a few layers to the end of the pre-trained model and trains them.

Line 7 essentially packages up the images so that they can be passed through the model and used to train it. Line 8 creates the learner object capable of actually doing the training. Line 9 actually performs the training with a learning rate and a number of epochs (cycles through all of the input data). With the fast.ai library, these three lines of code are all that is required to build and train a convolutional neural network. Fast.ai makes the process of training a neural network incredibly simple by taking care of preparing the pre-trained model, adding additional layers, and setting some hyper-parameters to sensible defaults if not specified. At the same time however, the source code of the fast.ai library is relatively concise and readable, so as you use fast.ai I encourage you to take a look at how it is leveraging PyTorch under the hood.

To train this model on your images, change the PATH variable to point at where your images reside. The PATH directory should have inside it a “train” and a “valid” directory. Inside these directories there should be a directory for each category of image with the images for that category inside. You should place roughly 80% of your images for each category in the train directory and the other 20% in the valid directory.

To get a prediction for a single image:

In the code above we get the transformations that are applied to the validation image set, apply them to our image, and get our model’s prediction. We then find the predicted category name from our data object’s classes attribute. Note that because inputs to our model are assumed to be tensors (rank 4 tensors in our case, or tensors with 4 dimensions), we must add another dimension to our 3-dimensional image with img[None].

The code in this post will work best on images that are similar to ImageNet images, which are primarily images of objects, animals, and other easily photographable things. If our images were something much different like X-rays or satellite images, the pretrained ResNet-34 model would likely benefit from some extra training that we have not done here and we might want to use different data augmentation transforms (the “transforms_side_on” variable from our ImageClassifierData instantiation). You may have noticed that during training, several rows of numbers get printed to the console.

[0. 0.04368 0.02732 0.99121]

[1. 0.03997 0.02237 0.99023]

[2. 0.04147 0.02266 0.99072]

The fourth number in each row is the accuracy of our model based on how well it categorizes the images in our validation set. The hope is that this number is as close as possible to 1.0 by your final training epoch. The number may vary greatly depending on your number of images, number of categories, and quality of images, but hopefully it is somewhere between 0.8 and 1.0.

There are several techniques we could employ to increase the accuracy of this model, but I will will leave them for a future post.

Machine learning (ML). Artificial intelligence (AI). Internet of Things (IoT). It seems these days there are so many complex terms flying about. In this blog, we’ll break down the basics and share how you’re experiencing these technologies in ways you may not have even realized.

Artificial Intelligence

We can’t talk about machine learning without first discussing artificial intelligence. Long before ML and IoT were commonplace, the concept of AI existed. The term was coined by John McCarthy decades ago in 1955 describing it as, “making a machine behave in ways that would be called intelligent if a human were so behaving.”

Simply put, AI is when machines handle tasks in an intelligent manner in the same ways humans would. Did you know you have AI all around you? Apple’s Siri, Amazon’s Alexa, Google’s personal assistant, and Tesla cars are all examples of AI at work.

Machine Learning

Machine learning is a product of artificial intelligence. Where AI is the ability of machines to handle tasks in an intelligent manner, ML takes it one step further and says that machines should not only handle complex tasks the same intelligent way humans would, but they should also have the ability to learn how to handle the complex tasks themselves. This concept is also decades old, dating back to 1959 when Arthur Samuel coined this term defining ML as, “the ability to learn without being explicitly programmed.”

Do you know that saying ‘all squares are rectangles, but not all rectangles are squares’? Well, the same is similarly true here: all ML is AI, but not all AI is ML.

Interested in the technical aspects of ML, including how neural networks are used? Our software engineer Taylor shared his thoughts and experiences on the blog before.

So how are you experiencing machine learning in your everyday life? Amazon and Netflix recommendations are great examples. As the two learn your buying or watching habits, they make recommendations of what you may be interested in next – and most times, they’re spot on! Or have you ever noticed that as you use your iPhone more and more for texting that it begins to pick up on your common spellings, acronyms and dialects? That’s machine learning.

My name is Ashlee. It’s spelled a little differently than most Ashleys, but my parents spelled it that way when I was born and I haven’t found it on keychains ever since, so what can I do? Apple didn’t program the iPhone to know that when I start typing Ashlee to not autocorrect it to the more popular Ashley, but after several times of me typing and then undoing the autocorrect to change the spelling back to Ashlee, my phone learned that when I type Ashlee, I mean Ashlee not Ashley. That’s machine learning.

And what about Internet of Things?

Artificial intelligence and/or machine learning can converge in the physical world with Internet of Things. IoT is the category of physical objects – like lightbulbs, thermostats, door locks, plugs, etc. – that are connected to the world via the Internet. When those IoT devices also employ ML, like when a Nest thermostat learns your temperature preferences and adjusts the temperature inside your house for you without your action, the two worlds are colliding.

At Oak City Labs, our mission is help businesses and organizations solve daily problems with technology. Utilizing AI and/or ML are excellent ways to accomplish that task. Do you have a problem that you need help solving? If so, let us know! We’d love to chat.

Read more about Computer Vision & Machine Vision here.

At Oak City Labs, technology excites us. We keep a keen eye on emerging technology and enjoy observing its impact on the world around us. For one of our last blog posts of 2017, I asked our developers for a technology that impressed them this year, and a technology they are excited about for in 2018. Read on for the results!

Jay Lyerly, CTO

Apple Face ID Oak City Labs Raleigh Durham Mobile App Development

Source: https://www.macrumors.com/2017/11/04/face-id-brothers-video/

Looking Back at 2017 – Face ID

Burgeoning from the recent announcement that Apple is investing $390 million into its Face ID and Airpod tech, the hype around Face ID has grown exponentially since its announcement at the iPhone X release event in September. Though it has had its troubles, Face ID is an exciting technology that pushes the boundaries of facial recognition and its plethora of applications. Jay is most excited about the idea of continuous authentication when it comes to Face ID.

 

Apple HomePod Oak City Labs Raleigh Durham Mobile App Development

Source: https://9to5mac.com/guides/homepod/

Looking Forward to in 2018 – HomePod

A new challenger has appeared in the market for smart home connectivity. The Apple HomePod is billed as a “breakthrough home speaker” by Apple’s VP of Worldwide Marketing, Phil Schiller. Unveiled at WWDC 2017, the HomePod houses an impressive woofer, tweeter array, microphone array, and A8 chip. All these parts were specially crafted to fulfill Schiller’s definition of a successful speaker, which must: “Rock the house”, have “spatial awareness”, and “be a musicologist.” According to Jay, he is looking forward to “Alexa done correctly.” Them’s fightin’ words, Jay.

Taylor Harrison, Software Engineer

Looking Back at 2017 – Kotlin

Kotlin Oak City Labs Raleigh Durham Mobile App Development

Source: https://kotlinlang.org/

For Android developers like Taylor, Kotlin is the newest hotness. Much like Swift is overtaking Objective-C as the programming language of choice for iOS apps, Kotlin is set to compete with Java as the main language for developing Android apps. In 2017, Kotlin gained 100% interoperability with the Java language and Android toolsets, and we are excited. By design, Kotlin is concise, safe, and gets along well with all popular IDEs. Taylor enjoys that Kotlin is less verbose than Java and is so excited about it that he wrote a blog post about it.

Looking Forward to in 2018 – Augmented Reality

AR Stickers Oak City Labs Raleigh Durham Mobile App Development

Source: https://thenextweb.com/google/2017/12/11/force-strong-google-pixels-new-ar-stickers/

With Google and Apple announcing their new, updated AR platforms at their respective conferences this year, the world of augmented reality seems full of possibility for 2018. Google recently launched a Star Wars AR for the Google Pixel 2 (seen above) that allows people to appear alongside everyone’s favorite Galactic Empire. Apple recently released an awesome commercial featuring their AR technology, which appears to include placing a piano and other objects in the area around the user. From games like Pokémon Go to architectural design solutions, we fully expect 2018 to be a year of rapid growth for AR. The wearable market for AR most excites Taylor for the upcoming year.

Trevor Brown, Software Engineer

Looking Back at 2017 – Apple Machine Learning for iOS

Machine Learning iOS Oak City Labs Raleigh Durham Mobile App Development

Source: https://hackernoon.com/swift-tutorial-native-machine-learning-and-machine-vision-in-ios-11-11e1e88aa397?gi=2c186ebbe699

What do Siri, your iPhone camera and your iPhone keyboard all have in common? All of these technologies use machine learning to create advanced user experiences. In order to bring these advancements to developers, Apple developed Core ML, a machine learning framework that can be used across all Apple devices. Core ML allows developers to create apps utilizing the thousands of hours of work and research that went into the machine learning used by Apple’s own products. Trevor is excited about the possibilities that machine learning opens up, including the ability to offload certain tasks to requests off the phone that get run through Apple’s machine learning framework. This, in turn, will open up the hardware on the device to be optimized and used efficiently while running the non-necessary tasks off the phone using machine learning.

Looking Forward to 2018 – Wearables Advancements

Apple Watch Oak City Labs Raleigh Durham Mobile App DevelopmentSmart watches are nothing new. According to certain estimates, the US wearables market is set to double by 2022. Announcements by Apple this year that the Apple Watch will contain more advanced sensors to be used to monitor health and promote good fitness practices. From keeping track of heart rate to measuring heart rhythm, the Apple Watch will provide an easy and convenient way of tracking many aspects of people’s health. Trevor is excited for the data gathered from these devices to provide more accurate data which can be used in studies to help the greater population. For instance, the tracking of blood pressure for diabetics and irregular heart rhythms to predict heart conditions.

Cory Scheviak (me!), Software Engineer

Looking Back at 2017 – Rust (Programming Language)

Rust Oak City Labs Mobile App DevelopmentWhile I, myself, am not a low-level programmer by trade, I have experience working with low-level languages such as C, C++ and Assembly. Through the grapevine, I learned about Rust, a systems programming language that, according to its docs, is focused on three goals: “safety, speed, and concurrency.” Rust was voted the Most Loved programming language of 2017 on Stack Overflow for the second year in a row, and continues to push to replace C++, the kingpin of low-level object-oriented programming. While not directly applicable to what we do here at Oak City Labs, it has been fun seeing an actual contender for replacing C++ come onto the programming scene.

Looking Forward to 2018 – Progressive Web Apps

Progressive Web Apps Oak City Labs Raleigh Durham Mobile App Development

Source: https://developers.google.com/web/progressive-web-apps/

My last blog post about Progressive Web Apps highlighted some of the pros and cons of the up-and-coming software ideology touted by Google as the future of webapps. Earlier this month, Google announced the release of an optimized version of Android called Android Go. Optimized for devices with less RAM, Android Go contains a version of Google Maps called Maps Go, which is a progressive web app. While not as fully-featured as Google Maps, Maps Go provides a lightweight, offline experience for its users that runs efficiently on their weaker devices. I am most excited to see if companies like Facebook, Twitter and Reddit turn their websites into progressive web apps instead of building native, or if they choose to use a progressive web app ideology alongside their native apps.

2018 and beyond

Gadget technology witnessed impressive gains in 2017, with a large emphasis on wearables that streamline everyday processes and provide value to users. Similarly, both Swift and Kotlin (the future of iOS and Android programming, respectively) saw a great increase in focus from developers looking to match pace with cutting-edge trends. Data gathered from devices such as smart watches and home automation devices will continue to provide valuable insight into human actions and needs that will further shape the development of such devices. It’s no question that 2017 was an incredible year in technological development, and we can expect much of the same in 2018. Viva la technology!

Machine learning can be an intimidating field to get into. In the event you want to dive right in, I’m providing a very brief and fairly high-level overview of how machine learning works. I’ll also cover a very simple image recognition project using Tensorflow that you can get up and running with a little bit of Python knowledge.

To begin, let’s discuss some of the basics. Machine learning is largely practiced through the use of neural networks. Neural networks are comprised of nodes that take an input and produce an output. These inputs and outputs are generally n-dimensional arrays. For example, an image you are providing as an input to a neural network would be represented as a 3-dimensional matrix with dimensions (width, height, number of color channels). Each node can be thought of as an operation that manipulates the data that comes into it. Operations can be as simple as addition or multiplication or as complicated as a multidimensional convolution. Each node also has variables (n-dimensional matrices that are often referred to as “weights”) that are used in whatever operation the node performs to transform the input. These weights on each node are the neural network’s “brain.” During training of a classification neural network, the weights are incrementally changed based on an error function using an algorithm like stochastic gradient descent (which attempts to minimize the error). This error function is a node that takes the neural network’s output value and the known correct output value (which is available during training) and calculates the error (often something like cross-entropy). After the training data set has been sent through the neural network many times and the error has been minimized by altering the weights, features of the training data are “remembered” in the weights. After training, when sending in new data that the neural network has not seen before, the network will use what it learned about the training data and apply it to this new data. In a classification neural network, this means detecting familiar features of the data and assigning probabilities that it belongs to each of the known classes. This is a high-level overview and isn’t quite the complete story, but hopefully, it provides some insight into how a neural network functions.

Machine Learning Oak City Labs Raleigh Durham Mobile App Development

The Inception-V3 neural network

So now let’s actually do some machine learning with a simple script that uses Tensorflow to detect bunnies in images. For the sake of brevity, I will leave it to the user to get set up with Python and Tensorflow and to become familiar with how Tensorflow is structured. Additionally, there is a wealth of example Tensorflow code available in the models Github repo. The classify_image.py file from this repo will be the basis for my bunny-finding script. The classify_image.py script takes an image file, loads a pre-trained neural network trained on the ImageNet data set, and classifies the image, outputting the top 5 probabilities and the corresponding image categories. I will discuss some of the code in this script and explain how it can be repurposed to find bunnies.  

This neural network (called the Inception-V3 model) has been pre-trained on the entire ImageNet database, a computationally intensive task that can take days depending on the resources available. With the GraphDef file in the classify_image.py file, the nodes of the Inception neural network and the trained weights associated with each node can be loaded, bypassing the intensive training step. In four lines of code, you have a trained graph capable of classifying images across 1000 categories.

Now a Tensorflow session can be started and inference run on an image. First, provide the image input tensor with a string containing the JPEG image data. Tensors in the loaded graph can be accessed by name. The tensors that need to be accessed here are ‘softmax:0’ and ‘DecodeJpeg/contents:0’.

The two code blocks above comprise essentially all of the “machine learning code” contained in the classify_image.py script. Most of the other code pertains to loading the image file into a string, downloading/extracting the zip file that contains the GraphDef file, and translating the indices of the output softmax tensor to human-readable categories. The predictions variable is a Tensor containing a 1×1000 array of the probabilities that the input image belongs to each of the classes. Translating the indices of this array to actual classes is just a matter of using the imagenet_2012_challenge_label_map_proto.pbtxt and imagenet_synset_to_human_label_map.txt files used in the NodeLookup class (unzipped with the GraphDef file). For simplicity’s sake, I skip that step in my bunny-finding script.

You can find my bunny finder script at the gist here

To run the script, call it through the command line and provide an image url as an argument.

python bunnyfinder.py image_url

The script instantiates the BunnyFinder class and passes in a list of urls to the BunnyFinder.findbunnies(url_list) method, which returns a tuple with the list of bunny-containing image urls and a list of their respective confidences. The script can be tweaked to find any of the ImageNet classes by changing the RABBIT_CATEGORY_IDS array defined at the top of the file to whatever object id’s you would like. As stated before, the label map and the human label map text files found in the GraphDef zip file can be used to find object id’s you might want to detect.

This has been an introductory tutorial on machine learning and neural networks with a focus on seeing the basics in action. From here, I encourage you to explore the Tensorflow documentation on the Tensorflow site. You could also extend the functionality of the bunnyfinder script to detect images of categories not included in the ImageNet classes by retraining Inception-V3 on a new class (here might be a good start).

We recently took part of our team to All Things Open in downtown Raleigh. It’s a great, affordable conference for developers and super inexpensive if you live in the Raleigh area. There were over 3,200 attendees, and the crowd has grown significantly since the first year. And so has female participation and speakers. Kudos to the crew that runs ATO for their efforts on diversity. If you weren’t able to attend, here are musings from my viewpoint:

Conference Takeaways

  1. As mentioned, the conference has grown significantly, from 750 in 2013 to over 3,000 in 2017. I’d love to see the geography stats for attendees to see how many are local to the Raleigh area versus out of state.
  2. There’s a little something for everyone. The tracks range from DevOps to UX/UI and Business. Check out the schedule and consider paying for only 1 of 2 days if you’re still unsure about both days.
  3. IMHO, the best sessions hit the high-level bits but then also gave practical advice and tactical actions for implementation. Craig Keirsten on Postgres Performance for Humans was one of my top 3 favorites. For those on MySQL, Valerie is a database rock star and covered a high-level approach to database upgrade testing. If you’ve never upgraded a production database, you haven’t lived life on the edge. You should follow her for all things MySQL.
  4. Kubernetes is the hotness this year. We heard it so many times it left one of our teammates rocking back and forth muttering it to himself over and over.
  5. It’s an incredible opportunity to learn at a high-level about other technologies that you don’t use day in and day out. For example, we’re dedicated to native app development for all of our mobile projects, but it gave us the opportunity to check out React Native and other frameworks. It didn’t change our minds, read more here about what we think about cross-platform.
  6. Machine learning interest continues to grow, and there was a dedicated track to it, however, most content was high level. The best teachings on machine learning came from a 15 minute conversation with Dave Anderson at Oracle (aka NetSuite aka Bronto). He has incredible hands-on experience with Spark at scale. Hopefully, he’ll submit a talk for next year. Dave echoed the teachings of another local machine learning expert John Haws: keep it simple. Don’t use things like Spark until you really need it. Most things can be handled with basic algorithms. Just guessing, but I imagine the same goes for Kubernetes.

And finally, ATO is like a mini-reunion where I was able to see some of the best developers and engineers I’ve worked with over the course of my career. It was also a refreshing reminder of how blessed I’ve been to work with those, like Dave, that mentored, taught and supported me as a young systems engineer to where I am today. We plan to attend again next year and hope to see you there.