,

Software or App Onboarding: How to Streamline Your User’s Experience

Software or App Onboarding

By: Carol Vercellino, CEO & Co-Founder

“Almost everyone will make a good first impression, but only a few will make a good lasting impression.” – Sonya Parker

What type of impression are you making when your users download and open up your app? 

User onboarding is one of the most critical elements of your app or software development and launch. It determines whether users stay, go, or even become raving fans of your product.

So, today we’re going to talk about how to streamline the user onboarding experience for your software product or app. Let’s dive in.

 

Jay: First, tell us what user onboarding is and why it’s important.

Carol: Onboarding is an experience a user has when they first open up your app, especially if they’re new to it. There are two different camps here. If your app is really well done and well built, you don’t need onboarding. But, some apps require your users to have a little bit of guidance the first time they’re opening up your app.

A lot of it depends on what type of app you have. If you have a social app, you probably don’t want to have a very extensive onboarding process. You want to make the app super simple and easy to use.

But an app that’s more of a utility or an educational tool, you want to add an onboarding process, so people know what they can do in the app.

 

Jay: How do you create an onboarding strategy?

Carol: You sit down with your team and talk about the goals of the app, the purpose of the app, and what the value is you’re delivering to the user. Then, you map out the first thing the user sees when they open up the app. Where do you want them to go? What do you want them to do while they’re there?

You can whiteboard out the steps or even wire frame it out. You really want to look at that layout and what would be intuitive for the user to click, tap, or do next.

 

Jay: What are some user onboarding best practices or tips?

Carol: If you take a step back from onboarding, there are best practices around just the app itself. For example, when a user signs up for your app, what is that account flow like? Is it super simple? Or do you have a lot of different steps?

You know, if you ask for a credit card upfront, you’re probably going to lose some users right there – depending on the app and depending on the target market. So, you really want to be strategic about the number of screens you use that the user has to go through.

Also, there’s also the experience of getting help from your company. Is it easy for your users to get help? Do they have to call a 1-800 number, is there a chatbot on your website, do you have an FAQ area or help center where they can easily find information. 

If you decide not to have a walk through of your app, can your user easily google your app and find a help center to get the information they need to be successful in using your app?

Jay: So it’s all about reducing friction during that process?

Carol: Yes.

 

Jay: Could you share an example onboarding process with us?

Carol: Any app you download from the app store, you can kind of get a feel of how the apps work. Just download one and take a look at it. 

Some of the best apps you download, you go straight into the app. One example is the Facebook app. When you download it, you go straight to the news feed. In some apps, you only have to sign up with a phone number – you don’t have to put in a whole bunch of account information.

Another way to think about it is when was the last time you went to a doctor’s office, and you walked in, and they handed you an iPad? You filled out the forms, and it was really simple. When that process is simple, you think, ‘oh, I like this doctor’s office; it makes it easy to schedule appointments and update my records.’

You want to translate that experience into your app as well.

**The above interview has been transcribed for clarity and brevity.**


Want to learn more about streamlining your users’ experience? Check out our video on How to Get Your App Users to Stick Around After the Free Trial here.

/by
, ,

How to Get Your App Users to Stick Around After the Free Trial

How to Get Your App Users to Stick Around After the Free Trial

By: Carol Vercellino, CEO & Co-Founder

Free trials are a great way to show users why they should choose your app or software product over anything else.

But how do you ensure your users will renew their subscriptions once the free trial is up?

Today, I’m going to share three ways you can get your users to commit after their free trial has ended. 

 

Nail your onboarding process

How do you nail the onboarding process? There are three ways you can focus on this.

First, you want to create a help center on your website or have a small support team. You could even have part-time folks who can answer technical questions. Make it easy for your users to find you; for example, your support email should be help@yourappname.com.

Second, send users a welcome email with all the information they need about their free trial and how to download the product to get started. You could even turn that process into a welcome cadence where you send them an email once every two days or every day for the first seven days.

Third, you also want to provide in-app native FAQs so your users can open the app, click Help, and quickly get the answers they need.

Empower your users

To get your app or product’s full benefits, your users need to know all the ins and outs. Empowering your customers with educational materials about your product can boost customer loyalty and reduce complaints.

So, first off, you want to offer your users hands-on experience with live demos. If you’re a startup, you want to do this person-to-person, for example, over Zoom, and ask your users how things are doing and if you can answer any questions.

Second, you can create and share useful content on your website, email, and social media. It can be simple how-to’s or tech tips to support your user in achieving their goals through your product.

You can also organize a virtual workshop or webinar. You can do those live or have them pre-recorded so your users can access them anytime via your website.

Make it easy for users to buy your app

This step often gets overlooked, but it’s the most important. Make it easy for your users to buy your app or software product once the trial is up.

Remind users their free trial is about to expire. You can send your users an email or maybe even call them if you’re a startup and have the team to do it. 

You can also provide an early discount if they buy before the expiration of the trial. For example, you can send them an email and say, ‘Hey, if you renew now (7 days before your trial is up), we’ll take 20% of the subscription!’

Also, send a final email that includes step-by-step how users can buy and activate the full version. Make the purchasing process really easy. Find the simplest checkout form that you can have them put their credit card information in (if you can, use Apple Pay). If the product is available in the app store, that makes it pretty easy too.

Conclusion

Your free trial phase is a great way to retain customers, build loyalty, and grow your business.

Make it easy for your free trial users to learn how to use your app or software to its fullest potential, and they’ll move right into the paying phase.

Now that you’ve got your users hooked, learn how to streamline your onboarding process, so your app users never want to leave. Read it here.

/by

Is Your Data “Really” Secure in the Cloud?

by Carol Vercellino, CEO of Oak City Labs

Are you following suit with the rest of the world and moving your data to the cloud?

Cloud storage can eclipse a traditional on-premise data warehouse when it comes to speed, reliability, flexibility, and security. But is your data really secure in the cloud? Many people are worried about security when they move to the cloud.

So, let’s look more closely at cloud storage security and the three things you need to consider when migrating your data to the cloud.

Data

What are you storing? Where are you storing it? And what’s the structure of that data and who needs to access it? These are all very important things to think about because you’ll want to design access controls into the storage infrastructure of the data in the beginning to make it easier to scale in the future.

For example, if you’re housing patient information, you may have critical protected health information (PHI) located in part of the cloud that you don’t want everyone to have access to, such as a developer.

However, you can break out your data into different environments. For example, in software development, it’s good practice to have several different environments for your work.

You might have a development environment, a Quality Assurance environment for testing, a staging environment to test changes before they’re promoted to production, and then finally production. Production should only be accessed by a limited number of individuals.

By separating your environments, you can gain control over the security of your data.

Also, the data in development doesn’t need to be the same data as in production – it just needs to look like it. In theory, your code – or the application you’re working on – should be the same in all those environments. And that can be a very helpful way to limit access to data while also helping with patching vulnerabilities when they pop up, whether it’s in the software itself, or it’s on the operating system you’re running these things on.

According to one report, 60% of breaches involve vulnerabilities where a patch was available, but not applied. And a lot of that is because people are scared to do patching; they’re afraid it’ll break production. 

By having multiple environments you can test your patches at each level, and then you’re reducing the risk that that patch will cause a problem in production while creating secure environments.

Documentation & Procedures

Documentation and procedures are very related to your data concerns. Not all startups will have much documentation, but in a more formal environment, like healthcare or government, you’ll have a ton and will want to get started on it very early on in the migration process.

Here are some documentations you need to think about:

Change management procedures and control
When a change happens, who’s notified about it? And who’s able to make that change and document that change?

Incident Response Plan
You’ll need to have a plan for what happens if you do have an incident. What kind of response does your team need to have? And what kind of documentation do you need to have a long way? An Incident Response Plan is especially important in healthcare.

A Patch and Vulnerability Plan
What is your patching schedule? How do you coordinate it and keep an audit trail of all the changes?

People

People. Humans. They have access to all this data.

And social engineering is used by most hackers in order to gain access to systems, whether it’s them trying to guess the password or sending out emails. In fact, 94% of malware is delivered by email. 

Our team members, our employees, our vendors…they need to have awareness and training on secure procedures and a way to think about security.

For example, if my organization is training people on security, I would train our team members to think twice about opening emails from the Prince of Nigeria who’s sending an Excel File about our company’s finances. Instead, they should not open the file and notify either the IT department or a security engineer or professional in our organization.

It’s also important to think about physical entry to buildings. While your data and/or applications are in the cloud, if somebody was to gain physical access to your facility or office, then they could hack somebody’s account and get access to the cloud.

In fact, there’s a story of a hacker who bought a $4 CISCO t-shirt and showed up at an organization saying he was the new CISCO engineer who was there to fix the phone system. They let him in, and he and another person were able to penetrate their network – company wide. Yikes.

So, physical security can be incredibly important as well – even while we’re all working from home. Having security awareness and training plans within your organization can help you tremendously.

Are you ready to migrate to the cloud?

It’s no surprise most companies are moving to the cloud – it’s flexible, reliable, and scalable. And, also very secure.

Many of the concerns we talked about here are not specific to the cloud; they’re specific to data in general. So, if you spend time focusing on your data, documentation, and people, you can feel confident in your secure migration to the cloud.

 

Have more questions about migrating to the cloud or tightening up your security? Check out our Q&A with Josh Wyatt, the Chief Information Security Officer with InfiniaML, to learn even more about data security and migration.

/by

Data Security: How to Safely Migrate to the Cloud

By: Carol Vercellino, CEO & Co-Founder

Because of COVID19, companies of all sizes are accelerating their digital transformation programs.

Many are moving their data to the cloud without all the security controls necessary to protect theirs and their customer’s data because it could slow down the digital transformation process.

But, this leaves them vulnerable to cyber attacks and being out of compliance with data protection regulations.

Our CEO and Co-Founder, Carol Vercellino hosted a Q&A with Josh Wyatt, the Chief Information Security Officer with InfiniaML, a machine learning business solutions company.

Watch the video below or read through our condensed transcription to learn what you need to know to manage your company’s data security as you migrate to Amazon Web Services or the cloud.
 

Oak City Labs: Could you give us a quick introduction to InfiniaML and what you do there?

Josh Wyatt: We’re a technology enabled services company.

We do a lot of high-level client projects focused around machine learning and other aspects of artificial intelligence, usually to solve some kind of business problem that [a company] has had trouble solving in conventional ways.

We also have a product suite that includes a library and some other toolings that we use in conjunction with delivery of these services.

We have a pretty healthy client base. We have an outstanding team, and what I would estimate is probably the top 0.5 % of machine learning talent in the country, if not the world.


OCL: What do you think, in terms of data and security, matters now more than ever?

JW:  Going back historically, there was a huge focus on centralized security. When you have people in a central location, it’s a lot easier to monitor activity on the network, for example. With server-centric infrastructure, it’s also very similar.

These days – and this necessarily wasn’t started by the COVID-19 situation, but it certainly made it more interesting – things are more decentralized. There’s Google’s initiatives that strongly espouse endpoint security, and I think that’s where things are headed. COVID-19 brought it even more to the forefront because employee bases are now 100% remote for a lot of technology organizations.

Monitoring activity in distributed scenarios is also pretty challenging. There are tool sets out there in the space that I would not call quite mature. I’m sure some vendors would disagree with that in regards to their offerings, but it’s more challenging than ever on that front.

And obviously, cloud has made that more challenging with people embracing Amazon Web Services and Infrastructure in the Cloud and distributed environments.


OCL: When you say endpoint security, what’s the definition of that?

JW: A great example is data loss prevention tooling. In the old days, that was pretty easy to do. You’d set some kind of sensor on a centralized point or a chokepoint, and watch for behavioral patterns, like large outbound data transfers or things of that nature.

That’s much, much more challenging with a distributed workforce where people are working from home or working from other places and there’s no single chokepoint.

The chokepoint actually moves out to the edge of the network where people are doing their work and in a distributed fashion versus more centralized.


OCL: Do you have a recommendation for companies who want to better manage their data security? Do they speak to a vendor, purchase software tools, or manage via their policies and procedures for staff?

JW: Great question. I think 100% start with culture and training. Let your organization know that this is something you value and is important. The tools don’t matter because you could evaluate and choose a tool, and then 6 months later, choose a different one based on who’s the frontrunner in the space. I’m a little loath to recommend specific tools because of the variability in the market right now.


OCL: Do you think there’s an opportunity in the data security space for better innovation when it comes to data security software?

JW: I do. The trend I see is that the larger entities that are big in the network space, they have solutions, but they generally tend to be heavy handed. And for someone like us with 35 employees, we’re probably not going to deploy some massive heavy-handed solution to basically spy on our employee base. We want to have an environment where we trust people to do their work and make sure they are aware of the consequences of any kind of failure on that front.

In your initial intro, you were talking about how folks are racing to move data or infrastructure out to the cloud to accommodate these remote working scenarios. That doesn’t “ad hoc” happen – it happens quickly – but it’s usually part of a plan, and that plan needs to include management around risk.


OCL: What do you think are some of the top security threats when moving to the cloud?

JW: The biggest one is you have to be an informed consumer of any kind of service or product before you give your money or your crown jewels, the data for your business, to someone. You need to understand the risk you’re assuming.

Every service offers some level of control, and so, there’s just diligence you need to do. It’s like anything else you buy, a car or insurance, you need to understand the value proposition and what the risks are, and have a plan for managing that risk as you make that move.


OCL: Is there anything specific you would look for in an AWS infrastructure to manage risks?

JW: As always, understand the architecture of what you’re moving. In the first place, it’s never just a lift and shift from on premises to the cloud. It frequently requires a re-architecture of your application. Remember, you’re in a shared environment instead of a dedicated environment, and that comes with some risks.

Always review the compliance standards from wherever you’re moving to. Amazon’s are pretty good, and they have to be. Ideally, anyone who is making this move also understands their compliance landscape too. Health organizations and CRO’s, they’re going to know about HIPAA, HITECH & HITRUST. Anyone that’s into ecommerce understands the implications of PCI, and things of that nature. You’re going to do some diligence.

Specific to Amazon, though, I’d recommend making sure that you have a threat assessment program that involves audits and testing. You can hire a consultant who can give you best practices, you implement them, put those best practices in your backlog, and then prioritize them and implement them. Or you can hire a penetration tester to come in every quarter to give you a full report of your exposure, which could also feed a backlog. 


OCL: Do you have recommendations for creating a data security budget?

JW: You need to understand your industry, business landscape, and what your competition looks like. That’s always the case with budgets too, right? Is your competitor spending more or less than you are?

For us, the nature of our business is data, and we deal with a lot of data, and our customers are very sensitive about it. You won’t see very many 35-person companies with a Chief Information Security Officer. You may want to position data security as a competitive advantage.

The budget also comes down to the value system of the leadership. What do they value? What is important? That’s frequently driven by the customer base. Some customers may have no requirements and some may have stringent requirements.

For a larger organization, you may have many, many clients that demand something from you, and they will drive what your trajectory looks like when it comes to data security.


OCL: What are some common mistakes companies make when shifting from on premise to the cloud?

JW: Like any other set of risks, the key piece is you want everybody on the same page. For example, you say, this product we’re building is going to be in production, and there’ll be no defects in it, security or otherwise.

You also don’t want a surprise on day 10 after a release that there’s a huge security breach. All your client data has been released and somebody knew about that defect, but it wasn’t managed. That’s shockingly common.


OCL: When should someone bring in a security consultant? 

JW: This is a question of generalization versus specialization. At the concept stage, someone needs to have a mindset around security. It may not be a dedicated individual, but it has to be a consideration. At the point where you perceive the risk is too high for you to manage with the expertise you have on hand, that’s when you start having a discussion internally with the leadership team about the strategy for managing the risks. 


OCL: Do you have any resources people check out if they want to learn more about security and data protection?

JW: A very important one is just to stay aware of what’s going on in the threat landscape. The U.S. Cybersecurity & Infrastructure Security Agency is an excellent resource for staying up to date.

Another great resource for me is Brian Kreb’s security blog. He’s a cross between a security technologist and an investigative journalist. His writing style is accessible, and he posts frequent updates.

If you’re in a sensitive space, most law enforcement type organizations will have resources. And it’s better to establish that relationship early and often, so if and when you have a security breach, you have an established relationship.


OCL: What’s one step a company or person could take right now to move forward in putting the proper protections or security controls in place?

JW: The very first thing is to do an assessment. You can do it yourself or hire an external entity to do it for you.

Assessment is a very broad term. It could mean anything from asking someone’s opinion of your environment or infrastructure configuration, all the way up to organized penetration tests.

But do some sort of assessment. Imagine if you walk into a dark room, and you have a flashlight and shine it around a bit. What are some things you can stump your toe on? What are the things you might fall over? Is there someone hiding in the corner?

There are lots of different flashlights and lots of different ways to do exploratory things. You probably already have an idea of what you’re concerned or not concerned about, and that’s a great place to start.

An assessment will give you something actionable to walk away with and plan some work around.

**The above interview has been transcribed for clarity and brevity.**

 

Enjoy this Q&A? Check out our interview with THX Rx Co-Founder Theresa Williamson, Ph.D. on fighting physician burnout through technology.

 

/by

Deep Learning and Neural Networks 101

Amazon Cloud 101

In this post I will explain in relatively simple terms what a neural network is in the context of machine learning. I’ll sprinkle in some machine learning terminology along the way with links to resources you can learn more from.

We’ll start with a simple statement:

A neural network takes input data and transforms it to a desired output format. It can be trained to do this in a way that reflects the real world.

And now we break it down.

What is the input data?

Numbers, essentially. These numbers can be in various forms and represent various things. An image as input data might be a 3-D array of red, green, and blue color channel values:

A three-dimensional array representation of an image with blue, green, and red color channels.

 

Input data could also be one-dimensional rows of a spreadsheet, database table, or other table-like structure:

UserId Hours Heart Rate (BPM) Temperature (F)
23 8.5 72 96

A table row that can be treated as one-dimensional array (vector).

What is the desired output format?

Again, it’s just numbers – but more specifically numbers that answer the problem you’re trying to solve. By “format” I just mean the shape of the numbers (array dimensionality and size). If your input data consists of images of different breeds of dogs and you want to know what kind of dog appears in each image, your desired output format could be a number that correlates to a breed (you would assign each breed a number before training). If you also want to know the location of the dog in the image, the output could then be the breed and the pixel coordinates of the top left and bottom right corners of the bounding box containing the dog (a vector of 5 numbers).

How does the neural network transform the input data?

There are several operations (or layers) commonly used to transform data in neural networks. Some operations are incredibly simple like ReLU, which looks at all of the input values and replaces all those that are less than zero with zero. Some are more complex like convolution layers that could, for example, apply a 2D array called a kernel to a 2D input array similar to  a weighted average. Regardless of complexity, all of the layers take numbers in and spit numbers out. Again, these numbers are arrays of varying dimensions.

This is an example of how a layer might transform its input. As you can see the shape changes from (100, 100, 3) to (50, 50, 3). The output array of each layer is called its activations.

 

The sequence of operations is what comprises the internals of the neural network (often called the “hidden layers”). FYI, the use of neural networks is considered “deep learning” when there are a bunch of hidden layers.

A key part of the layers with regards to training is the weights. Some layers have an array (or multiple) of numbers (dimensionality depends on the type of layer) called weights that they use to perform their transformation. For example, the kernel in the convolutional layer I mentioned earlier would be included in its weights. The values of these weights can be changed during training as we’ll see later. All the weights across the layers of the network are what allows it to store information and “learn.”

 

The above basic matrix calculation demonstrates how a weight matrix might be used to transform the input in a neural network layer (this specific example is part of what occurs inside a type of neural network layer called a fully-connected layer). For reference, the input matrix above could represent pixels of an image (black and white in this case since the matrix doesn’t have a third dimension for color channels), or more likely, the output activations of some layer before it.  

So how do you know what hidden layers to use and how to hook them up? To build your network, you need some understanding of the purpose that the different kinds of layers serve and how they might help you extract the important features from your input data and eventually output an answer in the desired form. That said, there are popular neural network architectures that deal with certain input data types and solve particular kinds of problems well, so you often don’t have to start from scratch. For example, image recognition is often done using a ResNet architecture.

How do we train?

To start with, the neural network, a.k.a. a series of matrix operations, doesn’t know how to do much of anything. All of the weights throughout its layers comprise a “brain” of sorts – a means of storing learned information. But those weights start out as random numbers, so this brain has no idea what it’s doing. The network needs to learn what the values of these weights should be.

During training we pass in input data samples to the network a few at a time. For each sample we pass in, we get an output. How do we take that output and learn from it?

Training relies on an error function which is a measure of how much the output (what the neural network produces when fed an input sample) differs from the actual answer. This “measure” is of course just a number. Error functions, also called loss functions, come in many different forms, but they all tell you how wrong the model is with respect to the training data.

The need for an “actual answer” brings up an important point I skipped over when introducing input data. For each sample of input data you train with, you must have a corresponding correct output data sample. For instance, each picture of a dog in the training data should be labeled with a breed (and possibly coordinates of the bounding box if that’s what you’re interested in). Providing the model with an input data sample and an answer is what makes this approach supervised learning. The network uses these sample and answer pairs to “learn.” That’s the “real world” piece of our original statement about neural networks. We tell the model how the real world behaves through the sample and answer pairs we train it on, and it learns how to emulate them.

So now we know what model uses to learn, but how does the model use it? As you feed in training samples to the neural network, each of the weights in the layers are slightly incremented or decremented to decrease the value of the error function. How do we know whether increasing or decreasing a specific weight will increase or decrease the error? This is possible because the error function and the layer operations are differentiable. In other words, you can take the function’s derivative with respect to a weight and determine if increasing or decreasing that weight will decrease or increase the error. If you’ve tuned out at this point because “derivative” is too much calculus you don’t remember, just think of the learning process as tweaking each of the weights up and down a small amount and keeping the “tweak” that decreases the error. That’s the basic idea behind stochastic gradient descent and backpropagation: scary machine learning terms that describe a relatively simple process. Little by little, sample by sample, this is how the network learns. Provide an input sample to the network, transform it to the output format, calculate the error based on what the output was supposed to be, calculate derivatives (backpropagation), increment or decrement each of the neural network weights, and repeat. Slightly simplified, but this is the basic idea of how neural networks are trained.

Once all of the training input data samples have been passed through the neural network once, a training epoch has completed. Training usually involves several epochs, and after each one the samples are usually shuffled.

This whole training process is automated (the grunt work math is buried in tools like TensorFlow, PyTorch, etc.), but it’s not foolproof. You won’t get a quality model by just training any neural network on any data. Here are a few of the challenges that machine learning engineers face to get a worthwhile model:

  • Choosing the right neural network architecture and adjusting it if need be (swapping out layers, introducing additional layers, etc.)
  • Providing the model with quality training data to learn from (things like missing data, too little data, all cause problems)
  • Tuning a myriad of training parameters and model parameters (the training process has tunable values such as the learning rate and many layers have some as well)
  • Changing the training schedule (how many epochs, varying the learning rate as training proceeds, temporarily “freezing” layers so their weights don’t get updated)

After the error is satisfactorily minimized, the neural network can be put to the test on real world data. If it performs well on real world input data it has never seen before, we say that the model generalizes well. If it performs poorly, the model has probably overfit to the training data. In the example of recognizing dog breeds, if the only Beagle images the model saw during training were those of Beagles panting, an overfit model may not be able to generalize to images of Beagles with closed mouths. The model has “learned” that if a dog isn’t panting it’s not a Beagle, but that’s a detail in the training images we didn’t necessarily want it to pick up on. It has overfit and learned too much from the training data, reducing its ability to generalize to the vast variety of Beagle images in the real world. We can’t be expected to train it with pictures of Beagles in every possible orientation, so we want it to learn enough about Beagles to distinguish them from other breeds, but not too much.

The beauty of neural networks is that the right architecture with the right training and model parameters will learn the important features and interactions from the input data on its own. It’s not magic – it’s just math, and there is certainly work required to tune neural networks correctly. However, manually pinpointing the important features and interactions in a data set and translating them into an algorithm is considerable more work than tuning some parameters and letting a neural network do its thing.

This is by no means everything you need to know about neural networks, but it hopefully provides a handhold for someone interested in how they work. There are a lot of details that were not covered here and I encourage you to dive deeper into the types of neural network architectures, what they’re used for, and how the training process works.

/by

Neural Networks with Structured Data

In my last post I shared how to train an image classifier on your own image using the fastai library. To continue our machine learning journey, we will look today at how to train a model on structured data like that found in a database or a spreadsheet. Specifically, we will look at how to turn a list of categorical and continuous variables into a prediction for a single response variable.

For example, say you have a spreadsheet with columns Month, State, Average Temperature, and Precipitation in inches. You want to be able to predict Precipitation in inches for a specific Month, State, and Temperature. In this case Month and State would be categorical variables (while their values can be represented as numbers, their relationships to each other are likely more complex than a one-dimensional continuous spectrum can represent), Temperature would be a continuous variable (because its a floating point number), and the precipitation in inches would be the response variable.

The code used in this post is based on Lesson 4 of the fast.ai deep learning course. You will need the fast.ai library to run it.

We will use the pandas library to load and manipulate our data into the proper format. It’s already imported in fastai for us.

You may need to clean up your dataframe and/or add some additional features (columns). I’ll leave that up to you, but here’s the the docs for the pandas library to get you started.

We need to convert our categorical variables into category types in the dataframe as well as standardize our continuous variable types to float32:

In order to train the neural network, the dataframe must contain only numeric values and we must separate out the response variable (the variable we are interested in training the model to calculate).

This proc_df function separates out our “y” variable and converts all columns of the dataframe to a numeric type.

Now we get the indexes for our validation set. Note that my example data here is times series data, so instead of randomly sampling the data set to get the validation set I take the last 25% of it. This ensures that the model will get trained to predict future values instead of random values interspersed along the timeline.

Next the model is constructed and trained.

I will briefly go through how the learner is created and what the parameters to the get_learner function mean. Fast AI assembles a model consisting of several fully connected layers (link) based on the data you provide to the get_learner method. Here is the definition of ColumnarModelData.from_data_frame from the fast.ai source code and a breakdown of some of the parameters:

def get_learner(self, emb_szs, n_cont, emb_drop, out_sz, szs, drops, y_range=None, use_bn=False, **kwargs)

  • emb_szs – a list of tuples that specifies how the embedding matrices for the categorical variables will be structured. Embedding matrices essentially take each category of a categorical variable and represent it as a vector of numbers. All categories in a categorical variable have vectors of equal length and the combination of all of the category vectors forms the embedding matrix for that variable. These vectors become part of the neural network “weights” that get trained through backpropagation and stochastic gradient descent, resulting in a rich representation of each category as a point in multidimensional space. The tuples hold the number of categories and the length of the embedding vectors as their two terms. The tuples are listed in the same order as the cat_vars variable defined earlier and passed to ColumnarModelData.from_data_frame.
  • n_cont – the number of continuous variables.
  • emb_drop – The third parameter is the dropout rate for the embedding matrices. Dropout is a technique used to avoid overfitting and introduce some randomness by taking some of the model’s weights and “dropping” them, setting their value to 0. In this case a dropout rate of 0.4 means that each embedding weight has a 0.4 chance of being zeroed out.
  • out_sz – the number of output variables of the model. We just have one, the response variable.
  • szs – an array of sizes of the model’s fully connected layers. The number of sizes determines the number of layers the model will have.
  • drops – an array of dropout rates for each fully connected layer.

To use your model to get a prediction after training it, you can use the following code


Here cat_values is a 2D array with column values that match the categorical variable indices (make sure the columns are in the same order as in your pandas dataframe) and cont_values is the same thing just with the continuous variables. Your single number prediction should be in the pred numpy array. You can also pass in multiple rows to get multiple predictions at once.

As an aside, you can get the categorical variable category indices by looking at the df.column_name.cat.categories attribute, where column_name is the name of the column you’re interested in and df is the dataframe (this will only work after the column has been set as categorical with the df[column_name].astype(‘category’).cat.as_ordered()). Just set the correct column in the cat_values array to the index of the category you want.

Feature engineering is arguably the hardest part of training any model. Getting the best model will rely heavily on how relevant the columns of data you have are to the problem you are trying to solve. If your model is performing poorly, rethink what data your training it on and experiment with extracting new features from your data and introducing other data that might be correlated.

/by

Computer Vision & Machine Vision 101

Today we’re back again sharing the basics of two not-so-emergent technology concepts and breaking down the basics between each: computer vision and machine vision.

It is possible that you’re reading this blog and have never heard of computer vision or machine vision. The concepts are well-known and discussed within the technology world, but the same can’t be said for the general public. Despite that unfamiliarity, the general public is already experiencing computer vision and machine vision in ways they may be surprised by. Read on to learn more.

Computer Vision

Computer vision falls under the Artificial Intelligence umbrella just like machine learning does. The goal is to utilize computers to acquire, process, analyze and understand digital images or videos. For instance, computer vision is being employed when a train station has a computer use security camera footage to count the number of people entering and exiting instead of manually counting with a turnstile. Or computer vision is at use when driverless cars use a live video feed to make decisions about turning, braking, speed, etc.

Have you seen the augmented reality capabilities from IKEA? The company encourages you to use your device to video your living room and then they virtually place sofas, coffee tables and chairs in real time for your consideration before making the big purchase. That’s possible because of computer vision. Summed up, computer vision is attempting to use a computer to emulate the human eye, visual cortex and brain when acquiring, processing, analyzing and understanding images.

We’ve talked about computer vision before here on our blog.

Machine Vision

When computer vision is put in place in an industrial (and sometimes non-industrial) setting to inform operations and functions of a machine, we call that machine vision. An inspection task at a manufacturing facility once performed by humans, can now be performed by machine vision.

Machine vision is at use when at a manufacturing facility, a machine will scan (read: computer vision) a bottle to ensure the liquid product (like cleaning solution, soda, medicine, etc.) is correct, the fill level is correct, the container is free of flaws, the correct label is placed (and placed straight!), the expiration date is correct, etc. And when one or more of these conditions aren’t met, machine vision has logic in place to tell the production line to reject the item. The beauty of machine vision is that all of the sample analyses I gave above are performed by one machine, with a high-degree of accuracy, over and over again.

At Oak City Labs, our mission is help businesses and organizations solve daily problems with technology. Utilizing computer vision and machine vision are excellent ways to accomplish that task. Do you have a problem that you need help solving? If so, let us know! We’d love to chat.

/by

Simple Image Classification with fast.ai

Image Classification with fast.ai

Today’s blog post will demonstrate some basic neural network code that uses the fast.ai library to train a model to categorize images into one of n categories and get predictions on single images. You can take this code and change it to train the model on any image categories you would like. I won’t go into too much detail on what is going on behind the scenes and will focus on showing you how to get a working model.

The code is based off of lesson 1 (and partially lesson 3) of the fast.ai course, which I strongly recommend taking a look at if you are interested in machine learning. To run the code, you will need the fast.ai library, which is built on top of the PyTorch deep learning framework. Additionally, to run the code in any reasonable amount of time you will likely want a GPU. Take a look at fast.ai lesson 1 to learn how to set up a GPU computer in the cloud using Paperspace.

The code above constructs a neural network using the ResNet-34 model, which is pre-trained on the ImageNet dataset. Our code simply adds a few layers to the end of the pre-trained model and trains them.

Line 7 essentially packages up the images so that they can be passed through the model and used to train it. Line 8 creates the learner object capable of actually doing the training. Line 9 actually performs the training with a learning rate and a number of epochs (cycles through all of the input data). With the fast.ai library, these three lines of code are all that is required to build and train a convolutional neural network. Fast.ai makes the process of training a neural network incredibly simple by taking care of preparing the pre-trained model, adding additional layers, and setting some hyper-parameters to sensible defaults if not specified. At the same time however, the source code of the fast.ai library is relatively concise and readable, so as you use fast.ai I encourage you to take a look at how it is leveraging PyTorch under the hood.

To train this model on your images, change the PATH variable to point at where your images reside. The PATH directory should have inside it a “train” and a “valid” directory. Inside these directories there should be a directory for each category of image with the images for that category inside. You should place roughly 80% of your images for each category in the train directory and the other 20% in the valid directory.

To get a prediction for a single image:

In the code above we get the transformations that are applied to the validation image set, apply them to our image, and get our model’s prediction. We then find the predicted category name from our data object’s classes attribute. Note that because inputs to our model are assumed to be tensors (rank 4 tensors in our case, or tensors with 4 dimensions), we must add another dimension to our 3-dimensional image with img[None].

The code in this post will work best on images that are similar to ImageNet images, which are primarily images of objects, animals, and other easily photographable things. If our images were something much different like X-rays or satellite images, the pretrained ResNet-34 model would likely benefit from some extra training that we have not done here and we might want to use different data augmentation transforms (the “transforms_side_on” variable from our ImageClassifierData instantiation). You may have noticed that during training, several rows of numbers get printed to the console.

[0. 0.04368 0.02732 0.99121]

[1. 0.03997 0.02237 0.99023]

[2. 0.04147 0.02266 0.99072]

The fourth number in each row is the accuracy of our model based on how well it categorizes the images in our validation set. The hope is that this number is as close as possible to 1.0 by your final training epoch. The number may vary greatly depending on your number of images, number of categories, and quality of images, but hopefully it is somewhere between 0.8 and 1.0.

There are several techniques we could employ to increase the accuracy of this model, but I will will leave them for a future post.

/by

Artificial Intelligence, Machine Learning & IoT 101

Machine learning (ML). Artificial intelligence (AI). Internet of Things (IoT). It seems these days there are so many complex terms flying about. In this blog, we’ll break down the basics and share how you’re experiencing these technologies in ways you may not have even realized.

Artificial Intelligence

We can’t talk about machine learning without first discussing artificial intelligence. Long before ML and IoT were commonplace, the concept of AI existed. The term was coined by John McCarthy decades ago in 1955 describing it as, “making a machine behave in ways that would be called intelligent if a human were so behaving.”

Simply put, AI is when machines handle tasks in an intelligent manner in the same ways humans would. Did you know you have AI all around you? Apple’s Siri, Amazon’s Alexa, Google’s personal assistant, and Tesla cars are all examples of AI at work.

Machine Learning

Machine learning is a product of artificial intelligence. Where AI is the ability of machines to handle tasks in an intelligent manner, ML takes it one step further and says that machines should not only handle complex tasks the same intelligent way humans would, but they should also have the ability to learn how to handle the complex tasks themselves. This concept is also decades old, dating back to 1959 when Arthur Samuel coined this term defining ML as, “the ability to learn without being explicitly programmed.”

Do you know that saying ‘all squares are rectangles, but not all rectangles are squares’? Well, the same is similarly true here: all ML is AI, but not all AI is ML.

Interested in the technical aspects of ML, including how neural networks are used? Our software engineer Taylor shared his thoughts and experiences on the blog before.

So how are you experiencing machine learning in your everyday life? Amazon and Netflix recommendations are great examples. As the two learn your buying or watching habits, they make recommendations of what you may be interested in next – and most times, they’re spot on! Or have you ever noticed that as you use your iPhone more and more for texting that it begins to pick up on your common spellings, acronyms and dialects? That’s machine learning.

My name is Ashlee. It’s spelled a little differently than most Ashleys, but my parents spelled it that way when I was born and I haven’t found it on keychains ever since, so what can I do? Apple didn’t program the iPhone to know that when I start typing Ashlee to not autocorrect it to the more popular Ashley, but after several times of me typing and then undoing the autocorrect to change the spelling back to Ashlee, my phone learned that when I type Ashlee, I mean Ashlee not Ashley. That’s machine learning.

And what about Internet of Things?

Artificial intelligence and/or machine learning can converge in the physical world with Internet of Things. IoT is the category of physical objects – like lightbulbs, thermostats, door locks, plugs, etc. – that are connected to the world via the Internet. When those IoT devices also employ ML, like when a Nest thermostat learns your temperature preferences and adjusts the temperature inside your house for you without your action, the two worlds are colliding.

At Oak City Labs, our mission is help businesses and organizations solve daily problems with technology. Utilizing AI and/or ML are excellent ways to accomplish that task. Do you have a problem that you need help solving? If so, let us know! We’d love to chat.

Read more about Computer Vision & Machine Vision here.

/by

Simple Image Recognition with Tensorflow

Machine learning

Machine learning can be an intimidating field to get into. In the event you want to dive right in, I’m providing a very brief and fairly high-level overview of how machine learning works. I’ll also cover a very simple image recognition project using Tensorflow that you can get up and running with a little bit of Python knowledge.

To begin, let’s discuss some of the basics. Machine learning is largely practiced through the use of neural networks. Neural networks are comprised of nodes that take an input and produce an output. These inputs and outputs are generally n-dimensional arrays. For example, an image you are providing as an input to a neural network would be represented as a 3-dimensional matrix with dimensions (width, height, number of color channels). Each node can be thought of as an operation that manipulates the data that comes into it. Operations can be as simple as addition or multiplication or as complicated as a multidimensional convolution.

Each node also has variables (n-dimensional matrices that are often referred to as “weights”) that are used in whatever operation the node performs to transform the input. These weights on each node are the neural network’s “brain.” During training of a classification neural network, the weights are incrementally changed based on an error function using an algorithm like stochastic gradient descent (which attempts to minimize the error). This error function is a node that takes the neural network’s output value and the known correct output value (which is available during training) and calculates the error (often something like cross-entropy).

After the training data set has been sent through the neural network many times and the error has been minimized by altering the weights, features of the training data are “remembered” in the weights. After training, when sending in new data that the neural network has not seen before, the network will use what it learned about the training data and apply it to this new data. In a classification neural network, this means detecting familiar features of the data and assigning probabilities that it belongs to each of the known classes. This is a high-level overview and isn’t quite the complete story, but hopefully, it provides some insight into how a neural network functions.

So now let’s actually do some machine learning with a simple script that uses Tensorflow to detect bunnies in images. For the sake of brevity, I will leave it to the user to get set up with Python and Tensorflow and to become familiar with how Tensorflow is structured. Additionally, there is a wealth of example Tensorflow code available in the models Github repo. The classify_image.py file from this repo will be the basis for my bunny-finding script. The classify_image.py script takes an image file, loads a pre-trained neural network trained on the ImageNet data set, and classifies the image, outputting the top 5 probabilities and the corresponding image categories. I will discuss some of the code in this script and explain how it can be repurposed to find bunnies.  

This neural network (called the Inception-V3 model) has been pre-trained on the entire ImageNet database, a computationally intensive task that can take days depending on the resources available. With the GraphDef file in the classify_image.py file, the nodes of the Inception neural network and the trained weights associated with each node can be loaded, bypassing the intensive training step. In four lines of code, you have a trained graph capable of classifying images across 1000 categories.

Now a Tensorflow session can be started and inference run on an image. First, provide the image input tensor with a string containing the JPEG image data. Tensors in the loaded graph can be accessed by name. The tensors that need to be accessed here are ‘softmax:0’ and ‘DecodeJpeg/contents:0’.

The two code blocks above comprise essentially all of the “machine learning code” contained in the classify_image.py script. Most of the other code pertains to loading the image file into a string, downloading/extracting the zip file that contains the GraphDef file, and translating the indices of the output softmax tensor to human-readable categories. The predictions variable is a Tensor containing a 1×1000 array of the probabilities that the input image belongs to each of the classes. Translating the indices of this array to actual classes is just a matter of using the imagenet_2012_challenge_label_map_proto.pbtxt and imagenet_synset_to_human_label_map.txt files used in the NodeLookup class (unzipped with the GraphDef file). For simplicity’s sake, I skip that step in my bunny-finding script.

You can find my bunny finder script at the gist here

To run the script, call it through the command line and provide an image url as an argument.

python bunnyfinder.py image_url

The script instantiates the BunnyFinder class and passes in a list of urls to the BunnyFinder.findbunnies(url_list) method, which returns a tuple with the list of bunny-containing image urls and a list of their respective confidences. The script can be tweaked to find any of the ImageNet classes by changing the RABBIT_CATEGORY_IDS array defined at the top of the file to whatever object id’s you would like. As stated before, the label map and the human label map text files found in the GraphDef zip file can be used to find object id’s you might want to detect.

This has been an introductory tutorial on machine learning and neural networks with a focus on seeing the basics in action. From here, I encourage you to explore the Tensorflow documentation on the Tensorflow site. You could also extend the functionality of the bunnyfinder script to detect images of categories not included in the ImageNet classes by retraining Inception-V3 on a new class (here might be a good start).

/by