Auto-Classify Images in SharePoint Online library via Flow for Free

Microsoft Flow's most recent update added ability to query and update SharePoint File property.  This is actually really timely, as I wanted to combine this with a few other techniques and built a Document Library Image Auto-Classifier Flow.

Is that a clickbait headline?  Well it's totally real, and we'll build it in a moment.

result-1.png

Steps:

  1. Set up your cognitive service account (understand the free bucket)
  2. Set up a SharePoint Online document library with Categories
  3. Set up the Flow file loop
  4. Do a fancy JSON array to concatenated string projection operation with Select and Join
  5. Viola, no code.  And pretty much *free*

This is part of a series on Microsoft Flow

Set up your Azure Cognitive Service instance

Follow these simple steps to create an Computer Vision API Cognitive Service in your Azure.  Computer Vision API has a free tier.

1. Create Computer Vision API

1. Create Computer Vision API

2. Scroll down and hit Create

3. Give this service a name, set up the region and select Free pricing tier

4. You need the endpoint url here

4. You need the endpoint url here

5. Also, copy the Name and key 1

5. Also, copy the Name and key 1

You will need the "Name" and a "Key" for the next step.

The free tier of Computer Vision API - first 5000 transactions free per month.

Note the service isn't available in all regions.  Most of my stuff is in Australia East, but for the Cognitive Service API it has to be hosted in Southeast Asia.  YMMV.

Then we need to set up the connection in Flow

1. Find the Computer Vision API action

1. Find the Computer Vision API action

2. Enter service name, key and the root site url to set up the initial connection

3. Created correctly, you get an action like this

 

Set up the SharePoint Document Library

My SharePoint document library is very simple - it is just a basic document library, but I added an extra site column "Categories". This is an out of the box field, and is just a simple text field.

This is a simple step

This is a simple step

Set up the Flow

I trigger the flow with a Scheduled Recurrence that runs once per day.
Using the new Get Files (properties only), I grab a list of all the files in a document library.
I then run for-each on the list of files.

Inside the for-each, I have a condition that checks if the Categories field is null.  If you type null directly into the field, you will get the string 'null'. 

Tip: To actually get the formulat/expression null, select Expressions and type null there.

If the Categories is null, then we proceed.

Grab the file content via Get file content
Call Computer Vision API with the image content.  Select the Image Source to binary, instead of URL.

Tip: I use a compose to see the debug results

I'll explain the array projection in the next section.

Select projection: JSON array to String array

We have an array of JSON objects:

[{
     'name': 'foo'
},
{
    'name': 'bar'
}]

flow-project-1.png

This default UI maps to:

tags -> [{ specified properties }…]

The result is that we would end up with a new array of (simpler) JSON objects.
Hit advanced text mode.

flow-project-2.png

Here, we can use Expression to say item('Tag_Image')?.name

flow-project-3.png

In this case the UI is smart enough to show Tag.Name as a dynamic content (as well as the Tag.ConfidenceScore property).  So we can select that.

This performs a projection of

tags -> [ names… ]

We now have an array of strings.  Combine them via Join with a comma (,) separator.
Update the file properties with this string.

flow-project-4.png

Lets see the results

I uploaded a few images to the library.
Note the categories field is blank.

result-2.png

Running the Flow

When it finishes, I'm checking the JSON - the picture is identified with a "person" with 99% confidence.
The combined string "person,young,posing" is updated into the File property.

The documents are updated.  When Flow runs tomorrow it will skip them.

 

The Final Flow

Speaking at Digital Workplace Conference Australia 2017

I'll be speaking at the Digital Workplace Conference Australia!  23-24 August in Sydney.

 

This is a conference that's near and dear to me - and I've had several opportunities in the past to present at this conference, where I covered Silverlight, JavaScript, TypeScript, Modern Office App-ins and now this year - I plan to present a supercharged talk on running Serverless with Office 365.

Parts of the talk - especially how to get started - may seem familiar to many of you that has started down this journey. 

I wanted to focus a bit less on the technical, and more about how this has changed people. 

Azure Functions democratized 'I need to run a bit of code' to everyone.  Suddenly, the cloud is not this scary place where there are a hundred things we don't know, and don't know where to start.  Suddenly, the toys that seems far out of reach are ours.  Suddenly, a cloud subscription that costs less than a coffee per month is something I don't even think about.

To me, that is the power of AzureFunctions and why Serverless is a game changer. 

Do you know there are now brand new categories of design patterns specifically rewritten for the Serverless world.

I will of course still cover the technical bits - but to see all 20+ demos I have with me, you'll have to come find me in the speaker area for a personal demo :-)

In Digital Workplace Conference 2017, I want to talk about Serverless.

And I want to talk about humans.  Us.

I think the future will be amazing.  I hope to see you at the DWC Australia.  Come and grab me and say hello!

Are you Cloud-Curious or Cloud-Serious? Azure Functions in DWCNZ 2017

I had a fantastic time at Digital Workplace Conference in NZ.

Highlight Sessions

There are many other great sessions, I wasn't able to be in multiple places at once!

My Own Session

I presented Azure Functions in Office 365 - Building Serverless Solutions

There were a few things that I didn't managed to get through.  I wanted to list them here, and hope you will accept my apologies.  I've had several conversations with you all over the two days of the conference, many wanted deeper details into certain aspects of using Azure Functions.

 

Demo: Timer Based Alert with Email

https://github.com/johnnliu/azure-functions-o365/blob/master/sharepoint-list-email.ps1

This demo outlines a very simple script that will connect to a SharePoint list (or document library), query and fetch list items, format them into HTML and email to user from the System Account.

Combined with a schedule, this is an extremely common scenario in SharePoint Online: you want to schedule a smart alert email once a week, based on a filter to a list.

 

Using Recurring event in Flow instead of Azure Functions Timer-Trigger

While you can schedule tasks in Azure Function via a Timer Trigger, Microsoft Flow's recurrent trigger has several benefits:

  • You can create a Team Flow - so multiple users can be owners and configure the recurrence trigger.
  • The UI for setting up a time for the trigger is more obvious for power users.
  • You can easily see past runs from within Flow
  • You can easily re-run a Flow

 

The Severless "Specturm"

From my own experiences and from reading and understanding the greater scope of Serverless solutions that are being designed in the world, I wanted to present the spectrum of Serverless solutions.  We start on one side - from the Cloud-Curious, to the experts - the Cloud-Serious.

 

Cloud Curious

The majority of the presentation is pitched for the cloud-curious.  You have heard of Azure Functions and Serverless.  The demos presented how to get going really quickly.

Functions are thus:

  1. Micro (web) services for everyone.  So many people I talked to has given up on programming, thinking writing microservices or complex architecture isn't for them.  It's for the young'in dev teams now.
    AzureFunctions, especially with PowerShell - flipped the whole thing upside down.  Now, many 'ex-developers' suddenly find themselves build amazing service end points, connecting them to webhooks and Azure Blob Queues.  It is an amazing resurgence and move to microservices.  And everyone's having fun playing with really cool new toys.
     
  2. Use your favourite language!
    C#?  JS?  PoSH?  F#?  You can even use TS or VB.NET compiled.  Nobody can tell you what language you can and can't use.
     
  3. Perfect solution for many problems in SharePoint customizations
    Elevate permissions
    Webhook and event receivers
    Timer Jobs
    Extending Flow (as custom workflow action)

    If you are bringing customizations in SharePoint On-Premises to SharePoint Online - Azure Functions is a solution that must be evaluated.  It fits so many scenarios that you need to bring your On-Premises customizations forward, without breaking the bank, or needing complex re-development.

 

Cloud Serious

For the cloud serious - you are already using simple functions.  You want to know what's next.

  1. "Idempotent" - this is the keyword that will define the entire Serverless Framework.  You want to design functions that has no side-effects if you rerun.  A function can fail, it will automatically retry until success.  Your function must be built to be retry-safe.
     
  2. Use message queues and service bus to scale your Function.
    In Serverless, you are bound by duration.  You are not bound by parallel compute.
    To scale your long running process, split into a Queue and spawn infinite parallel compute.
     
  3. In a serial code, we wanted to catch all our exceptions to speed up long running tasks.  When we convert to parallel compute - we no longer really care about exceptions.  If you fail, you want to fail fast.  Throw exceptions freely and as fast as possible.  Terminate the function.  Let Queue retry automatically.
     
  4. With the new Azure Functions Proxies, we can create Serverless Web Applications - which is essentially combining a CDN to host static resources, and Functions to run server side code.

    Future of Serverless web apps is basically: CDN + Functions
    Both scale in parallel infinitely, by default, by design.  But is easy to understand and accept in concept.

    You do no worry about scaling VMs, AppPools, IIS, WebJobs, WebSites... 

Your solutions sits on top of all of those things - but there is no fear.  A fast messaging queue with built-in retries and a thousand atomic hammers will carry your workload from now to infinity.  And it'll cost less than your coffee.

 

Slide downloads

https://github.com/johnnliu/pptx

 

If anyone asks is Microsoft still investing in SharePoint - show them this.

I was attending an MVP community event, and a few non-office MVPs asked me, hey is Microsoft still investing in SharePoint?  Is there anything new coming next?  There hasn't been a product for a few years right?  (This is so odd, we just had SP2016...  and SPO hasn't stopped having new features every month.)

I was also working at a client when one of our advanced users rushed over and asked John - what's going on with this new Teams thing - is Microsoft scrapping SharePoint.  (I was pretty shocked at this one).

Realizing that I'm probably directly connected to the flood of news regarding SharePoint - it is surprising people aren't aware of many of the best and greatest of SharePoint is still to come.  I decided to write this post.

April - SP Tech Con (lastest info as of this post)

This is the latest info as of this post.  Follow Mark Kashman - PM SharePoint. 

This was the SPTechCon keynote. 
https://www.slideshare.net/markkashman/reinventing-content-collaboration-the-future-of-sharepoint-is-now-sptechcon-austin-ms-keynote

 

Feburary - Internal Demo Day

Is there still a SharePoint team?  YES - and they are legion.

Directly from Jeff Teper the father of SharePoint

May - What's next?

2017 May 16 is the SharePoint Virtual Summit.  This is the big one.  Especially if we look back to 2016's May the 4th event when we first embarked on the next step of the Future of SharePoint, and now we see what the team has delivered within just one year.  It's been nothing but awesome. 

So no matter where you are in the world - you should register for the Virtual Summit.

In Australia

The Office 365 Saturday is upon us really soon.  The annual free community event will be coming to a city near you!  You need to sign up.  This helps us plan our catering needs.

March PnP special interest group call and Azure Functions demos

On March 22, I put my hand up to do demos for the Office Patterns and Practices Special Interest Group conference call.  We talked about Azure Functions with PnP, and Bert talked about a new Modern Page scanner that can be used to check if sites have customizations that would make migrating to modern pages difficult.

I had a lot of fun.  It probably was very obvious I was laughing most of the way.  This is a follow up post of various things that I had noted down but unsaid.

Demo files

All my Functions in the demo are published to: https://github.com/johnnliu/azure-functions-o365 

That Get-AccessToken typo

It turns out the problem is the dash - character that I copied from GitHub.  If I retrype the command as - (dash) then the cmdlet is fine.

Costs

Vesa mentioned this at the end.  Azure Function's pricing is very cheap.  It is actually because Functions don't run on VMs provisioned for you.  Azure finds an unused VM, copy your function on it, then run it, and then delete it.  The free monthly compute bucket is 400,000 Gb/s.  If you exceed the limit it is still dirt cheap.  I'm serious about the fact that I have turned off all my VMs, I only run Functions.

Also, I'm just as shocked as you are that Vesa seems to know exactly what I'm about to present next.

Docker

My platform as a service goes up.  I see Docker as going downwards and not the direction I want to go.  Functions as a Serverless platform needs to be above Web Server as a platform.  They could all live on top of Docker, but that's an implementation detail that I don't need to care about.

Configure Inputs and Outputs

Because Functions are serverless, it is important to understand that you don't really want to even write your own code for handling output.  If you want to write to Blob Storage or Azure Queue - there are output configuration you can set up.  Then the output from the function is written directly to those sources without you having to write bare minimum code.

Configuring Inputs also allows you to switch from a HttpTrigger to a TimerTrigger, if you need that.

Flow to Azure Functions

If you feel uncomfortable with having an open public URL exposed - you can have Flow upload a JSON directly to an Azure Queue, then execute the Azure Function from the queue.  This also provides automatic-retries.

Future announcement - PnP SIG JS Call

I've done a bit more work with pnp-js-core, azure-functions-cli and azure-functions-pack.  And have put up my hands to Patrick to do a PnP-JS-Core SIG call in the future.  Need a few pieces to sort out first.

Azure Functions has a very definite place with NodeJS as well.  We'll explore that hopefully soon.