A Power User friendly method of connecting hundreds of sites, list and libraries to a single Flow

Photo by Sebastian Boring on Unsplash

Photo by Sebastian Boring on Unsplash

We need to get this disclaimer out of the way first - this approach uses SharePoint Designer workflow as a glue - the main reason for this is because SharePoint Designer workflows (being old generation technology) can be deployed across site collections via PowerShell scripting without too much trouble, as long as they remain small and don’t need regular maintenance.

This approach isn’t necessary for deploying a Flow to a dozen sites. But if we are talking potentially hundreds of project sites. Those become scenarios that aren’t currently covered well, and would need workarounds such as this one.

Let me know what you think in the comments.

The Problem

There are many ways we can view this problem of “deploying Flows across many lists, libraries, sites and site collections”.

This problem has several issues we may not need to consider:

  • First, we have Export and Import, as well as Send Flow as a Copy. These are simple, manual steps. That can’t be automated.

  • But at some point - we will hit the 600 Flow limit (this isn’t per environment, this is per account)

  • We will need to reconfigure URLs specific to each Flow.

  • As these copied Flows are customized from the master version which makes updating and maintaining them very tricky.

  • When redeploy a Flow - existing connections must be maintained.

  • When deploying Flow to a new site - new connections must be configured (or existing connections re-used).

  • If we are ever in a situation where we need to deploy a new update an existing Flow but now with additional new Connections. God have mercy.

  • If we think perhaps just delete existing Flows and re-deploy as new, we’d kill any existing runs.

So there are alternate approaches as well… what if we just don’t deploy hundreds of copies of the same Flow?


A new solution?

The method in this blog post is a different idea - we will borrow SharePoint Designer workflow’s various tools for deploying cross site collections and have it invoke the same Flow to perform the main task.

This was posted as #FlowNinja hack 91

https://twitter.com/johnnliu/status/1121055884749053953


Let’s begin

Warning: this hack involves the use of SharePoint Designer. Now before you all run off screaming, there's a good reason why we want to try this method: mass deployments. So here we go

1. Make a Request trigger Flow. With a simple compose.
2. make SPd 2013 reusable workflow

3. first action - add SPd HTTP web request to the Flow request trigger
4. need to configure both request body and request header

5. If we don't blank out authorization this happens DirectApiAuthorizationRequired: The request must be authenticated only by Shared Access scheme

Add SPd Workflow to a Library and a List

6. now publish and add this to a List and a Document library
7. upload a file see SPd workflow trigger
8. see HTTP Request flow trigger

The Test is successful. Next copy the JSON we received in Flow

9. add Request schema from the previous successful run
10. change Compose to match request body

11. add a new item to a sharepoint list (that has the same SPd workflow attached)
12. the same Flow re-triggers.

Take a breather, have some tea

13. at this point, there's several more things to do - consider whether we want excessive logging. SPd workflow is simple - once it is working there's not a lot more to tweak, perhaps we want more context values. While still within this site collection, republish will update it.

Package the SPd Workflow for cross site collection

Let's package this workflow for cross site collection warp jump.

14 remove workflow association to list Achievements - because we don't have that list everywhere, only Documents. 15 export WSP to desktop

16 go to new site collection - this can be SharePoint or group sites or project sites.
17 site settings - upload solution to gallery - activate solution
18 site settings - site features -activate feature (this associates with Documents)

19 test with upload new file in new site collection
20 flow triggers, context of Flow is new site url.

21. note there was an error with the SPd workflow after it called HTTP - because I didn’t activate the SharePoint Workflow feature, Workflow History list isn’t available in the new site collection.

Consider Step #13 and whether you’d want to delete all the logging steps before exporting Flow


21. now we are in Flow. Re-fetch the item and go crazy.

Yes, in Step #4 we deliberately chose the context values that will allow us to re-configure Flow to pick up the item

Conclusion

Con:

  • The pain of writing a SharePoint Designer workflow again

  • Difficult to update deployed SPd workflows - but the one we have here is very simple (two build dictionary and a HTTP request)

Pros:

  • A simple lightweight SharePoint Designer workflow can be deployed across many site collections via PnP-PowerShell, or as part of PnP-Provisioning.

    Activate the solution, and then activate the feature (this will create the workflow associations)

  • All the events from associated SharePoint libraries will call a single Microsoft Flow - this is where we can customize to our heart’s content.

I don’t think this approach should be written off - it should be evaluated and may suit your situation well. I also think in a possible future when cross-site Flow deployments can be done easier - we may have different Flow triggers invoking our one Flow. So there exists an upgrade path forward.

Azure Global Bootcamp Sydney - this Saturday!

Photo by James Ree on Unsplash

Photo by James Ree on Unsplash

I’m presenting a talk on Azure Logic Apps (and Microsoft Flow) this Saturday at the Azure Global Bootcamp Sydney.

https://www.meetup.com/en-AU/Azure-Sydney-User-Group/events/256253065/



LogicApps & Flow for Developers - insane low-Code Serverless Automation

We make the case that every developer must understand what LogicApps and Microsoft Flow are, because they will make us rethink how we really write code.

There is always more code to write, but what if we can

  • Use out of the box actions when we don't need to write code

  • Connect to new systems painlessly

  • Connect to APIs that we have never used before

  • Not worry about how much it all costs

  • Build microservice architecture solutions

  • Fall into pit of success


Join this session to learn about how to write code, fast, without writing code.


This is a similar session (with more focus on Azure Logic Apps - there’s always room for tweaking) from a talk I’ve done previously, most notably during the MS Ignite Roadshow.

The talk is aimed at developers. There’s no assumption that you would know much about Logic Apps or Microsoft Flow, but think of this as a baptism by fire - an mid-level introduction that goes right into parallelism and HTTP requests.

These are what one would often consider to be difficult developer concepts - and as we will all see, these are extremely easy to achieve in Logic Apps / Microsoft Flow.

Join us this Saturday - how to write code fast without writing code.



Upload Image from PowerApps to Flow to SharePoint via an Unused Outlook connector

This is the simplest no code approach to the PowerApps image upload problem so far. Far simpler than with Azure Function, with custom connector, with hacked Flow button via Flow Studio, even simpler than Azure Blob Storage. All standard connectors so no premium required, and no risk of PowerApps trigger resetting and breaking the connection.

This is my simplest method to upload any image from PowerApps to SharePoint

  • No Swagger

  • No Edit JSON

  • No Azure Blob Storage

  • All Standard Connectors

  • No HTTP

  • Can easily add more arguments

Original

This blog post is a cleaned up version of the #Flow Ninja hack 87 thread which happened on Sunday night. https://twitter.com/johnnliu/status/1114863521525669888

Follow me on Twitter and catch the next live hack.

Steps - first a bit of study and exploration

I have a @MicrosoftFlow hack this evening to send files from @PowerApps to @SharePoint I have been thinking about this one for a while. So if you are still awake, follow along.

First - I check the Flow button trigger.
Then create a PowerApps trigger, use peek code to study

Double check SharePoint connector - I read this with FlowStudioApp - there's no method that takes format: byte. Everything wants format: binary.

I spent a while looking through various standard connectors looking for something that does format: byte - I found one. In the Outlook connector.
In send email with attachment. **cackle** **evil grin**

Evil twinkle in the eye acquired - we now execute the plan

So we hack the PowerApps trigger. by using a totally unrelated connector.
I can't hold back my dislike of the PowerApps trigger. Why can't it behave more like the Flow button trigger...

The argument sendanemail_attachmentscontent is ugly. Try using Flow Studio to rename them first before you go too far. This will also make the connection tidier when you take it over to PowerApps.

Finally

PowerApps time - this is probably my simplest method.
Don't need Azure blob storage
Don't need edit json
Don't need swagger
Can have multiple arguments

5-2.png

Just need to conditionally build a strange Flow that doesn't use the outlook connector but use it to lock the PowerApps trigger

  • See the condition is always false - it doesn’t run

  • See also the Size of the create file is much larger than a broken blob string

  • We need to keep the unused Send an email action even if we don’t use it - because it locks the PowerApps trigger in place so the trigger doesn’t reset.


And there we have it - the absolutely simplest no-code solution to send a File from PowerApps to SharePoint with ease.

We lock the PowerApps trigger to format: byte by using an otherwise unused Outlook send mail connector.

Future

There are a few things Microsoft could do that will make this even easier. If they ever get around to it:

  • Allow us to define PowerApps trigger directly either by using Flow Button UI or Request schema

  • Allow SharePoint connector to accept format: byte

  • Allow PowerApps to send format: binary, right now PowerApps converts that to string, dropping the non-character bytes from the data it sends to Flow






One Flow to handle them all - how to subscribe to multiple SharePoint lists with one Flow

One Flow to handle them all - how to subscribe to multiple SharePoint lists with one Flow

At some point - we think hmm how do we save a Flow as a template and deploy it with all our SharePoint site collections…

and did we just create a massive problem down the road where we have all these duplicate Flows…

and how do we manage versioning and changes to them hmm lets have a think and talk about this.

So instead of all that copying. Let’s try a different idea - what if we have One Flow that Rules them all. It handles events from All the Sites and Libraries, all within one Flow.

Depending on what our Flow actually do - this might be a better approach.


Live Tweet

This is a long overdue write up of an idea of two Flow done as a hack over twitter.  It was actually a Saturday morning live-tweet event and some joined in guessing where was John was going with all this.

Multi-Part Series: Deploy My Flow

This is a multi-part series called Deploy My Flow. I want to publish discussing the various ways to deploy Flows across SharePoint and Office 365, between dev, test and production environments. Across lists, sites, and tenants.

  • Export and Import

  • Automate - PowerShell, Flow Management and Flow Studio

  • Power Platform Solutions

  • Subscribe to multiple SharePoint Lists with One Flow

    • Part 1 - SharePoint Webhooks - you are reading this part

    • Part 2 - GetChanges - this was written in so much detail by lsspdev so I will link to his work

Part I - SharePoint Webhooks

A Flow can only listen/trigger from one list, so while in SharePoint as we provision sites - we consider having a way to provision templated Flows to handle each list. 

But let's turn the question around.  Can we have just one Flow to handle multiple lists?
Is it possible to have Flow connect directly to SharePoint's Webhooks and manage events directly with Flow?

 

Why this approach?

  • The Flow triggers listen to and raises events for each individual list or document library. So, out of the box, we can't create one Flow that listens to every list or library.

  • Cloning Flows multiple times has its own challenges:

    • 250 Flow limit (This limit is now 500, you can request a service ticket to raise this number) per account.

    • How do we manage multiple Flows, do we have one master? Does it get exported and imported over existing Flows when we upgrade? What happens with the connections?

    • Can we do these automatically?

  • There’s a lot to discuss, but for this blog post we will focus on a simple pattern

  1. Create a Handler Flow that handles list change events.

  2. Create a second Scheduler Flow that subscribes webhooks from multiple SharePoint lists onto the first Flow.

Reference

We don't start from scratch, we have the excellent materials on docs.microsoft.com from the SharePoint PnP team describing how to attach to SharePoint webhooks via code.

https://docs.microsoft.com/en-us/sharepoint/dev/apis/webhooks/overview-sharepoint-webhooks

We only need to translate this to Flow.

First Create the Scheduler Flow

First - here I’m reading all the lists from SharePoint that’s a document library. Then read the subscriptions on each document library. This end point shows me SharePoint’s webhooks.

TIP: Because I use odata=nometadata a LOT - I end up putting that header into a header_nometa JSON variable and insert it in every Send HTTP Request action.


Create Subscription and handle Resubscription

Next, we check if the webhooks contains has a clientState with the current Flow’s name (Flow names are unique guids).

If not, we create one. The check and filter for ClientState makes sure we don’t subscribe the same event handler to the same document library multiple times. It also makes this “scheduler” flow re-run safe. We can pretty much change the trigger to a recurrence trigger that runs as often as you’d like - say daily.

The Handler

We need a new Flow to handle the document updating. This is the One Flow that does all the work.

Save the Handler Flow and take the HTTP Reques URL back to our Subscriber Flow

And this is the final result - when we run the Subscriber Flow - it iterates through all my document libraries and connects them all to the handler Flow.

Now when documents are updated in these 12 document libraries, my “Handler Flow” will be triggered by SharePoint.

SharePoint webhook subscriptions are valid for 180 days. So at some point our Scheduler Flow will re-run on recurrence and re-subscribe the document libraries. (We can also handle the True condition in the Scheduler Flow to renew webhooks).

Part 2 - Get Changes

We need to go deeper into Get Changes. Your webhook fired, now you need to figure out what changed.

@ISSPDEV couldn’t wait for the next part he went ahead with this. And because he wrote in so much detail - I will basically link to his work for Part 2.

Apologies

This was a live hack in 06/2018 - I actually have three draft versions of this blog post never finished so never previously published. So moving that forward with this first post. 9 months late is still better than never.

Sadly, I can't make it to MVP Summit 2019

helena-lopes-592967-unsplash.jpg

I’ve been holding onto some slimmer of hope, may be I could still make the trip. Alas, it’s time to face reality and write down this blog post. I’m apologizing - I’m sorry.

I can’t make it to MVP Summit 2019, I’ll need to start cancelling my various plans - flight is cancelled, there’s some hotel bookings to cancel.

I have an immediate family member who wasn’t well through 2018, we fought it, and we thought we got over it, but the problem has returned in 2019. That means more treatments and wait, and hope.

The Silver Lining

I guess I could say at least there’s a silver lining that the Australian public health cover is fully covering the treatments, if I was in America I would be on Go Fund Me already, last year.

Reach out

I will be feeling a bit sad and lonely. If you’d like to say hi - we are only separated by a Twitter DM, a Facebook message or just a Skype call.

I know I was looking forward to catching up with many of you, and may be you wanted to say hello to me, well we can totally still do that, via Skype. P.S. Call me, okay?

P.S.S remember your NDA - no streaming summit content. I want to talk to you, I don’t need to know about the latest secrets I’m sure they’ll be announced fairly soon anyway. We don’t live in a 3-year cadence world anymore.

Please Remember the new MVPs

I remember my first MVP Summit and I have no idea where to go or where to be. Please remember MVPs from your country that’s their first time, make sure they aren’t left out, and make sure please, that they get back to their hotel safely.

Flow Studio

I had hoped to use the two weeks in Seattle to really catch up with many fellow MVP friends from across the globe, and perhaps some might want to see the next iteration (or the current iteration) of Flow Studio. That will have to be done remotely. Let me know if you want to help me perfect my pitch deck. I promise you’ll get the honest truth of where we are - and why Flow Studio exists. I can’t promise I can deliver the pitch deck in 5 minutes - still working on it.

I will totally and utterly miss

  • Randomly bumping into you at the LEGO store.

  • Enjoying American steak with you at a steak house.

  • Sipping long island ice tea unawares with you at a bar.

  • Walk long trek at night across empty streets with you because the hotel is so far away.

  • Getting kicked out of a karaoke bar because I’m not even drunk. But also it’s the 4th bar of the evening so perhaps it’s time to go back to hotel.

  • Enjoying hot wings, oh damn the hot wings are just so nice, with you.

  • The wonderful conversations I have with you.

  • You Microsoftie, putting up with this horde of barbarians that call themselves MVP - most vocal professionals. You are the best and so, so brave.

  • Did we just randomly wander across Seattle to the Space Needle at night and take a group selfie?

  • Bellevue is actually really cold, I’m glad you gave me a really warm Fellowship jacket.

  • Taking selfie photo and making them into stickers with you.

  • Talking selfie photo with you using maximum Samsung beautify mode because I really need it. You don’t, but now you look really pale Sorry!

  • Eating Wendy’s because the meat is square and it sticks out in the corners, and because we totally don’t have Wendy’s in Australia. Also the double dripping cheese is American heart society certified cause of death right?

  • Smuggling Tim Tams for you from Australia

  • Spinning around with you on the top of Space Needle

  • Seeing all your happy faces

  • The big giant hugs you give when you see me

  • Using up all my MS store voucher, and then take your voucher because you don’t know what to do with it, so I bought another bunch of Xbox controllers. You can never have enough Xbox Controllers.

  • Catching up with you about the Serverless Rag-Tag-Gang, next time it really is my shout

  • I was really looking forward to a redbowl selfie this time

  • I wanted to meet Mr Purple

  • Go hunting for a nice steak with you at 10pm and cursing why we didn’t leave the party earlier so we can go find a greater steak

  • Plotting with you over how we’d distract the Bellevue MS Store employee and run off with the first Microsoft Studio we saw there.

  • Accidentally yelling my introduction into the middle of a PnP monthly podcast recording because really I’m just a walking disaster waiting to make a fool of myself. And it turns out you all knew me anyway so there was no need to yell like that.

  • Impersonating a vendor employee while taking a group photo with you.

  • Discussing merits of bacon with you, and how we can always add more bacon salt to bacon.

  • Not knowing how to use a mic to ask a question in a full room full of people.

  • Sneaking out the back to get a thickshake with you.

You are one of a kind. You came from all over the world. We don’t even have the same native languages, but we are the same. The same kindred spirit. There’s dozens and hundreds of us. It is such a peculiar sight.

I hope to see you all instead in 2020.