A first look at the new Import Export Mailbox API in the Microsoft Graph Part 1
One of the missing pieces of the Microsoft Graph endpoints for Exchange Online has been the ability to import and export email and other exchange items. Outside of the obvious need to migrate and archive email this type of functionality has a variety of other uses such as copying and synchronizing messages (and other objects) between mailboxes either intra or between o365 tenants and importing from other sources.
If your migrating from EWS (because of its impending demise) this is what replaces the ExportItems and UploadItems operations in that it allows both volume (we will discuss this in detail at a later date) and the ability to migrate with full fidelity. If your new to Migrating items in Exchange (vs other email system) because of the way Exchange natively stores and handles Messages and other rich associated messaging objects (Calendars, task etc) you need to take some special considerations when copying, exporting and importing items. Back in the day the ability to have information (stored as properties) outside of the normal Mime Stream gave Exchange (and Outlook) a leading edge. What this now means if you copy a Message and just use the Mime (or Eml stream which is already available in the Graph ) this isn’t considered a full fidelity copy of the Message as it won’t include any information that is stored in Extended properties that aren’t part of the Mime stream. For email you can mostly get away with this but for richer objects like calendaring, task, notes etc if you need to copy these then you need a full fidelity copy of that Exchange store Item that will include all the extended properties associated with that item.
SDK Support
Currently there isn’t any direct SDK support for this yet as of 5.94.0-preview and the latest PowerShell Graph SDK. The examples in this post all use the PowerShell Graph SDK and the Invoke-GraphMethod but as soon as support is available I’ll migrate across as the direct support in the SDK makes things more reliable and readable.
Mailboxes Supported
The new endpoints support both the Mailbox and the Archive store which marks this as the only way you can currently access the Archive Store items using the Graph API. No Public folders support is available. Archive Mailboxes are an interesting challenge as the potential existence of Auto Expanding Archives makes the retrieval of that content much more complex then just a normal mailbox and its pretty complex and messy to use even in the legacy Exchange API’s. It will be interesting to see what level of support there is for this.
Permissions and Getting Started
The Import Export endpoints introduce some new permissions and like all the endPoints in the Graph it offers some good levels of granularity so if your app doesn’t need to do particular operations it can be restricted to just using the one it absolutely needs. The following is a list of the new permissions (note i’m just looking at delegate permissions there are App permissions as well)
MailboxItem.ImportExport - Required for createImportSession, exportItems,importItem
MailboxItem.Read - Required for listing and getting mailbox Items
MailboxFolder.Read Required for listing and getting Mailbox Folders
MailboxFolder.ReadWrite Allows you to do the above + create/update/delete of folders
If your using the Microsoft Graph SDK you want to include the following in your connect-mgGraph request eg
connect-mggraph -scope "MailboxItem.ImportExport,MailboxItem.Read,MailboxFolder.ReadWrite"
That will give you enough permissions to do everything you might want to do in these endpoints.
Mailbox Id’s
A big difference between the current graph endpoints and the import/export endpoint is the Id you use to access a particular mailbox (or archive store). In the regular endpoint you could use (a unique GUID), email address, or the me shortcut. In the Import/Export endpoints there is a new identifier that looks like
MBX:73418aa0-7efb-423d-805b-62b26bdacxxx@1c3a18bf-da31-4f6c-a404-2c06c9cfxxx
the format isn’t documented but its pretty easy to see this is
MBX:{ExchangeMailboxGuid@TenantId}
You don’t (or shouldn’t) try to infer this Id as they provide a simple Get Request endpoint for getting this for the Primary and Archive mailboxes eg
function Invoke-GetMailboxSettings{
[CmdletBinding()]
param (
[Parameter(Position = 0, Mandatory = $true)] [String]$Upn
)
Process{
$RequestURL = "https://graph.microsoft.com/beta/users/$Upn/settings/exchange"
$Results = Invoke-MgGraphRequest -Method Get -Uri $RequestURL
return $Results
}
}
which then returns the following
In this example that mailbox has both a Primary and Archive Store
Part 1 Folders
I’ve split these posts up into a series as there is actually a lot of ground to cover across multiple topics. In the first part I’ll just cover off folders
The Graph already has the ability to enumerate mailbox folders but it’s restrictive in what it returns eg you can only enumerate Mail folders in the Mail Endpoint and Contacts folders with the contacts endpoint etc. For other types of folders like Tasks, Journal or any type of custom folder you can’t enumerate these at all or do other CRUD operations. So this new endpoint gives you the ability to enumerate the Mailbox Folder tree like you would in EWS or Mapi. It does have some restrictions like not allowing you to get into the ApplicationDataRoot folder in the non_ipm_subtree which is home for most of the new features like flexible work hours, pronouns and signatures. But it allows you to enumerate all the folders where there would be user migratable data.
Hidden Folders are also available using the includeHiddenFolders=true parameter that has been available on the regular Mail Endpoints for some time. Hidden folders have a number of uses in Mailboxes generally its application driven. Moving Application data using a Migration endpoint is pretty fraught with danger as eg something like custom quicksteps in outlook will have source specific Id’s that won’t work if you just push the data without doing an application specific migration. It' is possible to do it by modifying the destination items properties but this can be a lot of work and a lot of these application specific properties don’t have good documentation.
Deep Traversal
If your coming from EWS there was the ability in the FindFolders operation to do a deep traversal which means in one request you can return the whole folder tree. This isn’t available in the Graph which is a little disappointing as it does make getting a folder tree more complex as you need to check and enumerate at each child level. Overall that’s more an annoyance but there is some performance loss if you have a mailbox with a very large and complex folder hierarchy.
Delta’s
Delta’s in the Graph are the EWS equivalent of syncfolderhierarchy this can save you time when it comes to synchronizing a folder tree. If you are synchronizing a folder tree pretty often then it is worth utilizing this feature.
FolderId’s
The FolderId’s are the same regular REST Id’s that have always been used in the Graph which are just a Base64Url safe version of the EWSid. FolderId’’s aren’t immutable so if a folder is moved (or deleted) then it will have a new FolderId in it’s new location. Also binding using the WellknownfolderEnum works so thing like
https://graph.microsoft.com/beta/admin/exchange/mailboxes/MBX:xxx6d@1c3a14xxxx/folders/inbox/
or
https://graph.microsoft.com/beta/admin/exchange/mailboxes/MBX:xxx6d@1c3a14xxxx/folders/deletedItems/
allows you to directly target the specific mailbox folders and more importantly if you have different regionalized mailbox it will work across the different languages.
Properties
EWS and Graph have both Strongly typed properties on folders like the DisplayName and type and also the ability to get and set extended properties eg for a Calendar folder it maybe the datetime a Folder was created that is important to you. In the sample script I added three extended properties to the Mailbox Folder enumeration samples. The first is the FolderPath because I always find this useful when you trying to visualize where a folder might be and what it represents. The other is the CreatedDateTime (pidTagCreationTime) this one is useful if you just want to see new folders created after a certain time and you haven’t used the delta feature to track changes. The Delta operations don’t support extended properties.
In the example script for this post I use this request to do some extended property expansion and get all the hiddenfolders
$RequestURL = "https://graph.microsoft.com/beta/admin/exchange/mailboxes/$MailboxId/folders?`$Top=999&`$count=true&`$expand=singleValueExtendedProperties(`$filter=(id eq 'String 0x66b5') or (id eq 'SystemTime 0x3007'))&includeHiddenFolders=true"
SearchFolders
The Graph already supports search folders and has its own endpoint for creating and enumerating them. However when you are enumerating folders using the Folders endpoint it will return both normal and search folders. Currently the properties you get back for a Search Folder vs a Normal folder are exactly the same eg everything is "@odata.type": "#microsoft.graph.mailboxFolder", so unless you specify some extra extended properties that tell you if a folder is a searchfolder or you filter out searchfolders you do need to be careful you don’t mistake a searchfolder as a real mailbox folder. One way to tell what type of folder your dealing with is is use the PidTagFolderType property so you can either include that in you folder enumerations like i have in the sample scripts. Or you can filter based on this property eg
eg
https://graph.microsoft.com/beta/admin/exchange/mailboxes/MBX:xxx6d@1c3a14xxxx/folders/inbox/childfolders?$filter=singleValueExtendedProperties/any(ep:ep/id eq 'Integer 0x3601' and cast(ep/value, Edm.Int32) eq 1)
This means only regular folders (FOLDER_GENERIC) will be returned by your folder/child folder query.
Batching for speed
If you migrating mailboxes that have large complex folder hierarchy’s then your going to be creating lots of Mailbox folders. If you do these creates 1 request at a time you folder sync code will be both slow and more likely to get throttled. To speed things up you can use the batching endpoint in Graph which allows you to submit up to 20 folder creates in one requests. eg this is what batched folder create looks likes
Throttling in batching *trigger warning*
Throttling in the Graph is not designed well for the mail endpoints and is really one of the biggest challenges when migrating from EWS. It appears there has been some work on it recently (undocumented currently of course) and it has improved things and if your only using a Single Thread which for folder sync is ample but for items it’s another story but I’ll save that for another post. The crux of the issue is in EWS you can have 27 concurrent connection and the only time I’ve seen someone hit that limit is with streaming subscriptions. In the Graph this was reduced to 4 which causes a lot of problems I wrote about this https://gsexdev.blogspot.com/2020/09/the-mailboxconcurrency-limit-and-using.html . The good news is this problem when you have only one batch request appears fixed but how they addressed it is a little less clear eg did they just up the limit to 20 concurrent connections for everyone or did they fix the thread pool and the way Graph batches to Exchange. If I try using concurrent threads then I can still get 429 concurrency issues so it appears maybe the quick option was chosen rather then engineering a proper solution which will also involve upping the concurrent connection. The frustrating things this is not an Exchange problem because EWS doesn’t have this issue so that points the finger at a one size fits all Graph solution which can be fixed relatively easily . Also concurrent connection limits isn’t a throttling limit it’s a more DOS (denial of service) limit you impose. In EWS migration you can start spinning up your threads but your going to hit the resource throttling limits which are then one’s you want to hit as they are the effective speed limit to which you must adhere. Having an arbitrary low concurrent connection limit to cater for a bodgy Graph batch implementation for Exchange helps no one.
Examples
I’ve put together a PowerShell module that uses the Graph SDK as part of this post and posted it https://github.com/gscales/Powershell-Scripts/blob/master/Graph101/GraphSDK/Import-ExportMod.ps1
Get your Mailbox settings (need for other ops)
$Mailbox = Invoke-GetMailboxSettings -Upn gscales@datarumble.com
Enumerate all folders in a Mailbox this will also return the FolderPath, Folder CreationTime and FolderType extended properties
Invoke-EnumerateChildMailFolders -MailboxId $Mailbox.primaryMailboxId
Create a New Mailbox Folder in the Inbox
Invoke-CreateNewFolder -ParentFolderId (Get-MgUserMailFolder -MailFolderId inbox -UserId gscales@xxx.com).Id -MailboxId Mailbox.primaryMailboxId -FolderName Testnew1234 -FolderType IPF.Note
Final word
For the Mailbox Folder side of this new endpoint it ticks all the boxes and there is really nothing apart from the searchfolders and if you have multi threaded you folder creation that is going to cause any huge issue if your migrating from EWS based folder sync. In the next post well look at migrating items.