These are some notes that I typed up during an Azure Machine Learning - Data Camp that I attended at the Microsoft learning center in the Aon building. It was entertaining, informative, and all around a great session; unfortunately, it was entirely useless for my job, since this really has nothing to do with the work that I perform. On the plus side - I was able to recommend to my co-workers on the Analytics team that they should attend it.
Ross Loforte, Data Scientist, Microsoft
https://github.com/melzoghbi/datacamp
<ul>
<li>
Data science is about using data to make decisions that drive actions, involving Data Selection, Preprocessing, Transformation, Data Mining, Delivering value from data (interpretation and evaluation).
</li>
<li>
Challenges have been extremely complex overhead, expensive startup, and steep learning curves. The goal of Azure Machine Learning Services is to provide a simpler and scalable method.
</li>
<li>
Machine Learning is using known data to develop a model to predict unknown data.
<ul>
<li>Known Data: Big enough archive, previous observations, past data</li>
<li>Model: Known data + algorithms (ML algorithms)</li>
<li>Unknown Data: Missing, unseen, not existing, future data</li>
</ul>
</li>
<li>
Microsoft's usage of ML: 1997 Hotmail spam filters, 2008 Bing maps routing, 2009 Bing search, 2010 Kinect
</li>
<li>
Azure Machine Learning's goal - Make machine learning accessible to every enterprise, data scientist, developer, information worker, consumer, and device anywhere in the world.
</li>
<li>
Expect that 75% of the time will be building the model and testing validity of predictions.
</li>
<li>
Cycle of Steps to building a machine learning solution
<ul>
<li>Frame the problem</li>
<li>Get and Prepare Data</li>
<li>Develop the Model
<ul>
<li>Analysis / Metric Definition</li>
<li>Feature Engineering</li>
<li>Model Training</li>
<li>Parameter Tuning</li>
<li>Evaluation</li>
</ul>
</li>
<li>Deploy Model</li>
<li>Evaluate / Track Performance</li>
</ul>
</li>
<li>
ML Algorithm defines how your model will react
<ul>
<li>Written in R or Python</li>
</ul>
</li>
<li>
Classification: answer yes/no
</li>
<li>
Regression: predict a numeric value
</li>
<li>
Anomaly Detection: value outside of bounds
</li>
<li>
Clustering: unsupervised learning, group like data
</li>
</ul>

Azure Machine Learning - Data Camp

Command Query Responsibility Segregation (<a href="https://martinfowler.com/bliki/CQRS.html">CQRS</a>) was not something that I easily understood. Martin Fowler did a thorough job of explaining it in often-linked post from 2011. However, I still did not quite grasp it at the time. I also wasn't working on any project large enough or complex enough that it made sense to use a pattern with that level of depth. When I came back to the concept many years later while working on a significantly more complex project, I could put together all of the puzzle pieces but I still couldn't quite see the whole picture. While researching the topic, I stumbled on a blog post by Cesar de la Torre from Microsoft - <a href="https://blogs.msdn.microsoft.com/cesardelatorre/2012/02/22/cqrs-bus-and-windows-azure-technologies/">CQRS BUS and Windows Azure technologies</a>. His picture was well worth the thousands of words that I had already read.
<img src="/images/cqrs-diagram.jpg" alt="CQRS - Basic patterns">
I am limiting the scope of this step of the project to just creating a Message Queue in Azure, writing a command to it, and triggering an appropriate <code>Command Handler</code>. As soon as I started working on the problem, though, I was sidetracked with an important and timely concern — how do I safely store connection information to a remote server? Considering that I work on this project on at least three different machines, I need a highly portable and secure way to store the connection information to my instance of Azure without exposing that connection information to anyone who may look at the code on <a href="https://github.com/drovani/Vigil">GitHub</a>. Now, instead of having three steps to this stage, I have four; maybe more.
<h2>Storing Connection Information</h2>
<img src="/images/vs2017-inavord-manage-user-secrets.png" alt="User Secrets in Context Menu">
When this migrates to a production ready environment, I will probably use <a href="https://docs.microsoft.com/en-us/azure/key-vault/key-vault-whatis">Azure Key Vault</a> or some similar service to manage these types of important secrets. However, for the limited amount of information that I need to keep, I am just going to use the built-in "User Secrets" that comes with Visual Studio 2017. Using the context menu on a Web Application project provides the option to "Manage User Secrets". This pulls up a <code>secrets.json</code> file that lives (unencrypted) on the local filesystem. There are plenty of caveats to its use, namely that it provides no security outside of making sure these settings exist entirely separate from your code. In order to store the information I needed, my <code>secrets.json</code> file looks similar to this.
<pre><code>{
 "vigil-storage": "vigilstorage",
 "vigil-storage-queue": "commandqueue",
 // sample key - the real one is much longer
 "vigil-storage-key1": "ocJFUuUrYgVppKtO726cC1S..."
}
</code></pre>
In order to access the user secrets to the configuration builder, a small piece is added to <code>Startup.cs</code>.
<pre><code>public Startup(IHostingEnvironment env)
{
 var builder = new ConfigurationBuilder()
 .SetBasePath(env.ContentRootPath)
 .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
 .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
 .AddEnvironmentVariables();

 if (env.IsDevelopment())
 {
 builder.AddUserSecrets("6bf18fc4-44a1-4c9b-98ea-22cc92ed1c94");
 }

 Configuration = builder.Build();
}
</code></pre>
Consuming the secrets is as simple as any other configuration value. Since I only need it right now for the <code>StorageCredentials</code> that get injected into the <code>AzureCommandQueue</code> constructor, I have the creation of the credentials as an extension method on the <code>IServiceCollection</code>. My <code>StartupExtensions</code> class can live anywhere; I have decided to put it in the same project that needs it, <code>Vigil.Azure</code>.
<pre><code>public static class StartupExtensions
{
 public static IServiceCollection AddVigilAzureServices(this IServiceCollection services, IConfigurationRoot configuration)
 {
 services.AddTransient&#x3C;IEventBus, AzureEventBus>()
 .AddTransient&#x3C;ICommandQueue, AzureCommandQueue>()
 .AddTransient(srvProvider =>
 {
 var storageCredentials = new StorageCredentials(
 configuration["vigil-storage"],
 configuration["vigil-storage-key1"]
 );
 CloudStorageAccount storageAccount = new CloudStorageAccount(storageCredentials, true);
 CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
 var commandQueue = queueClient.GetQueueReference(configuration["vigil-storage-queue"]);
 return commandQueue;
 });

 return services;
 }
}
</code></pre>
<h2>Create a Storage Account in Azure</h2>
This was the easiest step of them all, because there are many wonderful tutorials already created on how to do get started with <a href="https://docs.microsoft.com/en-us/azure/storage/storage-introduction">Azure Storage</a>. It comes down to three simple steps that can all be done in under a minute (once you know your way around).
<ol>
<li>Create a <img src="/images/azure-resource-group-icon.png" alt="icon"> Resource group</li>
<li>Create a <img src="/images/azure-storage-account-icon.png" alt="icon"> Storage Account</li>
<li>View <img src="/images/azure-access-key-icon.png" alt="icon"> Access keys</li>
</ol>
The name of the storage account (<code>vigilstorage</code>) and the value of one of the keys goes into the <code>User Secrets</code>, and voilà, the storage account is created. One item that tripped me up was that I expected to need to create the Queue within the portal. As I eventually figured out, that piece is performed exclusively in code.
<h2>Sending a Message to Azure Queue Storage</h2>
After many hours of digging through tutorials, hunting down dotnet core versions of the libraries, and trial and error - I finally got to a point where my code compiled and executed and I could test it with a <a href="https://www.getpostman.com/">Postman</a> query. I was surprised by how little code it ended up requiring. This is the entirety of what the <code>AzureCommandQueue</code> class looks like:
<pre><code>public class AzureCommandQueue : ICommandQueue
{
 private CloudQueue commandQueue;

 public AzureCommandQueue(CloudQueue commandQueue)
 {
 this.commandQueue = commandQueue;
 }

 public void Publish&#x3C;TCommand>(TCommand command) where TCommand : ICommand
 {
 PublishAsync(command).Wait();
 }

 public async Task PublishAsync&#x3C;TCommand>(TCommand command) where TCommand : ICommand
 {
 await commandQueue.CreateIfNotExistsAsync();

 var newCmd = new Command(command)
 {
 DispatchedOn = DateTime.UtcNow
 };
 var message = new CloudQueueMessage(JsonConvert.SerializeObject(newCmd));
 await commandQueue.AddMessageAsync(message);
 }
}
</code></pre>
<h2>HOLY SHIT I GOT IT TO WORK</h2>
<img src="/images/azure-queue-data.png" alt="Data in an Azure Queue">
This was one of those "really cool moments" when it comes together how fantastic my job is and how amazing technology truly is. I just had a tool on my computer simulate a user pushing a non-existant button that sends information to another tool on my computer that is pretending to be a web server which then executed code that I wrote to take that message and securely send it up to the internet and sit on some unknown computer where it will stay and wait for me to call upon it.

Rovani's Sandbox

azure

Azure Machine Learning - Data Camp

Vigil's Azure Command Queue