GCP: Machine Learning

Label = true answer

Input = predictor variable(s): what you can use to predict the label

Example = input + corresponding label

Model = math function that takes input variables and creates approximation to
label

Training = adjusting model to minimize error

Prediction = using model on unlabeled data

Batch: a group of training data set or samples (estimated to 100-500 samples by batch).

During the learning and optimization process, instead of loading all the training dataset at once, they are loaded in sets of specific size. These sets are called batches.

Supervised learning: Data have label with the input

Unsupervised learning: Data does not have labels

Structured data consist of rows and columns

Regression model: Linear data representation (f : x -> y ), used when label is continuous. (e.g., tip: 3,8 baby weight : 7.9, etc)

Regression Algorithms

  • Linear Regression
  • Regression Trees(e.g. Random Forest)
  • Support Vector Regression (SVR)
  • …etc

Classification model curvy data representation, used when label has a discrete number of values or classes. Discrete number can only take certain values like rolling dice (1, 2, 3, 4, 5, 6)

Algorithms for classification

  • Decision Trees
  • Logistic Regression
  • Naive Bayes
  • K Nearest Neighbors
  • Linear SVC (Support vector Classifier)
  • …..etc

AlgoUseML

image source url here

Learning process to find the label: considering input value (x1, x2,…, xn)

Training :

  1. Attribute random weight to input value
  2. Run and compare the output of this batch an example (label + input value)
  3. Calculate error on labeled dataset
  4. Change the weights so that the error goes down. This change determines the learning rate
  5. If the error goes down, Evaluate the overall dataset

 

 

weights

Note

Weights/bias = parameters we optimize
Batch size = the amount of data we compute error on
Epoch = one pass through entire dataset ?????
Gradient descent = process of reducing error sed to find the best parameters related to weights
Evaluation = is the model good enough? Has to be done on full dataset
Training = process of optimizing the weights; includes gradient descent +
evaluation

Neuron = one unit of combining inputs
Hidden layer = set of neurons that operate on the same set of inputs
Features = transformations of inputs, such as x^2
Feature engineering = coming up with what transformations to include

Mean Square Error: the loss measure for regression problems
Cross-entropy: the loss measure for classification problems

Confusion matrix

confusionmatrix

Accuracy = ∑Correct answer of ML/ Total

Precision = Positive Predictive Value : When ML says it is true :

∑ (True Positive + True Negative) / Total

Recall is true positive rate : ∑ True Positive/Total

More info : ML introduction

Publicités

GCP: Dataflow

Pipeline: BigQuery + Cloud Storage

A Pipeline is a directed graph of steps: The beginning of this graph is called source, any performed step within this graph is called transformer, the end of this graph is called sink.

Source –> Transformer –> Sink

A pipeline is executed on the cloud by a Runner; each step is elastically scaled.

A pipeline process :

  1. Read in data,
  2. transform it,
  3. write out

A pipeline developer can branch, merge, use if-then statements, etc.

Data in a pipeline are represented by PCollection
○ Supports parallel processing
○ Not an in-memory collection; can be unbounded

ParDo allows for parallel processing acting on one  item at a time .

Useful for:
○ Filtering (choosing which inputs to emit)
○ Converting one Java type to another
○ Extracting parts of an input (e.g., fields of TableRow)
○ Calculating values from different parts of inputs

Qwik Start: DataLab

Setting Up DataLab

gcloud auth list

gcloud config set core/project <PROJECT_ID>

gcloud config set compute/zone us-central1-f

datalab create –no-create-repository –machine-type n1-standard-2 image-class

portDataLab

 

 

 

 

 

 

 

If you lose connection to Datalab for some reason, use this command to reconnect:

datalab connect imageclass

When creating a notebook, you need to define directories for

  • preprocessing
  • model
  • prediction

Preprocessing:

Preprocessing uses a Dataflow pipeline to convert the image format, resize images, and run the converted image through a pre-trained model to get the features or embeddings. You can also do this step using alternate technologies like Spark or plain Python code if you like.

Qwik Start: BigQuery

 

 

bq ls

bq ls publicdata:

mkdire

bq  mk

 

bq ls <tablename>

bq show <tablename>.<schemaname>

bq show publicdata:samples.shakespeare

bq help query

bq query « SELECT word, corpus, COUNT(word) FROM publicdata:samples.shakespeare WHERE word CONTAINS ‘huzzah’ GROUP BY word, corpus »

Behind the Fuzz: Kubernetes

This is my summary of the Google cloud quick start for Kubernetes found here.

In Kubernetes Engine, a container cluster consists of at least one cluster master and multiple worker machines called nodes. These master and node machines run the Kubernetes cluster orchestration system.

 

  • Cluster master runs the Kubernetes control plane processes, including the Kubernetes API server, scheduler, and core resource controllers. deciding what runs on all of the cluster’s nodes. e.g., scheduling workloads, like containerized applications, and managing the workloads’ lifecycle, scaling, and upgrades.
  • Nodes run (i) Docker runtime (ii) Kubernetes node agent (kubelet) which communicates with the master and responsible for starting and running Docker containers scheduled on that node. (Node machine type n1-standard-1, with 1 virtual CPU and 3.75 GB of memory, OS image can be specialized).
  • Kubernetes API calls : The master and nodes also communicate using Kubernetes APIs, directly via HTTP/gRPC, or indirectly, by running commands from the Kubernetes command-line client (kubectl) or interacting with the UI in the GCP Console.
  • In Kubernetes Engine, there are also a number of special containers that run as per-node agents to provide functionality such as log collection and intra-cluster network connectivity.

ECSA Example [In Progress]

Application EFM

AppEFM

Valid Configurations

#1: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Cloud
#2: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG SVGGR LiveStreaming [ Server = 1 Built-In Cloud
#3: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG FaceRecognition LiveStreaming [ Server = 1 Built-In Cloud
#4: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG LiveStreaming [ Server = 1 Built-In Cloud
#5: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 StandAlone Cloud
#6: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Cloud
#7: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR LiveStreaming [ Server = 1 StandAlone Cloud
#8: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR LiveStreaming [ Server = 1 Built-In Cloud
#9: SecurityApp MotionDetection FlowDecoding Recording IntervalBasedRecord Flow MPEG FaceRecognition LiveStreaming [ Server = 1 Built-In Cloud
#10: SecurityApp MotionDetection FlowDecoding Recording IntervalBasedRecord Flow MPEG LiveStreaming [ Server = 1 Built-In Cloud
#11: SecurityApp MotionDetection FlowDecoding Flow MPEG SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Cloud
#12: SecurityApp MotionDetection FlowDecoding Flow MPEG SVGGR LiveStreaming [ Server = 1 Built-In Cloud
#13: SecurityApp MotionDetection FlowDecoding Flow MPEG FaceRecognition LiveStreaming [ Server = 1 Built-In Cloud
#14: SecurityApp MotionDetection FlowDecoding Flow MPEG LiveStreaming [ Server = 1 Built-In Cloud
#15: SecurityApp MotionDetection FlowDecoding Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 StandAlone Cloud
#16: SecurityApp MotionDetection FlowDecoding Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Cloud
#17: SecurityApp MotionDetection FlowDecoding Flow SVGGR LiveStreaming [ Server = 1 StandAlone Cloud
#18: SecurityApp MotionDetection FlowDecoding Flow SVGGR LiveStreaming [ Server = 1 Built-In Cloud
#19: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#20: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG SVGGR LiveStreaming [ Server = 1 Built-In Storage Cloud
#21: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#22: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG LiveStreaming [ Server = 1 Built-In Storage Cloud
#23: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 StandAlone Storage Cloud
#24: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#25: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR LiveStreaming [ Server = 1 StandAlone Storage Cloud
#26: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR LiveStreaming [ Server = 1 Built-In Storage Cloud
#27: SecurityApp MotionDetection FlowDecoding Recording IntervalBasedRecord Flow MPEG FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#28: SecurityApp MotionDetection FlowDecoding Recording IntervalBasedRecord Flow MPEG LiveStreaming [ Server = 1 Built-In Storage Cloud
#29: SecurityApp MotionDetection FlowDecoding Flow MPEG SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#30: SecurityApp MotionDetection FlowDecoding Flow MPEG SVGGR LiveStreaming [ Server = 1 Built-In Storage Cloud
#31: SecurityApp MotionDetection FlowDecoding Flow MPEG FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#32: SecurityApp MotionDetection FlowDecoding Flow MPEG LiveStreaming [ Server = 1 Built-In Storage Cloud
#33: SecurityApp MotionDetection FlowDecoding Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 StandAlone Storage Cloud
#34: SecurityApp MotionDetection FlowDecoding Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#35: SecurityApp MotionDetection FlowDecoding Flow SVGGR LiveStreaming [ Server = 1 StandAlone Storage Cloud
#36: SecurityApp MotionDetection FlowDecoding Flow SVGGR LiveStreaming [ Server = 1 Built-In Storage Cloud
#37: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#38: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG SVGGR LiveStreaming [ Server = 1 Built-In Storage Cloud
#39: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#40: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow MPEG LiveStreaming [ Server = 1 Built-In Storage Cloud
#41: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 StandAlone Storage Cloud
#42: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#43: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR LiveStreaming [ Server = 1 StandAlone Storage Cloud
#44: SecurityApp MotionDetection FlowDecoding Recording LiveRecord Flow SVGGR LiveStreaming [ Server = 1 Built-In Storage Cloud
#45: SecurityApp MotionDetection FlowDecoding Recording IntervalBasedRecord Flow MPEG FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#46: SecurityApp MotionDetection FlowDecoding Recording IntervalBasedRecord Flow MPEG LiveStreaming [ Server = 1 Built-In Storage Cloud
#47: SecurityApp MotionDetection FlowDecoding Flow MPEG SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#48: SecurityApp MotionDetection FlowDecoding Flow MPEG SVGGR LiveStreaming [ Server = 1 Built-In Storage Cloud
#49: SecurityApp MotionDetection FlowDecoding Flow MPEG FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#50: SecurityApp MotionDetection FlowDecoding Flow MPEG LiveStreaming [ Server = 1 Built-In Storage Cloud
#51: SecurityApp MotionDetection FlowDecoding Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 StandAlone Storage Cloud
#52: SecurityApp MotionDetection FlowDecoding Flow SVGGR FaceRecognition LiveStreaming [ Server = 1 Built-In Storage Cloud
#53: SecurityApp MotionDetection FlowDecoding Flow SVGGR LiveStreaming [ Server = 1 StandAlone Storage Cloud
#54: SecurityApp MotionDetection FlowDecoding Flow SVGGR LiveStreaming [ Server = 1 Built-In Storage Cloud
54

Total time: 623ms

nb Constraints 26

 

Cloud EFM

CloudME

Valid Configuration Cloud

#1: Cloud [ CPU = 20000 LoadBalancer Server Apache Language Java DataBase MongoBD
#2: Cloud [ CPU = 20000 LoadBalancer Server Apache Language Java DataBase MySQL
#3: Cloud [ CPU = 20000 Server Apache Language Java DataBase MongoBD
#4: Cloud [ CPU = 20000 Server Apache Language Java DataBase MySQL
#5: Cloud [ CPU = 20000 Server Jonas Language Java DataBase MongoBD
#6: Cloud [ CPU = 20000 LoadBalancer Server Jonas Language Java DataBase MongoBD
#7: Cloud [ CPU = 20000 Server Jonas Language Java DataBase MySQL
#8: Cloud [ CPU = 20000 LoadBalancer Server Jonas Language Java DataBase MySQL
#9: Cloud [ CPU = 20000 Server Apache Language Python DataBase MongoBD
#10: Cloud [ CPU = 20000 Server Apache Language Java Python DataBase MongoBD
#11: Cloud [ CPU = 20000 LoadBalancer Server Apache Language Java Python DataBase MongoBD
#12: Cloud [ CPU = 20000 LoadBalancer Server Apache Language Python DataBase MongoBD
#13: Cloud [ CPU = 20000 Server Apache Language Python DataBase MySQL
#14: Cloud [ CPU = 20000 LoadBalancer Server Apache Language Python DataBase MySQL
#15: Cloud [ CPU = 20000 Server Apache Language Java Python DataBase MySQL
#16: Cloud [ CPU = 20000 LoadBalancer Server Apache Language Java Python DataBase MySQL
#17: Cloud [ CPU = 20000 Server Jonas Language Python DataBase MongoBD
#18: Cloud [ CPU = 20000 Server Jonas Language Java Python DataBase MongoBD
#19: Cloud [ CPU = 20000 LoadBalancer Server Jonas Language Python DataBase MongoBD
#20: Cloud [ CPU = 20000 LoadBalancer Server Jonas Language Java Python DataBase MongoBD
#21: Cloud [ CPU = 20000 Server Jonas Language Python DataBase MySQL
#22: Cloud [ CPU = 20000 Server Jonas Language Java Python DataBase MySQL
#23: Cloud [ CPU = 20000 LoadBalancer Server Jonas Language Java Python DataBase MySQL
#24: Cloud [ CPU = 20000 LoadBalancer Server Jonas Language Python DataBase MySQL
24

Nb Constraints : 20
Total time: 1.375s

 

HAB EFM

HABME

 

Valid Configurations HAB

#1: HAB [ OS = CPU = 1  RAM = 1] Apache VAMPWrappers VAMPConfig Library PIL VLI JDK7
#2: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 Apache VAMPWrappers VAMPConfig Library VLI JDK7
#3: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 Apache VAMPWrappers VAMPConfig Library PIL JDK7
#4: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 Apache VAMPWrappers VAMPConfig Library JDK7
#5: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 VAMPWrappers VAMPConfig Library PIL VLI JDK7
#6: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 VAMPWrappers VAMPConfig Library PIL JDK7
#7: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 VAMPWrappers VAMPConfig Library JDK7
#8: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 VAMPWrappers VAMPConfig Library VLI JDK7
#9: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 VAMPWrappers VAMPConfig
#10: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 Apache VAMPWrappers VAMPConfig
#11: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 Apache VAMPWrappers VAMPConfig Library PIL
#12: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 VAMPWrappers VAMPConfig Library PIL
#13: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 VAMPWrappers VAMPConfig Library PIL VLI
#14: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 VAMPWrappers VAMPConfig Library VLI
#15: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 Apache VAMPWrappers VAMPConfig Library PIL VLI
#16: HAB [ OS = 1 [ CPU = 1 [ RAM = 1 Apache VAMPWrappers VAMPConfig Library VLI
16
Total time: 1.194s
nb constraint14

Matching Algorithm

 

Reuse ECore models in other Ecore models?

To reference one Ecore model from another:

  1. Open the model needing the added reference in the Ecore model editor. In your example, open the file C.ecore.
  2. Right click on the top item in the editor. In the below example, you would click on theplatform:/resource/..../SHEA-F.ecore item.ecore
  3. Select « Load Resource », then browse the workspace to find the Ecore file you want to include.